Message-ID: <1597233762.2836.1485850953135.JavaMail.confluence@ip-10-127-227-164> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_2835_1780797846.1485850953135" ------=_Part_2835_1780797846.1485850953135 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Smart HTTP cache clearing refers to the ability to clea= r cache for Locations/content that is in relation with the content being cu= rrently cleared.
When published, any Content item usually has at least one Location, iden= tified by its URL. Therefore, HTTP cache being bound to URLs, if a Content = item is updated (a new version is published), we want HTTP cache for all it= s Locations to be cleared, so the content itself can be updated everywhere = it is supposed to be displayed. Sometimes, clearing cache for the content's= Locations is not sufficient. You can, for instance, have an excerpt of it = displayed in a list from the parent Location, or from within a relation. In= this case, cache for the parent Location and/or the relation need to be cl= eared as well (at least if an ESI is not used).
Smart HTTP cache clearing is an event-based mechanism. =
Whenever a content item needs its cache cleared, the cache purger service s=
ends an ezpublish.cache_clear.content
event (also identified b=
y eZ\Publish\Core\MVC\Symfony\MVCEvents::CACHE_CLEAR_CONTENT
c=
onstant) and passes an eZ\Publish\Core\MVC\Symfony\Event\ContentCache=
ClearEvent
event object. This object contains the ContentInfo object=
we need to clear the cache for. Every listener for this event can add Loca=
tion objects to the cache clear list.
Once the event is dispatched, the purger passes collected Location objec=
ts to the purge client, which will effectively send the cache BAN request.
Note
ezpublish.http_cache.event_dispatcher
.
By default, following Locations will be added to the cache clear list:= p>
AssignedLocationsListener)
ParentLocationsL=
istener
)R=
elatedLocationsListener
)By design, smart HTTP cache clearing is extensible. One can easily imple=
ment an event listener/subscriber to the ezpublish.cache_clear.conten=
t
event and add Locations to the cache clear list.
Here's a very simple custom listener example, adding an arbitrary Locati= on to the list.
Important
ezpublish.http_cache.event_subscriber
or=20
ezpublish.http_cache.event_listener
.
namespace Acme\AcmeTestBundle\EventListener; use eZ\Publish\API\Repository\LocationService; use eZ\Publish\Core\MVC\Symfony\Event\ContentCacheClearEvent; use eZ\Publish\Core\MVC\Symfony\MVCEvents; use Symfony\Component\EventDispatcher\EventSubscriberInterface; class ArbitraryLocationsListener implements EventSubscriberInterface { /** * @var LocationService */ private $locationService; public function __construct( LocationService $locationService ) { $this->locationService =3D $locationService; } public static function getSubscribedEvents() { return [MVCEvents::CACHE_CLEAR_CONTENT =3D> ['onContentCacheClea= r', 100]]; } public function onContentCacheClear( ContentCacheClearEvent $event ) { // $contentInfo is the ContentInfo object for the content being cle= ared. // You can extract information from it (e.g. ContentType from its c= ontentTypeId), using appropriate Repository services. $contentInfo =3D $event->getContentInfo(); // Adding arbitrary locations to the cache clear list. $event->addLocationToClear( $this->locationService->loadLo= cation( 123 ) ); $event->addLocationToClear( $this->locationService->loadLo= cation( 456 ) ); } }=20
parameters: acme.cache_clear.arbitrary_locations_listener.class: Acme\AcmeTestBundl= e\EventListener\ArbitraryLocationsListener services: acme.cache_clear.arbitrary_locations_listener: class: %acme.cache_clear.arbitrary_locations_listener.class% arguments: [@ezpublish.api.service.location] tags: - { name: ezpublish.http_cache.event_subscriber }=20
eZ Platform uses Symfony HttpCache to = manage content "view" cache with an expiration model. In addition it is extended = (using FOSHttpCache) to add several advanced features. For content com= ing from the CMS the following is taken advantage of out of the box:
X-Location-Id
header, which both Symfony and=
Varnish Proxy are able to invalidate cache on (for details see =
Cache purge.)X-User-Hash
to allow pages to va=
r by user rights (so not per unique user, that is better se=
rved by browser cache.)ezpublish: system: my_siteaccess: content: view_cache: true # Activates HttpCache for content ttl_cache: true # Activates expiration based HttpCach= e for content (very fast) default_ttl: 60 # Number of seconds an Http response = is valid in cache (if ttl_cache is true)=20
Sometimes you need your controller's cache to be invalidated at the same=
time as specific content changes (i.e. ESI sub-requests with render
twig helper,=
for a menu for instance). To be able to do that, you just need to add
use Symfony\Component\HttpFoundation\Response; // Inside a controller action // "Connects" the response to location #123 and sets a max age (TTL) of 1 h= our. $response =3D new Response(); $response->headers->set('X-Location-Id', 123); $response->setSharedMaxAge(3600);=20
If the content you're rendering depends on a user's permissions, then yo= u should make the response context-aware:
use Symfony\Component\HttpFoundation\Response; // Inside a controller action // Tells proxy configured to support this header to take the rights of a us= er (user hash) into account for the cache $response =3D new Response(); $response->setVary('X-User-Hash');=20
This page explains the content cache purge= (aka invalidate) mechanism used when publishing content from the = UI or from a container-aware script, resulting in cache being invalidated e= ither in the built-in Symfony Reverse Proxy, or on the much faster Varnish = reverse proxy.
eZ Platform returns content-related responses with an X-Location-I=
d
header that are stored together by the configured HTTP cache. This=
allows you to clear (invalidate) HTTP cache representing specific=
ally a given Content item. On publishing the content, a cache purger i=
s triggered with the Content ID in question, which in turn figures out affe=
cted content Locations based on Smar=
t HTTP cache clearing logic. The returned Location IDs are sent for pur=
ge using the purge type explained further below.
By default, invalidation requests will be emulated and sent to the Symfo= ny Proxy cache Store. Cache purge will thus be synchronous, meaning no= HTTP purge requests will be sent around when publishing.
ezpublish: http_cache: purge_type: local=20
With Varnish you can configure one or several servers that should be pur= ged over HTTP. This purge type is asynchronous, and flushed by the end of S= ymfony kernel-request/console cycle (during terminate event). = ;Settings for purge servers can be configured per site group or site access= :
ezpublish: http_cache: purge_type: http system: my_siteacess: http_cache: purge_servers: ["http://varnish.server1", "http://varnish.s= erver2", "http://varnish.server3"]=20
For further information on setting up Varnish, see Using Varnish.
While purging on content, updates are handled for you; on actions agains= t the eZ Platform APIs, there are times you might have to purge manually.= p>
Manual purging from code which takes Smart HTTP cache clearing logic into account, this is using the ser= vice also used internally for cache clearing on content updates:
// Purging cache based on content id, this will trigger cache clear= of all locations found by Smart HttpCache clear // typically self, parent, related, .. $container->get('ezpublish.http_cache.purger')->purgeForContent(55);<= /pre>=20
Symfony Proxy stores its cache in the Symfony cache directory, so a regu=
lar cache:clear
commands will clear it:
php app/console --env=3Dprod cache:clear=20
When using Varnish and in need to purge content directly, then the follo= wing examples show how this is done internally by our FOSPurgeCl= ient, and in turn FOSHttpCache Varnish proxy client:
For purging all:
BAN / HTTP 1.1=20 Host: localhost=20 X-Location-Id: .*=20
Or with given location ids (here 123 and 234):
BAN / HTTP 1.1=20 Host: localhost=20 X-Location-Id: ^(123|234)$=20
eZ Platform being built on top of Symfony, it uses standard HTTP cache h= eaders. By default the Symfony reverse proxy, written in PHP, is used to ha= ndle cache, but it can be easily replaced with any other reverse proxy like= Varnish.
Use of Varnish is a requirement for use in Clustering setup, for ove= rview of clustering feature see Clustering.
For Varnish to work properly with eZ, you'll need to use one of the prov= ided files as a basis:
Note: Http cache management is done with the help of FOSHttpCacheBundle. You may need to tweak your VCL f= urther on according to F= OSHttpCache documentation in order to use features supported by it.
Somehow we need to tell php process that we are behind a Varnish proxy a= nd not the built in Symfony Http Proxy. If you use fastcgi/fpm you can pass= these directly to php process, but you can in all cases also specify them = in your web server config.
<VirthualHost *:80> # Configure your VirtualHost with rewrite rules and stuff # Force front controller NOT to use built-in reverse proxy. SetEnv SYMFONY_HTTP_CACHE 0 =09# Configure IP of your Varnish server to be trusted proxy # Replace fake IP address below by your Varnish IP address SetEnv SYMFONY_TRUSTED_PROXIES "193.22.44.22" </VirtualHost>=20
fastcgi_param SYMFONY_HTTP_CACHE 0; # Configure IP of your Varnish server to be trusted proxy # Replace fake IP address below by your Varnish IP address fastcgi_param SYMFONY_TRUSTED_PROXIES "193.22.44.22";=20
Secondly we need to tell eZ Platform to change to use http based purge c= lient (specifically FosHttpCache Varnish purge client is used), an= d specify url Varnish can be reached on:
ezpublish: http_cache: purge_type: http system: # Assuming that my_siteaccess_group your frontend AND backend sitea= ccesses my_siteaccess_group: http_cache: # Fill in your Varnish server(s) address(es). purge_servers: [http://my.varnish.server:8081]=20
As it is based on Symfony 2, eZ Platform uses HTTP cache extended with features like content awareness.= However, this cache management is only available for anonymous users due t= o HTTP restrictions.
It is of course possible to make HTTP cache vary thanks to the Accept-Encoding
). Thus, to=
make the cache vary on a specific context (for example a hash based on=
a user roles and limitations), this context must be present in the or=
iginal request.
As the response can vary on a request header, the base solution is to ma=
ke the kernel do a sub-request in order to retrieve the user context hash (=
aka user hash). Once the user hash =
;has been retrieved, it's injected in the original request in the
<?php use Symfony\Component\HttpFoundation\Response; // ... // Inside a controller action $response =3D new Response(); $response->setVary('X-User-Hash');=20
This solution is implemented in Symfony reverse proxy (aka Http= Cache) and is also accessible to dedicated reverse proxies like Varnis= h.
Note that sharing ESIs across SiteAccesses is not possible by design (se=
e EZP-22535 -
Cached ESI can not be shared across pages/siteaccesse=
s due to "pathinfo" property=20
Closed for techni=
cal details)
In cases where you need to deliver content uniquely to a given user, and= tricks like using JavaScript and cookie values, hinclude, or disabling cac= he is not an option. Then remaining option is to vary response by cookie:= p>
$response->setVary('Cookie');=20
Unfortunately this is not optimal as it will by default vary by all cook= ies, including those set by add trackers, analytics tools, recommendation s= ervices, etc. However, as long as your application backend does no= t need these cookies, you can solve this by stripping everything but the se= ssion cookie. Example for Varnish can be found in the default VCL examples = in part dealing with User Hash, for single-server setup this can easily be = accomplished in Apache / Nginx as well.
As eZ Platform uses FOSHttpCacheBundle, th= is impacts the following features:
Varnish proxy client from FOSHttpCache lib is used for clearing eZ HTTP =
cache, even when using Symfony HttpCache. A single BAN
request=
is sent to registered purge servers, containing a X-Location-Id header. This header contains all Location IDs for which objects in cache=
need to be cleared.
Symfony reverse proxy (aka HttpCache) is supported out of the box, all y= ou have to do is to activate it.
FOSHttpCacheBun= dle User Context feature is activated by default.
As the response can vary on a request header, the base solution is to ma=
ke the kernel do a sub-request in order to retrieve the context (aka X-User-Hash
h=
eader, making it possible to vary the HTTP response on this header=
:
**X-User-Hash**
.
<?php=20 use Symfony\Component\HttpFoundation\Response; =20 // ... =20 // Inside a controller action $response =3D new Response(); $response->setVary('X-User-Hash');=20
This solution is implemented in Symfony reverse proxy (aka HttpCache) a= nd is also accessible to dedicated reverse proxies like Varnish.
eZ Platform already interferes with the hash generation process by addin= g current user permissions and limitations. You can also interfere in this = process by implementing custom context provider(s).
The behavior described here comes out of the box with Symfony reverse pr= oxy, but it's of course possible to use Varnish to achieve the same.
# Varnish 3 style for eZ Platform # Our Backend - We assume that eZ Platform Web server listen on port 80 on = the same machine. backend ezplatform { .host =3D "127.0.0.1"; .port =3D "80"; } # Called at the beginning of a request, after the complete request has been= received sub vcl_recv { # Set the backend set req.backend =3D ezplatform; # ... # Retrieve client user hash and add it to the forwarded request. call ez_user_hash; # If it passes all these tests, do a lookup anyway; return (lookup); } # Sub-routine to get client user hash, for context-aware HTTP cache. # Don't forget to correctly set the backend host for the Curl sub-request. sub ez_user_hash { # Prevent tampering attacks on the hash mechanism if (req.restarts =3D=3D 0 && (req.http.accept ~ "application/vnd.fos.user-context-has= h" || req.http.x-user-context-hash ) ) { error 400; } if (req.restarts =3D=3D 0 && (req.request =3D=3D "GET" || req.r= equest =3D=3D "HEAD")) { # Get User (Context) hash, for varying cache by what user has acces= s to. # https://doc.ez.no/display/EZP/Context+aware+HTTP+cach # Anonymous user w/o session =3D> Use hardcoded anonymous hash t= o avoid backend lookup for hash if (req.http.Cookie !~ "eZSESSID" && !req.http.authorizatio= n) { # You may update this hash with the actual one for anonymous us= er # to get a better cache hit ratio across anonymous users. # Note: Then needs update every time anonymous user role assign= ments change. set req.http.X-User-Hash =3D "38015b703d82206ebc01d17a39c727e5"= ; } # Pre-authenticate request to get shared cache, even when authentic= ated else { set req.http.x-fos-original-url =3D req.url; set req.http.x-fos-original-accept =3D req.http.accept; set req.http.x-fos-original-cookie =3D req.http.cookie; # Clean up cookie for the hash request to only keep session coo= kie, as hash cache will vary on cookie. set req.http.cookie =3D ";" + req.http.cookie; set req.http.cookie =3D regsuball(req.http.cookie, "; +", ";"); set req.http.cookie =3D regsuball(req.http.cookie, ";(eZSESSID[= ^=3D]*)=3D", "; \1=3D"); set req.http.cookie =3D regsuball(req.http.cookie, ";[^ ][^;]*"= , ""); set req.http.cookie =3D regsuball(req.http.cookie, "^[; ]+|[; ]= +$", ""); set req.http.accept =3D "application/vnd.fos.user-context-hash"= ; set req.url =3D "/_fos_user_context_hash"; # Force the lookup, the backend must tell not to cache or vary = on all # headers that are used to build the hash. return (lookup); } } # Rebuild the original request which now has the hash. if (req.restarts > 0 && req.http.accept =3D=3D "application/vnd.fos.user-context= -hash" ) { set req.url =3D req.http.x-fos-original-url; set req.http.accept =3D req.http.x-fos-original-accept; set req.http.cookie =3D req.http.x-fos-original-cookie; unset req.http.x-fos-original-url; unset req.http.x-fos-original-accept; unset req.http.x-fos-original-cookie; # Force the lookup, the backend must tell not to cache or vary on t= he # user hash to properly separate cached data. return (lookup); } } sub vcl_fetch { # ... if (req.restarts =3D=3D 0 && req.http.accept ~ "application/vnd.fos.user-context-hash= " && beresp.status >=3D 500 ) { error 503 "Hash error"; } } sub vcl_deliver { # On receiving the hash response, copy the hash header to the original # request and restart. if (req.restarts =3D=3D 0 && resp.http.content-type ~ "application/vnd.fos.user-conte= xt-hash" && resp.status =3D=3D 200 ) { set req.http.x-user-hash =3D resp.http.x-user-hash; return (restart); } # If we get here, this is a real response that gets sent to the client. # Remove the vary on context user hash, this is nothing public. Keep al= l # other vary headers. set resp.http.Vary =3D regsub(resp.http.Vary, "(?i),? *x-user-hash *", = ""); set resp.http.Vary =3D regsub(resp.http.Vary, "^, *", ""); if (resp.http.Vary =3D=3D "") { remove resp.http.Vary; } # Sanity check to prevent ever exposing the hash to a client. remove resp.http.x-user-hash; }=20
// Varnish 4 style for eZ Platform // Complete VCL example vcl 4.0; // Our Backend - Assuming that web server is listening on port 80 // Replace the host to fit your setup backend ezplatform { .host =3D "127.0.0.1"; .port =3D "80"; } // Called at the beginning of a request, after the complete request has bee= n received sub vcl_recv { // Set the backend set req.backend_hint =3D ezplatform; // ... // Retrieve client user hash and add it to the forwarded request. call ez_user_hash; // If it passes all these tests, do a lookup anyway. return (hash); } // Called when the requested object has been retrieved from the backend sub vcl_backend_response { if (bereq.http.accept ~ "application/vnd.fos.user-context-hash" && beresp.status >=3D 500 ) { return (abandon); } // ... } // Sub-routine to get client user hash, for context-aware HTTP cache. sub ez_user_hash { // Prevent tampering attacks on the hash mechanism if (req.restarts =3D=3D 0 && (req.http.accept ~ "application/vnd.fos.user-context-has= h" || req.http.x-user-hash ) ) { return (synth(400)); } if (req.restarts =3D=3D 0 && (req.method =3D=3D "GET" || req.me= thod =3D=3D "HEAD")) { // Get User (Context) hash, for varying cache by what user has acce= ss to. // https://doc.ez.no/display/EZP/Context+aware+HTTP+cache // Anonymous user w/o session =3D> Use hardcoded anonymous hash = to avoid backend lookup for hash if (req.http.Cookie !~ "eZSESSID" && !req.http.authorizatio= n) { // You may update this hash with the actual one for anonymous u= ser // to get a better cache hit ratio across anonymous users. // Note: You should then update it every time anonymous user ri= ghts change. set req.http.X-User-Hash =3D "38015b703d82206ebc01d17a39c727e5"= ; } // Pre-authenticate request to get shared cache, even when authenti= cated else { set req.http.x-fos-original-url =3D req.url; set req.http.x-fos-original-accept =3D req.http.accept; set req.http.x-fos-original-cookie =3D req.http.cookie; // Clean up cookie for the hash request to only keep session co= okie, as hash cache will vary on cookie. set req.http.cookie =3D ";" + req.http.cookie; set req.http.cookie =3D regsuball(req.http.cookie, "; +", ";"); set req.http.cookie =3D regsuball(req.http.cookie, ";(eZSESSID[= ^=3D]*)=3D", "; \1=3D"); set req.http.cookie =3D regsuball(req.http.cookie, ";[^ ][^;]*"= , ""); set req.http.cookie =3D regsuball(req.http.cookie, "^[; ]+|[; ]= +$", ""); set req.http.accept =3D "application/vnd.fos.user-context-hash"= ; set req.url =3D "/_fos_user_context_hash"; // Force the lookup, the backend must tell how to cache/vary re= sponse containing the user hash return (hash); } } // Rebuild the original request which now has the hash. if (req.restarts > 0 && req.http.accept =3D=3D "application/vnd.fos.user-context= -hash" ) { set req.url =3D req.http.x-fos-original-url; set req.http.accept =3D req.http.x-fos-original-accept; set req.http.cookie =3D req.http.x-fos-original-cookie; unset req.http.x-fos-original-url; unset req.http.x-fos-original-accept; unset req.http.x-fos-original-cookie; // Force the lookup, the backend must tell not to cache or vary on = the // user hash to properly separate cached data. return (hash); } } sub vcl_deliver { // On receiving the hash response, copy the hash header to the original // request and restart. if (req.restarts =3D=3D 0 && resp.http.content-type ~ "application/vnd.fos.user-conte= xt-hash" ) { set req.http.x-user-hash =3D resp.http.x-user-hash; return (restart); } // If we get here, this is a real response that gets sent to the client= . // Remove the vary on context user hash, this is nothing public. Keep a= ll // other vary headers. set resp.http.Vary =3D regsub(resp.http.Vary, "(?i),? *x-user-hash *", = ""); set resp.http.Vary =3D regsub(resp.http.Vary, "^, *", ""); if (resp.http.Vary =3D=3D "") { unset resp.http.Vary; } // Sanity check to prevent ever exposing the hash to a client. unset resp.http.x-user-hash; if (client.ip ~ debuggers) { if (obj.hits > 0) { set resp.http.X-Cache =3D "HIT"; set resp.http.X-Cache-Hits =3D obj.hits; } else { set resp.http.X-Cache =3D "MISS"; } } }=20
The anonymous X-User-Hash is generated based on the anonymous user=
em>, group and role. The 38015b703d82206ebc01d=
17a39c727e5
hash that is provided in the code above will work only w=
hen these three variables are left unchanged. Once you change the default p=
ermissions and settings, the X-User-Hash will change and Varnish won't be a=
ble to effectively handle cache anymore.
In that case you need to find out the new anonymous X-User-Hash and chan= ge the VCL accordingly, else Varnish will return a no-cache header.
The easiest way to find the new hash is:
1. Connect to your server (shh should be enoug= h)
2. Add <your-domain.com>
to you=
r /etc/hosts
file
3. Execute the following command:
curl -I -H "Accept: application/vnd.fos.user-context-hash" http://=
<your-domain.com>/_fos_user_context_hash
You should get a result like this:
HTTP/1.1 200 OK Date: Mon, 03 Oct 2016 15:34:08 GMT Server: Apache/2.4.18 (Ubuntu) X-Powered-By: PHP/7.0.8-0ubuntu0.16.04.2 X-User-Hash: b1731d46b0e7a375a5b024e950fdb8d49dd25af85a5c7dd5116ad2a18cda82= cb Cache-Control: max-age=3D600, public Vary: Cookie,Authorization Content-Type: application/vnd.fos.user-context-hash=20
4. Now, replace the existing X-User-Hash value with the= new one:
# Note: This needs update every time anonymous user role assignment= s change. set req.http.X-User-Hash =3D "b1731d46b0e7a375a5b024e950fdb8d49dd25af85a5c7= dd5116ad2a18cda82cb";=20
5. Restart the Varnish server and everything should wor= k fine.
The following configuration is defined in eZ by default for FOSHttpCache= Bundle. You may override these settings.
fos_http_cache:=20 proxy_client:=20 # "varnish" is used, even when using Symfony HttpCache. default: varnish varnish:=20 # Means http_cache.purge_servers defined for current SiteAccess= . servers: [$http_cache.purge_servers$] user_context:=20 enabled: true # User context hash is cached during 10min hash_cache_ttl: 600 user_hash_header: X-User-Hash=20
This feature is based on Context aware HTTP caching post by asm89.