[Unit] Description=Varnish Cache, a high-performance HTTP accelerator [Service] Type=forking # Maximum number of open files (for ulimit -n) LimitNOFILE=131072 # Locked shared memory - should suffice to lock the shared memory log # (varnishd -l argument) # Default log size is 80MB vsl + 1M vsm + header -> 82MB # unit is bytes LimitMEMLOCK=85983232 # On systemd >= 228 enable this to avoid "fork failed" on reload. #TasksMax=infinity # Maximum size of the corefile. LimitCORE=infinity # Set WARMUP_TIME to force a delay in reload-vcl between vcl.load and vcl.use # This is useful when backend probe definitions need some time before declaring # configured backends healthy, to avoid routing traffic to a non-healthy backend. #WARMUP_TIME=0 ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/ secret -s malloc,256m ExecReload=/usr/share/varnish/reload-vcl [Install] WantedBy=multi-user.target Debian Stretch with systemd
✓ Request header ✓ Sent by client (~browser) ✓ Sent on every request ✓ Describes state ✓ Is not cached ✓ 3rd party tracking cookies vs own cookies About cookies Cookie: key=value;key2=value2
✓ Response header ✓ Sent by backend server ✓ Only when backend is called ✓ Changes state ✓ Is not cached ✓ Blacklisted for 120s (by default) About cookies Set-Cookie: key=another_value
✓vcl_recv: receive request ✓vcl_hash: compose cache key ✓vcl_miss: not found in cache ✓vcl_hit: found in cache ✓vcl_pass: don’t store in cache ✓vcl_pipe: bypass cache ✓vcl_backend_fetch: connect to backend ✓vcl_backend_response: response from backend ✓vcl_backend_error: backend fetch failed
✓ req: incoming request object ✓ req_top: top level esi request ✓ bereq: request object to send to backend ✓ beresp: backend response ✓ resp: response to send back to client ✓ obj: cached object ✓ client: client information ✓ server: server information ✓ local: local TCP information ✓ remote: remote TCP information ✓ storage: storage information VCL Objects
sub vcl_purge { return (synth(200, "Purged")); } sub vcl_hit { if (obj.ttl >= 0s) { return (deliver); } if (obj.ttl + obj.grace > 0s) { return (deliver); } return (miss); } sub vcl_miss { return (fetch); } sub vcl_deliver { return (deliver); } Remove from cache Found in cache Not found in cache Return HTTP response to client
sub vcl_recv { std.timestamp("Before std.log"); std.log("This should appear in the Shared Memory Log"); std.timestamp("After std.log"); std.syslog(9,"This should appear in the Syslog"); std.timestamp("After std.syslog"); return(synth(200,"OK")); } Logging
sub vcl_init { new vdir = directors.round_robin(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(); } Loadbalancing: round robin
sub vcl_init { new vdir = directors.random(); vdir.add_backend(server1,2); vdir.add_backend(server2,3); } sub vcl_recv { set req.backend_hint = vdir.backend(); } Loadbalancing: random
sub vcl_init { new vdir = directors.fallback(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(); } Loadbalancing: fallback
sub vcl_init { new vdir = directors.hash(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(client.identity); } Loadbalancing: IP hash
vcl 4.0; import cookie; sub vcl_recv { cookie.parse(req.http.cookie); if(cookie.isset("PHPSESSID")) { return(synth(200,"PHPSESSID is set and contains the following value: " + cookie.get("PHPSESSID") + "")); } } Other vmod_cookie examples
vcl 4.0; sub vcl_recv { if (req.http.cookie) { cookie.parse(req.http.cookie); cookie.filter_except("PHPSESSID"); set req.http.cookie = cookie.get_string(); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; } } } The VMOD way
sub vcl_recv { set req.http.Surrogate-Capability = "key=ESI/1.0"; } sub vcl_backend_response { if (beresp.http.Surrogate-Control ~ "ESI/1.0") { unset beresp.http.Surrogate-Control; set beresp.do_esi = true; } } Edge Side Includes
header("Cache-Control: public,must-revalidate,s-maxage=10"); echo "Date in the ESI tag: ".date('Y-m-d H:i:s').' '; header("Cache-Control: no-store"); header(“Surrogate-Control: content='ESI/1.0'"); echo ''.PHP_EOL; echo "Date in the main page: ".date('Y-m-d H:i:s').' '; Main page ESI frame: esi.php Cached for 10 seconds Not cached
✓ Server-side ✓ Standardized ✓ Processed on the “edge”, no in the browser ✓ Generally faster Edge-Side Includes - Sequential - One fails, all fail - Limited implementation in Varnish
{{ render_esi(controller('AppBundle:News:latest', { 'maxPerPage': 5 })) }} {{ render_esi(url('latest_news', { 'maxPerPage': 5 })) }} In your Twig templates Silex or Symfony Falls back to internal subrequests on failure Does ESI
sub vcl_backend_response { if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } Control Time To Live
Banning curl -XPURGE "http://example.com/products" curl -XPURGE -H "x-purge-regex:/products" "http://example.com" Add to ban list, remove on next request Remove patterns via “X-PURGE-REGEX”
Ban lurker Object User Varnish Server Sends HTTP response Response stored in object Sends BAN to Varnish Ban lurker thread Ban list Reads ban list Removes object ban req.http.host == localhost && req.url ~ /products
Ban lurker ban req.http.host == localhost && req.url ~ /products Lurker can’t match request info Lurker can only match what’s in the object Next visitor triggers cache remove
Lurker-friendly bans ban obj.http.x-host == localhost && obj.http.x-url ~ /products Lurker can match response info Ban lurker removes item from cache async
varnish> param.show ban_lurker_age 200 ban_lurker_age Value is: 60.000 [seconds] (default) Minimum is: 0.000 The ban lurker will ignore bans until they are this old. When a ban is added, the active traffic will be tested against it as part of object lookup. Because many applications issue bans in bursts, this parameter holds the ban-lurker off until the rush is over. This should be set to the approximate time which a ban-burst takes.
usage: varnishstat [-1lV] [-f field] [-t seconds|] [-n varnish_name] [-N filename] -1 # Print the statistics to stdout. -f field # Field inclusion glob # If it starts with '^' it is used as an exclusion list. -l # Lists the available fields to use with the -f option. -n varnish_name # The varnishd instance to get logs from. -N filename # Filename of a stale VSM instance. -t seconds| # Timeout before returning error on initial VSM connection. -V # Display the version number and exit. -x # Print statistics to stdout as XML. -j # Print statistics to stdout as JSON. Varnishstat usage
-- RFC 500 10 -1 1501598872 1501598869 1501598872 0 500 ✓ Don’t screw with the age header ✓ Only 497 second effective TTL ✓ Custom Age header (3 seconds off)
%s: %f %f %f | | | | | | | +- Time since last timestamp | | +---- Time since start of work unit | +------- Absolute time of event +----------- Event label Timestamp tag
✓ All response headers ✓ All tags that start with “Req” ✓ Exclude “x-“ response headers ✓ Include “response time” timestamp ✓ Exclude request protocol log lines, and the request accountancy log lines All-in-one
JWT Cookie:jwt_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9. eyJpc3MiOiJodHRwOlwvXC9sb2NhbGhvc3RcLyIsImlhdCI6MTUwOD c0NzM2MywiZXhwIjoxNTA4NzUwOTYzLCJ1aWQiOjEsInVzZXJuYW1l IjoiVGhpanMifQ.EPemqBaH74Sbs0bZqAaR8i7uVYO389VOlJvWC3o cL7g ✓Stored in a cookie ✓Can be validated by Varnish ✓Payload can be processed by any language (e.g. Javascript)
sub vcl_synth { #301 & 302 synths should be actual redirects if (resp.status == 301 || resp.status == 302) { set resp.http.location = resp.reason; set resp.reason = "Moved"; return (deliver); } }
var.set(“rawPayload",regsub(var.get("token"),"[^\.]+\.([^\.]+)\.[^\.]+ $","\1")); var.set("payload", digest.base64url_decode(var.get("rawPayload"))); Second chunk contains the raw payload Base64 decode to get JSON payload
var.set("exp",regsub(var.get("payload"),{"^.*?"exp"\s*:\s*([0-9]+).*? $"},"\1")); var.set("userId",regsub(var.get("payload"),{"^.*?"uid"\s*:\s*([0-9]+).*? $"},"\1")); Extract expiration time from JSON payload Extract user ID from JSON payload
thijsferyn
toshiaizawa
takahashikazutaro
hirokiotsuka
oogfranz
tatematsu_san
miura55
bonjiri_niwa
soachr
ganota
yoshinagaiwnl
oekazuma
jkebertz
nonsquared
jonyablonski
pauljervisheath
holman
rasmusluckow
jeffersonlam
kneath
lara
hatefulcrawdad
jensimmons
destraynor
thekraken