Varnish training - VarnishCon 2016

Slides for my 3 hour workshop at VarnishCon 2016. Course material at http://github.com/thijsferyn/varnishtraining

Transcript

1. Varnish training Varnishcon Amsterdam 2016 By Thijs Feryn
2. Slow websites suck
3. Web performance is an essential part of the user experience
4. Web performance matters
5. Varnish could solve that problem
6. Hi, I’m Thijs
7. I’m @ThijsFeryn on Twitter
8. I’m an Evangelist At
9. I’m an at Evangelist
10. I’m a at board member
11. Varnish
12. Cache?
13. Loadbalancer?
14. Proxy?
15. Web application firewall?
16. HTTP accelerator
17. Why Cache?
18. Run out of ways to optimize their application stack
19. Infrastructure or code optimization would cost too much money
20. Don’t recompute if the data hasn’t changed
21. Varnish is really fast !!!
22. In memory
23. Threading
24. Varnish Configuration Language
25. Reverse proxy
26. Normally User Server
27. With forward proxy User Proxy Server Office
28. With reverse proxy User Proxy Server Datacenter
29. Install & configure
30. apt-get install apt-transport-https curl https://repo.varnish-cache.org/GPG- key.txt | apt-key add - echo "deb https://repo.varnish-cache.org/ debian/ jessie varnish-4.1"\ >> /etc/apt/sources.list.d/varnish- cache.list apt-get update apt-get install varnish Debian
31. apt-get install apt-transport-https curl https://repo.varnish-cache.org/GPG- key.txt | apt-key add - echo "deb https://repo.varnish-cache.org/ ubuntu/ trusty varnish-4.1"\ >> /etc/apt/sources.list.d/varnish- cache.list apt-get update apt-get install varnish Ubuntu
32. yum install epel-release rpm --nosignature -i https://repo.varnish- cache.org/redhat/varnish-4.1.el7.rpm yum install varnish RHEL & CentOS 7
33. yum install epel-release rpm --nosignature -i https://repo.varnish- cache.org/redhat/varnish-4.1.el6.rpm yum install varnish RHEL & CentOS 6
34. Config file
36. SysV Systemd Ubuntu/Debian /etc/default/varnish /etc/systemd/system/ varnish.service RHEL/CentOS /etc/sysconfig/varnish /etc/varnish/ varnish.params
37. cp /lib/systemd/system/varnish.service \ /etc/systemd/system/ Copy defaults file Ubuntu & Debian with systemd
38. Startup options
39. [Unit] Description=Varnish Cache, a high- performance HTTP accelerator [Service] Type=forking LimitNOFILE=131072 LimitMEMLOCK=82000 ExecStart=/usr/sbin/varnishd -a :80 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,256m ExecReload=/usr/share/varnish/reload-vcl [Install] WantedBy=multi-user.target Debian Jessie with systemd
40. vim /etc/systemd/system/varnish.service systemctl daemon-reload service varnish reload Reload with systemd
41. DAEMON_OPTS="-a :80 \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s malloc,256m" Ubuntu Trusty without systemd
42. vim /etc/default/varnish service varnish reload Reload with sysv
43. -a: binding address & port -T: admin binding -f: VCL file -S: secret file -s: storage -j: jailing -l: shared memory log size -t: default TTL -p: runtime parameters Options
44. DAEMON_OPTS="-j unix,user=www-data \ -a :80 \ -a :81,PROXY \ -T localhost:6082 \ -f /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -l 100m,10m \ -t 60 \ -p feature=+esi_disable_xml_check \ -p connect_timeout=20s \ -p first_byte_timeout=100s \ -p between_bytes_timeout=5s \ -s malloc,3g"
45. The backend
47. Point of entry? Bind to 80 Otherwise 6081
48. 8080 HTTP-ALT
49. Listen 8080 Apache <VirtualHost *:8080> ports.conf vhost
50. listen 8080; Nginx vhost
51. ✓In VCL file ✓In config file using "-b" Link backend to Varnish
52. DAEMON_OPTS="-j unix,user=www-data \ -a :80 \ -a :81,PROXY \ -T localhost:6082 \ -b 127.0.0.1:8080 \ -S /etc/varnish/secret \ -s malloc,3g" Can't have -f In config file
53. vcl 4.0;
 
 backend default {
 .host = "127.0.0.1";
 .port = "8080";
 } In VCL file Minimal VCL
55. There are rules
56. ✓ Idempotence ✓ State ✓ TTL ✓ Variations Varnish and HTTP
57. Idempotence Execute multiple times Result doesn't change
58. ✓ GET ✓ HEAD - POST - PUT - DELETE - PATCH Idempotence Result changes Result doesn't change Don't cache
59. State
60. State ~ user specific data Cookies Auth headers
61. About cookies
62. ✓ Request header ✓ Sent by client (~browser) ✓ Sent on every request ✓ Describes state ✓ Is not cached ✓ 3rd party tracking cookies vs own cookies About cookies Cookie: key=value;key2=value2
63. ✓ Response header ✓ Sent by backend server ✓ Only when backend is called ✓ Changes state ✓ Is not cached ✓ Blacklisted for 120s (by default) About cookies Set-Cookie: key=another_value
64. Time To Live
65. ✓ 120s by default ✓ Respects HTTP cache-control header ✓ Respects expires header ✓ Override in VCL file Time to live
66. 1.VCL 2.s-maxage 3.max-age 4.expires 5.Default value 120s TTL Order
67. Use TTL not to cache Cache-Control: max-age=0 Cache-Control: s-maxage=0 Cache-Control: private Cache-Control: no-cache Cache-Control: no-store Expires: Fri, 1 Jan 1971 00:00:00 GMT
68. Combine values Cache-Control: public, max- age=1000, s-maxage=3600
69. Age Age: 10 How old is the cached object?
70. Real cache duration max-age - Age s-maxage - Age
71. Variations
72. Cache is not the same for everyone
73. ✓ URL ✓ Hostname ✓ IP if hostname is not set ✓ Vary header Basic variations
74. Vary header Vary: Accept-Encoding Vary: Accept-Language Vary: Cookie Watch out!
75. The flow
78. Varnish Configuration Language
79. Hooks & subroutines
80. VCL Recv VCL Hash VCL Miss VCL Hit VCL Backend Response VCL Deliver VCL Purge VCL Error VCL Pipe VCL Pass VCL Synth VCL Backend Error
81. ✓vcl_recv: receive request ✓vcl_hash: compose cache key ✓vcl_miss: not found in cache ✓vcl_hit: found in cache ✓vcl_pass: don’t store in cache ✓vcl_pipe: bypass cache ✓vcl_backend_fetch: connect to backend ✓vcl_backend_response: response from backend ✓vcl_backend_error: backend fetch failed
82. ✓vcl_purge: after successful purge ✓vcl_synth: send synthetic output ✓vcl_deliver: return data to client ✓vcl_init: initialize VMODs ✓vcl_fini: discard VMODs
83. Actions
84. ✓ hash: lookup in cache ✓ pass: don't cache ✓ synth: synthetic HTML output ✓ pipe: bypass cache ✓ purge: remove from cache VCL_RECV
85. ✓ synth: synthetic HTML output ✓ pipe: bypass cache VCL_PIPE
86. ✓ fetch: fetch data from backend, don't cache ✓ restart: restart transaction ✓ synth: synthetic HTML output VCL_PASS
87. ✓ deliver: send cached object ✓ miss: synchronous refresh despite hit ✓ pass: fetch data from backend despite hit, don't cache ✓ restart: restart transaction ✓ synth: synthetic HTML output VCL_HIT
88. ✓ fetch: fetch data from backend ✓ pass: fetch data from backend, don't cache ✓ restart: restart transaction ✓ synth: synthetic HTML output VCL_MISS
89. ✓ lookup: look for cached object by using hash key VCL_HASH
90. ✓ restart: restart transaction ✓ synth: synthetic HTML output VCL_PURGE
91. ✓ deliver: deliver object to client ✓ restart: restart transaction ✓ synth: synthetic HTML output VCL_DELIVER
92. ✓ deliver: deliver synthetic output to client ✓ restart: restart transaction VCL_SYNTH
93. ✓ fetch: fetch object from backend ✓ abandon: abandon request and send HTTP 503 error VCL_BACKEND_FETCH
94. ✓ deliver: send fetched data to client ✓ abandon: abandon request and send HTTP 503 error ✓ retry: retry backend request VCL_BACKEND_RESPONSE
95. ✓ deliver: send fetched data to client ✓ retry: retry backend request VCL_BACKEND_ERROR
96. Objects
97. ✓ req: incoming request object ✓ req_top: top level esi request ✓ bereq: request object to send to backend ✓ beresp: backend response ✓ resp: response to send back to client ✓ obj: cached object ✓ client: client information ✓ server: server information ✓ local: local TCP information ✓ remote: remote TCP information ✓ storage: storage information VCL Objects
98. Variables
99. ✓ req.url ✓ req.http.host ✓ req.http.user-agent ✓ req.backend_hint ✓ req.method ✓ … req variables
100. ✓ bereq.backend ✓ bereq.http.user-agent ✓ bereq.method ✓ bereq.url ✓ … bereq variables
101. ✓ beresp.age ✓ beresp.backend.ip ✓ beresp.backend.name ✓ beresp.do_esi ✓ beresp.grace ✓ beresp.keep ✓ beresp.http.set-cookie ✓ beresp.ttl ✓ beresp.status ✓ beresp.uncacheable beresp variables
102. ✓ beresp.age ✓ beresp.backend.ip ✓ beresp.backend.name ✓ beresp.do_esi ✓ beresp.grace ✓ beresp.keep ✓ beresp.http.set-cookie ✓ beresp.ttl ✓ beresp.status ✓ beresp.uncacheable beresp variables
103. ✓ client.ip ✓ client.identity client variables local variables ✓ local.ip remote variables ✓ remote.ip
104. ✓ obj.age ✓ obj.grace ✓ obj.hits ✓ obj.http.cache-control ✓ obj.reason ✓ obj.status ✓ obj.ttl object variables
105. ✓ resp.http.user-agent ✓ resp.is_streaming ✓ resp.reason ✓ resp.status resp variables
106. ✓ storage.<name>.free_space ✓ storage.<name>.used_space ✓ storage.<name>.happy storage variables
107. Default behaviour
108. vcl 4.0; sub vcl_recv { if (req.method == "PRI") { return (synth(405)); } if (req.method != "GET" && req.method != "HEAD" && req.method != "PUT" && req.method != "POST" && req.method != "TRACE" && req.method != "OPTIONS" && req.method != "DELETE") { return (pipe); } if (req.method != "GET" && req.method != "HEAD") { return (pass); } if (req.http.Authorization || req.http.Cookie) { return (pass); } return (hash); } Idempotence State Action Receive request
109. sub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (lookup); } Lookup in cache Variations Action
110. sub vcl_purge { return (synth(200, "Purged")); } sub vcl_hit { if (obj.ttl >= 0s) { return (deliver); } if (obj.ttl + obj.grace > 0s) { return (deliver); } return (miss); } sub vcl_miss { return (fetch); } sub vcl_deliver { return (deliver); } Remove from cache Found in cache Not found in cache Return HTTP response to client
111. sub vcl_synth { set resp.http.Content-Type = "text/html; charset=utf-8"; set resp.http.Retry-After = "5"; synthetic( {"<!DOCTYPE html> <html> <head> <title>"} + resp.status + " " + resp.reason + {"</title> </head> <body> <h1>Error "} + resp.status + " " + resp.reason + {"</h1> <p>"} + resp.reason + {"</p> <h3>Guru Meditation:</h3> <p>XID: "} + req.xid + {"</p> <hr> <p>Varnish cache server</p> </body> </html> "} ); return (deliver); } Send custom HTML
112. sub vcl_backend_fetch { return (fetch); } sub vcl_backend_response { if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Surrogate-control ~ "no-store" || (!beresp.http.Surrogate-Control && beresp.http.Cache-Control ~ "no-cache|no-store| private") || beresp.http.Vary == "*") { set beresp.ttl = 120s; set beresp.uncacheable = true; } return (deliver); } Send backend request Receive backend response TTL State
113. sub vcl_backend_error { set beresp.http.Content-Type = "text/html; charset=utf-8"; set beresp.http.Retry-After = "5"; synthetic( {"<!DOCTYPE html> <html> <head> <title>"} + beresp.status + " " + beresp.reason + {"</ title> </head> <body> <h1>Error "} + beresp.status + " " + beresp.reason + {"</ h1> <p>"} + beresp.reason + {"</p> <h3>Guru Meditation:</h3> <p>XID: "} + bereq.xid + {"</p> <hr> <p>Varnish cache server</p> </body> </html> "} ); return (deliver); } Send custom HTML on backend error
114. Minimal VCL
115. vcl 4.0; backend default { .host = "127.0.0.1"; .port = "8080"; } Minimal VCL
116. vcl 4.0; backend default { .host = "127.0.0.1"; .port = "80"; .max_connections = 300; .first_byte_timeout = 300s; .connect_timeout = 5s; .between_bytes_timeout = 2s; } More backend work
117. vcl 4.0; backend default { .host = "127.0.0.1"; .port = "80"; .max_connections = 300; .first_byte_timeout = 300s; .connect_timeout = 5s; .between_bytes_timeout = 2s; .probe = { .url = "/"; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; } } Even more backend work
118. vcl 4.0; backend default { .host = "127.0.0.1"; .port = "80"; .max_connections = 300; .first_byte_timeout = 300s; .connect_timeout = 5s; .between_bytes_timeout = 2s; .probe = { .request = "HEAD / HTTP/1.1" "Host: localhost" "Connection: close" "User-Agent: Varnish Health Probe"; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; } } Even more backend work
119. Load balancing
120. vcl 4.0; import directors; backend server1 { .host = “1.2.3.4”; .port = "80"; .max_connections = 300; .first_byte_timeout = 300s; .connect_timeout = 5s; .between_bytes_timeout = 2s; .probe = { .request = "HEAD / HTTP/1.1" "Host: localhost" "Connection: close" "User-Agent: Varnish Health Probe"; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; } } Loadbalancing: server 1
121. backend server2 { .host = “1.2.3.5”; .port = "80"; .max_connections = 300; .first_byte_timeout = 300s; .connect_timeout = 5s; .between_bytes_timeout = 2s; .probe = { .request = "HEAD / HTTP/1.1" "Host: localhost" "Connection: close" "User-Agent: Varnish Health Probe"; .interval = 5s; .timeout = 1s; .window = 5; .threshold = 3; } } Loadbalancing: server 2
122. sub vcl_init { new vdir = directors.round_robin(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(); } Loadbalancing: round robin
123. sub vcl_init { new vdir = directors.random(); vdir.add_backend(server1,2); vdir.add_backend(server2,3); } sub vcl_recv { set req.backend_hint = vdir.backend(); } Loadbalancing: random
124. sub vcl_init { new vdir = directors.fallback(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(); } Loadbalancing: fallback
125. sub vcl_init { new vdir = directors.hash(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(req.url); } Loadbalancing: URL hash
126. sub vcl_init { new vdir = directors.hash(); vdir.add_backend(server1); vdir.add_backend(server2); } sub vcl_recv { set req.backend_hint = vdir.backend(client.identity); } Loadbalancing: IP hash
127. sub vcl_recv { if(req.url ~ “^/products”) { set req.backend_hint = server1; } else { set req.backend_hint = server2; } } Conditional loadbalacning
128. Normalize
129. vcl 4.0; import std; sub vcl_recv { set req.http.Host = regsub(req.http.Host, ":[0-9]+", ""); set req.url = std.querysort(req.url); if (req.url ~ "\#") { set req.url = regsub(req.url, "\#.*$", ""); } if (req.url ~ "\?$") { set req.url = regsub(req.url, "\?$", ""); } if (req.restarts == 0) { if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } } } Normalize
130. Static assets
131. vcl 4.0; sub vcl_recv { if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac| flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf| ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf| txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") { unset req.http.Cookie; return (hash); } } sub vcl_backend_response { if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac| flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf| ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf| txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") { unset beresp.http.set-cookie; } if (bereq.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4| mpeg|mpg|ogg|ogm|opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") { unset beresp.http.set-cookie; set beresp.do_stream = true; set beresp.do_gzip = false; } } Cache static assets
132. Do you really want to cache static assets?
133. Nginx or Apache can be fast enough for that
134. vcl 4.0; import std; sub vcl_recv { if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac| flv|gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf| ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf| txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") { unset req.http.Cookie; return (pass); } } Don’t cache static assets
135. URL whitelist/blacklist
136. sub vcl_recv { if (req.url ~ "^/status\.php$" || req.url ~ "^/update\.php$" || req.url ~ "^/admin$" || req.url ~ "^/admin/.*$" || req.url ~ "^/user$" || req.url ~ "^/user/.*$" || req.url ~ "^/flag/.*$" || req.url ~ "^.*/ajax/.*$" || req.url ~ "^.*/ahah/.*$") { return (pass); } } URL blacklist
137. sub vcl_recv { if (req.url ~ "^/products/?" return (hash); } } URL whitelist
138. Those damn cookies again!
139. vcl 4.0; sub vcl_recv { set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", ""); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; } } Remove tracking cookies
140. vcl 4.0; sub vcl_recv { if (req.http.Cookie) { set req.http.Cookie = ";" + req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; } } } Only keep session cookie
141. sub vcl_recv { if (req.http.Cookie) { set req.http.Cookie = ";" + req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(language)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; return(pass); } return(hash); } } sub vcl_hash { hash_data(regsub( req.http.Cookie, "^.*language=([^;]*);*.*$", "\1" )); } Language cookie cache variation
142. Alternative language cache variation
143. sub vcl_hash { hash_data(req.http.Accept-Language); } Language cookie cache variation Or just send a “Vary:Accept-Language” header
144. sub vcl_hash { hash_data(req.http.Cookie); } Hash all cookies
145. Edge Side Includes
146. sub vcl_recv { set req.http.Surrogate-Capability = "key=ESI/1.0"; } sub vcl_backend_response { if (beresp.http.Surrogate-Control ~ "ESI/1.0") { unset beresp.http.Surrogate-Control; set beresp.do_esi = true; return(deliver); } } Edge Side Includes
147. <?php header("Cache-Control: public,must-revalidate,s-maxage=10"); echo "Date in the ESI tag: ".date('Y-m-d H:i:s').'<br />'; <?php header("Cache-Control: no-store"); header(“Surrogate-Control: content='ESI/1.0'"); echo '<esi:include src="/esi.php" />'.PHP_EOL; echo "Date in the main page: ".date('Y-m-d H:i:s').'<br />'; Main page ESI frame: esi.php Cached for 10 seconds Not cached
148. ESI_xmlerror No ESI processing, first char not '<'. (See feature esi_disable_xml_check) Expects HTML/ XML tags
149. -p feature=+esi_disable_xml_check Add as startup option
150. Control Time To Live
151. sub vcl_backend_response { set beresp.ttl = 3h; } Control Time To Live
152. sub vcl_backend_response { if (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Vary == "*") { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } Control Time To Live
153. Debugging
154. sub vcl_deliver { if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; } else { set resp.http.X-Cache = "MISS"; } } Debugging
155. Anonymize
156. sub vcl_deliver { unset resp.http.X-Powered-By; unset resp.http.Server; unset resp.http.X-Drupal-Cache; unset resp.http.X-Varnish; unset resp.http.Via; unset resp.http.Link; unset resp.http.X-Generator; } Anonymize
157. Purging
158. acl purge { "localhost"; "127.0.0.1"; "::1"; } sub vcl_recv { if (req.method == "PURGE") { if (!client.ip ~ purge) { return (synth(405, “Not allowed.”)); } return (purge); } } Purging
159. acl purge { "localhost"; "127.0.0.1"; "::1"; } sub vcl_backend_response { set beresp.http.x-url = bereq.url; set beresp.http.x-host = bereq.http.host; } sub vcl_deliver { unset resp.http.x-url; unset resp.http.x-host; } sub vcl_recv { if (req.method == "PURGE") { if (!client.ip ~ purge) { return (synth(405, "Not allowed")); } if(req.http.x-purge-regex) { ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url ~ " + req.http.x-purge-regex); } else { ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url == " + req.url); } return (synth(200, "Purged")); } } Banning
160. Banning curl -XPURGE -H "x-purge-regex:/products" "http://example.com" curl -XPURGE "http://example.com/products"
161. Banning through Varnishadm
162. varnishadm> ban obj.http.x-host == varnishtraining.dev && obj.http.x-url ~ /exercises/8 Varnishadm banning
163. varnishtop varnishtop -i ReqURL varnishtop -c -i VCL_call varnishtop -b -i VCL_call varnishtop -i ReqUrl -q "VCL_call eq 'HIT'" varnishtop -I VCL_call:PASS -I VCL_call:MISS -i ReqUrl -q "VCL_call eq 'MISS' or VCL_call eq 'PASS'" varnishtop -i ReqUrl -q "Timestamp:Process[2] > 0.8 Varnishtop filtering
164. Header-based invalidation
165. Header-based invalidation If-Modified-Since: Tue, 14 Jun 2016 11:49:18 GMT Last-Modified: Tue, 14 Jun 2016 11:49:32 GMT Cache-Control: max-age=10 HTTP 200: OK HTTP 304: Not modified
166. Header-based invalidation If-None-Match: 57601698ae1e2 Etag: 57601698ae1e2 Cache-Control: max-age=10 HTTP 200: OK HTTP 304: Not modified
167. beresp.keep Keep serving stale object while conditional backend request happens
168. Grace mode
169. beresp.grace Keep servering stale object while fetching data from backend Avoid excessive request queueing
170. sub vcl_backend_response { set beresp.grace = 6h; } Grace mode Monitoring & logging
171. Varnishstat Realtime statistics
173. usage: varnishstat [-1lV] [-f field] [-t seconds|<off>] [-n varnish_name] [-N filename] -1 # Print the statistics to stdout. -f field # Field inclusion glob # If it starts with '^' it is used as an exclusion list. -l # Lists the available fields to use with the -f option. -n varnish_name # The varnishd instance to get logs from. -N filename # Filename of a stale VSM instance. -t seconds|<off> # Timeout before returning error on initial VSM connection. -V # Display the version number and exit. -x # Print statistics to stdout as XML. -j # Print statistics to stdout as JSON. Varnishstat usage
174. ~# varnishstat -f MAIN.cache_hit -1 MAIN.cache_hit 13049135 5.39 Cache hits Varnishstat usage ~# varnishstat -f MAIN.cache_hit -j -1 { "timestamp": "2016-06-14T16:10:32", "MAIN.cache_hit": { "description": "Cache hits", "type": "MAIN", "flag": "c", "format": "i", "value": 13050992 } }
175. Varnishstat usage ~# varnishstat -f MAIN.n_object -f MAIN.n_lru_nuked -j { "timestamp": "2016-06-14T16:14:49", "MAIN.n_object": { "description": "object structs made", "type": "MAIN", "flag": "g", "format": "i", "value": 46295 }, "MAIN.n_lru_nuked": { "description": "Number of LRU nuked objects", "type": "MAIN", "flag": "g", "format": "i", "value": 0 } }
176. varnishstat -f MAIN.cache*
177. ✓ Session ✓ Client ✓ Uptime ✓ Hit/miss ✓ Backend ✓ Fetch ✓ Threading ✓ Cache objects ✓ Memory ✓ Invalidation Varnishstat counters
178. Varnishlog In-memory logs of Varnish activity
179. * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1466002470.571781 0.000000 0.000000 - Timestamp Req: 1466002470.571781 0.000000 0.000000 - ReqStart 10.10.10.1 51519 - ReqMethod GET - ReqURL /exercises/1/max.php - ReqProtocol HTTP/1.1 - ReqHeader Host: varnishtraining.dev - ReqHeader Connection: keep-alive - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 - ReqHeader Accept-Encoding: gzip, deflate, sdch - ReqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6 - ReqHeader X-Forwarded-For: 10.10.10.1 - VCL_call RECV - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate, sdch - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1466002470.574188 0.002407 0.002407 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx/1.10.1 - RespHeader Date: Wed, 15 Jun 2016 14:54:30 GMT - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader Cache-Control: max-age= 10 - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1466002470.574226 0.002445 0.000038 - RespHeader Accept-Ranges: bytes - RespHeader Content-Length: 106 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1466002470.574287 0.002507 0.000061 - ReqAcct 416 0 416 256 106 362 - End Client info: a MISS
180. * << BeReq >> 3 - Begin bereq 2 fetch - Timestamp Start: 1466002470.571910 0.000000 0.000000 - BereqMethod GET - BereqURL /exercises/1/max.php - BereqProtocol HTTP/1.1 - BereqHeader Host: varnishtraining.dev - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 - BereqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6 - BereqHeader X-Forwarded-For: 10.10.10.1 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 3 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 25 boot.default 127.0.0.1 8080 127.0.0.1 35536 - Timestamp Bereq: 1466002470.571998 0.000088 0.000088 - Timestamp Beresp: 1466002470.574088 0.002178 0.002090 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx/1.10.1 - BerespHeader Date: Wed, 15 Jun 2016 14:54:30 GMT - BerespHeader Content-Type: text/html; charset=UTF-8 - BerespHeader Transfer-Encoding: chunked - BerespHeader Connection: keep-alive - BerespHeader Cache-Control: max-age= 10 - TTL RFC 10 10 -1 1466002471 1466002471 1466002470 0 10 - VCL_call BACKEND_RESPONSE - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx/1.10.1 - ObjHeader Date: Wed, 15 Jun 2016 14:54:30 GMT - ObjHeader Content-Type: text/html; charset=UTF-8 - ObjHeader Cache-Control: max-age= 10 - Fetch_Body 2 chunked stream - BackendReuse 25 boot.default - Timestamp BerespBody: 1466002470.574162 0.002252 0.000073 - Length 106 - BereqAcct 420 0 420 198 106 304 - End Backend info: fetching
181. * << Request >> 7 - Begin req 4 rxreq - Timestamp Start: 1466002762.802285 0.000000 0.000000 - Timestamp Req: 1466002762.802285 0.000000 0.000000 - ReqStart 10.10.10.1 51560 - ReqMethod GET - ReqURL /exercises/1/max.php - ReqProtocol HTTP/1.1 - ReqHeader Host: varnishtraining.dev - ReqHeader Connection: keep-alive - ReqHeader Cache-Control: max-age=0 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 - ReqHeader Accept-Encoding: gzip, deflate, sdch - ReqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6 - ReqHeader X-Forwarded-For: 10.10.10.1 - VCL_call RECV - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate, sdch - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - Hit 6 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx/1.10.1 - RespHeader Date: Wed, 15 Jun 2016 14:59:21 GMT - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader Cache-Control: max-age= 10 - RespHeader X-Varnish: 7 6 - RespHeader Age: 1 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1466002762.802336 0.000051 0.000051 - RespHeader Accept-Ranges: bytes - RespHeader Content-Length: 106 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1466002762.802414 0.000129 0.000078 - ReqAcct 442 0 442 258 106 364 - End Client info: a HIT
182. varnishlog varnishlog -i ReqURL varnishlog -g raw -i ReqURL varnishlog -g request -i ReqURL -i ReqHeader varnishlog -c varnishlog -b varnishlog -i ReqUrl -q "VCL_call eq 'HIT'" varnishlog -I VCL_call:PASS -I VCL_call:MISS -i ReqUrl -q "VCL_call eq 'MISS' or VCL_call eq 'PASS'" varnishlog -i ReqUrl -q "Timestamp:Process[2] > 0.8 Varnishlog filtering
183. Varnishtop Incremental in-memory Varnish logs
184. varnishtop varnishtop -i ReqURL varnishtop -c -i VCL_call varnishtop -b -i VCL_call varnishtop -i ReqUrl -q "VCL_call eq 'HIT'" varnishtop -I VCL_call:PASS -I VCL_call:MISS -i ReqUrl -q "VCL_call eq 'MISS' or VCL_call eq 'PASS'" varnishtop -i ReqUrl -q "Timestamp:Process[2] > 0.8 Varnishtop filtering
186. VCL & ban module