Slide 1
Slide 1 text
Varnish workshop
By Thijs Feryn
Slide 2
Slide 2 text
Slow websites suck
Slide 3
Slide 3 text
Web performance is
an essential part of
the user experience
Slide 4
Slide 4 text
Slow ~ Down
Slide 5
Slide 5 text
Saturated
market
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Heavy load
Slide 11
Slide 11 text
Mo money
Mo servers
Slide 12
Slide 12 text
Refactor slow code
Write fast code
Slide 13
Slide 13 text
After a while you
hit the limits
Slide 14
Slide 14 text
Cache
Slide 15
Slide 15 text
Don’t
recompute
if the data
hasn’t
changed
Slide 16
Slide 16 text
Caching is not a
compensation for
poor code
Slide 17
Slide 17 text
Caching is an
essential
architectural
strategy
Slide 18
Slide 18 text
Hi, I’m Thijs
Slide 19
Slide 19 text
I’m
@ThijsFeryn
on Twitter
Slide 20
Slide 20 text
I’m an
Evangelist
At
Slide 21
Slide 21 text
I’m an
Evangelist
At
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
Normally
User Server
Slide 25
Slide 25 text
With Varnish
User Varnish Server
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
✓ Page cache
✓ Dynamic page cache
✓ Render cache
✓ BigPipe
✓ BigPipe sessionless
Slide 29
Slide 29 text
Install & configure
Slide 30
Slide 30 text
https://packagecloud.io/varnishcache/varnish60
https://packagecloud.io/varnishcache/varnish5
https://packagecloud.io/varnishcache/varnish41
Slide 31
Slide 31 text
$ curl -s https://packagecloud.io/install/repositories/
varnishcache/varnish6/script.deb.sh | sudo bash
$ apt-get install varnish
Debian & Ubuntu
Slide 32
Slide 32 text
$ apt-cache policy varnish
varnish:
Installed: 6.0.0-1~stretch
Candidate: 6.0.0-1~stretch
Version table:
*** 6.0.0-1~stretch 500
500 https://packagecloud.io/varnishcache/varnish60/debian stretch/main
amd64 Packages
100 /var/lib/dpkg/status amd64 Packages
5.0.0-7+deb9u1 500
500 http://deb.debian.org/debian stretch/main amd64 Packages
500 http://security.debian.org stretch/updates/main amd64 Packages
Slide 33
Slide 33 text
$ apt-get install varnish=6.0.0-1~stretch
Slide 34
Slide 34 text
$ curl -s https://packagecloud.io/install/repositories/
varnishcache/varnish60/script.rpm.sh | sudo bash
$ yum install varnish
RHEL & CentOS
Slide 35
Slide 35 text
Config
file
Slide 36
Slide 36 text
-a: binding address & port
-T: admin binding
-f: VCL file
-S: secret file
-s: storage
-j: jailing
-l: shared memory log size
-t: default TTL
-p: runtime parameters
Options
Slide 37
Slide 37 text
DAEMON_OPTS="-j unix,user=www-data \
-a main=0000:80 \
-a proxy=0.0.0.0:81,PROXY \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-l 100m,10m \
-t 60 \
-p feature=+esi_disable_xml_check \
-p feature=+http2 \
-p connect_timeout=20s \
-p first_byte_timeout=100s \
-p between_bytes_timeout=5s \
-s malloc,3g"
Slide 38
Slide 38 text
The backend
Slide 39
Slide 39 text
Other
port
Other IP
Other
server
Slide 40
Slide 40 text
✓In VCL file
✓In config file using "-b"
Link backend to Varnish
Slide 41
Slide 41 text
DAEMON_OPTS="-j unix,user=www-data \
-a main=0.0.0.0:80 \
-a proxy=0.0.0.0:81,PROXY \
-T localhost:6082 \
-b 127.0.0.1:8080 \
-S /etc/varnish/secret \
-s malloc,3g"
Can't
have -f
In config file
Slide 42
Slide 42 text
vcl 4.0;
backend default {
.host = "127.0.0.1";
.port = "8080";
}
In VCL file
Minimal
VCL
Slide 43
Slide 43 text
What about TLS/SSL?
Slide 44
Slide 44 text
✓HAProxy
✓Nginx
✓Pound
✓Hitch
Terminate TLS/SSL
Slide 45
Slide 45 text
HAProxy
Varnish
Nginx
HTTP & HTTPS
HTTP
HTTP
Certificates go here
Public
Slide 46
Slide 46 text
How do you know if it
was HTTP or HTTPS?
Slide 47
Slide 47 text
X-Forwarded-Proto: https
X-Forwarded-Proto: http
Slide 48
Slide 48 text
What’s the real IP?
Slide 49
Slide 49 text
X-Forwarded-For
Slide 50
Slide 50 text
Nginx
Public
172.18.0.1
Client IP: 172.18.0.1
X-Forwarded-For:
Slide 51
Slide 51 text
Varnish
172.18.0.8
Nginx
Public
172.18.0.1
Client IP: 172.18.0.8
X-Forwarded-For: 172.18.0.1
Slide 52
Slide 52 text
HAProxy
172.18.0.3
Varnish
172.18.0.8
Nginx
Public
172.18.0.1
Client IP: 172.18.0.8
X-Forwarded-For: 172.18.0.3
Slide 53
Slide 53 text
PROXY Protocol
Slide 54
Slide 54 text
Add original IP as
TCP preamble
Slide 55
Slide 55 text
DAEMON_OPTS="-j unix,user=www-data \
-a :80 \
-a :81,PROXY \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-l 100m,10m \
-t 60 \
-p feature=+esi_disable_xml_check \
-p connect_timeout=20s \
-p first_byte_timeout=100s \
-p between_bytes_timeout=5s \
-s malloc,3g"
Slide 56
Slide 56 text
HAProxy
172.18.0.3
Varnish
172.18.0.8
Nginx
Public
172.18.0.1
Client IP: 172.18.0.8
X-Forwarded-For: 172.18.0.1
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
Slide 59
Slide 59 text
// Initialize proxy settings.
if ($settings->get('reverse_proxy', FALSE)) {
$ip_header = $settings->get('reverse_proxy_header', 'X_FORWARDED_FOR');
$request::setTrustedHeaderName($request::HEADER_X_FORWARDED_FOR, $ip_header);
$proto_header = $settings->get('reverse_proxy_proto_header', 'X_FORWARDED_PROTO');
$request::setTrustedHeaderName($request::HEADER_X_FORWARDED_PROTO, $proto_header);
$host_header = $settings->get('reverse_proxy_host_header', 'X_FORWARDED_HOST');
$request::setTrustedHeaderName($request::HEADER_X_FORWARDED_HOST, $host_header);
$port_header = $settings->get('reverse_proxy_port_header', 'X_FORWARDED_PORT');
$request::setTrustedHeaderName($request::HEADER_X_FORWARDED_PORT, $port_header);
$forwarded_header = $settings->get('reverse_proxy_forwarded_header', 'FORWARDED');
$request::setTrustedHeaderName($request::HEADER_FORWARDED, $forwarded_header);
$proxies = $settings->get('reverse_proxy_addresses', []);
if (count($proxies) > 0) {
$request::setTrustedProxies($proxies, Request::HEADER_X_FORWARDED_ALL |
Request::HEADER_FORWARDED);
}
}
Drupal\Core\StackMiddleware\ReverseProxyMiddleware
Slide 60
Slide 60 text
There are rules
Slide 61
Slide 61 text
✓Idempotence
✓State
✓Expiration
✓Conditional requests
✓Cache variations
Varnish speaks HTTP
Slide 62
Slide 62 text
Idempotence
Execute
multiple
times
Result
doesn't
change
Slide 63
Slide 63 text
✓ GET
✓ HEAD
- POST
- PUT
- DELETE
- PATCH
Idempotence
Result
changes
Result
doesn't
change
Don't
cache
Slide 64
Slide 64 text
State
Slide 65
Slide 65 text
State
~
user specific
data
Cookies
Auth
headers
Slide 66
Slide 66 text
About cookies
Slide 67
Slide 67 text
✓ Request header
✓ Sent by client (~browser)
✓ Sent on every request
✓ Describes state
✓ Is not cached
✓ 3rd party tracking cookies vs own
cookies
About cookies
Cookie: key=value;key2=value2
Slide 68
Slide 68 text
✓ Response header
✓ Sent by backend server
✓ Only when backend is called
✓ Changes state
✓ Is not cached
✓ Blacklisted for 120s (by default)
About cookies
Set-Cookie: key=another_value
Slide 69
Slide 69 text
Time
To
Live
Slide 70
Slide 70 text
1.VCL
2.s-maxage
3.max-age
4.expires
5.Default value 120s
TTL Order
Slide 71
Slide 71 text
Cache-Control:
public,
max-age=1000,
s-maxage=3600
Slide 72
Slide 72 text
Use TTL not to cache
Cache-Control: max-age=0
Cache-Control: s-maxage=0
Cache-Control: private
Cache-Control: no-cache
Cache-Control: no-store
Expires: Fri, 1 Jan 1971 00:00:00 GMT
Slide 73
Slide 73 text
Age
Age: 10
How old is the
cached object?
Slide 74
Slide 74 text
Real cache duration
max-age - Age
s-maxage - Age
Slide 75
Slide 75 text
Don’t set the Age
header yourself!
Slide 76
Slide 76 text
Conditional
requests
Slide 77
Slide 77 text
Only fetch
payload that has
changed
Slide 78
Slide 78 text
Otherwise:
HTTP/1.1 304 Not Modified
Slide 79
Slide 79 text
✓ Conserve bandwidth
✓ Reduce load on the backend when
revalidating
Conditional requests
Slide 80
Slide 80 text
Conditional requests
HTTP/1.1 200 OK
Host: localhost
Etag: 7c9d70604c6061da9bb9377d3f00eb27
Content-type: text/html; charset=UTF-8
Hello world output
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
Slide 81
Slide 81 text
Conditional requests
HTTP/1.0 304 Not Modified
Host: localhost
Etag: 7c9d70604c6061da9bb9377d3f00eb27
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-None-Match:
7c9d70604c6061da9bb9377d3f00eb27
Slide 82
Slide 82 text
Conditional requests
HTTP/1.1 200 OK
Host: localhost
Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT
Content-type: text/html; charset=UTF-8
Hello world output
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
Slide 83
Slide 83 text
Conditional requests
HTTP/1.0 304 Not Modified
Host: localhost
Last-Modified: Fri, 22 Jul 2016 10:11:16 GMT
GET / HTTP/1.1
Host: localhost
User-Agent: curl/7.48.0
If-Last-Modified: Fri, 22 Jul 2016 10:11:16
GMT
Slide 84
Slide 84 text
Store etag or
modification date
Slide 85
Slide 85 text
Validate early on
Slide 86
Slide 86 text
Exit quickly
Slide 87
Slide 87 text
No content
Slide 88
Slide 88 text
$last_modified = $response->getLastModified();
if ($last_modified) {
// See if the client has provided the required HTTP headers.
$if_modified_since = $request->server->has('HTTP_IF_MODIFIED_SINCE') ? strtotime($request->server-
>get('HTTP_IF_MODIFIED_SINCE')) : FALSE;
$if_none_match = $request->server->has('HTTP_IF_NONE_MATCH') ? stripslashes($request->server-
>get('HTTP_IF_NONE_MATCH')) : FALSE;
if ($if_modified_since && $if_none_match
// etag must match.
&& $if_none_match == $response->getEtag()
// if-modified-since must match.
&& $if_modified_since == $last_modified->getTimestamp()) {
$response->setStatusCode(304);
$response->setContent(NULL);
// In the case of a 304 response, certain headers must be sent, and the
// remaining may not (see RFC 2616, section 10.3.5).
foreach (array_keys($response->headers->all()) as $name) {
if (!in_array($name, ['content-location', 'expires', 'cache-control', 'vary'])) {
$response->headers->remove($name);
}
}
}
}
Drupal\page_cache\StackMiddleware\PageCache
Slide 89
Slide 89 text
Varnish can revalidate
aynchronously
Slide 90
Slide 90 text
In both directions
User Varnish Server
304 Not Modified 304 Not Modified
Slide 91
Slide 91 text
And serve stale data while
that happens
Slide 92
Slide 92 text
Grace mode
Slide 93
Slide 93 text
Effective TTL = TTL + grace
Slide 94
Slide 94 text
Cache-Control: public, max-age=100, s-maxage=3600,
stale-while-revalidate=7200
Stale-While-Revalidate
Set grace mode
Slide 95
Slide 95 text
Variations
Slide 96
Slide 96 text
Cache is
not the
same for
everyone
Slide 97
Slide 97 text
✓ URL
✓ Hostname
✓ IP if hostname is not set
✓ Vary header
Basic variations
Slide 98
Slide 98 text
Vary header
Vary: Accept-Encoding
Vary: Accept-Language
Vary: Cookie
Watch out!
Slide 99
Slide 99 text
The flow
Slide 100
Slide 100 text
No content
Slide 101
Slide 101 text
No content
Slide 102
Slide 102 text
Varnish
Configuration
Language
Slide 103
Slide 103 text
DSL compiled
and linked as
shared object
Slide 104
Slide 104 text
/etc/varnish/default.vcl
Slide 105
Slide 105 text
Hooks
&
subroutines
Slide 106
Slide 106 text
✓vcl_recv: receive request
✓vcl_hash: compose cache key
✓vcl_miss: not found in cache
✓vcl_hit: found in cache
✓vcl_pass: don’t store in cache
✓vcl_pipe: bypass cache
✓vcl_backend_fetch: connect to
backend
✓vcl_backend_response: response from
backend
✓vcl_backend_error: backend fetch
failed
Slide 107
Slide 107 text
✓vcl_purge: after successful purge
✓vcl_synth: send synthetic output
✓vcl_deliver: return data to client
✓vcl_init: initialize VMODs
✓vcl_fini: discard VMODs
✓vcl_fail: stop execution
Slide 108
Slide 108 text
Actions
Slide 109
Slide 109 text
✓ hash: lookup in cache
✓ pass: don't cache
✓ synth: synthetic HTML output
✓ pipe: bypass cache
✓ purge: remove from cache
VCL_RECV
Slide 110
Slide 110 text
✓ synth: synthetic HTML output
✓ pipe: bypass cache
VCL_PIPE
Slide 111
Slide 111 text
✓ fetch: fetch data from
backend, don't cache
✓ restart: restart transaction
✓ synth: synthetic HTML output
VCL_PASS
Slide 112
Slide 112 text
✓ deliver: send cached object
✓ miss: synchronous refresh
despite hit
✓ pass: fetch data from
backend despite hit, don't
cache
✓ restart: restart transaction
✓ synth: synthetic HTML output
VCL_HIT
Slide 113
Slide 113 text
✓ fetch: fetch data from
backend
✓ pass: fetch data from
backend, don't cache
✓ restart: restart transaction
✓ synth: synthetic HTML output
VCL_MISS
Slide 114
Slide 114 text
✓ lookup: look for cached
object by using hash key
VCL_HASH
Slide 115
Slide 115 text
✓ restart: restart transaction
✓ synth: synthetic HTML output
VCL_PURGE
Slide 116
Slide 116 text
✓ deliver: deliver object to
client
✓ restart: restart transaction
✓ synth: synthetic HTML output
VCL_DELIVER
Slide 117
Slide 117 text
✓ deliver: deliver synthetic
output to client
✓ restart: restart transaction
VCL_SYNTH
Slide 118
Slide 118 text
✓ fetch: fetch object from
backend
✓ abandon: abandon request
and send HTTP 503 error
VCL_BACKEND_FETCH
Slide 119
Slide 119 text
✓ deliver: send fetched data to
client
✓ abandon: abandon request
and send HTTP 503 error
✓ retry: retry backend request
VCL_BACKEND_RESPONSE
Slide 120
Slide 120 text
✓ deliver: send fetched data to
client
✓ retry: retry backend request
VCL_BACKEND_ERROR
Slide 121
Slide 121 text
Typical flows
Slide 122
Slide 122 text
✓ vcl_recv: hash
✓ vcl_hash: lookup
✓ vcl_miss: fetch
✓ vcl_backend_request: fetch
✓ vcl_backend_response: deliver
✓ vcl_deliver: deliver
MISS
Slide 123
Slide 123 text
✓ vcl_recv: hash
✓ vcl_hash: lookup
✓ vcl_hit: deliver
✓ vcl_deliver: deliver
HIT
Slide 124
Slide 124 text
✓ vcl_recv: pass
✓ vcl_pass: fetch
✓ vcl_backend_request: fetch
✓ vcl_backend_response: deliver
✓ vcl_deliver: deliver
PASS
Slide 125
Slide 125 text
Objects
Slide 126
Slide 126 text
✓ req: incoming request object
✓ req_top: top level esi request
✓ bereq: request object to send to
backend
✓ beresp: backend response
✓ resp: response to send back to
client
✓ obj: cached object
✓ client: client information
✓ server: server information
✓ local: local TCP information
✓ remote: remote TCP information
✓ storage: storage information
VCL Objects
Slide 127
Slide 127 text
Variables
Slide 128
Slide 128 text
✓ req.url
✓ req.http.host
✓ req.http.user-agent
✓ req.backend_hint
✓ req.method
✓ …
req variables
Slide 129
Slide 129 text
✓ bereq.backend
✓ bereq.http.user-agent
✓ bereq.method
✓ bereq.url
✓ …
bereq variables
Slide 130
Slide 130 text
✓ beresp.age
✓ beresp.backend.ip
✓ beresp.backend.name
✓ beresp.do_esi
✓ beresp.grace
✓ beresp.keep
✓ beresp.http.set-cookie
✓ beresp.ttl
✓ beresp.status
✓ beresp.uncacheable
beresp variables
Slide 131
Slide 131 text
✓ client.ip
✓ client.identity
client variables
local variables
✓ local.ip
remote variables
✓ remote.ip
Slide 132
Slide 132 text
✓ obj.age
✓ obj.grace
✓ obj.hits
✓ obj.http.cache-control
✓ obj.reason
✓ obj.status
✓ obj.ttl
object variables
Slide 133
Slide 133 text
✓ resp.http.user-agent
✓ resp.is_streaming
✓ resp.reason
✓ resp.status
resp variables
Slide 134
Slide 134 text
Default
behaviour
Slide 135
Slide 135 text
vcl 4.0;
sub vcl_recv {
if (req.method != "GET" &&
req.method != "HEAD" &&
req.method != "PUT" &&
req.method != "POST" &&
req.method != "TRACE" &&
req.method != "OPTIONS" &&
req.method != “PATCH" &&
req.method != "DELETE") {
return (pipe);
}
if (req.method != "GET" && req.method != "HEAD") {
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
return (pass);
}
return (hash);
}
Idempotence
State
Action
Receive
request
Slide 136
Slide 136 text
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
return (lookup);
}
Lookup in
cache
Variations
Action
Slide 137
Slide 137 text
sub vcl_purge {
return (synth(200, "Purged"));
}
sub vcl_hit {
if (obj.ttl >= 0s) {
return (deliver);
}
if (obj.ttl + obj.grace > 0s) {
return (deliver);
}
return (miss);
}
sub vcl_miss {
return (fetch);
}
sub vcl_deliver {
return (deliver);
}
Remove
from cache
Found in
cache
Not found in
cache
Return
HTTP response to
client
Slide 138
Slide 138 text
sub vcl_synth {
set resp.http.Content-Type = "text/html; charset=utf-8";
set resp.http.Retry-After = "5";
synthetic( {"
"} + resp.status + " " + resp.reason + {"
Error "} + resp.status + " " + resp.reason + {"
"} + resp.reason + {"
Guru Meditation:
XID: "} + req.xid + {"
Varnish cache server
"} ); return (deliver); } Send custom HTMLSlide 139
Slide 139 text
sub vcl_backend_fetch {
return (fetch);
}
sub vcl_backend_response {
if (bereq.uncacheable) {
return (deliver);
} else if (beresp.ttl <= 0s ||
beresp.http.Set-Cookie ||
beresp.http.Surrogate-control ~ "no-store" ||
(!beresp.http.Surrogate-Control &&
beresp.http.Cache-Control ~ "no-cache|no-store|private") ||
beresp.http.Vary == "*") {
set beresp.ttl = 120s;
set beresp.uncacheable = true;
}
return (deliver);
}
Send
backend
request
Receive
backend
response
TTL
State
Receive
backend
response
Slide 140
Slide 140 text
sub vcl_backend_error {
set beresp.http.Content-Type = "text/html; charset=utf-8";
set beresp.http.Retry-After = "5";
synthetic( {"
"} + beresp.status + " " + beresp.reason + {"
Error "} + beresp.status + " " + beresp.reason + {"
"} + beresp.reason + {"
Guru Meditation:
XID: "} + bereq.xid + {"
Varnish cache server
"} ); return (deliver); } Send custom HTML on backend errorSlide 141
Slide 141 text
In an ideal world
Slide 142
Slide 142 text
HTTP best practices
>
custom VCL
Slide 143
Slide 143 text
Reality
sucks
Slide 144
Slide 144 text
No content
Slide 145
Slide 145 text
No content
Slide 146
Slide 146 text
Don’t trust the end-user
Slide 147
Slide 147 text
Cache-control ?
Slide 148
Slide 148 text
Legacy
Slide 149
Slide 149 text
Write VCL
Slide 150
Slide 150 text
Normalize
Slide 151
Slide 151 text
set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
set req.url = std.querysort(req.url);
if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|
cof|siteurl)=") {
set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|
utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|
utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
set req.url = regsub(req.url, "\?&", "?");
set req.url = regsub(req.url, "\?$", "");
}
if (req.url ~ "\#") {
set req.url = regsub(req.url, "\#.*$", "");
}
if (req.url ~ "\?$") {
set req.url = regsub(req.url, "\?$", "");
}
Normalize
Slide 152
Slide 152 text
Static assets
Slide 153
Slide 153 text
vcl 4.0;
sub vcl_recv {
if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|jpeg|
jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|rar|rtf|
svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)
(\?.*)?$") {
unset req.http.Cookie;
return (hash);
}
}
sub vcl_backend_response {
if (bereq.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|gif|gz|ico|
jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|ogm|opus|pdf|png|ppt|pptx|
rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|
zip)(\?.*)?$") {
unset beresp.http.set-cookie;
}
if (bereq.url ~ "^[^?]*\.(7z|avi|bz2|flac|flv|gz|mka|mkv|mov|mp3|mp4|mpeg|mpg|ogg|ogm|
opus|rar|tar|tgz|tbz|txz|wav|webm|xz|zip)(\?.*)?$") {
unset beresp.http.set-cookie;
set beresp.do_stream = true;
set beresp.do_gzip = false;
}
}
Cache static assets
Slide 154
Slide 154 text
Do you really
want to cache
static assets?
Slide 155
Slide 155 text
Nginx or
Apache can
be fast
enough for
that
Slide 156
Slide 156 text
vcl 4.0;
import std;
sub vcl_recv {
if (req.url ~ "^[^?]*\.(7z|avi|bmp|bz2|css|csv|doc|docx|eot|flac|flv|
gif|gz|ico|jpeg|jpg|js|less|mka|mkv|mov|mp3|mp4|mpeg|mpg|odt|otf|ogg|
ogm|opus|pdf|png|ppt|pptx|rar|rtf|svg|svgz|swf|tar|tbz|tgz|ttf|txt|txz|
wav|webm|webp|woff|woff2|xls|xlsx|xml|xz|zip)(\?.*)?$") {
unset req.http.Cookie;
return (pass);
}
}
Don’t cache static assets
Slide 157
Slide 157 text
Blacklist/whitelist
Slide 158
Slide 158 text
sub vcl_recv {
if (req.url ~ "^/status\.php$" ||
req.url ~ "^/update\.php$" ||
req.url ~ "^/admin$" ||
req.url ~ "^/admin/.*$" ||
req.url ~ "^/flag/.*$" ||
req.url ~ "^.*/ajax/.*$" ||
req.url ~ "^.*/ahah/.*$") {
return (pass);
}
}
URL blacklist
Slide 159
Slide 159 text
sub vcl_recv {
if (req.url ~ "^/some-page/?"
return (hash);
}
}
URL whitelist
Slide 160
Slide 160 text
if (beresp.http.Surrogate-Control ~ "BigPipe/1.0") {
set beresp.do_stream = true;
set beresp.ttl = 0s;
}
Don't cache BigPipe
responses
Slide 161
Slide 161 text
Those damn
cookies again!
Slide 162
Slide 162 text
vcl 4.0;
sub vcl_recv {
if (req.http.Cookie) {
set req.http.Cookie = ";" + req.http.Cookie;
set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
set req.http.Cookie = regsuball(req.http.Cookie,
";(SESS[a-z0-9]+|SSESS[a-z0-9]+|NO_CACHE)=", "; \1=");
set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
if (req.http.Cookie == "") {
unset req.http.Cookie;
}
else {
return (pass);
}
}
}
Only keep Drupal cookies
Slide 163
Slide 163 text
sub vcl_recv {
if (req.http.Cookie) {
set req.http.Cookie = ";" + req.http.Cookie;
set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";");
set req.http.Cookie = regsuball(req.http.Cookie, ";(language)=", "; \1=");
set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", "");
set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", "");
if (req.http.cookie ~ "^\s*$") {
unset req.http.cookie;
return(pass);
}
return(hash);
}
}
sub vcl_hash {
hash_data(regsub( req.http.Cookie, "^.*language=([^;]*);*.*$", "\1" ));
}
Language cookie cache variation
Slide 164
Slide 164 text
sub vcl_recv {
return(hash);
}
Cache, even with cookies
Slide 165
Slide 165 text
sub vcl_hash {
hash_data(req.http.Cookie);
}
Hash all cookies
Slide 166
Slide 166 text
Block caching
Slide 167
Slide 167 text
No content
Slide 168
Slide 168 text
Code
renders
single HTTP
response
Slide 169
Slide 169 text
Lowest
denominator:
no cache
Slide 170
Slide 170 text
Edge Side Includes
✓Placeholder
✓Parsed by Varnish
✓Output is a composition of blocks
✓State per block
✓TTL per block
Slide 171
Slide 171 text
sub vcl_recv {
set req.http.Surrogate-Capability = "key=ESI/1.0";
}
sub vcl_backend_response {
if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
unset beresp.http.Surrogate-Control;
set beresp.do_esi = true;
}
}
Edge Side Includes
Slide 172
Slide 172 text
';
'.PHP_EOL;
echo "Date in the main page: ".date('Y-m-d H:i:s').'
'; Main page ESI frame: esi.php Cached for 10 seconds Not cached
'; Main page ESI frame: esi.php Cached for 10 seconds Not cached
Slide 173
Slide 173 text
ESI_xmlerror No ESI processing, first char not '<'. (See
feature esi_disable_xml_check)
Expects HTML/
XML tags
Slide 174
Slide 174 text
-p feature=+esi_disable_xml_check
Add as
startup option
Slide 175
Slide 175 text
req_top
Get
information about
parent request in an
ESI call
Slide 176
Slide 176 text
vcl 4.0;
backend default {
.host = "nginx";
.port = "80";
}
sub vcl_recv {
if (req.esi_level > 0) {
set req.http.x-parent-url = req_top.url;
}
}
sub vcl_backend_response {
if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
unset beresp.http.Surrogate-Control;
set beresp.do_esi = true;
}
}
Get parent URL
Slide 177
Slide 177 text
ESI
vs
AJAX
Slide 178
Slide 178 text
✓ Server-side
✓ Standardized
✓ Processed on the
“edge”, no in the
browser
✓ Generally faster
Edge-Side Includes
- Sequential
- One fails, all fail
- Limited
implementation in
Varnish
- Not that common
Slide 179
Slide 179 text
✓ Client-side
✓ Common knowledge
✓ Parallel processing
✓ Graceful
degradation
AJAX
- Processed by the
browser
- Extra roundtrips
- Somewhat slower
Slide 180
Slide 180 text
Choose wisely
Slide 181
Slide 181 text
No content
Slide 182
Slide 182 text
Auto-placeholdering
Slide 183
Slide 183 text
BigPipe
Slide 184
Slide 184 text
Slide 185
Slide 185 text
[{"command":"insert","method":"replaceWith","selector":"[data-big-
pipe-placeholder-
id=\u0022callback=Drupal%5CCore%5CRender%5CElement%5CStatusMessages%
3A%3ArenderMessages\u0026args%5B0%5D\u0026token=_HAdUpwWmet0TOTe2PSi
JuMntExoshbm1kh2wQzzzAA\u0022]","data":"","settings":null}]
Slide 186
Slide 186 text
https://github.com/ThijsFeryn/drupal8_esi_placeholders
Slide 187
Slide 187 text
services:
esi:
class: Symfony\Component\HttpKernel\HttpCache\Esi
esi_placeholders.esi_subscriber:
class: Drupal\esi_placeholders\EventSubscriber\EsiSubscriber
arguments: ['@esi']
tags:
- { name: event_subscriber }
esi_placeholders.placeholder_strategy.esi:
class: Drupal\esi_placeholders\Render\Placeholder\EsiStrategy
arguments: ['@request_stack','@esi']
tags:
- { name: placeholder_strategy, priority: 0 }
Slide 188
Slide 188 text
esi_placeholders.esi_block_rendering:
path: '/esi/block'
defaults:
_controller:
'\Drupal\esi_placeholders\Controller\EsiController:returnEsiBlo
ckContent'
_title: 'ESI content'
requirements:
_permission: 'access content'
Slide 189
Slide 189 text
requestStack = $request_stack;
$this->esi = $esi;
}
/**
Slide 190
Slide 190 text
/**
* @param array $placeholders
* @return array
*/
public function processPlaceholders(array $placeholders)
{
$request = $this->requestStack->getCurrentRequest();
$overridenPlaceHolder = [];
foreach ($placeholders as $placeholder => $placeholder_elements) {
if (isset($placeholder_elements['#lazy_builder']) && $this->esi->hasSurrogateCapability($request)) {
$overridenPlaceHolder[$placeholder] = [
'#markup' =>
Markup::create(
$this->esi->renderIncludeTag(
'/esi/block/?'.
$this->generateBigPipePlaceholderId($placeholder,$placeholder_elements),
null,
false
)
)
];
}
}
return $overridenPlaceHolder;
}
}
Slide 191
Slide 191 text
[
'#lazy_builder' => [
$request->get('callback'),
$request->get('args'),
]
]
];
$output = \Drupal::service('renderer')->renderRoot($build);
$response = new CacheableResponse($output);
$response->addCacheableDependency(CacheableMetadata::createFromRenderArray($build));
return $response;
}
}
Slide 192
Slide 192 text
esi = $esi;
}
/**
* @param FilterResponseEvent $event
* @return \Symfony\Component\HttpFoundation\Response
*/
Slide 193
Slide 193 text
/**
* @param FilterResponseEvent $event
* @return \Symfony\Component\HttpFoundation\Response
*/
public function onRespond(FilterResponseEvent $event)
{
$request = $event->getRequest();
$response = $event->getResponse();
if($this->esi->hasSurrogateCapability($request)){
$this->esi->addSurrogateControl($response);
return $response;
}
}
/**
* {@inheritdoc}
*/
public static function getSubscribedEvents()
{
$events[KernelEvents::RESPONSE][] = ['onRespond', -10000];
return $events;
}
}
Slide 194
Slide 194 text
https://github.com/ThijsFeryn/drupal8_esi_placeholders
Slide 195
Slide 195 text
Breaking news isn't breaking
Slide 196
Slide 196 text
Purging
Slide 197
Slide 197 text
acl purgers {
"localhost";
"127.0.0.1";
"::1";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purgers) {
return (synth(405, "Method not allowed"));
}
return (purge);
}
}
Purging
Slide 198
Slide 198 text
Purging
curl -XPURGE "http://example.com/products"
Immediately
remove from
memory
Slide 199
Slide 199 text
acl purge {
"localhost";
"127.0.0.1";
"::1";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purgers) {
return (synth(405, "Method not allowed"));
}
if(req.http.x-purge-regex) {
ban("req.http.host == " + req.http.host + " && req.url ~ " + req.http.x-purge-regex);
} else {
ban("req.http.host == " + req.http.host + " && req.url == " + req.url);
}
return (synth(200, "Purged"));
}
}
Banning
Slide 200
Slide 200 text
Banning
curl -XPURGE "http://example.com/products"
curl -XPURGE -H "x-purge-regex:/products" "http://example.com"
Add to ban list, remove
on next request
Remove patterns via
“X-PURGE-REGEX”
Slide 201
Slide 201 text
Lurker-friendly bans
Slide 202
Slide 202 text
Ban lurker
Object
User
Varnish
Server
Sends HTTP
response
Response
stored in
object
Sends BAN to
Varnish
Ban lurker
thread
Ban list
Reads
ban list
Removes
object
ban req.http.host == localhost && req.url ~ /products
Slide 203
Slide 203 text
Ban lurker
ban req.http.host == localhost && req.url ~ /products
Lurker can’t match
request info
Lurker can only
match what’s in the
object
Next visitor
triggers cache
remove
Slide 204
Slide 204 text
acl purge {
"localhost";
"127.0.0.1";
"::1";
}
sub vcl_recv {
if (req.method == "PURGE") {
if (client.ip !~ purgers) {
return (synth(405, "Method not allowed"));
}
if(req.http.x-purge-regex) {
ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url ~ " + req.http.x-purge-regex);
} else {
ban("obj.http.host == " + req.http.host + " && obj.http.x-url == " + req.url);
}
return (synth(200, "Purged"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
Lurker-friendly bans
Store request info
in response object
Slide 205
Slide 205 text
Lurker-friendly bans
ban obj.http.x-host == localhost && obj.http.x-url ~ /products
Lurker can match
response info
Ban lurker
removes item from
cache async
Slide 206
Slide 206 text
Banning
through
Varnishadm
Slide 207
Slide 207 text
varnishadm> ban obj.http.x-host == localhost && obj.http.x-url ~ /products
Varnishadm banning
Slide 208
Slide 208 text
Banning
through
socket
Slide 209
Slide 209 text
Varnishadm is a
binary on top of
the socket
Slide 210
Slide 210 text
Varnishadm uses
the secret file to
authenticate
automatically
Slide 211
Slide 211 text
$ telnet varnish 6082
Connected to localhost.
Escape character is '^]'.
107 59
cohvooigdtqvkpwewhdxkqiwkfkpwsly
Authentication required.
auth 5a9c5722f31cc3c92f0e4616571624df7bddde2f8e42aaffe795dc80fb8c91dd
200 240
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.9.49-moby,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-5.2.0 revision 4c4875cbf
Type 'help' for command list.
Type 'quit' to close CLI session.
ban obj.http.x-host == localhost && obj.http.x-url ~ /products
200 0
Slide 212
Slide 212 text
Slide 213
Slide 213 text
$ telnet varnish 6082
Connected to localhost.
Escape character is '^]'.
107 59
cohvooigdtqvkpwewhdxkqiwkfkpwsly
Authentication required.
auth 5a9c5722f31cc3c92f0e4616571624df7bddde2f8e42aaffe795dc80fb8c91dd
200 240
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,4.9.49-moby,x86_64,-junix,-smalloc,-smalloc,-hcritbit
varnish-5.2.0 revision 4c4875cbf
Type 'help' for command list.
Type 'quit' to close CLI session.
ban obj.http.x-host == localhost && obj.http.x-url ~ /products
200 0
Slide 214
Slide 214 text
Secure your access the the admin socket
Slide 215
Slide 215 text
DAEMON_OPTS="-j unix,user=www-data \
-a :80 \
-a :81,PROXY \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-l 100m,10m \
-t 60 \
-p feature=+esi_disable_xml_check \
-p connect_timeout=20s \
-p first_byte_timeout=100s \
-p between_bytes_timeout=5s \
-s malloc,3g"
Slide 216
Slide 216 text
No content
Slide 217
Slide 217 text
✓purge
✓purge_purger_http
Required modules
Slide 218
Slide 218 text
No content
Slide 219
Slide 219 text
No content
Slide 220
Slide 220 text
No content
Slide 221
Slide 221 text
No content
Slide 222
Slide 222 text
# Only allow BAN requests from IP addresses in the 'purge' ACL.
if (req.method == "BAN") {
# Same ACL check as above:
if (!client.ip ~ purge) {
return (synth(403, "Not allowed."));
}
if (req.http.Purge-Cache-Tags) {
ban("obj.http.Purge-Cache-Tags ~ " + req.http.Purge-Cache-Tags);
}
else {
return (synth(403, "Purge-Cache-Tags header missing."));
}
# Throw a synthetic page so the request won't go to the backend.
return (synth(200, "Ban added."));
}
Slide 223
Slide 223 text
acl purge {
"127.0.0.1";
}
Slide 224
Slide 224 text
Purge-Cache-Tags: block:the_date_time block_view
config:block.block.bartik_account_menu config:block.block.bartik_branding
config:block.block.bartik_breadcrumbs config:block.block.bartik_content
config:block.block.bartik_footer config:block.block.bartik_help
config:block.block.bartik_local_actions
config:block.block.bartik_local_tasks config:block.block.bartik_main_menu
config:block.block.bartik_messages config:block.block.bartik_page_title
config:block.block.bartik_powered config:block.block.bartik_search
config:block.block.bartik_tools config:block.block.thedateandtimeblock
config:block_list config:color.theme.bartik config:filter.format.basic_html
config:search.settings config:system.menu.account config:system.menu.footer
config:system.menu.main config:system.menu.tools config:system.site
config:user.role.anonymous config:views.view.frontpage http_response node:1
node:2 node_list node_view rendered user:1
Slide 225
Slide 225 text
https://feryn.eu
https://twitter.com/ThijsFeryn
https://instagram.com/ThijsFeryn
Slide 226
Slide 226 text
No content
Slide 227
Slide 227 text
No content
Slide 228
Slide 228 text
sub vcl_backend_response {
set beresp.grace = 6h;
}
Grace mode
Monitoring & logging
Slide 229
Slide 229 text
Varnishstat
Realtime
statistics
Slide 230
Slide 230 text
No content
Slide 231
Slide 231 text
usage: varnishstat [-1lV] [-f field] [-t seconds|] [-n varnish_name] [-N filename]
-1 # Print the statistics to stdout.
-f field # Field inclusion glob
# If it starts with '^' it is used as an exclusion list.
-l # Lists the available fields to use with the -f option.
-n varnish_name # The varnishd instance to get logs from.
-N filename # Filename of a stale VSM instance.
-t seconds| # Timeout before returning error on initial VSM connection.
-V # Display the version number and exit.
-x # Print statistics to stdout as XML.
-j # Print statistics to stdout as JSON.
Varnishstat usage
Slide 232
Slide 232 text
~# varnishstat -f MAIN.cache_hit -1
MAIN.cache_hit 13049135 5.39 Cache hits
Varnishstat usage
~# varnishstat -f MAIN.cache_hit -j -1
{
"timestamp": "2016-06-14T16:10:32",
"MAIN.cache_hit": {
"description": "Cache hits",
"type": "MAIN", "flag": "c", "format": "i",
"value": 13050992
}
}
Slide 233
Slide 233 text
Varnishstat usage
~# varnishstat -f MAIN.n_object -f MAIN.n_lru_nuked -j
{
"timestamp": "2016-06-14T16:14:49",
"MAIN.n_object": {
"description": "object structs made",
"type": "MAIN", "flag": "g", "format": "i",
"value": 46295
},
"MAIN.n_lru_nuked": {
"description": "Number of LRU nuked objects",
"type": "MAIN", "flag": "g", "format": "i",
"value": 0
}
}
Slide 234
Slide 234 text
varnishstat -f MAIN.cache*
Slide 235
Slide 235 text
✓ Session
✓ Client
✓ Uptime
✓ Hit/miss
✓ Backend
✓ Fetch
✓ Threading
✓ Cache
objects
✓ Memory
✓ Invalidation
Varnishstat counters
Slide 236
Slide 236 text
VSL
Slide 237
Slide 237 text
Varnish
Shared memory
Logging
Slide 238
Slide 238 text
✓ In-memory logs
✓ Generated by varnishd
✓ 81 MB by default
✓ Customize with “-l” setting
✓ Varnishlog command
✓ Varnishtop command
VSL
Slide 239
Slide 239 text
* << Request >> 10973258
- Begin req 10973257 rxreq
- Timestamp Start: 1501507281.942533 0.000000 0.000000
- Timestamp Req: 1501507281.942533 0.000000 0.000000
- ReqStart 127.0.0.1 59753
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: feryn.eu
Slide 240
Slide 240 text
Transactions
Slide 241
Slide 241 text
✓ Items of work
✓ Identified by VXID
✓ 2 kinds:
✓ Sessions
✓ Requests
Transactions
Slide 242
Slide 242 text
✓ Identifies TCP
connection
✓ Contains multiple
requests
Transactions
✓ Client request
✓ Backend request
✓ ESI subrequest
Session Request
Slide 243
Slide 243 text
No content
Slide 244
Slide 244 text
✓ VXID (default)
✓ Session
✓ Request
✓ Raw
Transactions grouping
Slide 245
Slide 245 text
Example
composition that will
be monitored
Slide 246
Slide 246 text
varnishlog -i Begin,ReqUrl,Link,BereqURL
Slide 247
Slide 247 text
* << BeReq >> 98318
- Begin bereq 98317 fetch
- BereqURL /
* << BeReq >> 98320
- Begin bereq 98319 fetch
- BereqURL /header
* << Request >> 98319
- Begin req 98317 esi
- ReqURL /header
- Link bereq 98320 fetch
* << BeReq >> 98322
- Begin bereq 98321 fetch
- BereqURL /nav
* << Request >> 98321
- Begin req 98317 esi
- ReqURL /nav
- Link bereq 98322 fetch
* << Request >> 98317
- Begin req 98316 rxreq
- ReqURL /
- Link bereq 98318 fetch
- Link req 98319 esi
- Link req 98321 esi
* << BeReq >> 98324
- Begin bereq 98323 fetch
- BereqURL /footer
* << Request >> 98323
- Begin req 98316 rxreq
- ReqURL /footer
- Link bereq 98324 fetch
* << Session >> 98316
- Begin sess 0 HTTP/1
- Link req 98317 rxreq
- Link req 98323 rxreq
Slide 248
Slide 248 text
varnishlog -i Begin,ReqUrl,Link,BereqURL -g session
Slide 249
Slide 249 text
* << Session >> 14
- Begin sess 0 HTTP/1
- Link req 65539 rxreq
- Link req 65545 rxreq
** << Request >> 65539
-- Begin req 14 rxreq
-- ReqURL /
-- Link bereq 65540 fetch
-- Link req 65541 esi
-- Link req 65543 esi
** << Request >> 65545
-- Begin req 14 rxreq
-- ReqURL /footer
-- Link bereq 65546 fetch
*** << BeReq >> 65540
--- Begin bereq 65539 fetch
--- BereqURL /
*** << Request >> 65541
--- Begin req 65539 esi
--- ReqURL /header
--- Link bereq 65542 fetch
*** << Request >> 65543
--- Begin req 65539 esi
--- ReqURL /nav
--- Link bereq 65544 fetch
*** << BeReq >> 65546
--- Begin bereq 65545 fetch
--- BereqURL /footer
**** << BeReq >> 65542
---- Begin bereq 65541 fetch
---- BereqURL /header
**** << BeReq >> 65544
---- Begin bereq 65543 fetch
---- BereqURL /nav
Slide 250
Slide 250 text
No content
Slide 251
Slide 251 text
Tags
Slide 252
Slide 252 text
Request tags
Slide 253
Slide 253 text
✓ ReqMethod
✓ ReqUrl
✓ ReqProtocol
✓ ReqHeader
Request tags
Slide 254
Slide 254 text
- ReqStart 127.0.0.1 56312
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X
10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/
59.0.3071.115 Safari/537.36
- ReqHeader Accept: text/html,application/xhtml+xml,application/
xml;q=0.9,image/webp,image/apng,*/*;q=0.8
- ReqHeader Accept-Encoding: gzip, deflate, br
- ReqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6
- ReqHeader X-Forwarded-For: 127.0.0.1
Slide 255
Slide 255 text
Response tags
Slide 256
Slide 256 text
✓ RespProtocol
✓ RespStatus
✓ RespReason
✓ RespHeader
Response tags
Slide 257
Slide 257 text
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader Host: localhost
- RespHeader Cache-Control: public, s-maxage=500
- RespHeader Date: Tue, 01 Aug 2017 08:56:44 GMT
- RespHeader ETag: "c5afddc587599a72d467caca23e980bf"
- RespHeader Vary: Accept-Language
- RespHeader Content-Length: 3098
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader X-Varnish: 32770
- RespHeader Age: 10
Slide 258
Slide 258 text
Backend tags
Slide 259
Slide 259 text
- BackendOpen 19 boot.default 127.0.0.1 8080 127.0.0.1 62552
- BackendClose 19 boot.default
Slide 260
Slide 260 text
Backend request tags
Slide 261
Slide 261 text
✓ BereqMethod
✓ BereqUrl
✓ BereqProtocol
✓ BereqHeader
Backend request tags
Slide 262
Slide 262 text
Backend response
tags
Slide 263
Slide 263 text
✓ BerespProtocol
✓ BerespStatus
✓ BerespReason
✓ BerespHeader
Backens response tags
Slide 264
Slide 264 text
Object tags
Slide 265
Slide 265 text
✓ ObjProtocol
✓ ObjStatus
✓ ObjReason
✓ ObjHeader
Object tags
Slide 266
Slide 266 text
VCL tags
Slide 267
Slide 267 text
✓ VCL_Call
✓ VCL_Return
✓ VCL_Error
✓ VCL_Log
✓ VCL_Acl
VCL tags
Slide 268
Slide 268 text
* << Request >> 5
- ReqURL /
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call MISS
- VCL_return fetch
- VCL_call DELIVER
- VCL_return deliver
** << BeReq >> 6
-- BereqURL /
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- VCL_call BACKEND_RESPONSE
-- VCL_return deliver
Slide 269
Slide 269 text
* << Request >> 6
- ReqURL /
- VCL_call RECV
- ReqURL /
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call HIT
- VCL_return deliver
- VCL_call DELIVER
- VCL_return deliver
Slide 270
Slide 270 text
* << Request >> 32789
- ReqURL /header
- ExpBan 98355 banned lookup
ExpBan tag
Slide 271
Slide 271 text
* << Request >> 98369
- ReqURL /footer
- Hit 65597
Hit tag
varnishlog -i "ReqUrl,Hit"
Slide 272
Slide 272 text
** << Request >> 6
-- ReqURL /footer
-- HitPass 3
HitPass tag
Slide 273
Slide 273 text
TTL tag
Slide 274
Slide 274 text
%s %d %d %d %d [ %d %d %u %u ]
| | | | | | | | |
| | | | | | | | +- Max-Age from Cache-Control header
| | | | | | | +---- Expires header
| | | | | | +------- Date header
| | | | | +---------- Age (incl Age: header value)
| | | | +--------------- Reference time for TTL
| | | +------------------ Keep
| | +--------------------- Grace
| +------------------------ TTL
+--------------------------- "RFC" or "VCL"
TTL tag
Slide 275
Slide 275 text
-- TTL VCL 120 10 0 1501597242
✓ TTL decided by the VCL
✓ 120 seconds cached
✓ 10 seconds grace time
✓ 0 seconds keep time
✓ Reference time: 1501597242
(2017-08-01 14:20:42)
Slide 276
Slide 276 text
-- RFC 500 10 -1 1501598872 1501598872 1501598872 0 500
✓ 500 seconds TTL (via headers)
✓ 10 seconds grace
✓ No keep value
✓ 2017-08-01 14:47:52 date, age & reference time
✓ Cache-control headers sets 500 second TTL
Slide 277
Slide 277 text
-- RFC 500 10 -1 1501598872 1501598869 1501598872 0 500
✓ Don’t screw with the age header
✓ Only 497 second effective TTL
✓ Custom Age header (3 seconds off)
Slide 278
Slide 278 text
Begin tag
Slide 279
Slide 279 text
- Begin bereq 98317 fetch
- Begin req 98317 esi
- Begin req 98316 rxreq
- Begin sess 0 HTTP/1
Slide 280
Slide 280 text
Link tag
Slide 281
Slide 281 text
- Link bereq 98312 fetch
- Link req 98311 rxreq
- Link req 98318 esi
Slide 282
Slide 282 text
Timestamp tag
Slide 283
Slide 283 text
%s: %f %f %f
| | | |
| | | +- Time since last timestamp
| | +---- Time since start of work unit
| +------- Absolute time of event
+----------- Event label
Timestamp tag
Slide 284
Slide 284 text
* << Request >> 65539
- Timestamp Start: 1501601912.758662 0.000000 0.000000
- Timestamp Req: 1501601912.758662 0.000000 0.000000
- Timestamp Fetch: 1501601912.806733 0.048071 0.048071
- Timestamp Process: 1501601912.806750 0.048088 0.000017
- Timestamp Resp: 1501601912.806787 0.048125 0.000037
** << BeReq >> 65540
-- Timestamp Start: 1501601912.758753 0.000000 0.000000
-- Timestamp Bereq: 1501601912.758952 0.000199 0.000199
-- Timestamp Beresp: 1501601912.806677 0.047924 0.047725
-- Timestamp BerespBody: 1501601912.806749 0.047996 0.000072
Slide 285
Slide 285 text
Filtering output
Slide 286
Slide 286 text
✓ -i: include tags
✓ -I: include tags by regex
✓ -x: exclude tags
✓ -X: exclude by regex
Filtering output
Slide 287
Slide 287 text
Include tags
Slide 288
Slide 288 text
varnishlog -i ReqUrl,VCL_call,VCL_return -g session
varnishlog -i "ReqUrl,VCL_*" -g session
Slide 289
Slide 289 text
* << Session >> 252394
** << Request >> 252395
-- ReqURL /
-- VCL_call RECV
-- VCL_return hash
-- VCL_call HASH
-- VCL_return lookup
-- VCL_call HIT
-- VCL_return deliver
-- VCL_call DELIVER
-- VCL_return deliver
*** << Request >> 252397
--- ReqURL /header
--- VCL_call RECV
--- VCL_return hash
--- VCL_call HASH
--- VCL_return lookup
--- VCL_call HIT
--- VCL_return deliver
--- VCL_call DELIVER
--- VCL_return deliver
*** << Request >> 252399
--- ReqURL /nav
--- VCL_call RECV
--- VCL_return hash
--- VCL_call HASH
--- VCL_return lookup
--- VCL_call HIT
--- VCL_return deliver
--- VCL_call DELIVER
--- VCL_return deliver
*** << BeReq >> 252396
--- VCL_call BACKEND_FETCH
--- VCL_return fetch
--- VCL_call BACKEND_RESPONSE
--- VCL_return deliver
**** << BeReq >> 252398
---- VCL_call BACKEND_FETCH
---- VCL_return fetch
---- VCL_call BACKEND_RESPONSE
---- VCL_return deliver
**** << BeReq >> 252400
---- VCL_call BACKEND_FETCH
---- VCL_return fetch
---- VCL_call BACKEND_RESPONSE
---- VCL_return deliver
Slide 290
Slide 290 text
Exclude tags
Slide 291
Slide 291 text
varnishlog -i "Req*" -x ReqHeader,ReqUnset
Slide 292
Slide 292 text
* << Request >> 314125
- ReqStart 127.0.0.1 64585
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqAcct 476 0 476 311 0 311
* << Request >> 314126
- ReqStart 127.0.0.1 64585
- ReqMethod GET
- ReqURL /footer
- ReqProtocol HTTP/1.1
- ReqAcct 370 0 370 309 0 309
Slide 293
Slide 293 text
Include tags by regex
Slide 294
Slide 294 text
varnishlog -I "reqheader:Accept-Language" -i requrl
Slide 295
Slide 295 text
* << Request >> 374378
- ReqURL /
- ReqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6
* << Request >> 374379
- ReqURL /footer
- ReqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6
Slide 296
Slide 296 text
Exclude tags by regex
Slide 297
Slide 297 text
* << Request >> 374384
- ReqURL /footer
- RespHeader Host: localhost
- RespHeader X-Powered-By: PHP/7.0.15
- RespHeader Cache-Control: public, s-maxage=500
- RespHeader Date: Wed, 02 Aug 2017 11:27:21 GMT
- RespHeader ETag: "d47ac09f5351f8f4c97c99ef5b3d2ecd"
- RespHeader Vary: Accept-Language
- RespHeader Content-Length: 80
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader X-Varnish: 374384 15367
- RespHeader Age: 334
- RespHeader Via: 1.1 varnish-v4
- RespHeader Connection: keep-alive
Slide 298
Slide 298 text
varnishlog -i RespHeader,ReqUrl -X "RespHeader:(x|X)-"
Slide 299
Slide 299 text
* << Request >> 374384
- ReqURL /footer
- RespHeader Host: localhost
- RespHeader Cache-Control: public, s-maxage=500
- RespHeader Date: Wed, 02 Aug 2017 11:27:21 GMT
- RespHeader ETag: "d47ac09f5351f8f4c97c99ef5b3d2ecd"
- RespHeader Vary: Accept-Language
- RespHeader Content-Length: 80
- RespHeader Content-Type: text/html; charset=UTF-8
- RespHeader Age: 334
- RespHeader Via: 1.1 varnish-v4
- RespHeader Connection: keep-alive
Slide 300
Slide 300 text
All-in-one
Slide 301
Slide 301 text
varnishlog -i "RespHeader,Req*" -X "RespHeader:(x|X)-" -I
"timestamp:Resp" -x reqprotocol,reqacct -g request
Slide 302
Slide 302 text
* << Request >> 59383
- ReqStart 127.0.0.1 53195
- ReqMethod GET
- ReqURL /
- ReqHeader Host: localhost
- ReqHeader Connection: keep-alive
- ReqHeader Cache-Control: max-age=0
- ReqHeader Upgrade-Insecure-Requests:
1
- ReqHeader User-Agent: Mozilla/5.0
(Macintosh; Intel Mac OS X 10_12_5)
AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/59.0.3071.115 Safari/537.36
- ReqHeader Accept: text/
html,application/
xhtml+xml,application/xml;q=0.9,image/
webp,image/apng,*/*;q=0.8
- ReqHeader Accept-Encoding: gzip,
deflate, br
- ReqHeader Accept-Language: nl,en-
US;q=0.8,en;q=0.6
- ReqHeader If-None-Match:
W/"27f341f8e459dd35f1087c55351cacda"
- ReqHeader X-Forwarded-For: 127.0.0.1
- ReqUnset Accept-Language: nl,en-
US;q=0.8,en;q=0.6
- ReqHeader accept-language: nl
- ReqHeader Surrogate-Capability:
key=ESI/1.0
- ReqUnset Accept-Encoding: gzip,
deflate, br
- ReqHeader Accept-Encoding: gzip
- RespHeader Host: localhost
- RespHeader Cache-Control: public, s-
maxage=500
- RespHeader Date: Wed, 02 Aug 2017
11:46:49 GMT
- RespHeader ETag:
"27f341f8e459dd35f1087c55351cacda"
- RespHeader Vary: Accept-Language
- RespHeader Content-Length: 3098
- RespHeader Content-Type: text/html;
charset=UTF-8
- RespHeader Age: 152
- RespHeader Via: 1.1 varnish-v4
- RespHeader ETag:
W/"27f341f8e459dd35f1087c55351cacda"
- RespHeader Connection: keep-alive
- Timestamp Resp: 1501674561.472358
0.000068 0.000020
Slide 303
Slide 303 text
✓ All response headers
✓ All tags that start with “Req”
✓ Exclude “x-“ response headers
✓ Include “response time” timestamp
✓ Exclude request protocol log lines,
and the request accountancy log
lines
All-in-one
Slide 304
Slide 304 text
VSL queries
Slide 305
Slide 305 text
Filtering fields from
all transactions
Filtering transactions
Slide 306
Slide 306 text
{level}taglist:record-prefix[field]
Slide 307
Slide 307 text
ReqUrl eq ‘/‘
{level}taglist:record-prefix[field]
Slide 308
Slide 308 text
Timestamp:Resp[2] > 1.0
{level}taglist:record-prefix[field]
Slide 309
Slide 309 text
{2}Timestamp:Resp[2] > 1.0
{level}taglist:record-prefix[field]
Slide 310
Slide 310 text
varnishlog -i VCL_call,VCL_return -g request -q "ReqURL eq '/'"
Slide 311
Slide 311 text
* << Request >> 374400
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call HIT
- VCL_return deliver
- VCL_call DELIVER
- VCL_return deliver
Slide 312
Slide 312 text
varnishlog -i ReqUrl -q "VCL_call eq 'MISS' or VCL_call eq 'PASS'"
varnishlog -i ReqUrl -I "Timestamp:Resp" -q "Timestamp:Resp[2] > 1.0"
Slide 313
Slide 313 text
Other options
Slide 314
Slide 314 text
varnishlog -n myVarnishInstance
varnishlog -d
varnishlog -w file
varnishlog -r file
varnishlog -A -a -w file
varnishlog -i "ReqUrl,VCL_*" -D -a -A -w /var/log/varnish/custom.log -
P /var/run/custom_varnishlog.pid
Slide 315
Slide 315 text
Varnishtop
Slide 316
Slide 316 text
Same syntax as varnishlog.
Output is incremental
Slide 317
Slide 317 text
$ varnishtop -I reqheader:Accept-Language -q "ReqUrl eq '/'"
list length 4
0.86 ReqHeader Accept-Language: en-US
0.80 ReqHeader Accept-Language: nl-NL,nl;q=0.8,en-
US;q=0.6,en;q=0.4
0.54 ReqHeader Accept-Language: nl,en-US;q=0.8,en;q=0.6
0.39 ReqHeader Accept-Language: nl-BE
Slide 318
Slide 318 text
Common Varnishlog
scenarios
Slide 319
Slide 319 text
Why didn’t Varnish serve this request from cache?
Slide 320
Slide 320 text
Request not
cacheable
Slide 321
Slide 321 text
varnishlog -i "Req*,VCL*" -x ReqAcct,ReqStart -q "VCL_call
eq 'PASS'"
Slide 322
Slide 322 text
* << Request >> 12
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.48.0
- ReqHeader Accept: */*
- ReqHeader Cookie: myCookie=bla
- ReqHeader X-Forwarded-For: 127.0.0.1
- VCL_call RECV
- VCL_return pass
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- VCL_call DELIVER
- VCL_return deliver
Slide 323
Slide 323 text
<< Request >> 32779
- ReqMethod POST
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader User-Agent: curl/7.48.0
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 127.0.0.1
- VCL_call RECV
- VCL_return pass
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- VCL_call DELIVER
- VCL_return deliver
Slide 324
Slide 324 text
* << Request >> 15
- ReqMethod GET
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost
- ReqHeader Authorization: Basic dGhpanM6ZmVyeW4=
- ReqHeader User-Agent: curl/7.48.0
- ReqHeader Accept: */*
- ReqHeader X-Forwarded-For: 127.0.0.1
- VCL_call RECV
- VCL_return pass
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- VCL_call DELIVER
- VCL_return deliver
Slide 325
Slide 325 text
Response not
cacheable
Slide 326
Slide 326 text
varnishlog -i ReqUrl,VCL_*,Beresp*,TTL -q "HitPass" -g
request
Slide 327
Slide 327 text
* << Request >> 19010384
- ReqURL /my-url
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- VCL_call DELIVER
- VCL_return deliver
** << BeReq >> 19010385
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- BerespProtocol HTTP/1.1
-- BerespStatus 200
-- BerespReason OK
-- BerespHeader Date: Thu, 03 Aug
2017 08:15:22 GMT
-- BerespHeader Server: Apache/
2.4.10 (Debian)
-- BerespHeader Last-Modified: Tue,
01 Aug 2017 07:21:00 GMT
-- BerespHeader ETag:
"5c0d-555abfd3f422f-gzip"
-- BerespHeader Vary: Accept-
Encoding
-- BerespHeader Content-Encoding:
gzip
-- BerespHeader Cache-Control: max-
age=0
-- BerespHeader Expires: Thu, 03 Aug
2017 08:15:22 GMT
-- BerespHeader Content-Length: 7686
-- BerespHeader Content-Type:
application/json
-- TTL RFC 0 10 -1 1501748123
1501748123 1501748122 1501748122 0
-- VCL_call BACKEND_RESPONSE
-- TTL VCL 120 10 0 1501748123
-- VCL_return deliver
Slide 328
Slide 328 text
* << Request >> 65551
- ReqURL /set-cookie
- VCL_call RECV
- VCL_return hash
- VCL_call HASH
- VCL_return lookup
- VCL_call PASS
- VCL_return fetch
- VCL_call DELIVER
- VCL_return deliver
** << BeReq >> 65552
-- VCL_call BACKEND_FETCH
-- VCL_return fetch
-- BerespProtocol HTTP/1.1
-- BerespStatus 200
-- BerespReason OK
-- BerespHeader Cache-control: s-maxage=10
-- BerespHeader Set-Cookie: myCookie=bla
-- BerespHeader Content-type: text/html; charset=UTF-8
-- BerespHeader Date: Thu, 03 Aug 2017 08:39:04 GMT
-- TTL RFC 10 10 -1 1501749545 1501749545 1501749544 0 10
-- VCL_call BACKEND_RESPONSE
-- TTL VCL 120 10 0 1501749545
-- VCL_return deliver
Slide 329
Slide 329 text
https://feryn.eu
https://twitter.com/ThijsFeryn
https://instagram.com/ThijsFeryn
Slide 330
Slide 330 text
No content