Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Varnish In Action - PHP.gent 2017

Varnish In Action - PHP.gent 2017

Varnish talk at the PHP.gent meetup. More information: https://talks.feryn.eu/talks/167/varnish-in-action-ghent-php-meetup-gent

Ca901ddcea38854b9783781c91fc87c9?s=128

Thijs Feryn

March 21, 2017
Tweet

Transcript

  1. Varnish In Action By Thijs Feryn

  2. Slow websites suck

  3. Web performance is an essential part of the user experience

  4. Down Slowdown ~ downtime

  5. Hi, I’m Thijs

  6. I’m @ThijsFeryn on Twitter

  7. I’m an Evangelist At

  8. None
  9. None
  10. https://www.facebook.com/fcbruges.support/

  11. Mo money Mo servers

  12. Cache

  13. Don’t recompute if the data hasn’t changed

  14. None
  15. Normally User Server

  16. With Varnish User Varnish Server

  17. None
  18. Why is it so good?

  19. Smart kernel tricks

  20. Stores HTTP output in memory*

  21. Respects HTTP best practices

  22. Varnish Configuration Language

  23. What about TLS/SSL?

  24. I'll talk about Varnish 4

  25. Quick install & configure

  26. apt-get install apt-transport-https curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add - echo

    "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.1"\ >> /etc/apt/sources.list.d/varnish-cache.list apt-get update apt-get install varnish Install on Linux (Debian) There's stuff for Ubuntu, RHEL & CentOS too
  27. -a :80 \ -a :81,PROXY \ -T localhost:6082 \ -f

    /etc/varnish/default.vcl \ -S /etc/varnish/secret \ -s malloc,3g Simple config
  28. vcl 4.0; backend default { .host = "localhost"; .port =

    "8080"; } Minimal VCL
  29. Varnish speaks HTTP

  30. ✓Idempotence ✓State ✓Expiration ✓Conditional requests ✓Cache variations Varnish speaks HTTP

  31. ✓GET ✓HEAD -POST -PUT -PATCH -DELETE -TRACE -OPTIONS Idempotence Only

    cache GET & HEAD
  32. ✓Doesn't cache when cookies are present ✓Doesn't cache when cookies

    are set ✓Doesn't cache when Authorization headers are present State Avoids caching user- specific content Affects hit rate
  33. Expiration Expires: Sat, 09 Sep 2017 14:30:00 GMT Cache-control: public,

    max-age=3600, s-maxage=86400 Cache-control: private, no-cache, no-store
  34. Expiration precedence 1. beresp.ttl (VCL) 2. s-maxage 3. max-age 4.

    expires 5. 120 seconds by default
  35. Conditional requests HTTP/1.1 200 OK Host: localhost Etag: 7c9d70604c6061da9bb9377d3f00eb27 Content-type:

    text/html; charset=UTF-8 Hello world output GET /if_none_match.php HTTP/1.1 Host: localhost User-Agent: curl/7.48.0
  36. Conditional requests HTTP/1.0 304 Not Modified Host: localhost Etag: 7c9d70604c6061da9bb9377d3f00eb27

    GET /if_none_match.php HTTP/1.1 Host: localhost User-Agent: curl/7.48.0 If-None-Match: 7c9d70604c6061da9bb9377d3f00eb27
  37. Conditional requests HTTP/1.1 200 OK Host: localhost Last-Modified: Fri, 22

    Jul 2016 10:11:16 GMT Content-type: text/html; charset=UTF-8 Hello world output GET /if_none_match.php HTTP/1.1 Host: localhost User-Agent: curl/7.48.0
  38. Conditional requests HTTP/1.0 304 Not Modified Host: localhost Last-Modified: Fri,

    22 Jul 2016 10:11:16 GMT GET /if_none_match.php HTTP/1.1 Host: localhost User-Agent: curl/7.48.0 If-Modified-Since: Fri, 22 Jul 2016 10:11:16 GMT
  39. ✓Hostname (or IP) ✓URL Cache variations Identify object in cache

  40. Cache variations GET / HTTP/1.1 Host: localhost Accept-Language: nl HTTP/1.1

    200 OK Host: localhost Vary: Accept-Language Hallo, deze pagina is in het Nederlands geschreven.
  41. Built-in VCL cheat sheet

  42. ✓Only GET & HEAD ✓No Cookie ✓No Authorization Built-in VCL

    (frontend) Lookup in cache Otherwise pass and don't cache
  43. ✓No Set-Cookie ✓TTL > 0 ✓No "no-cache", "private", "no-store" Built-in

    VCL (backend) Store in cache Otherwise blacklist for 120s
  44. The actual VCL code

  45. sub vcl_recv { if (req.method == "PRI") { /* We

    do not support SPDY or HTTP/2.0 */ return (synth(405)); } if (req.method != "GET" && req.method != "HEAD" && req.method != "PUT" && req.method != "POST" && req.method != "TRACE" && req.method != "OPTIONS" && req.method != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); } if (req.method != "GET" && req.method != "HEAD") { /* We only deal with GET and HEAD by default */ return (pass); } if (req.http.Authorization || req.http.Cookie) { /* Not cacheable by default */ return (pass); } return (hash); }
  46. sub vcl_pipe { # By default Connection: close is set

    on all piped requests, to stop # connection reuse from sending future requests directly to the # (potentially) wrong backend. If you do want this to happen, you can undo # it here. # unset bereq.http.connection; return (pipe); } sub vcl_pass { return (fetch); } sub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (lookup); }
  47. sub vcl_purge { return (synth(200, "Purged")); } sub vcl_hit {

    if (obj.ttl >= 0s) { // A pure unadultered hit, deliver it return (deliver); } if (obj.ttl + obj.grace > 0s) { // Object is in grace, deliver it // Automatically triggers a background fetch return (deliver); } // fetch & deliver once we get the result return (miss); } sub vcl_miss { return (fetch); } sub vcl_deliver { return (deliver); }
  48. sub vcl_backend_fetch { return (fetch); } sub vcl_backend_response { if

    (beresp.ttl <= 0s || beresp.http.Set-Cookie || beresp.http.Surrogate-control ~ "no-store" || (!beresp.http.Surrogate-Control && beresp.http.Cache-Control ~ "no-cache|no- store|private") || beresp.http.Vary == "*") { /* * Mark as "Hit-For-Pass" for the next 2 minutes */ set beresp.ttl = 120s; set beresp.uncacheable = true; } return (deliver); }
  49. Reality sucks

  50. None
  51. None
  52. Cache-control ?

  53. Legacy

  54. Write VCL

  55. Normalize

  56. vcl 4.0; import std; sub vcl_recv { set req.http.Host =

    regsub(req.http.Host, ":[0-9]+", ""); set req.url = std.querysort(req.url); if (req.url ~ "\#") { set req.url = regsub(req.url, "\#.*$", ""); } if (req.url ~ "\?$") { set req.url = regsub(req.url, "\?$", ""); } if (req.restarts == 0) { if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } } } }
  57. sub vcl_recv { if (req.url ~ "^/status\.php$" || req.url ~

    "^/update\.php$" || req.url ~ "^/admin$" || req.url ~ "^/admin/.*$" || req.url ~ "^/user$" || req.url ~ "^/user/.*$" || req.url ~ "^/flag/.*$" || req.url ~ "^.*/ajax/.*$" || req.url ~ "^.*/ahah/.*$") { return (pass); } } URL blacklist
  58. sub vcl_recv { if (req.url ~ "^/products/?" return (hash); }

    } URL whitelist
  59. Cookies

  60. vcl 4.0; sub vcl_recv { set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(;

    )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", ""); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; } } Remove tracking cookies
  61. vcl 4.0; sub vcl_recv { if (req.http.Cookie) { set req.http.Cookie

    = ";" + req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(PHPSESSID)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; } } } Only keep session cookie
  62. sub vcl_recv { if (req.http.Cookie) { set req.http.Cookie = ";"

    + req.http.Cookie; set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(language)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.cookie ~ "^\s*$") { unset req.http.cookie; return(pass); } return(hash); } } sub vcl_hash { hash_data(regsub( req.http.Cookie, "^.*language=([^;]*);*.*$", "\1" )); } Language cookie cache variation
  63. Or just use vmod_cookie

  64. Alternative language cache variation

  65. sub vcl_hash { hash_data(req.http.Accept-Language); } HTTP/1.1 200 OK Host: localhost

    Vary: Accept-Language Hash language Or just use
  66. Cookie guidelines

  67. ✓Don't throw everything in a session ✓Use dedicated cookies ✓Strip

    off tracking cookies in VCL ✓Match cookies in VCL & make decisions ✓Perform cache variations on cookie values ✓Only use session cookies when you really have to ✓Use JWT to store session state client-side Cookie guidelines
  68. Block caching

  69. None
  70. Code renders single HTTP response

  71. Lowest common denominator: no cache

  72. <esi:include src="/header" /> Edge Side Includes ✓Placeholder ✓Parsed by Varnish

    ✓Output is a composition of blocks ✓State per block ✓TTL per block
  73. sub vcl_recv { set req.http.Surrogate-Capability = "key=ESI/1.0"; } sub vcl_backend_response

    { if (beresp.http.Surrogate-Control ~ "ESI/1.0") { unset beresp.http.Surrogate-Control; set beresp.do_esi = true; } } Edge Side Includes
  74. ESI vs AJAX

  75. <esi:include src="/header" /> Choose wisely <hx:include src="/header"></hx:include> ESI, parsed by

    Varnish HInclude, parsed by Javascript
  76. Measure the pressure

  77. ✓Varnishlog ✓Varnishtop ✓Varnishstat Measure the pressure

  78. Breaking news isn't breaking

  79. Purging

  80. acl purge { "localhost"; "127.0.0.1"; "::1"; } sub vcl_recv {

    if (req.method == "PURGE") { if (!client.ip ~ purge) { return (synth(405, “Not allowed.”)); } return (purge); } } Purging
  81. acl purge { "localhost"; "127.0.0.1"; "::1"; } sub vcl_backend_response {

    set beresp.http.x-url = bereq.url; set beresp.http.x-host = bereq.http.host; } sub vcl_deliver { unset resp.http.x-url; unset resp.http.x-host; } sub vcl_recv { if (req.method == "PURGE") { if (!client.ip ~ purge) { return (synth(405, "Not allowed")); } if(req.http.x-purge-regex) { ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url ~ " + req.http.x-purge-regex); } else { ban("obj.http.x-host == " + req.http.host + " && obj.http.x-url == " + req.url); } return (synth(200, "Purged")); } } Banning
  82. Banning curl -XPURGE "http://example.com/products" curl -XPURGE -H "x-purge-regex:/products" \ "http://example.com"

    varnishadm> ban obj.http.x-host == example.com && obj.http.x-url ~ ^/product/[0-9]+/details
  83. Write less VCL Write more code

  84. https://github.com/ThijsFeryn/ cacheable-site-silex/tree/v2

  85. None
  86. None
  87. https://twitter.com/thijsferyn https://instagram.com/thijsferyn https://blog.feryn.eu https://talks.feryn.eu https://book.feryn.eu https://youtube.com/thijsferyn https://soundcloud.com/thijsferyn http://itunes.feryn.eu