Fastly - Inaugural NYC Varnish Meetup

July 23, 2014. Fastly's CEO, Artur Bergman, spoke about how he got started with Varnish and why he chose to build a high performance content delivery network on top of Varnish. Come to a Varnish Meetup in NYC - full event list here: http://www.meetup.com/NYC-Varnish-Meetup/

Transcript

1. Varnish NYC Meetup | 23 July 2014 •  First major user of Varnish •  Varnish contributor •  Fastly Founder and CEO •  @crucially •  artur@fastly.com •  Please ask questions Artur Bergman
2. Varnish NYC Meetup | 23 July 2014 •  Large distributed Varnish •  Realtime •  Purging •  Logfiles •  Config •  Stats • Since July 2011 Fastly
3. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  Reverse proxy   •  Sits between you and origin/backends •  Caches content •  Load balancer •  Advanced configuration language Varnish
4. Varnish NYC Meetup | 23 July 2014 FASTLY VCL
5. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  Simple scripting language •  Compiles into c code •  Shared library dynamically loaded into app VCL
6. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 sub  vcl_recv  {          if  (req.http.Accept-­‐Encoding)  {              if  (req.http.User-­‐Agent  ~  "MSIE  6")  {                  unset  req.http.Accept-­‐Encoding;              }  elsif  (req.http.Accept-­‐Encoding  ~  "gzip")  {                  set  req.http.Accept-­‐Encoding  =  "gzip";              }  elsif  (req.http.Accept-­‐Encoding  ~  "deflate")  {                  set  req.http.Accept-­‐Encoding  =  "deflate";              }  else  {                  unset  req.http.Accept-­‐Encoding;              }          }   VCL
7. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  “The Vary field value indicates the set of request-header fields that fully determines, while the response is fresh, whether a cache is permitted to use the response to reply to a subsequent request without revalidation. For uncacheable or stale responses, the Vary field value advises the user agent about the criteria that were used to select the representation" •  Second level cache key •  You need to look up the object, get the vary signature, then compute the vary and match it Vary
8. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  varnishlog -i RxHeader | grep Accept-Encoding: | head -1000 | cut - c 22- | sort | uniq -c | sort -rn •  399 Accept-Encoding: gzip, deflate •  285 Accept-Encoding: gzip •  262 Accept-Encoding: gzip,deflate,sdch •  42 Accept-Encoding: gzip,deflate •  3 Accept-Encoding: identity •  3 Accept-Encoding: deflate, gzip •  1 Accept-Encoding: identity,gzip •  1 Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3 •  1 Accept-Encoding: gzip,deflate,lzma,sdch •  1 Accept-Encoding: gzip, deflate, identity •  1 Accept-Encoding: gzip, deflate, compress •  1 Accept-Encoding:gzip Vary - Accept Encoding
9. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 Varnishlog varnishlog -i RxHeader | grep Accept-Encoding: | head -1000 | cut -c 22- | sort | uniq -c | sort -rn        
10. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  So should be 1-4 different versions •  Vary is literal •  10 different version! •  Massive DOS vector •  Linear scan across all Varys Vary - Accept Encoding
11. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 sub  vcl_recv  {          if  (req.http.Accept-­‐Encoding)  {              if  (req.http.User-­‐Agent  ~  "MSIE  6")  {                  unset  req.http.Accept-­‐Encoding;              }  elsif  (req.http.Accept-­‐Encoding  ~  "gzip")  {                  set  req.http.Accept-­‐Encoding  =  "gzip";              }  elsif  (req.http.Accept-­‐Encoding  ~  "deflate")  {                  set  req.http.Accept-­‐Encoding  =  "deflate";              }  else  {                  unset  req.http.Accept-­‐Encoding;              }          }   VCL
12. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  Bounded number of possible combinations •  Vary: User-Agent, Referer, Accept-Encoding, Cookie •  Seen in the wild! •  Be careful with what you let into you vary Vary - Accept Encoding
13. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 sub  vcl_recv  {          set  req.http.Vary-­‐Cookie  =  regsub(req.http.Cookie,   “.*(userid=[^;]+).*”,  “\1”);   }     sub  vcl_fetch  {    set  beresp.http.Vary  =  “Vary-­‐Cookie”;   }   VCL
14. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  Ignore cookies from •  ad networks •  google analytics •  random 3rd party tracking cookies •  Cache-Control: private for cookie content •  Now you have hitpass objects for logged in users Vary - Cookie
15. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  return(pass) in vcl_recv •  bypass all caching •  bypass request collapsing •  bypass hash lookup •  return(pass) in vcl_fetch •  Cache that the object is uncacheable •  On next request will closely follow vcl_recv return(pass) •  Except it does go into vcl_hash •  Useful to avoid bad request collapsing Hitpass
16. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 Inline C C{   asm  (   "\t.LD2496:\n"   "\t.string  \”xxxx.services.fastly.net\"\n"   "\t.LD2497:\n"   "\t.string  \”yyy.services.fastly.net\"\n"   "\t.LD2498:\n"   "\t.string  \”zzzz.services.fastly.net\"\n"   "\t.LD2499:\n"   \tcmpb  $103,  %r14b  #  g\n"   "\tleaq  .LS5230(%rip),  %r8  #  xxxxxx  -­‐>  5705\n"   "\tleaq  .LD23896(%rip),  %r9\n"   "\tje  compare  #  foo\n"   "\tcmpb  $114,  %r14b  #  r\n"   "\tje  L177\n"   "\tcmpb  $48,  %r14b  #  0\n"   "\tleaq  .LS1407(%rip),  %r8  #  zzzzzzzzzz  -­‐>  2264\n"   "\tleaq  .LD9381(%rip),  %r9\n"   "\tje  compare  #  foo\n"  
17. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 Inline C
18. Varnish NYC Meetup | 23 July 2014 FASTLY Purging
19. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  Cache forever •  Purge on changes •  Hook into CMS/Wordpress/Whatever •  Fastly supports it :)   Cache invalidation
20. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  Tag responses •  Surrogate-Key: objectid userid templateid •  Purge by tag •  ban obj.http.surrogate-key ~ “objectid” Surrogate key purging
21. Varnish NYC Meetup | 23 July 2014 FASTLY Running Varnish
22. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  -p thread_pool_stack=524288 •  Save ram •  -p syslog_cli_traffic=off •  Don’t block if syslog hangs •  -p thread_pool_max=32000 •  Depends on your count of CPU •  -p thread_pools=8 •  -p thread_pool_min=1000 •  -p thread_pool_add_delay=0 •  Don’t spend time waiting for threads when you need them Start parameters
23. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  -s file,<dir_or_file>,<size>   •  Has concurrency problems •  Holds lock while paging (reading) from disk! •  Solution, use many per disk (or rewrite it) •  (And obviously use a SSD) Disk storage
24. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  vm.dirty_background_bytes  =  531838208   •  vm.dirty_expire_cenYsecs  =  3000   •  vm.dirty_raYo  =  50   •  vm.dirty_writeback_cenYsecs  =  500   •  vm.lowmem_reserve_raYo  =  1  1  32   •  vm.max_map_count  =  524288   •  vm.min_free_kbytes  =  15000000   •  vm.mmap_min_addr  =  65536   •  vm.swappiness  =  0   •  vm.zone_reclaim_mode  =  0   sysctl
25. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  vm.dirty_background_bytes  =  531838208   •  vm.dirty_expire_cenYsecs  =  3000   •  vm.dirty_raYo  =  50   •  vm.dirty_writeback_cenYsecs  =  500   •  vm.lowmem_reserve_raYo  =  1  1  32   •  vm.max_map_count  =  524288   •  vm.min_free_kbytes  =  15000000   •  vm.mmap_min_addr  =  65536   •  vm.swappiness  =  0   •  vm.zone_reclaim_mode  =  0   Control writeback
26. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  vm.dirty_background_bytes  =  531838208   •  vm.dirty_expire_cenYsecs  =  3000   •  vm.dirty_raYo  =  50   •  vm.dirty_writeback_cenYsecs  =  500   •  vm.lowmem_reserve_raYo  =  1  1  32   •  vm.max_map_count  =  524288   •  vm.min_free_kbytes  =  15000000                  <<<  WARNING   •  vm.mmap_min_addr  =  65536   •  vm.swappiness  =  0   •  vm.zone_reclaim_mode  =  0   keep memory free
27. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  vm.dirty_background_bytes  =  531838208   •  vm.dirty_expire_cenYsecs  =  3000   •  vm.dirty_raYo  =  50   •  vm.dirty_writeback_cenYsecs  =  500   •  vm.lowmem_reserve_raYo  =  1  1  32   •  vm.max_map_count  =  524288   •  vm.min_free_kbytes  =  15000000   •  vm.mmap_min_addr  =  65536   •  vm.swappiness  =  0   •  vm.zone_reclaim_mode  =  0   Don’t fucking swap
28. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  2 Intel 2690 v2 (Sandy Bridge) •  10 Cores @ 3 Ghz   •  768 GB of Ram •  4x10Gb Ethernet EB82599 •  24*500GB SSD •  Intel 3500 •  Samsung 840 Pro Hardware
29. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014 •  16 Servers   •  12 TB Ram •  192 TB of SSD •  640 Gbit/sec Rack
30. Varnish NYC Meetup | 23 July 2014 Varnish NYC Meetup | 23 July 2014
31. Varnish NYC Meetup | 23 July 2014 FASTLY Questions?
32. Varnish NYC Meetup | 23 July 2014 FASTLY Bonus!
33. Varnish NYC Meetup | 23 July 2014 FASTLY Thank you.