Varnish workshop - ZCEU13

Varnish workshop at ZendConEU 2013

Transcript

1. Varnish
   Workshop
   By Thijs Feryn
   

   View Slide

2. Hi, my name is Thijs
   

   View Slide

3. Et j’ai un joli accent
   belge
   

   View Slide

4. I’m @ThijsFeryn
   on Twitter
   

   View Slide

5. I’m an
   at
   Evangelist
   

   View Slide

6. I’m a
   at
   board member
   

   View Slide

7. h"ps://joind.in/9280
   Please&
   give&me&
   feedback
   

   View Slide

8. Varnish ?
   

   View Slide

9. Cache?
   

   View Slide

10. Reverse proxy?
    

    View Slide

11. Loadbalancer?
    

    View Slide

12. Loadbalancer?
    HTTP accelerator
    

    View Slide

13. In&front&of&the&
    webserver
    

    View Slide

14. Primary&purpose
    Caching
    

    View Slide

15. Reverse&caching&
    proxy
    

    View Slide

16. Install &
    configure
    

    View Slide

17. curl%http://repo.varnish3cache.org/debian/GPG3
    key.txt%|%apt3key%add%3
    apt3get%update
    echo%"deb%http://repo.varnish3cache.org/
    debian/%squeeze%varnish33.0"%>>%/etc/apt/
    sources.list
    apt3get%install%varnish
    

    View Slide

18. Daemon5op6ons
    h"ps://www.varnish:cache.org/docs/trunk/reference/varnishd.html
    

    View Slide

19. Daemon5op6ons
    DAEMON_OPTS=":a5127.0.0.1:80805\
    5555555555555:T5localhost:60825\
    5555555555555:f5/etc/varnish/default.vcl5\
    5555555555555:s5malloc,3G5\
    5555555555555:p5thread_pools=8
    5555555555555:p5thread_pool_min=1605\
    5555555555555:p5thread_pool_max=30005\
    5555555555555:p5thread_pool_add_delay=25\
    5555555555555:p5connect_6meout=200s5\
    5555555555555:p5first_byte_6meout=200s5\
    5555555555555:p5between_bytes_6meout=50s5\
    5555555555555:p5session_linger=1005\
    5555555555555:p5lru_interval=205\
    5555555555555:p5listen_depth=81925\
    5555555555555:p5sess_workspace=2621445\
    5555555555555:l5300M"
    In&“/etc/default/varnish”
    

    View Slide

20. Daemon5op6ons
    

    View Slide

21. varnishadm>%param.show%3l
    Shows&all&params
    Varnish
    parameters
    

    View Slide

22. Backend
    

    View Slide

23. Listen%8080
    In&“/etc/apache2/ports.conf”
    Backend
    

    View Slide

24. 
    In&vhost&config&when&Varnish&is&
    on&same&node&as&webserver
    Backend
    

    View Slide

25. backend%default%{
    %%%%%%.host%=%"127.0.0.1";
    %%%%%%.port%=%"8080";
    }
    In&“/etc/varnish/default.vcl”
    Backend
    

    View Slide

26. View Slide

27. GET / HTTP/1.1
    Host: localhost
    User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X
    10.5; fr; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
    Accept: text/html,application/xhtml+xml,application/
    xml;q=0.9,*/*;q=0.8
    Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
    Accept-Encoding: gzip,deflate
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Keep-Alive: 115
    Connection: keep-alive
    Cache-Control: max-age=0
    Request
    

    View Slide

28. HTTP/1.1 200 OK
    Server: Apache/2.2.14 (Ubuntu)
    X-Powered-By: PHP/5.3.2-1ubuntu4.7
    Cache-Control: public, max-age=86400
    Last-Modified: Mon, 04 Apr 2011 04:13:41 +0000
    Expires: Sun, 11 Mar 1984 12:00:00 GMT
    Vary: Cookie,Accept-Encoding
    ETag: "1301890421"
    Content-Type: text/html; charset=utf-8
    Content-Length: 23562
    Date: Mon, 04 Apr 2011 09:02:26 GMT
    X-Varnish: 1886109724 1886107902
    Age: 17324
    Via: 1.1 varnish
    Connection: keep-alive
    Response
    

    View Slide

29. Cache@control
    Cache@Control&“max@age=3600,&s@
    maxage=1000,&public,&must@revalidate”
    

    View Slide

30. Max@Age
    S@maxage
    Public
    Private
    No@cache
    No@store
    Must@revalidate
    Proxy@revalidate
    TTL&for&browsers&in&seconds
    TTL&for&proxies&in&seconds
    Proxies&&&browsers&can&cache
    Only&browsers&can&cache
    Revalidate&before&dropping&from&cache
    Don’t&cache&at&all
    Browser&revalidate&before&serving&from
    cache&
    Proxy&revalidate&before&serving&from&
    cache&
    

    View Slide

31. Age
    Age&“56”
    How&old&is&
    the&cached&
    object?
    

    View Slide

32. Real&cache&duration
    Real&cache&duraWon&for&clients&=&max@age&@&age
    Real&cache&duraWon&for&proxies&=&s@maxage&@&age
    

    View Slide

33. Expires
    Expires&"Wed,&1&Jan&2014&20:00:00&GMT"
    

    View Slide

34. Expiration
    1.Vcl5(beresp."l)
    2.Cache:control5s:maxage
    3.Cache:control5max:age
    4.Expires
    

    View Slide

35. The&flow
    

    View Slide

36. View Slide

37. View Slide

38. View Slide

39. Advanced&
    flow
    

    View Slide

40. View Slide

41. View Slide

42. View Slide

43. Exercise 1
    Testing HTTP cache headers
    

    View Slide

44. Exercise 2
    Overriding Time To Live in Varnish
    

    View Slide

45. Out of the box
    

    View Slide

46. View Slide

47. Varnish obeys caching headers
    

    View Slide

48. Only get & HEAD
    No cookie & auth headers
    No set-cookie headers
    cache ttl > 0
    No vary “*”
    When will varnish cache?
    

    View Slide

49. About cookies
    

    View Slide

50. About cookies
    HTTP cookie
    request header
    via browser
    HTTP set-cookie
    response header
    via webserver
    

    View Slide

51. 200: OK
    203: Non-Authoritative Information
    300: Multiple Choices
    301: Moved Permanently
    302: Moved Temporarily
    307: Temporary Redirect
    410: Gone
    404: Not Found
    Cacheable status codes
    

    View Slide

52. Exercise 3
    Testing default behaviour
    

    View Slide

53. Monitoring
    &
    Logging
    

    View Slide

54. Varnishstat
    Real&time&stats&for&
    Varnish&instance
    

    View Slide

55. 4+05:26:25
    Hitrate.ratio:.......10......100......254
    Hitrate.avg:.....0.8486...0.7619...0.7285
    .....1760818.........6.99.........4.82.client_conn.;.Client.connections.accepted
    ....11088687........25.96........30.36.client_req.;.Client.requests.received
    .....8042715........11.98........22.02.cache_hit.;.Cache.hits
    .....2609561........11.98.........7.15.cache_miss.;.Cache.misses
    .......47104.........1.00.........0.13.backend_conn.;.Backend.conn..success
    .........610.........0.00.........0.00.backend_fail.;.Backend.conn..failures
    .....2998265........12.98.........8.21.backend_reuse.;.Backend.conn..reuses
    .......12081.........0.00.........0.03.backend_toolate.;.Backend.conn..was.closed
    .....3010356........13.98.........8.24.backend_recycle.;.Backend.conn..recycles
    ..........13.........0.00.........0.00.backend_retry.;.Backend.conn..retry
    .........520.........0.00.........0.00.fetch_head.;.Fetch.head
    .....2857965........11.98.........7.83.fetch_length.;.Fetch.with.Length
    ......151309.........2.00.........0.41.fetch_chunked.;.Fetch.chunked
    ........4404.........0.00.........0.01.fetch_close.;.Fetch.wanted.close
    .........676.........0.00.........0.00.fetch_failed.;.Fetch.failed
    .......31164.........0.00.........0.09.fetch_304.;.Fetch.no.body.(304)
    .........220...........................n_sess_mem.;.N.struct.sess_mem
    ..........53...........................n_sess.;.N.struct.sess
    .......29540...........................n_object.;.N.struct.object
    .......29561...........................n_objectcore.;.N.struct.objectcore
    ........5058...........................n_objecthead.;.N.struct.objecthead
    ........2613...........................n_waitinglist.;.N.struct.waitinglist
    ...........3...........................n_vbc.;.N.struct.vbc
    ..........22...........................n_wrk.;.N.worker.threads
    ........1789.........0.00.........0.00.n_wrk_create.;.N.worker.threads.created
    

    View Slide

56. .......31164.........0.00.........0.09.fetch_304.;.Fetch.no.body.(304)
    .........220...........................n_sess_mem.;.N.struct.sess_mem
    ..........53...........................n_sess.;.N.struct.sess
    .......29540...........................n_object.;.N.struct.object
    .......29561...........................n_objectcore.;.N.struct.objectcore
    ........5058...........................n_objecthead.;.N.struct.objecthead
    ........2613...........................n_waitinglist.;.N.struct.waitinglist
    ...........3...........................n_vbc.;.N.struct.vbc
    ..........22...........................n_wrk.;.N.worker.threads
    ........1789.........0.00.........0.00.n_wrk_create.;.N.worker.threads.created
    .......17204.........0.00.........0.05.n_wrk_queued.;.N.queued.work.requests
    ...........1...........................n_backend.;.N.backends
    .....2579403...........................n_expired.;.N.expired.objects
    .....6920374...........................n_lru_moved.;.N.LRU.moved.objects
    .....8959782........26.96........24.53.n_objwrite.;.Objects.sent.with.write
    .....1760818.........5.99.........4.82.s_sess.;.Total.Sessions
    ....11088687........25.96........30.36.s_req.;.Total.Requests
    ......436411.........2.00.........1.20.s_pass.;.Total.pass
    .....3044686........13.98.........8.34.s_fetch.;.Total.fetch
    ..3817818737......8996.58.....10454.48.s_hdrbytes.;.Total.header.bytes
    .84764679157....218712.68....232114.35.s_bodybytes.;.Total.body.bytes
    ......196150.........0.00.........0.54.sess_closed.;.Session.Closed
    .......73512.........0.00.........0.20.sess_pipeline.;.Session.Pipeline
    .......50339.........0.00.........0.14.sess_readahead.;.Session.Read.Ahead
    ....10875541........25.96........29.78.sess_linger.;.Session.Linger
    .....7865547........25.96........21.54.sess_herd.;.Session.herd
    ...591440571......1649.54......1619.56.shm_records.;.SHM.records
    ....30844446.......121.82........84.46.shm_writes.;.SHM.writes
    ...........6.........0.00.........0.00.shm_flushes.;.SHM.flushes.due.to.overflow
    

    View Slide

57. $ varnishstat -1 -f n_lru_nuked
    n_lru_nuked 0 . N LRU nuked objects
    Varnishstat
    Removed&from&
    cache&due&to&lack&of&
    memory
    

    View Slide

58. Varnishlog
    In@memory&logs&of&
    Varnish&activity
    

    View Slide

59. ...11.SessionOpen..c.12.12.12.1.53727.:80
    ...11.ReqStart.....c.12.12.12.1.53727.1401010767
    ...11.RxRequest....c.GET
    ...11.RxURL........c./
    ...11.RxProtocol...c.HTTP/1.1
    ...11.RxHeader.....c.Host:.12.12.12.6
    ...11.RxHeader.....c.User;Agent:.Mozilla/5.0.(Macintosh;.Intel.Mac.OS.X.10.8;.rv:17.0).
    Gecko/20100101.Firefox/17.0
    ...11.RxHeader.....c.Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/
    *;q=0.8
    ...11.RxHeader.....c.Accept;Language:.nl,en;q=0.7,fr;be;q=0.3
    ...11.RxHeader.....c.Accept;Encoding:.gzip,.deflate
    ...11.RxHeader.....c.Connection:.keep;alive
    ...11.VCL_call.....c.recv.lookup
    ...11.VCL_call.....c.hash
    ...11.Hash.........c./
    ...11.Hash.........c.12.12.12.6
    ...11.VCL_return...c.hash
    ...11.VCL_call.....c.miss.fetch
    ...11.Backend......c.13.default.default
    ...11.TTL..........c.1401010767.RFC.0.;1.;1.1357920021.0.1357920020.0.0
    ...11.VCL_call.....c.fetch
    ...11.TTL..........c.1401010767.VCL.120.;1.;1.1357920021.;0
    ...11.VCL_return...c.hit_for_pass
    ...11.ObjProtocol..c.HTTP/1.1
    ...11.ObjResponse..c.OK
    ...11.ObjHeader....c.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...11.ObjHeader....c.Server:.Apache
    ...11.ObjHeader....c.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...11.ObjHeader....c.Cache;Control:.no;cache,.no;store,.max;age=0
    Client&
    

    View Slide

60. ...11.VCL_return...c.hit_for_pass
    ...11.ObjProtocol..c.HTTP/1.1
    ...11.ObjResponse..c.OK
    ...11.ObjHeader....c.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...11.ObjHeader....c.Server:.Apache
    ...11.ObjHeader....c.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...11.ObjHeader....c.Cache;Control:.no;cache,.no;store,.max;age=0
    ...11.ObjHeader....c.Vary:.Accept;Encoding
    ...11.ObjHeader....c.Content;Encoding:.gzip
    ...11.ObjHeader....c.Content;Length:.119
    ...11.ObjHeader....c.Content;Type:.text/html
    ...11.Gzip.........c.u.F.;.119.336.80.80.887
    ...11.VCL_call.....c.deliver.deliver
    ...11.TxProtocol...c.HTTP/1.1
    ...11.TxStatus.....c.200
    ...11.TxResponse...c.OK
    ...11.TxHeader.....c.Server:.Apache
    ...11.TxHeader.....c.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...11.TxHeader.....c.Cache;Control:.no;cache,.no;store,.max;age=0
    ...11.TxHeader.....c.Vary:.Accept;Encoding
    ...11.TxHeader.....c.Content;Encoding:.gzip
    ...11.TxHeader.....c.Content;Type:.text/html
    ...11.TxHeader.....c.Content;Length:.119
    ...11.TxHeader.....c.Accept;Ranges:.bytes
    ...11.TxHeader.....c.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...11.TxHeader.....c.X;Varnish:.1401010767
    ...11.TxHeader.....c.Age:.0
    ...11.TxHeader.....c.Via:.1.1.varnish
    ...11.TxHeader.....c.Connection:.keep;alive
    ...11.Length.......c.119
    ...11.ReqEnd.......c.1401010767.1357920020.712090731.1357920020.727306366.0.000087738.
    Client&
    

    View Slide

61. ...13.BackendClose.;.default
    ...13.BackendOpen..b.default.127.0.0.1.51597.127.0.0.1.8080
    ...13.TxRequest....b.GET
    ...13.TxURL........b./
    ...13.TxProtocol...b.HTTP/1.1
    ...13.TxHeader.....b.Host:.12.12.12.6
    ...13.TxHeader.....b.User;Agent:.Mozilla/5.0.(Macintosh;.Intel.Mac.OS.X.10.8;.rv:17.0).
    Gecko/20100101.Firefox/17.0
    ...13.TxHeader.....b.Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/
    *;q=0.8
    ...13.TxHeader.....b.Accept;Language:.nl,en;q=0.7,fr;be;q=0.3
    ...13.TxHeader.....b.X;Forwarded;For:.12.12.12.1
    ...13.TxHeader.....b.X;Varnish:.1401010767
    ...13.TxHeader.....b.Accept;Encoding:.gzip
    ...13.RxProtocol...b.HTTP/1.1
    ...13.RxStatus.....b.200
    ...13.RxResponse...b.OK
    ...13.RxHeader.....b.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...13.RxHeader.....b.Server:.Apache
    ...13.RxHeader.....b.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...13.RxHeader.....b.Cache;Control:.no;cache,.no;store,.max;age=0
    ...13.RxHeader.....b.Vary:.Accept;Encoding
    ...13.RxHeader.....b.Content;Encoding:.gzip
    ...13.RxHeader.....b.Content;Length:.119
    ...13.RxHeader.....b.Content;Type:.text/html
    ...13.Fetch_Body...b.4(length).cls.0.mklen.1
    ...13.Length.......b.119
    ...13.BackendReuse.b.default
    Backend&
    

    View Slide

62. :a!When!writing!to!a!file,!append!to!it!rather!than!overwrite!it.
    :b!Include!backend!communication!logs
    :c!Include!client!communication!logs
    :d!Process!old!logs!on!startup
    :I5Perform!regex!filter!on!output
    :i5Include!specific!tags!into!output
    :m5Perform!regex!filter!on!tags
    :r5Read!a!shared!memory!log!dump!from!file
    :w!Dump!shared!memory!logs!to!file
    :x!Exclude!specific!tags!from!output
    :X!Perform!exclusion!regex!filter!on!output
    :O!Don’t!group!log!entries.!Don’t!use!with!Bm
    Varnishlog
    

    View Slide

63. Backend
    BackendClose
    BackendOpen
    BackendReuse
    BackendXID
    CLI
    ClientAddr
    Debug
    Error
    ExpBan
    ExpKill
    ExpPick
    Hit
    HitPass
    HttpError
    HttpGarbage
    Length
    ObjHeader
    ObjLostHeader
    ObjProtocol
    ObjRequest
    ObjResponse
    ObjStatus
    ObjURL
    ReqEnd
    ReqStart
    RxHeader
    RxLostHeader
    RxProtocol
    RxRequest
    RxResponse
    RxStatus
    RxURL
    SessionClose
    SessionOpen
    StatAddr
    StatSess
    TTL
    TxHeader
    TxLostHeader
    TxProtocol
    TxRequest
    TxResponse
    TxStatus
    TxURL
    VCL_acl
    VCL_call
    VCL_return
    VCL_trace
    WorkThread
    Tags
    

    View Slide

64. varnishlog -c -m VCL_call:hit | grep -i 'RxURL'
    Varnishlog
    Get&the&URL&for&all&hits
    

    View Slide

65. varnishlog -b -i TxURL
    Varnishlog
    Track&all&“missed”&
    URL’s
    varnishlog -O -b -i TxURL
    Ungrouped
    

    View Slide

66. varnishlog -c -i RxHeader -I User-Agent
    Varnishlog
    Get&all&user&agents
    varnishlog -O -c -i RxHeader -I User-Agent
    Ungrouped
    

    View Slide

67. varnishlog -i RxHeader
    Varnishlog
    varnishlog -c -i RxHeader
    varnishlog -b -i RxHeader
    Get&all&request&
    headers
    

    View Slide

68. varnishlog -c -i RxURL -m VCL_call:hit
    Varnishlog
    Doesn’t&work
    @i&is&done&before&@m
    

    View Slide

69. Why&did&it&miss?
    

    View Slide

70. 11 VCL_return c hit_for_pass
    Why&did&it&miss?
    TTL&<=&0
    13 RxHeader - Cache-Control: no-cache, no-
    store, max-age=0
    

    View Slide

71. 11 RxRequest c POST
    11 RxURL c /exercises/3/post.php
    11 RxProtocol c HTTP/1.1
    11 RxHeader c Host: varnish.dev
    11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS
    X 10.8; rv:18.0) Gecko/20100101 Firefox/18.0
    11 RxHeader c Accept: text/html,application/xhtml
    +xml,application/xml;q=0.9,*/*;q=0.8
    11 RxHeader c Accept-Language: nl,en;q=0.7,fr-be;q=0.3
    11 RxHeader c Accept-Encoding: gzip, deflate
    11 RxHeader c Referer: http://varnish.dev/exercises/3/post.php
    11 RxHeader c Connection: keep-alive
    11 RxHeader c Content-Type: application/x-www-form-urlencoded
    11 RxHeader c Content-Length: 11
    11 VCL_call c recv pass
    Why&did&it&miss?
    HTTP&POST
    

    View Slide

72. Was&it&stored&to&
    cache?
    

    View Slide

73. 11 TTL c 324425260 RFC 5 -1 -1
    1358344107 0 1358344107 0 5
    11 VCL_call c fetch deliver
    Was&it&stored&to&cache?
    Yes
    

    View Slide

74. 11 TTL c 324425261 VCL 120 -1 -1
    1358344313 -0
    11 VCL_return c hit_for_pass
    Was&it&stored&to&cache?
    No,&for&
    120&seconds&in&
    blacklist
    

    View Slide

75. Exercise 4
    Using varnishlog
    

    View Slide

76. What’s&the&cache&key?
    

    View Slide

77. 11 Hash c /exercises/1/public.php
    11 Hash c varnish.dev
    What’s&the&cache&key?
    Host&+&URL
    

    View Slide

78. Varnishtop
    Incremental&in@
    memory&Varnish&logs
    

    View Slide

79. list.length.109
    .....8.12.VCL_return.....deliver
    .....4.19.RxRequest......GET
    .....4.19.RxProtocol.....HTTP/1.1
    .....4.19.RxHeader.......Host:.varnish.dev
    .....4.19.RxHeader.......Connection:.keep;alive
    .....4.19.VCL_call.......recv
    .....4.19.VCL_return.....lookup
    .....4.19.VCL_call.......hash
    .....4.19.Hash...........varnish.dev
    .....4.19.VCL_return.....hash
    .....4.19.VCL_call.......hit
    .....4.19.VCL_call.......deliver
    .....4.19.TxProtocol.....HTTP/1.1
    .....4.19.TxHeader.......Server:.Apache
    .....4.19.TxHeader.......Content;Encoding:.gzip
    .....4.19.TxHeader.......Accept;Ranges:.bytes
    .....4.19.TxHeader.......Via:.1.1.varnish
    .....4.19.TxHeader.......Connection:.keep;alive
    .....3.76.TxHeader.......Age:.26
    .....3.12.RxURL........../exercises/5/vary.php
    .....3.12.RxHeader.......Accept:.text/html,application/xhtml+xml,application/
    xml;q=0.9,*/*;q=0.8
    .....3.12.Hash.........../exercises/5/vary.php
    .....3.12.TxStatus.......200
    .....3.12.TxResponse.....OK
    .....3.12.TxHeader.......X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    .....3.12.TxHeader.......Vary:.User;Agent,Accept;Encoding
    .....3.12.TxHeader.......Cache;Control:.public,must;revalidate,s;maxage=.100
    Unfiltered&list
    

    View Slide

80. :b!Include!backend!communication!logs
    :c!Include!client!communication!logs
    :d!Process!old!logs!on!startup
    :I5Perform!regex!filter!on!output
    :i5Include!specific!tags!into!output
    :m5Perform!regex!filter!on!tags
    :r5Read!a!shared!memory!log!dump!from!file
    :p5Specifies!!the!!number!!of!!seconds!!to!!measure!!over
    :x!Exclude!specific!tags!from!output
    :X!Perform!exclusion!regex!filter!on!output
    :O!Don’t!group!log!entries.!Don’t!use!with!Bm
    :f!Sort!and!group!only!on!the!first!field!of!each!log!entry
    Varnishtop
    

    View Slide

81. varnishtop -c -i RxHeader -I ^User-Agent
    Varnishtop
    Get&top&user&agents
    

    View Slide

82. varnishtop -b -i TxURL
    Varnishtop
    Get&top&missed&URL’s
    

    View Slide

83. varnishtop -c -i VCL_call
    Varnishtop
    Get&top&VCL&calls
    

    View Slide

84. varnishtop -c -i RxURL
    Varnishtop
    Most&popular&URL’s
    

    View Slide

85. Exercise 5
    Using varnishtop
    

    View Slide

86. Cache variations
    

    View Slide

87. Vary: Accept-Language
    Cache&variaWons
    Keep&
    seperate&cache&objects&
    for&each&varation&of&the&
    header
    Vary: X-myHeader
    Vary: Accept-Encoding
    

    View Slide

88. •Don’t5vary5‘*’
    •Only5vary5request5headers
    •Custom5headers5only5work5if5browser5
    sends5them
    •VCL5magic5(coming!up!later)
    Cache&variaWons
    

    View Slide

89. Exercise 6
    Cache variations
    

    View Slide

90. Varnish&
    Configuration&
    Language
    

    View Slide

91. Varnish&
    Configuration&
    Language
    The&Varnish&ConfiguraWon&
    Language&allows&you&to&define&
    your&caching&policy.&You&write&
    VCL&code&which&Varnish&will&
    parse,&translate&to&C&code,&
    compile&and&link&to.
    

    View Slide

92. Out&of&the&box&behaviour
    Remember&
    the&flow!
    

    View Slide

93. sub vcl_recv {
    if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
    set req.http.X-Forwarded-For =
    req.http.X-Forwarded-For + “, ” + client.ip;
    } else {
    set req.http.X-Forwarded-For = client.ip;
    }
    }
    if (req.request != “GET” &&
    req.request != “HEAD” &&
    req.request != “PUT” &&
    req.request != “POST” &&
    req.request != “TRACE” &&
    req.request != “OPTIONS” &&
    req.request != “DELETE”) {
    /* Non-RFC2616 or CONNECT which is weird. */
    return (pipe);
    }
    if (req.request != “GET” && req.request != “HEAD”) {
    /* We only deal with GET and HEAD by default */
    return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
    }
    return (lookup);
    }
    

    View Slide

94. sub vcl_pipe {
    # Note that only the first request to the backend will have
    # X-Forwarded-For set. If you use X-Forwarded-For and want to
    # have it set for all requests, make sure to have:
    # set bereq.http.connection = "close";
    # here. It is not set by default as it might break some broken web
    # applications, like IIS with NTLM authentication.
    return (pipe);
    }
    sub vcl_pass {
    return (pass);
    }
    sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
    hash_data(req.http.host);
    } else {
    hash_data(server.ip);
    }
    return (hash);
    }
    

    View Slide

95. sub vcl_hit {
    return (deliver);
    }
    sub vcl_miss {
    return (fetch);
    }
    sub vcl_fetch {
    if (beresp.ttl <= 0s ||
    beresp.http.Set-Cookie ||
    beresp.http.Vary == "*") {
    ! ! /*
    ! ! * Mark as "Hit-For-Pass" for the next 2 minutes
    ! ! */
    ! ! set beresp.ttl = 120 s;
    ! ! return (hit_for_pass);
    }
    return (deliver);
    }
    sub vcl_deliver {
    return (deliver);
    }
    

    View Slide

96. sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";
    set obj.http.Retry-After = "5";
    synthetic {"
    
    br/>"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    
    
    "} + obj.status + " " + obj.response + {"
    
    
    Error "} + obj.status + " " + obj.response + {"
    "} + obj.response + {"
    Guru Meditation:
    XID: "} + req.xid + {"
    
    Varnish cache server
    
    
    "};
    return (deliver);
    }
    sub vcl_init {
    ! return (ok);
    }
    sub vcl_fini {
    ! return (ok);
    }
    

    View Slide

97. VCL&
    syntax
    

    View Slide

98. //This is comment
    #This is comment too
    /* More comment
    A lot more comment*/
    Comment
    

    View Slide

99. if(req.url == "/") {
    ! error 200 "OK dude";
    } elseif(req.url == "/notfound") {
    ! error 404 "Not found man";
    } else {
    ! error 200 "Else";
    }
    CondiWonal
    

    View Slide

100. if(req.url == "/") {
     ! set req.backend = otherBackend;
     }
     Set&variables
     

     View Slide

101. Common&objects
     ✓Req:5request5object
     ✓Bereq:5backend5request5object
     ✓Beresp:5backend5response5object
     ✓Obj:5cache5object
     ✓Resp:5output5object
     ✓Client:5client5information
     ✓Server:5server5information
     

     View Slide

102. Variable&availability
     

     View Slide

103. .host
     Host5name5or5IP5address5of5a5backend.
     .port
     Service5name5or5port5number5of5a5backend.
     .connect_timeout
     Backend5marked5“unhealthy”5when5a5connection5fails5after5x5time.
     .first_byte_timeout
     Backend5marked5“unhealthy”5when5the5first5byte5isn’t5received5after5x5time.
     .between_byte_timeout
     Backend5marked5“unhealthy”5when5the5byte5sequence5isn’t5coming5in5fast5
     enough.
     Backend&variables
     

     View Slide

104. .url
     Specify5a5URL5to5request5from5the5backend.5Defaults5to5"/".
     .request
     Specify5a5full5HTTP5request5using5multiple5strings.5.request5will5have5\r\n5
     automatically5inserted5after5every5string.5If5specified,5.request5will5take5
     precedence5over5.url.
     .window
     How5many5of5the5latest5polls5we5examine5to5determine5backend5health.5
     Defaults5to58.
     .threshold
     How5many5of5the5polls5in5.window5must5have5succeeded5for5us5to5consider5the
     backend5healthy.5Defaults5to53.
     Probe&variables&(1)
     

     View Slide

105. .initial
     How5many5of5the5probes5are5considered5good5when5Varnish5starts.5Defaults5to
     the5same5amount5as5the5threshold.
     .expected_response
     The5expected5backend5HTTP5response5code.5Defaults5to5200.
     .interval
     Defines5how5often5the5probe5should5check5the5backend.5Default5is5every555
     seconds.
     .timeout
     How5fast5each5probe5times5out.5Default5is525seconds.
     Probe&variables&(2)
     

     View Slide

106. client.ip
     The5client's5IP5address.
     client.identity
     Identification5of5the5client,5used5to5load5balance5in5the5client5director.
     server.hostname
     The5host5name5of5the5server.
     server.identity
     The5identity5of5the5server,5as5set5by5the5:i5parameter.5If5the5:i5parameter5is5not5
     passed5to5varnishd,5server.identity5will5be5set5to5the5name5of5the5instance,5as5
     specified5by5the5:n5parameter.
     server.ip
     The5IP5address5of5the5socket5on5which5the5client5connection5was5received.
     server.port
     The5port5number5of5the5socket5on5which5the5client5connection5was5received.
     Client&&&server&variables
     

     View Slide

107. req.request
     The5request5type5(e.g.5"GET",5"HEAD").
     req.url
     The5requested5URL.
     req.proto
     The5HTTP5protocol5version5used5by5the5client.
     req.backend
     The5backend5to5use5to5service5the5request.
     req.backend.healthy
     Whether5the5backend5is5healthy5or5not.5Requires5an5active5probe5to5be5set5
     on5the5backend.
     req.http.header
     The5corresponding5HTTP5header.
     Request&variables&(1)
     

     View Slide

108. req.hash_always_miss
     Force5a5cache5miss5for5this5request.5If5set5to5true5Varnish5will5disregard5any5
     existing5objects5and5always5(re)fetch5from5the5backend.
     req.hash_ignore_busy
     Ignore5any5busy5object5during5cache5lookup.5You5would5want5to5do5this5if5
     you5have5two5server5looking5up5content5from5each5other5to5avoid5potential5
     deadlocks.
     req.can_gzip
     Does5the5client5accept5the5gzip5transfer5encoding.
     req.restarts
     A5count5of5how5many5times5this5request5has5been5restarted.
     Unique5ID5of5this5request.
     Request&variables&(2)
     

     View Slide

109. req.esi
     Boolean.5Set5to5false5to5disable5ESI5processing5regardless5of5any5value5in5
     beresp.do_esi.5Defaults5to5true.5This5variable5is5subject5to5change5in5future5
     versions,5you5should5avoid5using5it.
     req.esi_level
     A5count5of5how5many5levels5of5ESI5requests5we're5currently5at.
     req.grace
     Set5to5a5period5to5enable5grace.
     req.xid
     Unique5ID5of5this5request.
     req.hash
     The5hash5key5used5to5refer5to5an5object5in5the5cache.5Used5when5both5
     reading5from5and5writing5to5the5cache.
     Request&variables&(3)
     

     View Slide

110. bereq.request
     The5request5type5(e.g.5"GET",5"HEAD").
     bereq.url
     The5requested5URL.
     bereq.proto
     The5HTTP5protocol5version5used5to5talk5to5the5server.
     bereq.http.header
     The5corresponding5HTTP5header.
     bereq.connect_timeout
     The5time5in5seconds5to5wait5for5a5backend5connection.
     bereq.first_byte_timeout
     The5time5in5seconds5to5wait5for5the5first5byte5from5the5backend.5Not5
     available5in5pipe5mode.
     bereq.between_bytes_timeout
     The5time5in5seconds5to5wait5between5each5received5byte5from5the5backend.5
     Not5available5in5pipe5mode.
     Backend&request&variables
     

     View Slide

111. beresp.do_stream
     Deliver5the5object5to5the5client5directly5without5fetching5the5whole5object5
     into5varnish.5If5this5request5is5pass'ed5it5will5not5be5stored5in5memory.5As5of5
     Varnish5Cache53.05the5object5will5marked5as5busy5as5it5is5delivered5so5only5
     client5can5access5the5object.
     beresp.do_esi
     Boolean.5ESI:process5the5object5after5fetching5it.5Defaults5to5false.5Set5it5to5
     true5to5parse5the5object5for5ESI5directives.5Will5only5be5honored5if5req.esi5is5
     true.
     beresp.do_gzip
     Boolean.5Gzip5the5object5before5storing5it.5Defaults5to5false.
     beresp.do_gunzip
     Boolean.5Unzip5the5object5before5storing5it5in5the5cache.5Defaults5to5false.
     beresp.proto
     The5HTTP5protocol5version5used5the5backend5replied5with.
     beresp.storage
     Set5to5force5Varnish5to5save5this5object5to5a5particular5storage5backend.
     Backend&response&variables&(1)
     

     View Slide

112. beresp.status
     The5HTTP5status5code5returned5by5the5server.
     beresp.response
     The5HTTP5status5message5returned5by5the5server.
     beresp.ttl
     The5object's5remaining5time5to5live,5in5seconds.5beresp.ttl5is5writable.
     beresp.grace
     Set5to5a5period5to5enable5grace.
     beresp.saintmode
     Set5to5a5period5to5enable5saint5mode.
     beresp.backend.name
     Name5of5the5backend5this5response5was5fetched5from.
     beresp.backend.ip
     IP5of5the5backend5this5response5was5fetched5from.
     beresp.backend.port
     Port5of5the5backend5this5response5was5fetched5from.
     Backend&response&variables&(2)
     

     View Slide

113. obj.proto
     The5HTTP5protocol5version5used5when5the5object5was5retrieved.
     obj.status
     The5HTTP5status5code5returned5by5the5server.
     obj.response
     The5HTTP5status5message5returned5by5the5server.
     obj.ttl
     The5object's5remaining5time5to5live,5in5seconds.5obj.ttl5is5writable.
     obj.lastuse
     The5approximate5time5elapsed5since5the5object5was5last5requests,5in5
     seconds.5This5variable5is5also5available5in5vcl_deliver.
     obj.hits
     The5approximate5number5of5times5the5object5has5been5delivered.5A5value5of5
     05indicates5a5cache5miss.5This5variable5is5also5available5in5vcl_deliver.
     obj.grace
     The5object's5grace5period5in5seconds.5obj.grace5is5writable.
     obj.http.header
     The5corresponding5HTTP5header.
     Cache&object&variables
     

     View Slide

114. resp.proto
     The5HTTP5protocol5version5to5use5for5the5response.
     resp.status
     The5HTTP5status5code5that5will5be5returned.
     resp.response
     The5HTTP5status5message5that5will5be5returned.
     resp.http.header
     The5corresponding5HTTP5header.
     Client&response&variables
     

     View Slide

115. Reminder!
     

     View Slide

116. Working&
     with&
     backends
     

     View Slide

117. backend default {
     .host = "1.2.3.4";
     .port = "http";
     }
     Backend&config
     backend default {
     .host = "localhost";
     .port = "8080";
     }
     Simple
     

     View Slide

118. backend default {
     .host = "1.2.3.4";
     .port = "http";
     .connect_timeout = 1s;
     .first_byte_timeout = 5s;
     .between_bytes_timeout = 2s;
     }
     Backend&config
     More
     

     View Slide

119. backend default {
     .host = "1.2.3.4";
     .port = "http";
     }
     backend other {
     .host = "1.2.3.5";
     .port = "http";
     }
     sub vcl_recv {
     if (req.http.host ~ "^(www.)?example.com$") {
     set req.backend = other;
     }
     }
     Select&backend
     

     View Slide

120. Probes
     

     View Slide

121. backend default {
     .host = "1.2.3.4";
     .port = "http";
     .probe = {
     .url = "/test.jpg";
     .timeout = 0.3 s;
     .window = 8;
     .threshold = 3;
     .initial = 3;
     }
     }
     Probes
     Test&if&
     healthy
     

     View Slide

122. probe healthcheck {
     .url = "/status.cgi";
     .interval = 60s;
     .timeout = 0.3 s;
     .window = 8;
     .threshold = 3;
     .initial = 3;
     .expected_response = 200;
     }
     backend www {
     .host = "www.example.com";
     .port = "http";
     .probe = healthcheck;
     }
     Probes
     

     View Slide

123. probe rawprobe {
     # NB: \r\n automatically inserted after each
     string!
     .request =
     "GET / HTTP/1.1"
     "Host: www.foo.bar"
     "Connection: close";
     }
     Probes
     

     View Slide

124. Directors
     

     View Slide

125. A!director!is!a!logical!group!of!
     backend!servers!clustered!
     together!for!redundancy.!The!
     basic!role!of!the!director!is!to!
     let!Varnish!choose!a!backend!
     server!amongst!several!so!if!one!
     is!down!another!can!be!used.
     

     View Slide

126. Directors
     ✓Round:robin
     ✓Random
     ✓Client
     ✓Hash
     ✓Fallback
     ✓DNS
     

     View Slide

127. backend.one.{
     ....host.=.“localhost”;
     ....port.=.“80”;
     }
     backend.two.{
     ....host.=.“127.0.0.1”;
     ....port.=.“81”;
     }
     director.localhosts.round;robin.{
     ....{..backend.=.one;.}
     ....{..backend.=.two;.}
     ....{..backend.=.{.
     .............host.=.“localhost”;.
     .............port.=.“82”;.
     ........}.
     ....}
     }
     sub.vcl_recv.{
     ....set.req.backend.=.localhosts;
     }
     Round@robin&director
     

     View Slide

128. backend.one.{
     ....host.=.“localhost”;
     ....port.=.“80”;
     }
     backend.two.{
     ....host.=.“127.0.0.1”;
     ....port.=.“81”;
     }
     director.localhosts.random.{
     ....{..backend.=.one;..weight=4;}
     ....{..backend.=.two;..weight=6;}
     ....{..backend.=.{.
     .............host.=.“localhost”;.
     .............port.=.“82”;.
     ........}.
     ....}
     }
     sub.vcl_recv.{
     ....set.req.backend.=.localhosts;
     }
     Random&director
     

     View Slide

129. director.localhosts.client.{
     ....{..backend.=.one;..weight=1;.}
     ....{..backend.=.two;..weight=1;.}
     }
     sub.vcl_recv.{
     ....set.req.backend.=.localhosts;
     ....//Load.balance.by.URL
     ....set.client.identity.=.req.url;
     ....//Load.balance.by.client.IP,.this.is.the.default.
     ....set.client.identity.=.client.ip;
     ....//Load.balance.by.user.agent
     ....set.client.identity.=.req.http.user;agent;
     }
     Client&director
     Uses&
     client.identity
     

     View Slide

130. director.localhosts.hash.{
     ....{..backend.=.one;..weight=1;.}
     ....{..backend.=.two;..weight=1;.}
     }
     sub.vcl_recv.{
     ....set.req.backend.=.localhosts;
     }
     Hash&director
     It&will&use&the&value&of&req.hash,&just&as&the&normal&
     cache@lookup&methods
     This&is&useful&is&you&are&using&Varnish&to&load&
     balance&in&front&of&other&Varnish&caches&or&other&
     web&accelerators&as&objects&won't&be&duplicated&
     across&caches.
     

     View Slide

131. director.localhosts.fallback.{
     ....{..backend.=.one;}
     ....{..backend.=.two;}
     }
     sub.vcl_recv.{
     ....set.req.backend.=.localhosts;
     }
     Fallback&director
     Picks&first&
     healthy&
     backend
     Order&
     matters
     

     View Slide

132. director.directorname.dns.{
     .........list.=.{
     .................host_header.=."www.example.com";
     .................port.=."80";
     .................connect_timeout.=.0.4s;
     ................"192.168.15.0"/24;
     ................"192.168.16.128"/25;
     ........}
     .........ttl.=.5m;
     .........suffix.=."internal.example.net";
     }
     DNS&director
     Autogenerates&backends&in&IP&ranges
     

     View Slide

133. director.directorname.dns.{
     .........list.=.{
     .................host_header.=."www.example.com";
     .................port.=."80";
     .................connect_timeout.=.0.4s;
     ................"192.168.15.0"/24;
     ................"192.168.16.128"/25;
     ........}
     .........ttl.=.5m;
     .........suffix.=."internal.example.net";
     }
     DNS&director
     Hostname&=&host_header&+&suffix
     www.example.com.internal.example.net
     

     View Slide

134. director.directorname.dns.{
     .........list.=.{
     .................host_header.=."www.example.com";
     .................port.=."80";
     .................connect_timeout.=.0.4s;
     ................"192.168.15.0"/24;
     ................"192.168.16.128"/25;
     ........}
     .........ttl.=.5m;
     .........suffix.=."internal.example.net";
     }
     DNS&director
     Resolves&hostname&&&caches&result
     Backend&selecWon&based&on&result&of&DNS&resolving
     

     View Slide

135. Exercise 7
     Playing with backends, probes &
     directors
     

     View Slide

136. Cache
     invalidation
     

     View Slide

137. Don’t wait for
     cache objects
     to expire
     

     View Slide

138. There!are!only!two!hard!
     things!in!Computer!Science:!
     cache!invalidaKon,!naming!
     things,!and!offBbyBone!errors.
     

     View Slide

139. ✓Purge
     ✓Ban
     ✓Always5miss
     Cache&invalidaWon
     

     View Slide

140. ✓Purge5VCL5method
     ✓Removes5specific5cache5item
     ✓Frees5up5memory
     ✓Evicts5variants
     ✓Next5client5triggers5the5refresh
     ✓No5backup5if5backend5is5down
     Purge
     

     View Slide

141. sub.vcl_recv.{
     ........if.(req.request.==."PURGE").{
     ................return.(lookup);
     ........}
     }
     sub.vcl_hit.{
     ........if.(req.request.==."PURGE").{
     ................purge;
     ................error.200."Purged";
     ........}
     }
     sub.vcl_miss.{
     ........if.(req.request.==."PURGE").{
     ................error.404."Not.in.cache";
     ........}
     }
     Purge
     HTTP&call&
     to&purge
     

     View Slide

142. ✓Implement5in5your5CMS
     ✓HTTP52005:5removed5from5cache
     ✓HTTP5404:5not5in5cache
     ✓Some5CMS5plugins5require52005for5miss()
     Purge
     curl.;X.PURGE.http://varnish.dev/your;page
     

     View Slide

143. ✓Ban5VCL5method5
     ✓Ban5command5line5action
     ✓Ban.url5command5line5action
     ✓Pattern5(regex)5based
     ✓Adds5ban5to5ban:list
     ✓No5memory5freed5up5immediately
     ✓Evicts5variants
     ✓Extremely5fast5and5powerful
     Ban
     

     View Slide

144. sub.vcl_recv.{
     ........if.(req.request.==."PURGE").{
     ban("req.http.host.==.".+.req.http.host.+.".
     &&.req.url.==.".+.req.url);
     error.200."Banned";
     ........}
     }
     Ban
     sub.vcl_recv.{
     ........if.(req.request.==."PURGE").{
     ............ban("req.http.host.==.".+.req.http.host.+.
     ".&&.req.url.~.".+.req.url);
     error.200."Banned";
     ........}
     } Ban&URL&
     pattern
     Ban&URL
     Name&
     doesn’t&
     matter
     

     View Slide

145. ✓Removes5old5bans5from5the5ban:list
     ✓Separate5thread
     ✓Do5ban.list5on5CLI5to5see5the5ban:list
     ✓ban_lurker_sleep5parameter
     ✓Has5no5access5to5“req”5object
     ✓Smart5bans5are5ban5lurker5friendly
     Ban&lurker
     

     View Slide

146. ✓Ban5lurker5friendly
     ✓Use5beresp.x:url5&5beresp.x:host5instead5of5
     req.url5&5req.http.host
     Smart&bans
     

     View Slide

147. sub.vcl_recv.{
     ........if.(req.request.==."PURGE").{
     ............ban("obj.http.x;host.==.".+.req.http.host.+.
     ".&&.obj.http.x;url.==.".+.req.url);
     ............error.200."Banned";
     ........}
     }
     sub.vcl_fetch.{
     ........set.beresp.http.x;url.=.req.url;
     ........set.beresp.http.x;host.=.req.http.host;
     }
     sub.vcl_deliver.{
     ........unset.resp.http.x;url;
     ........unset.resp.http.x;host;
     }
     Smart&bans
     Ban&lurker&
     friendly
     

     View Slide

148. ✓Used5for5reloading5content,5not5evicting
     ✓Uses5set5req.hash_always_miss5=5true;
     ✓Looks5up5object,5but5ignores5it
     ✓Doesn’t5work5well5with5variations
     ✓Because5return(pass);5doesn’t5store5fetch5
     result5in5cache
     ✓Because5purge();5will5fail5when5your5
     backend5is5down
     Always&miss&
     

     View Slide

149. sub.vcl_recv.{
     ........if.(req.request.==."REFRESH").{
     ................set.req.request.=."GET";
     ................set.req.hash_always_miss.=.true;
     ........}
     }
     }
     Refresh/always&miss
     

     View Slide

150. varnishstat.;1.;f.n_object
     Measure&with&varnishstat
     Number&
     of&objects&in&
     cache
     

     View Slide

151. Protect&your&purges
     

     View Slide

152. acl.purge.{
     .."localhost";
     .."217.21.177.0"/24;
     .."10.10.10.1";
     .."some.host.com";
     }
     sub.vcl_recv.{....
     ..if.(req.request.==."PURGE").{
     ....if.(!client.ip.~.purge).{
     ......error.405."Not.allowed.";
     ....}....
     ....ban("obj.http.x;url.==.".+.req.url.+".&&
     ...obj.http.x;host.==.".+.req.http.host);
     ....error.200."Purged";
     ..}
     }
     Protect&your&purges
     Use&an&ACL
     

     View Slide

153. Exercise 8
     Invalidation
     

     View Slide

154. RELOAD !
     

     View Slide

155. ✓/etc/init.d/varnish5reload
     ✓/etc/init.d/varnish5restart
     ✓varnishadm5(vcl.load5&5vcl.use)
     ✓telnet5(vcl.load5&5vcl.use)
     Reloading&Varnish&VCL
     

     View Slide

156. ✓/etc/init.d/varnish5reload
     ✓New5VCL5is5loaded
     ✓Keeps5objects5in5memory
     Service&reload
     

     View Slide

157. ✓/etc/init.d/varnish5restart
     ✓New5VCL5is5loaded
     ✓All5objects5are5removed5from5memory
     Service&restart
     

     View Slide

158. ✓Call5varnishadm5on5CLI
     ✓Call5vcl.list5for5an5overview
     ✓call5vcl.load5bla5/path/vcl/default.vcl
     ✓On5compilation5failure,5errors5are5shown
     ✓Call5vcl.use5bla5to5confirm5new5VCL
     Varnishadm
     

     View Slide

159. ✓Open5telnet5connection
     ✓Call5vcl.list5for5an5overview
     ✓call5vcl.load5bla5/path/vcl/default.vcl
     ✓On5compilation5failure,5errors5are5shown
     ✓Call5vcl.use5bla5to5confirm5new5VCL
     Telnet
     See&
     next&slide
     

     View Slide

160. /usr/sbin/varnishd.;P./var/run/
     varnishd.pid.;a.:80.;f./home/data/
     default.vcl.;T.127.0.0.1:6082.;u.
     varnish.;g.varnish.;t.120.;p.vcl_dir./
     home/data.;w.5,500,300.;s.malloc,256m.;
     S./etc/varnish/secret.;n.lucid32
     Telnet Running&
     process
     

     View Slide

161. $.telnet.127.0.0.1.6082
     Telnet
     

     View Slide

162. $.telnet.127.0.0.1.6082
     Trying.::1...
     Trying.127.0.0.1...
     Connected.to.localhost.
     Escape.character.is.'^]'.
     107.59......
     ayfmdmirgrluoqtukkueawfbrpmmvtkw
     Authentication.required.
     Telnet&authenWcaWon
     Auth&
     challenge
     

     View Slide

163. Telnet&authenWcaWon
     $.cat./etc/varnish/secret.
     ab9cae99;7905;42e8;9f28;
     b405e684ce93
     Secret&
     key
     

     View Slide

164. Telnet&authenWcaWon
     ✓Challenge
     ✓Newline5character5(\x0A)
     ✓Secret5key
     ✓Newline5character5(\x0A)
     ✓Challenge
     ✓Newline5character5(\x0A)
     ✓SHA2565string
     Compose&
     auth&string
     

     View Slide

165. Telnet&authenWcaWon
     $challenge = $argv[1];
     $secret = trim(fgets(STDIN));
     $pack = $challenge . "\x0A" . $secret .
     "\x0A" . $challenge . "\x0A";
     $key = hash('sha256', $pack);
     echo $key.PHP_EOL;
     Challenge Secret&
     key
     Result
     

     View Slide

166. Telnet&authenWcaWon
     $.cat./etc/varnish/secret.|.php.
     secret.php.
     ayfmdmirgrluoqtukkueawfbrpmmvtkw.
     4fac6b3e568d8b307300d6fa7a1b92ce4e764ce
     46cabed4783ffb0e937d86ff3
     

     View Slide

167. $.telnet.127.0.0.1.6082
     Trying.::1...
     Trying.127.0.0.1...
     Connected.to.localhost.
     Escape.character.is.'^]'.
     107.59......
     ayfmdmirgrluoqtukkueawfbrpmmvtkw
     Authentication.required.
     auth.
     4fac6b3e568d8b307300d6fa7a1b92ce4e764ce
     46cabed4783ffb0e937d86ff3
     200.204
     Telnet&authenWcaWon
     

     View Slide

168. Exercise 9
     Load VCL via Varnishadm & telnet
     

     View Slide

169. Custom&
     VCL
     

     View Slide

170. sub.vcl_recv.{
     unset.req.http.cookie;
     }
     sub.vcl_fetch.{
     unset.beresp.http.set;cookie;
     }
     Remove&all&cookies
     

     View Slide

171. sub.vcl_recv.{
     set.req.http.Cookie.=.regsuball(req.http.Cookie,.
     "__utm.=[^;]+(;.)?",."");
     set.req.http.Cookie.=.regsuball(req.http.Cookie,.
     "utmctr=[^;]+(;.)?",."");
     set.req.http.Cookie.=.regsuball(req.http.Cookie,.
     "utmcmd.=[^;]+(;.)?",."");
     set.req.http.Cookie.=.regsuball(req.http.Cookie,.
     "utmccn.=[^;]+(;.)?",."");
     if.(req.http.cookie.~."^.*$").{
     unset.req.http.cookie;
     }
     }
     Remove&GA&cookies
     

     View Slide

172. sub.vcl_recv.{
     ...return(lookup);
     }
     sub.vcl_hash.{
     ...hash_data(req.http.cookie);
     }
     Cache&cookies
     

     View Slide

173. sub.vcl_hash.{
     ....if(req.http.Cookie.~."lang"){
     ........hash_data(regsuball(req.http.Cookie,."^.
     +;?.?lang=([a;zA;Z0;9]+)(.|;|.;).*$","\1"));
     ....}
     }
     Cache&some&cookies
     

     View Slide

174. sub.vcl_recv.{
     ...return(lookup);
     }
     sub.vcl_fetch.{
     ...return(deliver);
     }
     Ignore&cookies
     

     View Slide

175. sub.vcl_recv.{
     . if.(req.url.~."^[^?]*\.(bmp|bz2|css|doc|
     eot|flv|gif|gz|ico|jpeg|jpg|js|less|mp[34]|
     pdf|png|rar|rtf|swf|tar|tgz|txt|wav|woff|
     xml|zip)(\?.*)?$").{
     ........unset.req.http.Cookie;
     ........return.(lookup);
     ....}
     }
     Cache&staWc&files
     

     View Slide

176. sub.vcl_fetch.{
     . if.(req.url.~."^[^?]*\.(bmp|bz2|css|doc|
     eot|flv|gif|gz|ico|jpeg|jpg|js|less|mp[34]|
     pdf|png|rar|rtf|swf|tar|tgz|txt|wav|woff|
     xml|zip)(\?.*)?$").{
     ........unset.beresp.http.set;cookie;
     ....}
     }
     Cache&staWc&files
     

     View Slide

177. sub.vcl_deliver.{
     ....set.resp.http.x;cache;hits.=.obj.hits;
     ....if.(obj.hits.>.0).{
     ........set.resp.http.x;cache.=."hit";
     ....}.else.{
     ........set.resp.http.x;cache.=."miss";
     ....}
     }
     Debug
     

     View Slide

178. sub.vcl_recv.{
     set.req.http.Host.=.regsub(req.http.Host,.":[0;9]+",."");
     if.(req.url.~."\#").{
     set.req.url.=.regsub(req.url,."\#.*$",."");
     }
     if.(req.url.~."\?$").{
     set.req.url.=.regsub(req.url,."\?$",."");
     }
     }
     SaniWze&URL
     

     View Slide

179. sub.vcl_recv.{
     if.(req.url.~."(\?|&)(utm_source|utm_medium|utm_campaign|
     gclid|cx|ie|cof|siteurl)=").{
     set.req.url.=.regsuball(req.url,."&(utm_source|
     utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A;
     z0;9_\;\.%25]+)",."");
     ...set.req.url.=.regsuball(req.url,."\?(utm_source|
     utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A;z0;9_
     \;\.%25]+)",."?");
     }
     }
     SaniWze&URL&(2)
     

     View Slide

180. Exercise 10
     Custom VCL
     

     View Slide

181. Grace mode
     

     View Slide

182. ✓A&graced&object&is&an&object&that&has&expired,&but&is&
     still&kept&in&cache
     ✓Grace&mode&is&when&Varnish&uses&a&graced&object
     ✓Used&when&backend&is&slow&or&down
     ✓req.grace&defines&how&long&an&overdue&an&object&can&
     be&for&Varnish&to&still&consider&it&for&grace&mode.
     ✓beresp.grace&defines&how&long&past&the&beresp.ttl@
     time&Varnish&will&keep&an&object
     

     View Slide

183. sub.vcl_recv{
     ....if.(req.backend.healthy).{
     ........set.req.grace.=.0s;
     ....}.else.{
     ........set.req.grace.=.20s;
     ....}
     }
     sub.vcl_fetch{
     ....set.beresp.grace.=.20s;
     }
     

     View Slide

184. probe.healthcheck.{
     ....url.=."/some/url";
     ....interval.=.1s;
     ....timeout.=.1s;
     ....window.=.1;
     ....threshold.=.1;
     ....initial.=.1;
     ....expected_response.=.200;
     }
     backend.default.{
     ...host.=."localhost";
     ...port.=."8080";
     ...probe.=.healthcheck;
     }
     sub.vcl_recv{
     ....if.(req.backend.healthy).{
     ........set.req.grace.=.0s;
     ....}.else.{
     ........set.req.grace.=.20s;
     ....}
     }
     sub.vcl_fetch{
     ....set.beresp.grace.=.20s;
     }
     With&
     backend&
     probe
     

     View Slide

185. Exercise 11
     Grace mode
     

     View Slide

186. Edge&Side&Includes
     

     View Slide

187. 
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

     View Slide

188. header.php
     menu.php main.php
     footer.php
     TTL&5s
     No&caching
     TTL&10s
     TTL&2s
     

     View Slide

189. function esi($file){
     if(isset($_SERVER['Surrogate-Capability'])
     && preg_match('/^(.+)="ESI/1.0$/',$_SERVER['Surrogate-Capability'])){
     header("Surrogate-Control: content='ESI/1.0'");
     $url = 'http://'.
     $_SERVER['HTTP_HOST'].substr($_SERVER['SCRIPT_NAME'],
     0,1+strrpos($_SERVER['SCRIPT_NAME'],'/')).$file;
     echo "".PHP_EOL;
     } else {
     include($file);
     }
     }
     ESI&PHP&helper&funcWon&(1)
     

     View Slide

190. function esi($file){
     if(isset($_SERVER['Surrogate-Capability'])
     && preg_match('/^(.+)="ESI/1.0$/',$_SERVER['Surrogate-Capability'])){
     header("Surrogate-Control: content='ESI/1.0'");
     $url = 'http://'.
     $_SERVER['HTTP_HOST'].substr($_SERVER['SCRIPT_NAME'],
     0,1+strrpos($_SERVER['SCRIPT_NAME'],'/')).$file;
     echo "".PHP_EOL;
     } else {
     include($file);
     }
     }
     ESI&PHP&helper&funcWon&(2)
     

     View Slide

191. 
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     
     

     View Slide

192. ESI&VCL
     sub.vcl_recv.{
     ....set.req.http.Surrogate;Capability="key=ESI/1.0";
     }
     sub.vcl_fetch.{
     . if(beresp.http.Surrogate;Control~"ESI/1.0").{
     .........unset.beresp.http.Surrogate;Control;
     . ....set.beresp.do_esi=true;
     ....}
     }.
     

     View Slide

193. require_once __DIR__.'/vendor/autoload.php';
     use Symfony\Component\HttpKernel\HttpCache\Esi;
     use Symfony\Component\HttpFoundation\Request;
     use Symfony\Component\HttpFoundation\Response;
     $esi = new Esi;
     $request = Request::createFromGlobals();
     $response = new Response();
     Symfony&components&(1)
     

     View Slide

194. $content = 'Exercise 12';
     $content .= 'Edge Side Includes';
     $content .= date("Y-m-d H:i:s").'
     ';
     if($esi->hasSurrogateEsiCapability($request)){
     $content .= $esi->renderIncludeTag(
     $request->getBasePath().'/edge.php?ttl=2',
     null,false).'
     ';
     $content .= $esi->renderIncludeTag(
     $request->getBasePath().'/edge.php?ttl=0',
     null,false).'
     ';
     }
     $content .= '
     Back to index';
     Symfony&components&(2)
     

     View Slide

195. $response->setContent($content);
     $esi->addSurrogateControl($response);
     $response->setSharedMaxAge(10);
     $response->send();
     Symfony&components&(3)
     

     View Slide

196. And&
     on&that&
     bombshell&...
     

     View Slide

197. View Slide

198. h"ps://joind.in/9280
     Please&
     give&me&
     feedback
     

     View Slide

199. Thanks
     

     View Slide

200. View Slide