Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Varnish workshop - ZCEU13

Thijs Feryn
November 18, 2013

Varnish workshop - ZCEU13

Varnish workshop at ZendConEU 2013

Thijs Feryn

November 18, 2013
Tweet

More Decks by Thijs Feryn

Other Decks in Technology

Transcript

  1. Varnish
    Workshop
    By Thijs Feryn

    View Slide

  2. Hi, my name is Thijs

    View Slide

  3. Et j’ai un joli accent
    belge

    View Slide

  4. I’m @ThijsFeryn
    on Twitter

    View Slide

  5. I’m an
    at
    Evangelist

    View Slide

  6. I’m a
    at
    board member

    View Slide

  7. h"ps://joind.in/9280
    Please&
    give&me&
    feedback

    View Slide

  8. Varnish ?

    View Slide

  9. Cache?

    View Slide

  10. Reverse proxy?

    View Slide

  11. Loadbalancer?

    View Slide

  12. Loadbalancer?
    HTTP accelerator

    View Slide

  13. In&front&of&the&
    webserver

    View Slide

  14. Primary&purpose
    Caching

    View Slide

  15. Reverse&caching&
    proxy

    View Slide

  16. Install &
    configure

    View Slide

  17. curl%http://repo.varnish3cache.org/debian/GPG3
    key.txt%|%apt3key%add%3
    apt3get%update
    echo%"deb%http://repo.varnish3cache.org/
    debian/%squeeze%varnish33.0"%>>%/etc/apt/
    sources.list
    apt3get%install%varnish

    View Slide

  18. Daemon5op6ons
    h"ps://www.varnish:cache.org/docs/trunk/reference/varnishd.html

    View Slide

  19. Daemon5op6ons
    DAEMON_OPTS=":a5127.0.0.1:80805\
    5555555555555:T5localhost:60825\
    5555555555555:f5/etc/varnish/default.vcl5\
    5555555555555:s5malloc,3G5\
    5555555555555:p5thread_pools=8
    5555555555555:p5thread_pool_min=1605\
    5555555555555:p5thread_pool_max=30005\
    5555555555555:p5thread_pool_add_delay=25\
    5555555555555:p5connect_6meout=200s5\
    5555555555555:p5first_byte_6meout=200s5\
    5555555555555:p5between_bytes_6meout=50s5\
    5555555555555:p5session_linger=1005\
    5555555555555:p5lru_interval=205\
    5555555555555:p5listen_depth=81925\
    5555555555555:p5sess_workspace=2621445\
    5555555555555:l5300M"
    In&“/etc/default/varnish”

    View Slide

  20. Daemon5op6ons

    View Slide

  21. varnishadm>%param.show%3l
    Shows&all&params
    Varnish
    parameters

    View Slide

  22. Backend

    View Slide

  23. Listen%8080
    In&“/etc/apache2/ports.conf”
    Backend

    View Slide


  24. In&vhost&config&when&Varnish&is&
    on&same&node&as&webserver
    Backend

    View Slide

  25. backend%default%{
    %%%%%%.host%=%"127.0.0.1";
    %%%%%%.port%=%"8080";
    }
    In&“/etc/varnish/default.vcl”
    Backend

    View Slide

  26. View Slide

  27. GET / HTTP/1.1
    Host: localhost
    User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X
    10.5; fr; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
    Accept: text/html,application/xhtml+xml,application/
    xml;q=0.9,*/*;q=0.8
    Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3
    Accept-Encoding: gzip,deflate
    Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
    Keep-Alive: 115
    Connection: keep-alive
    Cache-Control: max-age=0
    Request

    View Slide

  28. HTTP/1.1 200 OK
    Server: Apache/2.2.14 (Ubuntu)
    X-Powered-By: PHP/5.3.2-1ubuntu4.7
    Cache-Control: public, max-age=86400
    Last-Modified: Mon, 04 Apr 2011 04:13:41 +0000
    Expires: Sun, 11 Mar 1984 12:00:00 GMT
    Vary: Cookie,Accept-Encoding
    ETag: "1301890421"
    Content-Type: text/html; charset=utf-8
    Content-Length: 23562
    Date: Mon, 04 Apr 2011 09:02:26 GMT
    X-Varnish: 1886109724 1886107902
    Age: 17324
    Via: 1.1 varnish
    Connection: keep-alive
    Response

    View Slide

  29. View Slide

  30. [email protected]
    [email protected]
    Public
    Private
    [email protected]
    [email protected]
    [email protected]
    [email protected]
    TTL&for&browsers&in&seconds
    TTL&for&proxies&in&seconds
    Proxies&&&browsers&can&cache
    Only&browsers&can&cache
    Revalidate&before&dropping&from&cache
    Don’t&cache&at&all
    Browser&revalidate&before&serving&from
    cache&
    Proxy&revalidate&before&serving&from&
    cache&

    View Slide

  31. Age
    Age&“56”
    How&old&is&
    the&cached&
    object?

    View Slide

  32. Real&cache&duration
    Real&cache&duraWon&for&clients&=&[email protected]&@&age
    Real&cache&duraWon&for&proxies&=&[email protected]&@&age

    View Slide

  33. Expires
    Expires&"Wed,&1&Jan&2014&20:00:00&GMT"

    View Slide

  34. Expiration
    1.Vcl5(beresp."l)
    2.Cache:control5s:maxage
    3.Cache:control5max:age
    4.Expires

    View Slide

  35. The&flow

    View Slide

  36. 1

    View Slide

  37. 2

    View Slide

  38. 3

    View Slide

  39. Advanced&
    flow

    View Slide

  40. 1

    View Slide

  41. 2

    View Slide

  42. 3

    View Slide

  43. Exercise 1
    Testing HTTP cache headers

    View Slide

  44. Exercise 2
    Overriding Time To Live in Varnish

    View Slide

  45. Out of the box

    View Slide

  46. View Slide

  47. Varnish obeys caching headers

    View Slide

  48. Only get & HEAD
    No cookie & auth headers
    No set-cookie headers
    cache ttl > 0
    No vary “*”
    When will varnish cache?

    View Slide

  49. About cookies

    View Slide

  50. About cookies
    HTTP cookie
    request header
    via browser
    HTTP set-cookie
    response header
    via webserver

    View Slide

  51. 200: OK
    203: Non-Authoritative Information
    300: Multiple Choices
    301: Moved Permanently
    302: Moved Temporarily
    307: Temporary Redirect
    410: Gone
    404: Not Found
    Cacheable status codes

    View Slide

  52. Exercise 3
    Testing default behaviour

    View Slide

  53. Monitoring
    &
    Logging

    View Slide

  54. Varnishstat
    Real&time&stats&for&
    Varnish&instance

    View Slide

  55. 4+05:26:25
    Hitrate.ratio:.......10......100......254
    Hitrate.avg:.....0.8486...0.7619...0.7285
    .....1760818.........6.99.........4.82.client_conn.;.Client.connections.accepted
    ....11088687........25.96........30.36.client_req.;.Client.requests.received
    .....8042715........11.98........22.02.cache_hit.;.Cache.hits
    .....2609561........11.98.........7.15.cache_miss.;.Cache.misses
    .......47104.........1.00.........0.13.backend_conn.;.Backend.conn..success
    .........610.........0.00.........0.00.backend_fail.;.Backend.conn..failures
    .....2998265........12.98.........8.21.backend_reuse.;.Backend.conn..reuses
    .......12081.........0.00.........0.03.backend_toolate.;.Backend.conn..was.closed
    .....3010356........13.98.........8.24.backend_recycle.;.Backend.conn..recycles
    ..........13.........0.00.........0.00.backend_retry.;.Backend.conn..retry
    .........520.........0.00.........0.00.fetch_head.;.Fetch.head
    .....2857965........11.98.........7.83.fetch_length.;.Fetch.with.Length
    ......151309.........2.00.........0.41.fetch_chunked.;.Fetch.chunked
    ........4404.........0.00.........0.01.fetch_close.;.Fetch.wanted.close
    .........676.........0.00.........0.00.fetch_failed.;.Fetch.failed
    .......31164.........0.00.........0.09.fetch_304.;.Fetch.no.body.(304)
    .........220...........................n_sess_mem.;.N.struct.sess_mem
    ..........53...........................n_sess.;.N.struct.sess
    .......29540...........................n_object.;.N.struct.object
    .......29561...........................n_objectcore.;.N.struct.objectcore
    ........5058...........................n_objecthead.;.N.struct.objecthead
    ........2613...........................n_waitinglist.;.N.struct.waitinglist
    ...........3...........................n_vbc.;.N.struct.vbc
    ..........22...........................n_wrk.;.N.worker.threads
    ........1789.........0.00.........0.00.n_wrk_create.;.N.worker.threads.created

    View Slide

  56. .......31164.........0.00.........0.09.fetch_304.;.Fetch.no.body.(304)
    .........220...........................n_sess_mem.;.N.struct.sess_mem
    ..........53...........................n_sess.;.N.struct.sess
    .......29540...........................n_object.;.N.struct.object
    .......29561...........................n_objectcore.;.N.struct.objectcore
    ........5058...........................n_objecthead.;.N.struct.objecthead
    ........2613...........................n_waitinglist.;.N.struct.waitinglist
    ...........3...........................n_vbc.;.N.struct.vbc
    ..........22...........................n_wrk.;.N.worker.threads
    ........1789.........0.00.........0.00.n_wrk_create.;.N.worker.threads.created
    .......17204.........0.00.........0.05.n_wrk_queued.;.N.queued.work.requests
    ...........1...........................n_backend.;.N.backends
    .....2579403...........................n_expired.;.N.expired.objects
    .....6920374...........................n_lru_moved.;.N.LRU.moved.objects
    .....8959782........26.96........24.53.n_objwrite.;.Objects.sent.with.write
    .....1760818.........5.99.........4.82.s_sess.;.Total.Sessions
    ....11088687........25.96........30.36.s_req.;.Total.Requests
    ......436411.........2.00.........1.20.s_pass.;.Total.pass
    .....3044686........13.98.........8.34.s_fetch.;.Total.fetch
    ..3817818737......8996.58.....10454.48.s_hdrbytes.;.Total.header.bytes
    .84764679157....218712.68....232114.35.s_bodybytes.;.Total.body.bytes
    ......196150.........0.00.........0.54.sess_closed.;.Session.Closed
    .......73512.........0.00.........0.20.sess_pipeline.;.Session.Pipeline
    .......50339.........0.00.........0.14.sess_readahead.;.Session.Read.Ahead
    ....10875541........25.96........29.78.sess_linger.;.Session.Linger
    .....7865547........25.96........21.54.sess_herd.;.Session.herd
    ...591440571......1649.54......1619.56.shm_records.;.SHM.records
    ....30844446.......121.82........84.46.shm_writes.;.SHM.writes
    ...........6.........0.00.........0.00.shm_flushes.;.SHM.flushes.due.to.overflow

    View Slide

  57. $ varnishstat -1 -f n_lru_nuked
    n_lru_nuked 0 . N LRU nuked objects
    Varnishstat
    Removed&from&
    cache&due&to&lack&of&
    memory

    View Slide

  58. Varnishlog
    [email protected]&logs&of&
    Varnish&activity

    View Slide

  59. ...11.SessionOpen..c.12.12.12.1.53727.:80
    ...11.ReqStart.....c.12.12.12.1.53727.1401010767
    ...11.RxRequest....c.GET
    ...11.RxURL........c./
    ...11.RxProtocol...c.HTTP/1.1
    ...11.RxHeader.....c.Host:.12.12.12.6
    ...11.RxHeader.....c.User;Agent:.Mozilla/5.0.(Macintosh;.Intel.Mac.OS.X.10.8;.rv:17.0).
    Gecko/20100101.Firefox/17.0
    ...11.RxHeader.....c.Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/
    *;q=0.8
    ...11.RxHeader.....c.Accept;Language:.nl,en;q=0.7,fr;be;q=0.3
    ...11.RxHeader.....c.Accept;Encoding:.gzip,.deflate
    ...11.RxHeader.....c.Connection:.keep;alive
    ...11.VCL_call.....c.recv.lookup
    ...11.VCL_call.....c.hash
    ...11.Hash.........c./
    ...11.Hash.........c.12.12.12.6
    ...11.VCL_return...c.hash
    ...11.VCL_call.....c.miss.fetch
    ...11.Backend......c.13.default.default
    ...11.TTL..........c.1401010767.RFC.0.;1.;1.1357920021.0.1357920020.0.0
    ...11.VCL_call.....c.fetch
    ...11.TTL..........c.1401010767.VCL.120.;1.;1.1357920021.;0
    ...11.VCL_return...c.hit_for_pass
    ...11.ObjProtocol..c.HTTP/1.1
    ...11.ObjResponse..c.OK
    ...11.ObjHeader....c.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...11.ObjHeader....c.Server:.Apache
    ...11.ObjHeader....c.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...11.ObjHeader....c.Cache;Control:.no;cache,.no;store,.max;age=0
    Client&

    View Slide

  60. ...11.VCL_return...c.hit_for_pass
    ...11.ObjProtocol..c.HTTP/1.1
    ...11.ObjResponse..c.OK
    ...11.ObjHeader....c.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...11.ObjHeader....c.Server:.Apache
    ...11.ObjHeader....c.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...11.ObjHeader....c.Cache;Control:.no;cache,.no;store,.max;age=0
    ...11.ObjHeader....c.Vary:.Accept;Encoding
    ...11.ObjHeader....c.Content;Encoding:.gzip
    ...11.ObjHeader....c.Content;Length:.119
    ...11.ObjHeader....c.Content;Type:.text/html
    ...11.Gzip.........c.u.F.;.119.336.80.80.887
    ...11.VCL_call.....c.deliver.deliver
    ...11.TxProtocol...c.HTTP/1.1
    ...11.TxStatus.....c.200
    ...11.TxResponse...c.OK
    ...11.TxHeader.....c.Server:.Apache
    ...11.TxHeader.....c.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...11.TxHeader.....c.Cache;Control:.no;cache,.no;store,.max;age=0
    ...11.TxHeader.....c.Vary:.Accept;Encoding
    ...11.TxHeader.....c.Content;Encoding:.gzip
    ...11.TxHeader.....c.Content;Type:.text/html
    ...11.TxHeader.....c.Content;Length:.119
    ...11.TxHeader.....c.Accept;Ranges:.bytes
    ...11.TxHeader.....c.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...11.TxHeader.....c.X;Varnish:.1401010767
    ...11.TxHeader.....c.Age:.0
    ...11.TxHeader.....c.Via:.1.1.varnish
    ...11.TxHeader.....c.Connection:.keep;alive
    ...11.Length.......c.119
    ...11.ReqEnd.......c.1401010767.1357920020.712090731.1357920020.727306366.0.000087738.
    Client&

    View Slide

  61. ...13.BackendClose.;.default
    ...13.BackendOpen..b.default.127.0.0.1.51597.127.0.0.1.8080
    ...13.TxRequest....b.GET
    ...13.TxURL........b./
    ...13.TxProtocol...b.HTTP/1.1
    ...13.TxHeader.....b.Host:.12.12.12.6
    ...13.TxHeader.....b.User;Agent:.Mozilla/5.0.(Macintosh;.Intel.Mac.OS.X.10.8;.rv:17.0).
    Gecko/20100101.Firefox/17.0
    ...13.TxHeader.....b.Accept:.text/html,application/xhtml+xml,application/xml;q=0.9,*/
    *;q=0.8
    ...13.TxHeader.....b.Accept;Language:.nl,en;q=0.7,fr;be;q=0.3
    ...13.TxHeader.....b.X;Forwarded;For:.12.12.12.1
    ...13.TxHeader.....b.X;Varnish:.1401010767
    ...13.TxHeader.....b.Accept;Encoding:.gzip
    ...13.RxProtocol...b.HTTP/1.1
    ...13.RxStatus.....b.200
    ...13.RxResponse...b.OK
    ...13.RxHeader.....b.Date:.Fri,.11.Jan.2013.16:00:20.GMT
    ...13.RxHeader.....b.Server:.Apache
    ...13.RxHeader.....b.X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    ...13.RxHeader.....b.Cache;Control:.no;cache,.no;store,.max;age=0
    ...13.RxHeader.....b.Vary:.Accept;Encoding
    ...13.RxHeader.....b.Content;Encoding:.gzip
    ...13.RxHeader.....b.Content;Length:.119
    ...13.RxHeader.....b.Content;Type:.text/html
    ...13.Fetch_Body...b.4(length).cls.0.mklen.1
    ...13.Length.......b.119
    ...13.BackendReuse.b.default
    Backend&

    View Slide

  62. :a!When!writing!to!a!file,!append!to!it!rather!than!overwrite!it.
    :b!Include!backend!communication!logs
    :c!Include!client!communication!logs
    :d!Process!old!logs!on!startup
    :I5Perform!regex!filter!on!output
    :i5Include!specific!tags!into!output
    :m5Perform!regex!filter!on!tags
    :r5Read!a!shared!memory!log!dump!from!file
    :w!Dump!shared!memory!logs!to!file
    :x!Exclude!specific!tags!from!output
    :X!Perform!exclusion!regex!filter!on!output
    :O!Don’t!group!log!entries.!Don’t!use!with!Bm
    Varnishlog

    View Slide

  63. Backend
    BackendClose
    BackendOpen
    BackendReuse
    BackendXID
    CLI
    ClientAddr
    Debug
    Error
    ExpBan
    ExpKill
    ExpPick
    Hit
    HitPass
    HttpError
    HttpGarbage
    Length
    ObjHeader
    ObjLostHeader
    ObjProtocol
    ObjRequest
    ObjResponse
    ObjStatus
    ObjURL
    ReqEnd
    ReqStart
    RxHeader
    RxLostHeader
    RxProtocol
    RxRequest
    RxResponse
    RxStatus
    RxURL
    SessionClose
    SessionOpen
    StatAddr
    StatSess
    TTL
    TxHeader
    TxLostHeader
    TxProtocol
    TxRequest
    TxResponse
    TxStatus
    TxURL
    VCL_acl
    VCL_call
    VCL_return
    VCL_trace
    WorkThread
    Tags

    View Slide

  64. varnishlog -c -m VCL_call:hit | grep -i 'RxURL'
    Varnishlog
    Get&the&URL&for&all&hits

    View Slide

  65. varnishlog -b -i TxURL
    Varnishlog
    Track&all&“missed”&
    URL’s
    varnishlog -O -b -i TxURL
    Ungrouped

    View Slide

  66. varnishlog -c -i RxHeader -I User-Agent
    Varnishlog
    Get&all&user&agents
    varnishlog -O -c -i RxHeader -I User-Agent
    Ungrouped

    View Slide

  67. varnishlog -i RxHeader
    Varnishlog
    varnishlog -c -i RxHeader
    varnishlog -b -i RxHeader
    Get&all&request&
    headers

    View Slide

  68. varnishlog -c -i RxURL -m VCL_call:hit
    Varnishlog
    Doesn’t&work
    @i&is&done&before&@m

    View Slide

  69. Why&did&it&miss?

    View Slide

  70. 11 VCL_return c hit_for_pass
    Why&did&it&miss?
    TTL&<=&0
    13 RxHeader - Cache-Control: no-cache, no-
    store, max-age=0

    View Slide

  71. 11 RxRequest c POST
    11 RxURL c /exercises/3/post.php
    11 RxProtocol c HTTP/1.1
    11 RxHeader c Host: varnish.dev
    11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS
    X 10.8; rv:18.0) Gecko/20100101 Firefox/18.0
    11 RxHeader c Accept: text/html,application/xhtml
    +xml,application/xml;q=0.9,*/*;q=0.8
    11 RxHeader c Accept-Language: nl,en;q=0.7,fr-be;q=0.3
    11 RxHeader c Accept-Encoding: gzip, deflate
    11 RxHeader c Referer: http://varnish.dev/exercises/3/post.php
    11 RxHeader c Connection: keep-alive
    11 RxHeader c Content-Type: application/x-www-form-urlencoded
    11 RxHeader c Content-Length: 11
    11 VCL_call c recv pass
    Why&did&it&miss?
    HTTP&POST

    View Slide

  72. Was&it&stored&to&
    cache?

    View Slide

  73. 11 TTL c 324425260 RFC 5 -1 -1
    1358344107 0 1358344107 0 5
    11 VCL_call c fetch deliver
    Was&it&stored&to&cache?
    Yes

    View Slide

  74. 11 TTL c 324425261 VCL 120 -1 -1
    1358344313 -0
    11 VCL_return c hit_for_pass
    Was&it&stored&to&cache?
    No,&for&
    120&seconds&in&
    blacklist

    View Slide

  75. Exercise 4
    Using varnishlog

    View Slide

  76. What’s&the&cache&key?

    View Slide

  77. 11 Hash c /exercises/1/public.php
    11 Hash c varnish.dev
    What’s&the&cache&key?
    Host&+&URL

    View Slide

  78. Varnishtop
    Incremental&[email protected]
    memory&Varnish&logs

    View Slide

  79. list.length.109
    .....8.12.VCL_return.....deliver
    .....4.19.RxRequest......GET
    .....4.19.RxProtocol.....HTTP/1.1
    .....4.19.RxHeader.......Host:.varnish.dev
    .....4.19.RxHeader.......Connection:.keep;alive
    .....4.19.VCL_call.......recv
    .....4.19.VCL_return.....lookup
    .....4.19.VCL_call.......hash
    .....4.19.Hash...........varnish.dev
    .....4.19.VCL_return.....hash
    .....4.19.VCL_call.......hit
    .....4.19.VCL_call.......deliver
    .....4.19.TxProtocol.....HTTP/1.1
    .....4.19.TxHeader.......Server:.Apache
    .....4.19.TxHeader.......Content;Encoding:.gzip
    .....4.19.TxHeader.......Accept;Ranges:.bytes
    .....4.19.TxHeader.......Via:.1.1.varnish
    .....4.19.TxHeader.......Connection:.keep;alive
    .....3.76.TxHeader.......Age:.26
    .....3.12.RxURL........../exercises/5/vary.php
    .....3.12.RxHeader.......Accept:.text/html,application/xhtml+xml,application/
    xml;q=0.9,*/*;q=0.8
    .....3.12.Hash.........../exercises/5/vary.php
    .....3.12.TxStatus.......200
    .....3.12.TxResponse.....OK
    .....3.12.TxHeader.......X;Powered;By:.PHP/5.3.2;1ubuntu4.18
    .....3.12.TxHeader.......Vary:.User;Agent,Accept;Encoding
    .....3.12.TxHeader.......Cache;Control:.public,must;revalidate,s;maxage=.100
    Unfiltered&list

    View Slide

  80. :b!Include!backend!communication!logs
    :c!Include!client!communication!logs
    :d!Process!old!logs!on!startup
    :I5Perform!regex!filter!on!output
    :i5Include!specific!tags!into!output
    :m5Perform!regex!filter!on!tags
    :r5Read!a!shared!memory!log!dump!from!file
    :p5Specifies!!the!!number!!of!!seconds!!to!!measure!!over
    :x!Exclude!specific!tags!from!output
    :X!Perform!exclusion!regex!filter!on!output
    :O!Don’t!group!log!entries.!Don’t!use!with!Bm
    :f!Sort!and!group!only!on!the!first!field!of!each!log!entry
    Varnishtop

    View Slide

  81. varnishtop -c -i RxHeader -I ^User-Agent
    Varnishtop
    Get&top&user&agents

    View Slide

  82. varnishtop -b -i TxURL
    Varnishtop
    Get&top&missed&URL’s

    View Slide

  83. varnishtop -c -i VCL_call
    Varnishtop
    Get&top&VCL&calls

    View Slide

  84. varnishtop -c -i RxURL
    Varnishtop
    Most&popular&URL’s

    View Slide

  85. Exercise 5
    Using varnishtop

    View Slide

  86. Cache variations

    View Slide

  87. Vary: Accept-Language
    Cache&variaWons
    Keep&
    seperate&cache&objects&
    for&each&varation&of&the&
    header
    Vary: X-myHeader
    Vary: Accept-Encoding

    View Slide

  88. •Don’t5vary5‘*’
    •Only5vary5request5headers
    •Custom5headers5only5work5if5browser5
    sends5them
    •VCL5magic5(coming!up!later)
    Cache&variaWons

    View Slide

  89. Exercise 6
    Cache variations

    View Slide

  90. Varnish&
    Configuration&
    Language

    View Slide

  91. Varnish&
    Configuration&
    Language
    The&Varnish&ConfiguraWon&
    Language&allows&you&to&define&
    your&caching&policy.&You&write&
    VCL&code&which&Varnish&will&
    parse,&translate&to&C&code,&
    compile&and&link&to.

    View Slide

  92. Out&of&the&box&behaviour
    Remember&
    the&flow!

    View Slide

  93. sub vcl_recv {
    if (req.restarts == 0) {
    if (req.http.x-forwarded-for) {
    set req.http.X-Forwarded-For =
    req.http.X-Forwarded-For + “, ” + client.ip;
    } else {
    set req.http.X-Forwarded-For = client.ip;
    }
    }
    if (req.request != “GET” &&
    req.request != “HEAD” &&
    req.request != “PUT” &&
    req.request != “POST” &&
    req.request != “TRACE” &&
    req.request != “OPTIONS” &&
    req.request != “DELETE”) {
    /* Non-RFC2616 or CONNECT which is weird. */
    return (pipe);
    }
    if (req.request != “GET” && req.request != “HEAD”) {
    /* We only deal with GET and HEAD by default */
    return (pass);
    }
    if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
    }
    return (lookup);
    }

    View Slide

  94. sub vcl_pipe {
    # Note that only the first request to the backend will have
    # X-Forwarded-For set. If you use X-Forwarded-For and want to
    # have it set for all requests, make sure to have:
    # set bereq.http.connection = "close";
    # here. It is not set by default as it might break some broken web
    # applications, like IIS with NTLM authentication.
    return (pipe);
    }
    sub vcl_pass {
    return (pass);
    }
    sub vcl_hash {
    hash_data(req.url);
    if (req.http.host) {
    hash_data(req.http.host);
    } else {
    hash_data(server.ip);
    }
    return (hash);
    }

    View Slide

  95. sub vcl_hit {
    return (deliver);
    }
    sub vcl_miss {
    return (fetch);
    }
    sub vcl_fetch {
    if (beresp.ttl <= 0s ||
    beresp.http.Set-Cookie ||
    beresp.http.Vary == "*") {
    ! ! /*
    ! ! * Mark as "Hit-For-Pass" for the next 2 minutes
    ! ! */
    ! ! set beresp.ttl = 120 s;
    ! ! return (hit_for_pass);
    }
    return (deliver);
    }
    sub vcl_deliver {
    return (deliver);
    }

    View Slide

  96. sub vcl_error {
    set obj.http.Content-Type = "text/html; charset=utf-8";
    set obj.http.Retry-After = "5";
    synthetic {"

    br/>"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">


    "} + obj.status + " " + obj.response + {"


    Error "} + obj.status + " " + obj.response + {"
    "} + obj.response + {"
    Guru Meditation:
    XID: "} + req.xid + {"

    Varnish cache server


    "};
    return (deliver);
    }
    sub vcl_init {
    ! return (ok);
    }
    sub vcl_fini {
    ! return (ok);
    }

    View Slide

  97. VCL&
    syntax

    View Slide

  98. //This is comment
    #This is comment too
    /* More comment
    A lot more comment*/
    Comment

    View Slide

  99. if(req.url == "/") {
    ! error 200 "OK dude";
    } elseif(req.url == "/notfound") {
    ! error 404 "Not found man";
    } else {
    ! error 200 "Else";
    }
    CondiWonal

    View Slide

  100. if(req.url == "/") {
    ! set req.backend = otherBackend;
    }
    Set&variables

    View Slide

  101. Common&objects
    ✓Req:5request5object
    ✓Bereq:5backend5request5object
    ✓Beresp:5backend5response5object
    ✓Obj:5cache5object
    ✓Resp:5output5object
    ✓Client:5client5information
    ✓Server:5server5information

    View Slide

  102. Variable&availability

    View Slide

  103. .host
    Host5name5or5IP5address5of5a5backend.
    .port
    Service5name5or5port5number5of5a5backend.
    .connect_timeout
    Backend5marked5“unhealthy”5when5a5connection5fails5after5x5time.
    .first_byte_timeout
    Backend5marked5“unhealthy”5when5the5first5byte5isn’t5received5after5x5time.
    .between_byte_timeout
    Backend5marked5“unhealthy”5when5the5byte5sequence5isn’t5coming5in5fast5
    enough.
    Backend&variables

    View Slide

  104. .url
    Specify5a5URL5to5request5from5the5backend.5Defaults5to5"/".
    .request
    Specify5a5full5HTTP5request5using5multiple5strings.5.request5will5have5\r\n5
    automatically5inserted5after5every5string.5If5specified,5.request5will5take5
    precedence5over5.url.
    .window
    How5many5of5the5latest5polls5we5examine5to5determine5backend5health.5
    Defaults5to58.
    .threshold
    How5many5of5the5polls5in5.window5must5have5succeeded5for5us5to5consider5the
    backend5healthy.5Defaults5to53.
    Probe&variables&(1)

    View Slide

  105. .initial
    How5many5of5the5probes5are5considered5good5when5Varnish5starts.5Defaults5to
    the5same5amount5as5the5threshold.
    .expected_response
    The5expected5backend5HTTP5response5code.5Defaults5to5200.
    .interval
    Defines5how5often5the5probe5should5check5the5backend.5Default5is5every555
    seconds.
    .timeout
    How5fast5each5probe5times5out.5Default5is525seconds.
    Probe&variables&(2)

    View Slide

  106. client.ip
    The5client's5IP5address.
    client.identity
    Identification5of5the5client,5used5to5load5balance5in5the5client5director.
    server.hostname
    The5host5name5of5the5server.
    server.identity
    The5identity5of5the5server,5as5set5by5the5:i5parameter.5If5the5:i5parameter5is5not5
    passed5to5varnishd,5server.identity5will5be5set5to5the5name5of5the5instance,5as5
    specified5by5the5:n5parameter.
    server.ip
    The5IP5address5of5the5socket5on5which5the5client5connection5was5received.
    server.port
    The5port5number5of5the5socket5on5which5the5client5connection5was5received.
    Client&&&server&variables

    View Slide

  107. req.request
    The5request5type5(e.g.5"GET",5"HEAD").
    req.url
    The5requested5URL.
    req.proto
    The5HTTP5protocol5version5used5by5the5client.
    req.backend
    The5backend5to5use5to5service5the5request.
    req.backend.healthy
    Whether5the5backend5is5healthy5or5not.5Requires5an5active5probe5to5be5set5
    on5the5backend.
    req.http.header
    The5corresponding5HTTP5header.
    Request&variables&(1)

    View Slide

  108. req.hash_always_miss
    Force5a5cache5miss5for5this5request.5If5set5to5true5Varnish5will5disregard5any5
    existing5objects5and5always5(re)fetch5from5the5backend.
    req.hash_ignore_busy
    Ignore5any5busy5object5during5cache5lookup.5You5would5want5to5do5this5if5
    you5have5two5server5looking5up5content5from5each5other5to5avoid5potential5
    deadlocks.
    req.can_gzip
    Does5the5client5accept5the5gzip5transfer5encoding.
    req.restarts
    A5count5of5how5many5times5this5request5has5been5restarted.
    Unique5ID5of5this5request.
    Request&variables&(2)

    View Slide

  109. req.esi
    Boolean.5Set5to5false5to5disable5ESI5processing5regardless5of5any5value5in5
    beresp.do_esi.5Defaults5to5true.5This5variable5is5subject5to5change5in5future5
    versions,5you5should5avoid5using5it.
    req.esi_level
    A5count5of5how5many5levels5of5ESI5requests5we're5currently5at.
    req.grace
    Set5to5a5period5to5enable5grace.
    req.xid
    Unique5ID5of5this5request.
    req.hash
    The5hash5key5used5to5refer5to5an5object5in5the5cache.5Used5when5both5
    reading5from5and5writing5to5the5cache.
    Request&variables&(3)

    View Slide

  110. bereq.request
    The5request5type5(e.g.5"GET",5"HEAD").
    bereq.url
    The5requested5URL.
    bereq.proto
    The5HTTP5protocol5version5used5to5talk5to5the5server.
    bereq.http.header
    The5corresponding5HTTP5header.
    bereq.connect_timeout
    The5time5in5seconds5to5wait5for5a5backend5connection.
    bereq.first_byte_timeout
    The5time5in5seconds5to5wait5for5the5first5byte5from5the5backend.5Not5
    available5in5pipe5mode.
    bereq.between_bytes_timeout
    The5time5in5seconds5to5wait5between5each5received5byte5from5the5backend.5
    Not5available5in5pipe5mode.
    Backend&request&variables

    View Slide

  111. beresp.do_stream
    Deliver5the5object5to5the5client5directly5without5fetching5the5whole5object5
    into5varnish.5If5this5request5is5pass'ed5it5will5not5be5stored5in5memory.5As5of5
    Varnish5Cache53.05the5object5will5marked5as5busy5as5it5is5delivered5so5only5
    client5can5access5the5object.
    beresp.do_esi
    Boolean.5ESI:process5the5object5after5fetching5it.5Defaults5to5false.5Set5it5to5
    true5to5parse5the5object5for5ESI5directives.5Will5only5be5honored5if5req.esi5is5
    true.
    beresp.do_gzip
    Boolean.5Gzip5the5object5before5storing5it.5Defaults5to5false.
    beresp.do_gunzip
    Boolean.5Unzip5the5object5before5storing5it5in5the5cache.5Defaults5to5false.
    beresp.proto
    The5HTTP5protocol5version5used5the5backend5replied5with.
    beresp.storage
    Set5to5force5Varnish5to5save5this5object5to5a5particular5storage5backend.
    Backend&response&variables&(1)

    View Slide

  112. beresp.status
    The5HTTP5status5code5returned5by5the5server.
    beresp.response
    The5HTTP5status5message5returned5by5the5server.
    beresp.ttl
    The5object's5remaining5time5to5live,5in5seconds.5beresp.ttl5is5writable.
    beresp.grace
    Set5to5a5period5to5enable5grace.
    beresp.saintmode
    Set5to5a5period5to5enable5saint5mode.
    beresp.backend.name
    Name5of5the5backend5this5response5was5fetched5from.
    beresp.backend.ip
    IP5of5the5backend5this5response5was5fetched5from.
    beresp.backend.port
    Port5of5the5backend5this5response5was5fetched5from.
    Backend&response&variables&(2)

    View Slide

  113. obj.proto
    The5HTTP5protocol5version5used5when5the5object5was5retrieved.
    obj.status
    The5HTTP5status5code5returned5by5the5server.
    obj.response
    The5HTTP5status5message5returned5by5the5server.
    obj.ttl
    The5object's5remaining5time5to5live,5in5seconds.5obj.ttl5is5writable.
    obj.lastuse
    The5approximate5time5elapsed5since5the5object5was5last5requests,5in5
    seconds.5This5variable5is5also5available5in5vcl_deliver.
    obj.hits
    The5approximate5number5of5times5the5object5has5been5delivered.5A5value5of5
    05indicates5a5cache5miss.5This5variable5is5also5available5in5vcl_deliver.
    obj.grace
    The5object's5grace5period5in5seconds.5obj.grace5is5writable.
    obj.http.header
    The5corresponding5HTTP5header.
    Cache&object&variables

    View Slide

  114. resp.proto
    The5HTTP5protocol5version5to5use5for5the5response.
    resp.status
    The5HTTP5status5code5that5will5be5returned.
    resp.response
    The5HTTP5status5message5that5will5be5returned.
    resp.http.header
    The5corresponding5HTTP5header.
    Client&response&variables

    View Slide

  115. Reminder!

    View Slide

  116. Working&
    with&
    backends

    View Slide

  117. backend default {
    .host = "1.2.3.4";
    .port = "http";
    }
    Backend&config
    backend default {
    .host = "localhost";
    .port = "8080";
    }
    Simple

    View Slide

  118. backend default {
    .host = "1.2.3.4";
    .port = "http";
    .connect_timeout = 1s;
    .first_byte_timeout = 5s;
    .between_bytes_timeout = 2s;
    }
    Backend&config
    More

    View Slide

  119. backend default {
    .host = "1.2.3.4";
    .port = "http";
    }
    backend other {
    .host = "1.2.3.5";
    .port = "http";
    }
    sub vcl_recv {
    if (req.http.host ~ "^(www.)?example.com$") {
    set req.backend = other;
    }
    }
    Select&backend

    View Slide

  120. Probes

    View Slide

  121. backend default {
    .host = "1.2.3.4";
    .port = "http";
    .probe = {
    .url = "/test.jpg";
    .timeout = 0.3 s;
    .window = 8;
    .threshold = 3;
    .initial = 3;
    }
    }
    Probes
    Test&if&
    healthy

    View Slide

  122. probe healthcheck {
    .url = "/status.cgi";
    .interval = 60s;
    .timeout = 0.3 s;
    .window = 8;
    .threshold = 3;
    .initial = 3;
    .expected_response = 200;
    }
    backend www {
    .host = "www.example.com";
    .port = "http";
    .probe = healthcheck;
    }
    Probes

    View Slide

  123. probe rawprobe {
    # NB: \r\n automatically inserted after each
    string!
    .request =
    "GET / HTTP/1.1"
    "Host: www.foo.bar"
    "Connection: close";
    }
    Probes

    View Slide

  124. Directors

    View Slide

  125. A!director!is!a!logical!group!of!
    backend!servers!clustered!
    together!for!redundancy.!The!
    basic!role!of!the!director!is!to!
    let!Varnish!choose!a!backend!
    server!amongst!several!so!if!one!
    is!down!another!can!be!used.

    View Slide

  126. Directors
    ✓Round:robin
    ✓Random
    ✓Client
    ✓Hash
    ✓Fallback
    ✓DNS

    View Slide

  127. backend.one.{
    ....host.=.“localhost”;
    ....port.=.“80”;
    }
    backend.two.{
    ....host.=.“127.0.0.1”;
    ....port.=.“81”;
    }
    director.localhosts.round;robin.{
    ....{..backend.=.one;.}
    ....{..backend.=.two;.}
    ....{..backend.=.{.
    .............host.=.“localhost”;.
    .............port.=.“82”;.
    ........}.
    ....}
    }
    sub.vcl_recv.{
    ....set.req.backend.=.localhosts;
    }
    [email protected]&director

    View Slide

  128. backend.one.{
    ....host.=.“localhost”;
    ....port.=.“80”;
    }
    backend.two.{
    ....host.=.“127.0.0.1”;
    ....port.=.“81”;
    }
    director.localhosts.random.{
    ....{..backend.=.one;..weight=4;}
    ....{..backend.=.two;..weight=6;}
    ....{..backend.=.{.
    .............host.=.“localhost”;.
    .............port.=.“82”;.
    ........}.
    ....}
    }
    sub.vcl_recv.{
    ....set.req.backend.=.localhosts;
    }
    Random&director

    View Slide

  129. director.localhosts.client.{
    ....{..backend.=.one;..weight=1;.}
    ....{..backend.=.two;..weight=1;.}
    }
    sub.vcl_recv.{
    ....set.req.backend.=.localhosts;
    ....//Load.balance.by.URL
    ....set.client.identity.=.req.url;
    ....//Load.balance.by.client.IP,.this.is.the.default.
    ....set.client.identity.=.client.ip;
    ....//Load.balance.by.user.agent
    ....set.client.identity.=.req.http.user;agent;
    }
    Client&director
    Uses&
    client.identity

    View Slide

  130. director.localhosts.hash.{
    ....{..backend.=.one;..weight=1;.}
    ....{..backend.=.two;..weight=1;.}
    }
    sub.vcl_recv.{
    ....set.req.backend.=.localhosts;
    }
    Hash&director
    It&will&use&the&value&of&req.hash,&just&as&the&normal&
    [email protected]&methods
    This&is&useful&is&you&are&using&Varnish&to&load&
    balance&in&front&of&other&Varnish&caches&or&other&
    web&accelerators&as&objects&won't&be&duplicated&
    across&caches.

    View Slide

  131. director.localhosts.fallback.{
    ....{..backend.=.one;}
    ....{..backend.=.two;}
    }
    sub.vcl_recv.{
    ....set.req.backend.=.localhosts;
    }
    Fallback&director
    Picks&first&
    healthy&
    backend
    Order&
    matters

    View Slide

  132. director.directorname.dns.{
    .........list.=.{
    .................host_header.=."www.example.com";
    .................port.=."80";
    .................connect_timeout.=.0.4s;
    ................"192.168.15.0"/24;
    ................"192.168.16.128"/25;
    ........}
    .........ttl.=.5m;
    .........suffix.=."internal.example.net";
    }
    DNS&director
    Autogenerates&backends&in&IP&ranges

    View Slide

  133. director.directorname.dns.{
    .........list.=.{
    .................host_header.=."www.example.com";
    .................port.=."80";
    .................connect_timeout.=.0.4s;
    ................"192.168.15.0"/24;
    ................"192.168.16.128"/25;
    ........}
    .........ttl.=.5m;
    .........suffix.=."internal.example.net";
    }
    DNS&director
    Hostname&=&host_header&+&suffix
    www.example.com.internal.example.net

    View Slide

  134. director.directorname.dns.{
    .........list.=.{
    .................host_header.=."www.example.com";
    .................port.=."80";
    .................connect_timeout.=.0.4s;
    ................"192.168.15.0"/24;
    ................"192.168.16.128"/25;
    ........}
    .........ttl.=.5m;
    .........suffix.=."internal.example.net";
    }
    DNS&director
    Resolves&hostname&&&caches&result
    Backend&selecWon&based&on&result&of&DNS&resolving

    View Slide

  135. Exercise 7
    Playing with backends, probes &
    directors

    View Slide

  136. Cache
    invalidation

    View Slide

  137. Don’t wait for
    cache objects
    to expire

    View Slide

  138. There!are!only!two!hard!
    things!in!Computer!Science:!
    cache!invalidaKon,!naming!
    things,!and!offBbyBone!errors.

    View Slide

  139. ✓Purge
    ✓Ban
    ✓Always5miss
    Cache&invalidaWon

    View Slide

  140. ✓Purge5VCL5method
    ✓Removes5specific5cache5item
    ✓Frees5up5memory
    ✓Evicts5variants
    ✓Next5client5triggers5the5refresh
    ✓No5backup5if5backend5is5down
    Purge

    View Slide

  141. sub.vcl_recv.{
    ........if.(req.request.==."PURGE").{
    ................return.(lookup);
    ........}
    }
    sub.vcl_hit.{
    ........if.(req.request.==."PURGE").{
    ................purge;
    ................error.200."Purged";
    ........}
    }
    sub.vcl_miss.{
    ........if.(req.request.==."PURGE").{
    ................error.404."Not.in.cache";
    ........}
    }
    Purge
    HTTP&call&
    to&purge

    View Slide

  142. ✓Implement5in5your5CMS
    ✓HTTP52005:5removed5from5cache
    ✓HTTP5404:5not5in5cache
    ✓Some5CMS5plugins5require52005for5miss()
    Purge
    curl.;X.PURGE.http://varnish.dev/your;page

    View Slide

  143. ✓Ban5VCL5method5
    ✓Ban5command5line5action
    ✓Ban.url5command5line5action
    ✓Pattern5(regex)5based
    ✓Adds5ban5to5ban:list
    ✓No5memory5freed5up5immediately
    ✓Evicts5variants
    ✓Extremely5fast5and5powerful
    Ban

    View Slide

  144. sub.vcl_recv.{
    ........if.(req.request.==."PURGE").{
    ban("req.http.host.==.".+.req.http.host.+.".
    &&.req.url.==.".+.req.url);
    error.200."Banned";
    ........}
    }
    Ban
    sub.vcl_recv.{
    ........if.(req.request.==."PURGE").{
    ............ban("req.http.host.==.".+.req.http.host.+.
    ".&&.req.url.~.".+.req.url);
    error.200."Banned";
    ........}
    } Ban&URL&
    pattern
    Ban&URL
    Name&
    doesn’t&
    matter

    View Slide

  145. ✓Removes5old5bans5from5the5ban:list
    ✓Separate5thread
    ✓Do5ban.list5on5CLI5to5see5the5ban:list
    ✓ban_lurker_sleep5parameter
    ✓Has5no5access5to5“req”5object
    ✓Smart5bans5are5ban5lurker5friendly
    Ban&lurker

    View Slide

  146. ✓Ban5lurker5friendly
    ✓Use5beresp.x:url5&5beresp.x:host5instead5of5
    req.url5&5req.http.host
    Smart&bans

    View Slide

  147. sub.vcl_recv.{
    ........if.(req.request.==."PURGE").{
    ............ban("obj.http.x;host.==.".+.req.http.host.+.
    ".&&.obj.http.x;url.==.".+.req.url);
    ............error.200."Banned";
    ........}
    }
    sub.vcl_fetch.{
    ........set.beresp.http.x;url.=.req.url;
    ........set.beresp.http.x;host.=.req.http.host;
    }
    sub.vcl_deliver.{
    ........unset.resp.http.x;url;
    ........unset.resp.http.x;host;
    }
    Smart&bans
    Ban&lurker&
    friendly

    View Slide

  148. ✓Used5for5reloading5content,5not5evicting
    ✓Uses5set5req.hash_always_miss5=5true;
    ✓Looks5up5object,5but5ignores5it
    ✓Doesn’t5work5well5with5variations
    ✓Because5return(pass);5doesn’t5store5fetch5
    result5in5cache
    ✓Because5purge();5will5fail5when5your5
    backend5is5down
    Always&miss&

    View Slide

  149. sub.vcl_recv.{
    ........if.(req.request.==."REFRESH").{
    ................set.req.request.=."GET";
    ................set.req.hash_always_miss.=.true;
    ........}
    }
    }
    Refresh/always&miss

    View Slide

  150. varnishstat.;1.;f.n_object
    Measure&with&varnishstat
    Number&
    of&objects&in&
    cache

    View Slide

  151. Protect&your&purges

    View Slide

  152. acl.purge.{
    .."localhost";
    .."217.21.177.0"/24;
    .."10.10.10.1";
    .."some.host.com";
    }
    sub.vcl_recv.{....
    ..if.(req.request.==."PURGE").{
    ....if.(!client.ip.~.purge).{
    ......error.405."Not.allowed.";
    ....}....
    ....ban("obj.http.x;url.==.".+.req.url.+".&&
    ...obj.http.x;host.==.".+.req.http.host);
    ....error.200."Purged";
    ..}
    }
    Protect&your&purges
    Use&an&ACL

    View Slide

  153. Exercise 8
    Invalidation

    View Slide

  154. RELOAD !

    View Slide

  155. ✓/etc/init.d/varnish5reload
    ✓/etc/init.d/varnish5restart
    ✓varnishadm5(vcl.load5&5vcl.use)
    ✓telnet5(vcl.load5&5vcl.use)
    Reloading&Varnish&VCL

    View Slide

  156. ✓/etc/init.d/varnish5reload
    ✓New5VCL5is5loaded
    ✓Keeps5objects5in5memory
    Service&reload

    View Slide

  157. ✓/etc/init.d/varnish5restart
    ✓New5VCL5is5loaded
    ✓All5objects5are5removed5from5memory
    Service&restart

    View Slide

  158. ✓Call5varnishadm5on5CLI
    ✓Call5vcl.list5for5an5overview
    ✓call5vcl.load5bla5/path/vcl/default.vcl
    ✓On5compilation5failure,5errors5are5shown
    ✓Call5vcl.use5bla5to5confirm5new5VCL
    Varnishadm

    View Slide

  159. ✓Open5telnet5connection
    ✓Call5vcl.list5for5an5overview
    ✓call5vcl.load5bla5/path/vcl/default.vcl
    ✓On5compilation5failure,5errors5are5shown
    ✓Call5vcl.use5bla5to5confirm5new5VCL
    Telnet
    See&
    next&slide

    View Slide

  160. /usr/sbin/varnishd.;P./var/run/
    varnishd.pid.;a.:80.;f./home/data/
    default.vcl.;T.127.0.0.1:6082.;u.
    varnish.;g.varnish.;t.120.;p.vcl_dir./
    home/data.;w.5,500,300.;s.malloc,256m.;
    S./etc/varnish/secret.;n.lucid32
    Telnet Running&
    process

    View Slide

  161. $.telnet.127.0.0.1.6082
    Telnet

    View Slide

  162. $.telnet.127.0.0.1.6082
    Trying.::1...
    Trying.127.0.0.1...
    Connected.to.localhost.
    Escape.character.is.'^]'.
    107.59......
    ayfmdmirgrluoqtukkueawfbrpmmvtkw
    Authentication.required.
    Telnet&authenWcaWon
    Auth&
    challenge

    View Slide

  163. Telnet&authenWcaWon
    $.cat./etc/varnish/secret.
    ab9cae99;7905;42e8;9f28;
    b405e684ce93
    Secret&
    key

    View Slide

  164. Telnet&authenWcaWon
    ✓Challenge
    ✓Newline5character5(\x0A)
    ✓Secret5key
    ✓Newline5character5(\x0A)
    ✓Challenge
    ✓Newline5character5(\x0A)
    ✓SHA2565string
    Compose&
    auth&string

    View Slide

  165. Telnet&authenWcaWon
    $challenge = $argv[1];
    $secret = trim(fgets(STDIN));
    $pack = $challenge . "\x0A" . $secret .
    "\x0A" . $challenge . "\x0A";
    $key = hash('sha256', $pack);
    echo $key.PHP_EOL;
    Challenge Secret&
    key
    Result

    View Slide

  166. Telnet&authenWcaWon
    $.cat./etc/varnish/secret.|.php.
    secret.php.
    ayfmdmirgrluoqtukkueawfbrpmmvtkw.
    4fac6b3e568d8b307300d6fa7a1b92ce4e764ce
    46cabed4783ffb0e937d86ff3

    View Slide

  167. $.telnet.127.0.0.1.6082
    Trying.::1...
    Trying.127.0.0.1...
    Connected.to.localhost.
    Escape.character.is.'^]'.
    107.59......
    ayfmdmirgrluoqtukkueawfbrpmmvtkw
    Authentication.required.
    auth.
    4fac6b3e568d8b307300d6fa7a1b92ce4e764ce
    46cabed4783ffb0e937d86ff3
    200.204
    Telnet&authenWcaWon

    View Slide

  168. Exercise 9
    Load VCL via Varnishadm & telnet

    View Slide

  169. Custom&
    VCL

    View Slide

  170. sub.vcl_recv.{
    unset.req.http.cookie;
    }
    sub.vcl_fetch.{
    unset.beresp.http.set;cookie;
    }
    Remove&all&cookies

    View Slide

  171. sub.vcl_recv.{
    set.req.http.Cookie.=.regsuball(req.http.Cookie,.
    "__utm.=[^;]+(;.)?",."");
    set.req.http.Cookie.=.regsuball(req.http.Cookie,.
    "utmctr=[^;]+(;.)?",."");
    set.req.http.Cookie.=.regsuball(req.http.Cookie,.
    "utmcmd.=[^;]+(;.)?",."");
    set.req.http.Cookie.=.regsuball(req.http.Cookie,.
    "utmccn.=[^;]+(;.)?",."");
    if.(req.http.cookie.~."^.*$").{
    unset.req.http.cookie;
    }
    }
    Remove&GA&cookies

    View Slide

  172. sub.vcl_recv.{
    ...return(lookup);
    }
    sub.vcl_hash.{
    ...hash_data(req.http.cookie);
    }
    Cache&cookies

    View Slide

  173. sub.vcl_hash.{
    ....if(req.http.Cookie.~."lang"){
    ........hash_data(regsuball(req.http.Cookie,."^.
    +;?.?lang=([a;zA;Z0;9]+)(.|;|.;).*$","\1"));
    ....}
    }
    Cache&some&cookies

    View Slide

  174. sub.vcl_recv.{
    ...return(lookup);
    }
    sub.vcl_fetch.{
    ...return(deliver);
    }
    Ignore&cookies

    View Slide

  175. sub.vcl_recv.{
    . if.(req.url.~."^[^?]*\.(bmp|bz2|css|doc|
    eot|flv|gif|gz|ico|jpeg|jpg|js|less|mp[34]|
    pdf|png|rar|rtf|swf|tar|tgz|txt|wav|woff|
    xml|zip)(\?.*)?$").{
    ........unset.req.http.Cookie;
    ........return.(lookup);
    ....}
    }
    Cache&staWc&files

    View Slide

  176. sub.vcl_fetch.{
    . if.(req.url.~."^[^?]*\.(bmp|bz2|css|doc|
    eot|flv|gif|gz|ico|jpeg|jpg|js|less|mp[34]|
    pdf|png|rar|rtf|swf|tar|tgz|txt|wav|woff|
    xml|zip)(\?.*)?$").{
    ........unset.beresp.http.set;cookie;
    ....}
    }
    Cache&staWc&files

    View Slide

  177. sub.vcl_deliver.{
    ....set.resp.http.x;cache;hits.=.obj.hits;
    ....if.(obj.hits.>.0).{
    ........set.resp.http.x;cache.=."hit";
    ....}.else.{
    ........set.resp.http.x;cache.=."miss";
    ....}
    }
    Debug

    View Slide

  178. sub.vcl_recv.{
    set.req.http.Host.=.regsub(req.http.Host,.":[0;9]+",."");
    if.(req.url.~."\#").{
    set.req.url.=.regsub(req.url,."\#.*$",."");
    }
    if.(req.url.~."\?$").{
    set.req.url.=.regsub(req.url,."\?$",."");
    }
    }
    SaniWze&URL

    View Slide

  179. sub.vcl_recv.{
    if.(req.url.~."(\?|&)(utm_source|utm_medium|utm_campaign|
    gclid|cx|ie|cof|siteurl)=").{
    set.req.url.=.regsuball(req.url,."&(utm_source|
    utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A;
    z0;9_\;\.%25]+)",."");
    ...set.req.url.=.regsuball(req.url,."\?(utm_source|
    utm_medium|utm_campaign|gclid|cx|ie|cof|siteurl)=([A;z0;9_
    \;\.%25]+)",."?");
    }
    }
    SaniWze&URL&(2)

    View Slide

  180. Exercise 10
    Custom VCL

    View Slide

  181. Grace mode

    View Slide

  182. ✓A&graced&object&is&an&object&that&has&expired,&but&is&
    still&kept&in&cache
    ✓Grace&mode&is&when&Varnish&uses&a&graced&object
    ✓Used&when&backend&is&slow&or&down
    ✓req.grace&defines&how&long&an&overdue&an&object&can&
    be&for&Varnish&to&still&consider&it&for&grace&mode.
    ✓beresp.grace&defines&how&long&past&the&[email protected]
    time&Varnish&will&keep&an&object

    View Slide

  183. sub.vcl_recv{
    ....if.(req.backend.healthy).{
    ........set.req.grace.=.0s;
    ....}.else.{
    ........set.req.grace.=.20s;
    ....}
    }
    sub.vcl_fetch{
    ....set.beresp.grace.=.20s;
    }

    View Slide

  184. probe.healthcheck.{
    ....url.=."/some/url";
    ....interval.=.1s;
    ....timeout.=.1s;
    ....window.=.1;
    ....threshold.=.1;
    ....initial.=.1;
    ....expected_response.=.200;
    }
    backend.default.{
    ...host.=."localhost";
    ...port.=."8080";
    ...probe.=.healthcheck;
    }
    sub.vcl_recv{
    ....if.(req.backend.healthy).{
    ........set.req.grace.=.0s;
    ....}.else.{
    ........set.req.grace.=.20s;
    ....}
    }
    sub.vcl_fetch{
    ....set.beresp.grace.=.20s;
    }
    With&
    backend&
    probe

    View Slide

  185. Exercise 11
    Grace mode

    View Slide

  186. Edge&Side&Includes

    View Slide





















  187. View Slide

  188. header.php
    menu.php main.php
    footer.php
    TTL&5s
    No&caching
    TTL&10s
    TTL&2s

    View Slide

  189. function esi($file){
    if(isset($_SERVER['Surrogate-Capability'])
    && preg_match('/^(.+)="ESI/1.0$/',$_SERVER['Surrogate-Capability'])){
    header("Surrogate-Control: content='ESI/1.0'");
    $url = 'http://'.
    $_SERVER['HTTP_HOST'].substr($_SERVER['SCRIPT_NAME'],
    0,1+strrpos($_SERVER['SCRIPT_NAME'],'/')).$file;
    echo "".PHP_EOL;
    } else {
    include($file);
    }
    }
    ESI&PHP&helper&funcWon&(1)

    View Slide

  190. function esi($file){
    if(isset($_SERVER['Surrogate-Capability'])
    && preg_match('/^(.+)="ESI/1.0$/',$_SERVER['Surrogate-Capability'])){
    header("Surrogate-Control: content='ESI/1.0'");
    $url = 'http://'.
    $_SERVER['HTTP_HOST'].substr($_SERVER['SCRIPT_NAME'],
    0,1+strrpos($_SERVER['SCRIPT_NAME'],'/')).$file;
    echo "".PHP_EOL;
    } else {
    include($file);
    }
    }
    ESI&PHP&helper&funcWon&(2)

    View Slide




















  191. View Slide

  192. ESI&VCL
    sub.vcl_recv.{
    ....set.req.http.Surrogate;Capability="key=ESI/1.0";
    }
    sub.vcl_fetch.{
    . if(beresp.http.Surrogate;Control~"ESI/1.0").{
    .........unset.beresp.http.Surrogate;Control;
    . ....set.beresp.do_esi=true;
    ....}
    }.

    View Slide

  193. require_once __DIR__.'/vendor/autoload.php';
    use Symfony\Component\HttpKernel\HttpCache\Esi;
    use Symfony\Component\HttpFoundation\Request;
    use Symfony\Component\HttpFoundation\Response;
    $esi = new Esi;
    $request = Request::createFromGlobals();
    $response = new Response();
    Symfony&components&(1)

    View Slide

  194. $content = 'Exercise 12';
    $content .= 'Edge Side Includes';
    $content .= date("Y-m-d H:i:s").'
    ';
    if($esi->hasSurrogateEsiCapability($request)){
    $content .= $esi->renderIncludeTag(
    $request->getBasePath().'/edge.php?ttl=2',
    null,false).'
    ';
    $content .= $esi->renderIncludeTag(
    $request->getBasePath().'/edge.php?ttl=0',
    null,false).'
    ';
    }
    $content .= '
    Back to index';
    Symfony&components&(2)

    View Slide

  195. $response->setContent($content);
    $esi->addSurrogateControl($response);
    $response->setSharedMaxAge(10);
    $response->send();
    Symfony&components&(3)

    View Slide

  196. And&
    on&that&
    bombshell&...

    View Slide

  197. View Slide

  198. h"ps://joind.in/9280
    Please&
    give&me&
    feedback

    View Slide

  199. Thanks

    View Slide

  200. View Slide