Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Varnish Magento

Varnish Magento

Use varnish reverse proxy cache to increase performance and scalability of Magento. Delivered at Magento Developer Paradise #mdp2011

7ad4c8a7218e44fdf1600b4ebc451738?s=128

Alistair Stead

October 08, 2011
Tweet

Transcript

  1. Ibiza, June 4th – 7th 2011 Monday, 6 June 2011

  2. VARNISH YOUR MAGENTO STORE Make it fly! Monday, 6 June

    2011
  3. WHO AM I • Alistair Stead - @alistairstead • Technical

    Team Lead @ Ibuildings UK / Session Digital • Lead Magento projects for Dreams, 3663, Kookai and others • Author of MageTool and MageTest • Zend Certified Engineer • Over 11 years commercial experience developing in PHP Monday, 6 June 2011
  4. PERFORMANCE AND SCALABILITY Is a hot topic for clients and

    developers alike. Monday, 6 June 2011
  5. PERFORMANCE Vs CONVERSION RATE These are directly linked! Monday, 6

    June 2011
  6. PERCEIVED PERFORMANCE Vs PHYSICAL PERFORMANCE Is there an easy route

    to performance? Monday, 6 June 2011
  7. VARNISH http://www.varnish-cache.org/ Monday, 6 June 2011

  8. PRELIMINARY LOAD TESTS What can Varnish do for us? Monday,

    6 June 2011
  9. +4,000,000 Requests per hour Monday, 6 June 2011

  10. REVERSE PROXY? Yeah okay... tell me more! Monday, 6 June

    2011
  11. CACHING PROXY? It is... Monday, 6 June 2011

  12. LOAD BALANCER? It can do that too! Monday, 6 June

    2011
  13. HTTP ACCELERATOR It may just be magic! Monday, 6 June

    2011
  14. HOW TO CONTROL CACHING Additional caching layers need to be

    coordinated Monday, 6 June 2011
  15. RFC 2616 HTTP 1.1 http://www.w3.org/Protocols/rfc2616/rfc2616.html http://tools.ietf.org/wg/httpbis/ Monday, 6 June 2011

  16. HTTP 1.1 $ curl -I magentocommerce.com Monday, 6 June 2011

  17. HTTP/1.1 200 OK Server: nginx Date: Sun, 05 Jun 2011

    17:40:56 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive X-Powered-By: PHP/5.2.17 Set-Cookie: PHPSESSID=j1cdbj2617abbl8ccsbkd2mt44; path=/ Expires: Mon, 26 Jul 1997 05:00:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post- check=0, pre-check=0 Pragma: no-cache Last-Modified: Sun, 05 Jun 2011 17:40:56 GMT $ curl -I magentocommerce.com Monday, 6 June 2011
  18. HTTP 1.1 HEADERS Request modifiers / Response parameters Monday, 6

    June 2011
  19. CACHEING WITH HTTP 1.1 Monday, 6 June 2011

  20. EXPIRATION MODEL How long may I use this resource for?

    Monday, 6 June 2011
  21. EXPIRATION HEADERS • Cache-Control • Expires Monday, 6 June 2011

  22. VALIDATION MODEL Is this resource still okay to use? Monday,

    6 June 2011
  23. VALIDATION HEADERS • Last-Modified • If-Modified-Since • ETag • If-None-Match

    Monday, 6 June 2011
  24. HTTP CACHE HEADERS Only work with safe HTTP verbs GET

    & HEAD Monday, 6 June 2011
  25. HTTP 1.1 REQUEST FLOW Monday, 6 June 2011

  26. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET

    / HTTP1.1 Host: mystore.com HTTP1.1 200 OK HTTP1.1 200 OK Monday, 6 June 2011
  27. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET

    / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
  28. Client Application Client Cache GET / HTTP1.1 Host: mystore.com HTTP1.1

    200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
  29. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET

    / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Etag: abcde HTTP1.1 200 OK Etag: abcde HTTP1.1 200 OK Etag: abcde Monday, 6 June 2011
  30. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET

    / HTTP1.1 Host: mystore.com If-None-Match: abcde HTTP1.1 304 Not Modified HTTP1.1 304 Not Modified HTTP1.1 200 OK Etag: abcde Monday, 6 June 2011
  31. EXPIRATION ALLOWS YOU TO SCALE VALIDATION SAVES BANDWIDTH Fewer requests

    hitting your server and client speed is better too! Monday, 6 June 2011
  32. OUR GOAL IS TO NEVER GENERATE THE SAME RESPONSE TWICE

    HTTP caching can help Monday, 6 June 2011
  33. LEVERAGE HTTP CACHE IN MAGENTO With an event observer this

    is simple. Monday, 6 June 2011
  34. <?php class Disclosure_Varnish_Model_Observer extends Disclosure_Varnish_Model_Observer_Abstract { /** * Update the

    headers transmitted to allow greater varnish cache hits * * @param $observer */ public function processPreDispatch(Varien_Event_Observer $observer) { if (!$this->isVarnishEnabled()) { return $this; } $routeTtls = $this->_config->getRouteTtls(); $routeName = Mage::app()->getRequest()->getRequestedRouteName(); if (isset($routeTtls[$routeName])) { $observer->getResponse()->setHeader('Cache-Control', $routeTtls[$routeName]['cache_control'], false); } return $this; } } Monday, 6 June 2011
  35. <?xml version="1.0" encoding="UTF-8"?> <config> <ttls> <routes> <cms> <name>cms</name> <cache_control>public, max-age=3600</cache_control>

    </cms> <catalog> <name>catalog</name> <cache_control>public, must-revalidate, max-age=0</cache_control> <actions> <index> <cache_control>private, max-age=0</cache_control> </index> </actions> </catalog> </routes> </ttls> Monday, 6 June 2011
  36. USE VARNISH WITH MAGENTO Add shared cache to reduce the

    load on the application Client Browser Varnish Port 80 Backend Apache Port 8080 Monday, 6 June 2011
  37. VARNISH RESPECTS CACHE CONTROL Dispatch the headers that fit your

    application Monday, 6 June 2011
  38. Client Application Client Cache Reverse Proxy Cache GET / HTTP1.1

    Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
  39. Client Application Client Cache Reverse Proxy Cache GET / HTTP1.1

    Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
  40. Client Application Client Cache Reverse Proxy Cache GET / HTTP1.1

    Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Client Client Cache GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
  41. Client Application Client Cache Reverse Proxy Cache Client Client Cache

    Gateway Cache Network Proxy Cache max-age s-maxage max-age s-maxage max-age s-maxage max-age max-age Monday, 6 June 2011
  42. VARNISH IS CONFIGURED USING VCL Send the correct headers you

    should not need to do much. Monday, 6 June 2011
  43. sub vcl_recv { set req.http.Surrogate-Capability = "magento=ESI/1.0"; return (lookup); }

    sub vcl_fetch { if (beresp.http.Surrogate-Control ~ "ESI/1.0") { # unset beresp.http.Surrogate-Control; esi; } return (deliver); } sub vcl_deliver { # Set a cache header to allow us to inspect the response # headers during testing if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS"; } } Monday, 6 June 2011
  44. EDGE SIDE INCLUDES http://www.w3.org/TR/esi-lang Monday, 6 June 2011

  45. max-age=3600 max-age=86400 Monday, 6 June 2011

  46. max-age=3600 max-age=86400 Monday, 6 June 2011

  47. ESI TAGS <esi:include src="/varnish/esi?attributes=%s"/> Monday, 6 June 2011

  48. /** * Render ESI tags for the block if needed

    * * @param Varien_Event_Observer $observer */ public function renderBlockEsiTag(Varien_Event_Observer $observer) { if (!$this->isEsiEnabled() || !$this->isVarnishEnabled()) { return $this; } $block = $observer->getEvent()->getBlock(); $transport = $observer->getEvent()->getTransport(); $tag = Mage::getModel('varnish/esi_tag_factory')->create($block); if ($transport && $tag) { $transport->setHtml($tag->getHtml()); } return $this; } Monday, 6 June 2011
  49. /** * Render the html for the ESI tag *

    * @return string * @author Alistair Stead **/ public function getHtml() { return sprintf( '<esi:include src="/varnish/esi?attributes=%s"/>', $this->getAttributeHash() ); } Monday, 6 June 2011
  50. class Disclosure_Varnish_EsiController extends Mage_Core_Controller_Front_Action { public function indexAction() { $attributes

    = unserialize(base64_decode($this->getRequest()->getParam('attributes'))); $context = Mage::getModel('varnish/esi_context'); $context->setData($attributes); $processor = Mage::getModel('varnish/esi_processor', $context); // Initialize the layout $this->loadLayout(); Mage::dispatchEvent('varnish_controller_esi_appendbody_before', array("response" => $this->getResponse())); $this->getResponse()->appendBody($processor->render()); Mage::dispatchEvent('varnish_controller_esi_appendbody_after', array("response" => $this->getResponse())); } } Monday, 6 June 2011
  51. FINAL PERFORMANCE RESULTS How much traffic did these modifications let

    us serve Monday, 6 June 2011
  52. TRAFFIC SAMPLE • Product Details page: 58k views per hour

    • Homepage: 50k requests per hour • Main Search : 70k requests per hour • Refined Search (via facets): 20k requests per hour • Add to Basket: 11k requests per hour • Login: 11k requests per hour • Category Pages: 46k requests per hour • Checkout: 6k requests per hour Monday, 6 June 2011
  53. Homepage Category Listings Product Detail Add to cart Login Search

    Purchase 0 375000 750000 1125000 1500000 Client Baseline Magento Baseline Magento Compiler Magento + Varnish FINAL PERFORMANCE RESULTS Monday, 6 June 2011
  54. NON CACHEABLE TRANSACTIONS Add to cart Purchase 0 7500 15000

    22500 30000 Client Baseline Magento Baseline Magento Compiler Magento + Varnish Monday, 6 June 2011
  55. Monday, 6 June 2011

  56. THE TARGET ~ 300K REQUEST We achieved almost 14 x

    that with little effort. Monday, 6 June 2011
  57. IS VARNISH THE BEST SOLUTION? What do we need to

    consider? Monday, 6 June 2011
  58. VARNISH LIMITATIONS • Does not currently support GZIP • Cookies

    need to be managed carefully • Still need SSL endpoint • Routing and URL generation can be tricky • Beware cache stampede • ESI must have a TTL greater than the parent page • Shared cache consider using s-maxage with max-age Monday, 6 June 2011
  59. ADDITIONAL OPTIONS Other HTTP caching options and solutions Monday, 6

    June 2011
  60. CLIENT SIDE INCLUDES A common pattern using Ajax Monday, 6

    June 2011
  61. STALE WHILE Maintain a compromised UI while resolving content Monday,

    6 June 2011
  62. ALTERNATE WHILE Provide alternate content will content is being resolved

    Monday, 6 June 2011
  63. REFERENCES • http://blog.sessiondigital.com/post/5764403845/high-performance-magento • http://www.varnish-cache.org/ • http://www.w3.org/Protocols/rfc2616/rfc2616.html • http://tools.ietf.org/wg/httpbis/ •

    http://www.w3.org/TR/esi-lang • http://www.fabrizio-branca.de/make-your-magento-store-fly-using-varnish.html Monday, 6 June 2011
  64. IMAGE CREDITS http://www.flickr.com/photos/bondgirly/4520060395/sizes/o/in/photostream/ Monday, 6 June 2011

  65. THANK YOU! • Email: astead@ibuildings.com • Skype: astead-ibuildings • Twitter:

    @alistairstead Monday, 6 June 2011
  66. QUESTIONS? Monday, 6 June 2011

  67. WE ARE HIRING! http://www.ibuildings.co.uk/about/careers/ Monday, 6 June 2011

  68. Monday, 6 June 2011