Varnish Magento

Use varnish reverse proxy cache to increase performance and scalability of Magento. Delivered at Magento Developer Paradise #mdp2011

Transcript

1. Ibiza, June 4th – 7th 2011 Monday, 6 June 2011
2. VARNISH YOUR MAGENTO STORE Make it fly! Monday, 6 June 2011
3. WHO AM I • Alistair Stead - @alistairstead • Technical Team Lead @ Ibuildings UK / Session Digital • Lead Magento projects for Dreams, 3663, Kookai and others • Author of MageTool and MageTest • Zend Certified Engineer • Over 11 years commercial experience developing in PHP Monday, 6 June 2011
4. PERFORMANCE AND SCALABILITY Is a hot topic for clients and developers alike. Monday, 6 June 2011
5. PERFORMANCE Vs CONVERSION RATE These are directly linked! Monday, 6 June 2011
6. PERCEIVED PERFORMANCE Vs PHYSICAL PERFORMANCE Is there an easy route to performance? Monday, 6 June 2011
7. VARNISH http://www.varnish-cache.org/ Monday, 6 June 2011
8. PRELIMINARY LOAD TESTS What can Varnish do for us? Monday, 6 June 2011
9. +4,000,000 Requests per hour Monday, 6 June 2011
10. REVERSE PROXY? Yeah okay... tell me more! Monday, 6 June 2011
11. CACHING PROXY? It is... Monday, 6 June 2011
12. LOAD BALANCER? It can do that too! Monday, 6 June 2011
13. HTTP ACCELERATOR It may just be magic! Monday, 6 June 2011
14. HOW TO CONTROL CACHING Additional caching layers need to be coordinated Monday, 6 June 2011
15. RFC 2616 HTTP 1.1 http://www.w3.org/Protocols/rfc2616/rfc2616.html http://tools.ietf.org/wg/httpbis/ Monday, 6 June 2011
16. HTTP 1.1 $ curl -I magentocommerce.com Monday, 6 June 2011
17. HTTP/1.1 200 OK Server: nginx Date: Sun, 05 Jun 2011 17:40:56 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive X-Powered-By: PHP/5.2.17 Set-Cookie: PHPSESSID=j1cdbj2617abbl8ccsbkd2mt44; path=/ Expires: Mon, 26 Jul 1997 05:00:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post- check=0, pre-check=0 Pragma: no-cache Last-Modified: Sun, 05 Jun 2011 17:40:56 GMT $ curl -I magentocommerce.com Monday, 6 June 2011
18. HTTP 1.1 HEADERS Request modifiers / Response parameters Monday, 6 June 2011
19. CACHEING WITH HTTP 1.1 Monday, 6 June 2011
20. EXPIRATION MODEL How long may I use this resource for? Monday, 6 June 2011
21. EXPIRATION HEADERS • Cache-Control • Expires Monday, 6 June 2011
22. VALIDATION MODEL Is this resource still okay to use? Monday, 6 June 2011
23. VALIDATION HEADERS • Last-Modified • If-Modified-Since • ETag • If-None-Match Monday, 6 June 2011
24. HTTP CACHE HEADERS Only work with safe HTTP verbs GET & HEAD Monday, 6 June 2011
25. HTTP 1.1 REQUEST FLOW Monday, 6 June 2011
26. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK HTTP1.1 200 OK Monday, 6 June 2011
27. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
28. Client Application Client Cache GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
29. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Etag: abcde HTTP1.1 200 OK Etag: abcde HTTP1.1 200 OK Etag: abcde Monday, 6 June 2011
30. Client Application Client Cache GET / HTTP1.1 Host: mystore.com GET / HTTP1.1 Host: mystore.com If-None-Match: abcde HTTP1.1 304 Not Modified HTTP1.1 304 Not Modified HTTP1.1 200 OK Etag: abcde Monday, 6 June 2011
31. EXPIRATION ALLOWS YOU TO SCALE VALIDATION SAVES BANDWIDTH Fewer requests hitting your server and client speed is better too! Monday, 6 June 2011
32. OUR GOAL IS TO NEVER GENERATE THE SAME RESPONSE TWICE HTTP caching can help Monday, 6 June 2011
33. LEVERAGE HTTP CACHE IN MAGENTO With an event observer this is simple. Monday, 6 June 2011
34. <?php class Disclosure_Varnish_Model_Observer extends Disclosure_Varnish_Model_Observer_Abstract { /** * Update the headers transmitted to allow greater varnish cache hits * * @param $observer */ public function processPreDispatch(Varien_Event_Observer $observer) { if (!$this->isVarnishEnabled()) { return $this; } $routeTtls = $this->_config->getRouteTtls(); $routeName = Mage::app()->getRequest()->getRequestedRouteName(); if (isset($routeTtls[$routeName])) { $observer->getResponse()->setHeader('Cache-Control', $routeTtls[$routeName]['cache_control'], false); } return $this; } } Monday, 6 June 2011
35. <?xml version="1.0" encoding="UTF-8"?> <config> <ttls> <routes> <cms> <name>cms</name> <cache_control>public, max-age=3600</cache_control> </cms> <catalog> <name>catalog</name> <cache_control>public, must-revalidate, max-age=0</cache_control> <actions> <index> <cache_control>private, max-age=0</cache_control> </index> </actions> </catalog> </routes> </ttls> Monday, 6 June 2011
36. USE VARNISH WITH MAGENTO Add shared cache to reduce the load on the application Client Browser Varnish Port 80 Backend Apache Port 8080 Monday, 6 June 2011
37. VARNISH RESPECTS CACHE CONTROL Dispatch the headers that fit your application Monday, 6 June 2011
38. Client Application Client Cache Reverse Proxy Cache GET / HTTP1.1 Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
39. Client Application Client Cache Reverse Proxy Cache GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
40. Client Application Client Cache Reverse Proxy Cache GET / HTTP1.1 Host: mystore.com GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 HTTP1.1 200 OK Cache-Control: max- age=20 Client Client Cache GET / HTTP1.1 Host: mystore.com HTTP1.1 200 OK Cache-Control: max- age=20 Monday, 6 June 2011
41. Client Application Client Cache Reverse Proxy Cache Client Client Cache Gateway Cache Network Proxy Cache max-age s-maxage max-age s-maxage max-age s-maxage max-age max-age Monday, 6 June 2011
42. VARNISH IS CONFIGURED USING VCL Send the correct headers you should not need to do much. Monday, 6 June 2011
43. sub vcl_recv { set req.http.Surrogate-Capability = "magento=ESI/1.0"; return (lookup); } sub vcl_fetch { if (beresp.http.Surrogate-Control ~ "ESI/1.0") { # unset beresp.http.Surrogate-Control; esi; } return (deliver); } sub vcl_deliver { # Set a cache header to allow us to inspect the response # headers during testing if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS"; } } Monday, 6 June 2011
44. EDGE SIDE INCLUDES http://www.w3.org/TR/esi-lang Monday, 6 June 2011
45. max-age=3600 max-age=86400 Monday, 6 June 2011
46. max-age=3600 max-age=86400 Monday, 6 June 2011
47. ESI TAGS <esi:include src="/varnish/esi?attributes=%s"/> Monday, 6 June 2011
48. /** * Render ESI tags for the block if needed * * @param Varien_Event_Observer $observer */ public function renderBlockEsiTag(Varien_Event_Observer $observer) { if (!$this->isEsiEnabled() || !$this->isVarnishEnabled()) { return $this; } $block = $observer->getEvent()->getBlock(); $transport = $observer->getEvent()->getTransport(); $tag = Mage::getModel('varnish/esi_tag_factory')->create($block); if ($transport && $tag) { $transport->setHtml($tag->getHtml()); } return $this; } Monday, 6 June 2011
49. /** * Render the html for the ESI tag * * @return string * @author Alistair Stead **/ public function getHtml() { return sprintf( '<esi:include src="/varnish/esi?attributes=%s"/>', $this->getAttributeHash() ); } Monday, 6 June 2011
50. class Disclosure_Varnish_EsiController extends Mage_Core_Controller_Front_Action { public function indexAction() { $attributes = unserialize(base64_decode($this->getRequest()->getParam('attributes'))); $context = Mage::getModel('varnish/esi_context'); $context->setData($attributes); $processor = Mage::getModel('varnish/esi_processor', $context); // Initialize the layout $this->loadLayout(); Mage::dispatchEvent('varnish_controller_esi_appendbody_before', array("response" => $this->getResponse())); $this->getResponse()->appendBody($processor->render()); Mage::dispatchEvent('varnish_controller_esi_appendbody_after', array("response" => $this->getResponse())); } } Monday, 6 June 2011
51. FINAL PERFORMANCE RESULTS How much traffic did these modifications let us serve Monday, 6 June 2011
52. TRAFFIC SAMPLE • Product Details page: 58k views per hour • Homepage: 50k requests per hour • Main Search : 70k requests per hour • Refined Search (via facets): 20k requests per hour • Add to Basket: 11k requests per hour • Login: 11k requests per hour • Category Pages: 46k requests per hour • Checkout: 6k requests per hour Monday, 6 June 2011
53. Homepage Category Listings Product Detail Add to cart Login Search Purchase 0 375000 750000 1125000 1500000 Client Baseline Magento Baseline Magento Compiler Magento + Varnish FINAL PERFORMANCE RESULTS Monday, 6 June 2011
54. NON CACHEABLE TRANSACTIONS Add to cart Purchase 0 7500 15000 22500 30000 Client Baseline Magento Baseline Magento Compiler Magento + Varnish Monday, 6 June 2011
55. Monday, 6 June 2011
56. THE TARGET ~ 300K REQUEST We achieved almost 14 x that with little effort. Monday, 6 June 2011
57. IS VARNISH THE BEST SOLUTION? What do we need to consider? Monday, 6 June 2011
58. VARNISH LIMITATIONS • Does not currently support GZIP • Cookies need to be managed carefully • Still need SSL endpoint • Routing and URL generation can be tricky • Beware cache stampede • ESI must have a TTL greater than the parent page • Shared cache consider using s-maxage with max-age Monday, 6 June 2011
59. ADDITIONAL OPTIONS Other HTTP caching options and solutions Monday, 6 June 2011
60. CLIENT SIDE INCLUDES A common pattern using Ajax Monday, 6 June 2011
61. STALE WHILE Maintain a compromised UI while resolving content Monday, 6 June 2011
62. ALTERNATE WHILE Provide alternate content will content is being resolved Monday, 6 June 2011
63. REFERENCES • http://blog.sessiondigital.com/post/5764403845/high-performance-magento • http://www.varnish-cache.org/ • http://www.w3.org/Protocols/rfc2616/rfc2616.html • http://tools.ietf.org/wg/httpbis/ • http://www.w3.org/TR/esi-lang • http://www.fabrizio-branca.de/make-your-magento-store-fly-using-varnish.html Monday, 6 June 2011
64. IMAGE CREDITS http://www.flickr.com/photos/bondgirly/4520060395/sizes/o/in/photostream/ Monday, 6 June 2011
65. THANK YOU! • Email: astead@ibuildings.com • Skype: astead-ibuildings • Twitter: @alistairstead Monday, 6 June 2011
66. QUESTIONS? Monday, 6 June 2011
67. WE ARE HIRING! http://www.ibuildings.co.uk/about/careers/ Monday, 6 June 2011
68. Monday, 6 June 2011