Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Surviving a Prime Time TV Commercial (DPC2013 2013-06-07)

Surviving a Prime Time TV Commercial (DPC2013 2013-06-07)

Presentation given at Dutch PHP Conference 2013 in Amsterdam, The Netherlands.


David Zuelke

June 07, 2013

More Decks by David Zuelke

Other Decks in Programming



  2. David Zülke

  3. David Zuelke

  4. None
  5. http://en.wikipedia.org/wiki/File:München_Panorama.JPG

  6. Lead Architect

  7. None
  8. @dzuelke

  9. THE CHALLENGE (who doesn’t love a challenge?)

  10. None
  11. None
  12. happily running an off-the-shelf e-commerce solution

  13. suddenly: an ad on TV during prime time hours

  14. = up to 10k unique visitors within a minute or

  15. and before you can count to three...

  16. None
  17. None
  18. 0 5000 10000 15000 20000 20:44 20:45 20:46 20:47 20:48

    20:49 20:50 20:51 20:52 Requests per minute
  19. What will happen first? Apache eating up all RAM? Too

    many MySQL connections? I/O trash/wait halting everything?

  21. the good news:

  22. suddenly: an ad on TV during prime time hours ...

    airs at a pre-determined time ...
  23. that helps, but you still need to make it web

  24. OPTIONS FOR SCALING OUT 1.Add hardware & caching • and

    deal with “great” code • Recurring license costs... • ... per server, usually! • Rinse and repeat for the next shop you launch 2.Build it yourself • Investment up front • Needs maintenance • Hybrid approach makes a lot of sense: keep old software for cart, checkout, payment, ...
  25. OUR GOALS • Home page, product detail, category browser, search:

    <100 ms • Try to degrade, but not to fail, under extreme load • Prefer failures for some users over slowness for all users • Sacrifice freshness of data and features for performance • Replicate any change in product DB to front-end in <60 s

  27. FOR THAT, WE NEED: • A share-nothing architecture (or at

    least a “share little” one) • Failover configurations where needed (load balancer, cache, ...) • An FTS capable database that withstands a lot of parallel reads • Ideally, an approach where we don’t even hit an app server
  28. TRADITIONAL HOSTING OR THE CLOUD? A) Host everything yourself B)

    Only use SaaS and components like S3, CDNs and so forth C) Host some parts yourself, and others on AWS or similar D) Host everything in the cloud
  29. forget “random” scale-out from hardware to EC2 (data replication, latency,

  30. We chose option B. Then option C. Then B again

  31. SYMFONY delivers! :)

  32. it’s great, lots of bundles, community, blah... we all know

    that, right? :)
  33. SPECIAL MENTIONS • Bundle Inheritance • Base implementations for Search,

    Product, Category, ... • Sites (Kiveda, ...) may override templates, controllers, ... • Twig • Share templates (at least some) between “Fox” and OXID
  34. ELASTICSEARCH (or Solr, if you prefer that)

  35. not just for search

  36. why not store all data (products, categories, ...) in there?

  37. distribution & HA built-in

  38. REPLICATING MYSQL TO ES • Triggers on product/category/... tables write

    to a table with commands for replication • Script (Symfony CLI) runs periodically, reads replication commands and replays them into ElasticSearch • supervisord keeps it alive (crons could overtake each other) • Sends PURGE commands to Varnish
  39. No Gearman, your operations must be serialized

  40. CACHE everything as much as you can

  41. Varnish!

  42. use grace and saint modes

  43. done, problem solved?

  44. users have shopping carts

  45. LOGINS & SHOPPING CARTS A) ESI: Set cookie when adding

    to cart, detect cookie in VCL B) XHR: No server-side work, but latency on the client side, and you need to deal with CORS even for just HTTP vs HTTPS C) Signed cookies
  46. ESI TRICK FOR THE CART 1. Set marker cookie (with

    a lifetime) when adding to cart 2. Page contains <esi:include  href=”/cart”  /> 3. If cookie above exists, hit backend 4. If not, rewrite location to “/cart_empty.html” in vcl_fetch 5. Same can be done for logged in users
  47. ESIs are processed sequentially!

  48. AUTOMATE everything

  49. CI: Jenkins (or Travis, ...)

  50. Deployment: Capistrano

  51. Provisioning: Puppet (or Chef)

  52. bring new boxes, staging envs (LXC, Xen, ...) up quickly

  53. make sure stage mimics production (logical nodes, LB, ...)

  54. ADDED BENEFIT: DEV ENVS! • Build a Veewee base box

    that mirrors production config • Use Vagrant and Puppet manifests from production for dev! • Quickly test e.g. a new ElasticSearch node going up or down • For the future: • Run staging using https://github.com/fgrehm/vagrant-lxc • Devs can log in and kill the database, add web nodes, ...
  55. MEASURE, LOG AND TEST everything

  56. profile your stuff, with XHProf, not “some” debug toolbar ;)

  57. log errors, performance metrics, user flows

  58. and keep this data (disk, tape, Hadoop HDFS, whatever)

  59. give your developers access to this data (logstash, HDFS, ...)

  60. load test using this data (Siege/JMeter/Tsung)

  61. https://github.com/j2labs/microarmy

  62. https://github.com/newsapps/beeswithmachineguns

  63. OUTSOURCE everything anything tedious

  64. storage and backups (to S3 and friends)

  65. content delivery (CloudFront, CloudFlare, ...)

  66. transactional email (Mandrill, Postmark, SendGrid, ...)

  67. metrics/logging/monitoring (New Relic, Loggly, Silverline)

  68. !e End

  69. Questions?

  70. THANK YOU! This was http://joind.in/8440 by @dzuelke. Questions: dz@europeanmedia.com