Speed up your backend caches with Heisencache

Speed up your backend caches with Heisencache

Data caching is often over- or ill-used in complex Drupal projects. Until now, there was no good way to capture and log cache events to analyze cache behaviour. Enter Heisencache and reduce the uncertainty.

9770614c66b56331b6947c79b622a7af?s=128

Frédéric G. MARAND

April 02, 2014
Tweet

Transcript

  1. 1/58 | heisencache | © OSInet Frederic MARAND Speed up

    your backend caches with Heisencache Frédéric G. Marand
  2. 2/58 | heisencache | © OSInet Frederic MARAND Drupal DevDays

    Szeged, 24 30/3/2014 –
  3. 3/58 | heisencache | © OSInet Frederic MARAND CONTEXT MEASURING

    IMPLEMENTATION RESULTS
  4. 4/58 | heisencache | © OSInet Frederic MARAND CONTEXT: perception

    • Front-end dominates • Downloads start after the page is served • Not all sites are equal when it comes to the back/front performance ratio
  5. 5/58 | heisencache | © OSInet Frederic MARAND www.webpagetest.org Major

    press site example: • simple content • below 500 msec backend • tons of extras on page • result: 23 sec front-end time • +/- 50 * backend time CONTEXT: front-end
  6. 6/58 | heisencache | © OSInet Frederic MARAND www.webpagetest.org CONTEXT:

    back-end Major Drupal Commerce site example: • complex backend logic • optimized front-end • +/- 4 * backend time • backend impact = 25%
  7. 7/58 | heisencache | © OSInet Frederic MARAND CONTEXT: battle

    plan • Level1: page caching • Opcode cache • Varnish, CDN • Level 2: storage tuning • MySQL slow queries • MongoDB, Redis, ... • Level 3: cache tuning • On we go... CC BY­NC­ND 2.0 najbo/Flickr
  8. 8/58 | heisencache | © OSInet Frederic MARAND CONTEXT LESS

    DB QUERIES MEANS DATA CACHING Drupal needs caching Cache is a good thing «Cache is king» Steve Souders, author of High-Performance web sites http://fr.slideshare.net/souders/cache-is-king
  9. 9/58 | heisencache | © OSInet Frederic MARAND Photo by

    Alan Light Too much of a good thing is WONDERFUL
  10. 10/58 | heisencache | © OSInet Frederic MARAND (Not that

    kind of good thing) CONTEXT
  11. 11/58 | heisencache | © OSInet Frederic MARAND CONTEXT CACHING

    CAN BE BAD • Memcached speed +/- like good MySQL • Can be worse: clustering network issues • A miss costs more than uncached work
  12. 12/58 | heisencache | © OSInet Frederic MARAND CONTEXT HOW

    DO YOU KNOW WHEN YOU'VE HAD ENOUGH? MEASURE! CC BY­NC­ND 3.0 http://saintgasoline.com
  13. 14/58 | heisencache | © OSInet Frederic MARAND Introducing HEISENCACHE

  14. 15/58 | heisencache | © OSInet Frederic MARAND CONTEXT MEASURING

    IMPLEMENTATION RESULTS
  15. 16/58 | heisencache | © OSInet Frederic MARAND MEASURING WHY

    HEISENCACHE? CC BY 3.0 Gerhard Hund
  16. 17/58 | heisencache | © OSInet Frederic MARAND MEASURING •

    Reduce uncertainty • Minimize observer impact • Don't push the analogy too far
  17. 18/58 | heisencache | © OSInet Frederic MARAND MEASURING THE

    COST OF MEASUREMENT • Observer code runs within Drupal • Needs to be invoked more CPU → • Needs to store data more I/O →
  18. 19/58 | heisencache | © OSInet Frederic MARAND MEASURING CACHING

    BEHAVIOUR vs. LOAD PROFILE
  19. 20/58 | heisencache | © OSInet Frederic MARAND • Instant

    peaks for top content • Fast decay • TTL works wonders • Long tail issues MEASURING: Sports site CC BY 2.0 Ronnie Macdonald
  20. 21/58 | heisencache | © OSInet Frederic MARAND • Many

    content pieces • Few repeat hits • Complex cache policy • Avoid caching some data • Preserve cache memory for selected content MEASURING: Regional TV
  21. 22/58 | heisencache | © OSInet Frederic MARAND MEASURING OBSERVE

    CACHE IN PRODUCTION
  22. 23/58 | heisencache | © OSInet Frederic MARAND MEASURING in

    production • Precision vs. velocity? • Performance module • Cannot write to the DB in real-time • Cached pages • Observing bootstrap
  23. 24/58 | heisencache | © OSInet Frederic MARAND CONTEXT MEASURING

    IMPLEMENTATION RESULTS
  24. 25/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: early

    hits DRUPAL 7 BOOT SEQUENCE • Page cycle: index.php → drupal_bootstrap() • Configuration, Page Cache, DB, Variables, Session, Page Header, Language, Full • Cache handlers declaration: settings.php
  25. 26/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: early

    hits FIRST POSSIBLE CACHE HITS • Exotic: sites.php, settings.php, drupal_settings_initialize() CC BY­SA 3.0 Mdk572/Wikimedia Commons • Normally not (phew...)
  26. 28/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: early

    hits CATCHING EARLY HITS • _drupal_bootstrap_page_cache(): 2nd boot phase • So? Start during 1st bootstrap phase • No module system, no DB as this point • When measuring a cached page hit, don't open the DB
  27. 29/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: early

    hits SOLUTION Use a standalone event dispatching system
  28. 30/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: operation

    IDEAS • Doctrine, NodeJS and Symfony event systems • Use dependency injection, but no DIC (on D7)
  29. 31/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: operation

    BONUS • Composer for deployment – Before hooks, so no composer_manager • Easy unit testing decent code → coverage
  30. 32/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: coverage

  31. 33/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: cache

    driver DECORATOR PATTERN • Inspiration: authcache • Read existing cache configuration • Save it, declare as the sole cache provider • Handle requests per the original configuration, but enhance the service
  32. 34/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: cache

    driver EMIT EVENTS AROUND ALL OPERATIONS • Configuration • Initialization • Cache operations • …and page termination
  33. 35/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: late

    hits CATCHING LATE HITS 1/2 • hook_exit ? Lots of code after that @see drupal_page_footer() • Catching page caching ? – not triggered on AJAX callbacks @see ajax_deliver() – poormancron can run lots of code after that
  34. 36/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: late

    hits CATCHING LATE HITS 2/2 • Catching the session commit? – Only if a session was started – Dirty interactions – Session regeneration • => Shutdown function stack
  35. 37/58 | heisencache | © OSInet Frederic MARAND NEEDS •

    minimize I/O load aim for #writes <= 1 → • «Fingers crossed»-inspired strategy – Keep data in memory during the page lifecycle – Write it at end of page IMPLEMENTATION: storing data
  36. 38/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: storing

    data CHALLENGES • Writing after the last possible cache operation • Write while classes are still available • Passing information within an event-oriented procedural code base
  37. 39/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: events

    EventEmitter (à la Node) • narrowcast events created by sources • subscribers add/remove events on the fly – can further tighten narrowcasting
  38. 40/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: events

    EventSubscriberInterface • Doctrine/Symfony • + (add | remove)Event() EventSourceInterface • Define events a source can emit • Base source of events: Cache API
  39. 41/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: events

    SYNTHETIC EVENTS • API Limitations: post-operation cache events do not get the operation settings • Enable immediate event reconciliation • Event susbcribers can also be sources
  40. 42/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: events

    SYNTHETIC EVENT EXAMPLES • MissSubscriber: miss info for Cache::get() • PerformanceSubscriber: timing info for all ops • WriteSubscriber: single event for write and delete ops
  41. 43/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: setup

    heisencache.inc PLUGIN BOOTSTRAP • wrap $conf, create Config and EventEmitter instances • register a Drupal shutdown handler emitting an onShutdown event • setup the cache override, include settings.heisencache.inc
  42. 44/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: setup

    settings.heisencache.inc CONFIGURATION • Retrieve the EventEmitter from Config instance • Create EventSubscriber instances as chosen • Register them on chosen events • DebugSubscriber listen to all events – not in production ! • Don't forget to include a WriterSubscriber
  43. 45/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: extending

    HEISENCACHE IS A CODER TOOL
  44. 46/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: extending

    EASY TO EXTEND : • Write additional EventSubscriber classes • Add them to your configuration ALREADY EXTENDED : • WriterSubscriber is a user contribution
  45. 47/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: extending

    MOST TYPICAL • Create new subscriber for custom conditions • Create new writer classes • Target alternate stores for speed and ease • MongoDB • K/V or data structure store (Redis) • Message queue (Beanstalkd, RabbitMQ, ZeroMQ, etc)
  46. 48/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: extending

    CREATING A WRITER CLASS • Extend BaseWriterSubscriber • or implement BaseEventSubscriber yourself • Handle the onShutdown event • write $this->history – where you want – how you want
  47. 49/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: extending

    EXISTING WRITER CLASSES • WatchdogWriterSubscriber {watchdog} → • SqlWriterSubscriber specific table →
  48. 50/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: UI

    • ROLL YOUR OWN! • Use WatchdogSubscriber data ‒ Just use admin/reports/dblog ‒ Or rework them • Views integration ‒ Use the SqlWriterSubscriber data ‒ Two default views provided (SqlWS, WatchdogWS) ‒ Drop your own exported views into heisencache/views for automatic loading
  49. 51/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: UI

    • Data collecting: raw data, big volume • Three-step data processing ‒ Collect Heisencache → ‒ Cook Process data based on your needs → ‒ Consumer Visualize processed data → • Sweet spot ‒ Use a queue (Beanstalkd, etc) instead of cron ‒ Push to RRD for longitudinal analysis (Munin, etc)
  50. 52/58 | heisencache | © OSInet Frederic MARAND IMPLEMENTATION: UI

    Comparing with network analysis • Heisencache is like ‒ libpcap / tcpdump / iptrace / snoop ... • Someone has to design a Wireshark on top of it
  51. 53/58 | heisencache | © OSInet Frederic MARAND CONTEXT MEASURING

    IMPLEMENTATION RESULTS
  52. 54/58 | heisencache | © OSInet Frederic MARAND RESULTS: instant

    data TYPICAL USEFUL INSTANT RESULTS • Repeated misses – Usual suspects: default Memcached and big writes (prod) – Rewriting a variable on most pages (dev) • Many calls to same key – Usual suspect: missing or broken static cache • Many calls to related keys – Usual suspect: code loop instead of cache multiple
  53. 55/58 | heisencache | © OSInet Frederic MARAND RESULTS: longitudinal

    analysis TYPICAL USEFUL TIME-SERIES RESULTS • Size of known-to-be-growing keys. Usual suspects: – Translation cache Someone left a → t($foo) somewhere – Context cache Contexts are piled instead of refactored → – Views plugins Hard Views problem. Partial fix only. → • Miss rate shooting up from baseline on a bin – Call for instant analysis on that bin: likely a code regression • Response time shooting up on a normally stable key – Network/server problem, bin saturation – Call for instant analysis on the cache instance
  54. 56/58 | heisencache | © OSInet Frederic MARAND Drupal 8?

    SHOULD HAVE LESS CUSTOM CODE • Reuse Symfony components – Data visibiliy persisted via the Kernel – Ability to hook into boostrap – Configuration using services – Shutdown events
  55. 57/58 | heisencache | © OSInet Frederic MARAND Drupal 8?

    WHEN? • When bigger sites start to deploy D8 • …and need them to go faster :-)
  56. 58/58 | heisencache | © OSInet Frederic MARAND Drupal, faster

    http://www.osinet.fr/