Lessons Learnt. An In Production, Scaleable Django Stack

Lessons Learnt. An In Production, Scaleable Django Stack

My PyCon Ireland 2012 talk with an analysis and description of some of the various in production parts of our stack.

IE, Linux, MySQL, Memcache, Django, Apache, Solr, Django-sentry, Buildout/Fabric, statsd etc.

Some discussion also of alternative choices, up and coming technology, things to watch.

52bf15d69aec37a732663a1742006bb3?s=128

John G. Moylan

October 14, 2012
Tweet

Transcript

  1. Lessons Learnt. An In Production, Scaleable Django Stack PyCon ireland

    - Dublin, Oct 13th-14th 2012 john.moylan@rte.ie / john@8t8.eu / @redsnapper88 / www.8t8.eu
  2. Brief Intro to RTE.ie Website of Ireland's national broadcaster

  3. Problem Mostly W.O.R.M, but some stuff can't be cached :(

    Lot's of content (100's GB's of HTML) 10's TB of Audio and Video Lots of strands (News, Sport, Business, Ent., TV, Radio, Teletext, Archives, Young Peoples, Performing Groups...) Small team. Try to keep things as simple and efficient as possible. ...massive growth
  4. Traffic Growth Oct 2010 V's Oct 2011 Traffic nearly doubled.

  5. Flashmob Traffic 36Gb/s of Traffic. 41K Concurrent streams. Playlists were

    not cacheable........ But what a great fight.
  6. Static? Legacy system published to static files using XML/XSL Publisher.

    Multiple versions per type of view. SSI's inside SSI's inside SSI's Republishing a strand could take days.
  7. Dynamic Requirement Evaluated some systems: Wicket, Rails, Shale....Django Hmmm... nice

    All in package, sensible defaults. Similar problem domain.
  8. Try Before you buy 2008 Wrote prototype video upload sharing

    site... (Django 0.96) and then decided to make it live. http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/
  9. Try Before You Buy 2009 Original RTÉ Player - Written

    in Django. 2011 Let's use Django to make the whole site dynamic!
  10. ..continued 2012 Sport, About, RTÉ Player v.2, Radio, News?, Archives,

    playlists, lifestyle, SSO......and on and on 2013.. More
  11. Django Delivery Linux Apache with Event MPM and mod_wsgi Mysql

    and MHA (tip Innodb plugin) Solr for search and 'NoSQL' Varnish (used to use Squid) LVS-NAT and LVS-DR Load-balancers. Memcache Deploying Glusterfs
  12. Code Management Git Redmine Gitosis

  13. Deployment Cobbler Fabric Buildout

  14. Monitoring Monitor & Measure everything! Zenoss RRDTool MRTG Graphite Statsd

    - mozilla's Django-statsd Sentry Lot's of custom monitoring
  15. In House Monitoring projects Pyvarnish https://github.com/redsnapper8t8/pyvarnish

  16. In House Monitoring Projects Digital Dash - (open sourced soon)

  17. Benchmarking Tool agnostic but seem to use these alot. Bonnie++

    Netperf #HTTP Httperf Siege
  18. Performance Tips 1. Memory, memory, memory 2. Caching - in

    memory 3. Does it scale horizontally? 4. KISS
  19. Lessons Learnt. Read from memory whenever possible Cache, learned this

    one a long time ago but whenever we have issues it can usually be tracked down bad design and lack of caching. Don't hit your web servers unless you have to. Memcache, Varnish Long live Rest - GET's should not have side effects. Varnish KISS - and no resume driven development.
  20. Thank you Questions?