Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lessons Learnt. An In Production, Scaleable Django Stack

Lessons Learnt. An In Production, Scaleable Django Stack

My PyCon Ireland 2012 talk with an analysis and description of some of the various in production parts of our stack.

IE, Linux, MySQL, Memcache, Django, Apache, Solr, Django-sentry, Buildout/Fabric, statsd etc.

Some discussion also of alternative choices, up and coming technology, things to watch.

John G. Moylan

October 14, 2012
Tweet

More Decks by John G. Moylan

Other Decks in Technology

Transcript

  1. Problem Mostly W.O.R.M, but some stuff can't be cached :(

    Lot's of content (100's GB's of HTML) 10's TB of Audio and Video Lots of strands (News, Sport, Business, Ent., TV, Radio, Teletext, Archives, Young Peoples, Performing Groups...) Small team. Try to keep things as simple and efficient as possible. ...massive growth
  2. Static? Legacy system published to static files using XML/XSL Publisher.

    Multiple versions per type of view. SSI's inside SSI's inside SSI's Republishing a strand could take days.
  3. Dynamic Requirement Evaluated some systems: Wicket, Rails, Shale....Django Hmmm... nice

    All in package, sensible defaults. Similar problem domain.
  4. Try Before you buy 2008 Wrote prototype video upload sharing

    site... (Django 0.96) and then decided to make it live. http://web.archive.org/web/20090318064215/http://www.rte.ie/tv/theden/ugc/red/video/
  5. Try Before You Buy 2009 Original RTÉ Player - Written

    in Django. 2011 Let's use Django to make the whole site dynamic!
  6. ..continued 2012 Sport, About, RTÉ Player v.2, Radio, News?, Archives,

    playlists, lifestyle, SSO......and on and on 2013.. More
  7. Django Delivery Linux Apache with Event MPM and mod_wsgi Mysql

    and MHA (tip Innodb plugin) Solr for search and 'NoSQL' Varnish (used to use Squid) LVS-NAT and LVS-DR Load-balancers. Memcache Deploying Glusterfs
  8. Monitoring Monitor & Measure everything! Zenoss RRDTool MRTG Graphite Statsd

    - mozilla's Django-statsd Sentry Lot's of custom monitoring
  9. Performance Tips 1. Memory, memory, memory 2. Caching - in

    memory 3. Does it scale horizontally? 4. KISS
  10. Lessons Learnt. Read from memory whenever possible Cache, learned this

    one a long time ago but whenever we have issues it can usually be tracked down bad design and lack of caching. Don't hit your web servers unless you have to. Memcache, Varnish Long live Rest - GET's should not have side effects. Varnish KISS - and no resume driven development.