Website on cache: scalable, cheap and realiable by Samat Galimov

Transcript

2. Website on the Cache Dmitry Zakharov Samat Galimov Meduza
3. Meduza • exiled news outlet from Russia, built by the former team of the most popular news website Lenta.Ru (>10mln readers monthly, #1 cited in Russian Internet) • Website, iOS, Android and Windows Phone apps with consistent user experience, browser extensions & more • 7 developers in total, 30 people editorial team
5. Stats • Launched Oct 2014 • 4mln readers, 45mln pageviews in Feb 2016 • 200TB in traffic • 400 eur for edge servers, 1 (one) systems engineer
6. External requirements • traffic bursts 1:10 or more (news websites at 9/11) — we've seen in in Feb 2015, Nemtsov assassination — 500k daily users,100k users avg • no serious upfront investment in hardware or services • minimum fee possible • good latency & speed
7. We need cache
8. 'usual' nginx cache with 1min expire • editorial waiting to see content live (it's compiling!) • uneven distribution — contant 'you might be on the late server' excuses • website deployment delay (we do debug on prod )
9. NodeJS Nginx nginx nginx nginx nginx Edge Cluster this is bottlenect 2.5rps/core „
10. Custom cache control system (1k SLOC golang)
11. • ∞ TTL cache on all nginxex • custom service to clear the cache
12. ... NodeJS Nginx nginx CacheM nginx CacheM nginx CacheM CacheM master CacheM master Edge Cluster
13. • master - slave arch. • outer world communicates with master • sync API on master: 'purge' & 'heat' • master maintains list of all slaves and passes messages to them • clear nginx cache only if upstream returns 200
14. NodeJS Nginx CacheM Redis admin interface Nginx CacheM Edge Machine
15. hardware & hosting for edge • started with Amazon AWS (intel virtual) • then bare metal (intel xeon) • then scaleway+online.net (intel atom + arm) • main concern is bandwith per node per price
16. latency is everything • DNS resolve: 20ms-200ms (amazon vs cloudflare) • TLS negotiation: 0-200ms (RFC 5077) • backend page generation: 0-400ms (node vs nginx) • deep system tuning: scheduler, buffers, network cards, etc. • data transfer: closer to the clients (Moscow server) • dom loading: server side rendering for nodejs
17. operations • 1-click deployment of edge nodes via ansible • self-registration of edge CacheM to master • cacheM master notifies on issues with slaves • graphana graphs for traffic etc. • ELK for access logs
18. QA, please • Dmitry Zakharov zaharov@meduza.io • Samat Galimov @samat, samat@meduza.io