Scaling Expression Engine: So, your site got popular?

This talk will focus on the "behind the scenes" of what makes your Expression Engine site stay online during large traffic surges. After investing money in SEO and advertising for your site, making sure your hosting platform can handle the traffic is paramount. Proper load balancing, clustering, caching, and database optimizations will be covered so you know that your hosting provider is doing it "The Right Way."

Transcript

1. Scaling Expression Engine So, your site got popular? Brian James Hill
2. Hosting Challenges • Price / performance • Service standard • Application familiarity • Delineation between application and environment • Exception Handling – When things go awry • Perception vs. Reality – Buzzwords...
3. Developer / Hosting Relationship • Communication is key! • Is it a server problem or software problem? • Client satisfaction: to the max! – Ultimately, this is most important.
4. EE Hosting Caveats • Security – Directory permissions (i.e. the 777 curse) • Historical reasons for 777 • “Naked” Apache/Nginx issuesis • SuPHP and ultimately PHP-FPM – You can just be yourself – Breaches are limited to that user and only that user. – IDS / mod_security • All popular web applications are targets • Auto-scanners (beware of /admin and /phpMyAdmin)
5. EE Hosting Caveats • Performance & The Growth Arc – Quick “fixes” for dynamic applications in-a-pinch (static caching) – Two server clustering solutions (web + DB server) – Load balanced clustering – LBC + dynamic caching (Varnish) • CDN as a general rule – At least CSS / JS
6. Truthdig.org – An EE Case Study • An 8 year journey • Began hosting with us in 2004 • Started on a 2 node cluster • Frequent traffic bursts • Moved to a load balanced cluster • Left us :( • Came back!
7. Performance Issue #1 • Problem – Occasional traffic spikes – Google News – Reddit/Digg etc. • Solution – Static page caching – PRO • Simple, instant, effective – CON • Reactionary • Lag between detection and implementation
8. Performance Issue #2 • Problem – Consistent high traffic – More consistently on Google News and Social sites – Ambient traffic exceeds shared or dedicated hosting • Solution – Two server cluster (web server + DB) – PRO • Still relatively simple • Raises bar for general traffic patterns • Still requires static caching on large traffic events – CON • Still not able to instantly handle large spikes in traffic
9. Performance Issue #3 • Problem – Ever increasing high traffic demand – Regular on Google News and Social sites – Regular result on pertinent Google searches – Ambient traffic exceeds even dedicated web and DB servers • Solution – Load balanced cluster (web * 3 + DB + FS + LB) – PRO • Horizontally and predictably scalable • Good ambient excess capacity • May still require static caching on large traffic events – CON • More complex and harder to implement and maintain • Better able to handle spikes but still not perfect • Other problems are introduced (EE flock issue – irritating but worked through)
10. FLOCK flock($fp, LOCK_SH); // Read the first line (left a small buffer - just in case) $timestamp = trim(fgets($fp, 30)); …... if ((strlen($timestamp) != 10) OR ($timestamp !== ((string)(int) $timestamp))) // Integer check flock($fp, LOCK_UN);
11. Performance Issue #4 • Problem – THOSE MASSIVE SPIKES!! – Single source for massively popular story • OMG – LINDSAY LOHAN GOT DRUNK WHILE WEARING HER “30 DAYS SOBER” PENDANT!!! – Spike traffic/load exceeds reasonable excess capacity
12. Performance Issue #4 (cont.) • Solution – Dynamic caching (i.e. Varnish) – PRO • Memory based storage system for incredible speed • Very low cost – great price / performance. • Relatively easy to implement and generally plays nice – CON • Induces some management headaches with publishing • Without ESI the TTL is the ultimate gate-keeper for page refreshes
13. Varnish • Reverse caching proxy • Can live in RAM or on disk • VERY FAST!!!! • VCL
14. Varnish concurrency 20 no varnish Transactions: 2060 hits Availability: 100.00 % Elapsed time: 95.12 secs Data transferred: 36.64 MB Response time: 0.36 secs Transaction rate: 21.66 trans/sec Throughput: 0.39 MB/sec Concurrency: 7.71 Successful transactions: 2060 Failed transactions: 0 Longest transaction: 0.51 Shortest transaction: 0.29 concurrency 20 varnish Transactions: 2060 hits Availability: 100.00 % Elapsed time: 61.63 secs Data transferred: 36.64 MB Response time: 0.01 secs Transaction rate: 33.43 trans/sec Throughput: 0.59 MB/sec Concurrency: 0.22 Successful transactions: 2060 Failed transactions: 0 Longest transaction: 0.03 Shortest transaction: 0.00
15. Varnish concurrency 60 no varnish Transactions: 6180 hits Availability: 100.00 % Elapsed time: 124.17 secs Data transferred: 109.92 MB Response time: 0.63 secs Transaction rate: 49.77 trans/sec Throughput: 0.89 MB/sec Concurrency: 31.42 Successful transactions: 6180 Failed transactions: 0 Longest transaction: 1.51 Shortest transaction: 0.23 60 concurrent varnish Transactions: 6180 hits Availability: 100.00 % Elapsed time: 62.83 secs Data transferred: 109.92 MB Response time: 0.01 secs Transaction rate: 98.36 trans/sec Throughput: 1.75 MB/sec Concurrency: 1.38 Successful transactions: 6180 Failed transactions: 0 Longest transaction: 5.01 Shortest transaction: 0.00
16. Varnish concurrency 200 no varnish Transactions: 20600 hits Availability: 100.00 % Elapsed time: 401.65 secs Data transferred: 366.40 MB Response time: 3.34 secs Transaction rate: 51.29 trans/sec Throughput: 0.91 MB/sec Concurrency: 171.44 Successful transactions: 20600 Failed transactions: 0 Longest transaction: 8.47 Shortest transaction: 0.22 200 concurrent varnish Transactions: 20600 hits Availability: 100.00 % Elapsed time: 70.22 secs Data transferred: 366.40 MB Response time: 0.02 secs Transaction rate: 293.36 trans/sec Throughput: 5.22 MB/sec Concurrency: 6.43 Successful transactions: 20600 Failed transactions: 0 Longest transaction: 5.04 Shortest transaction: 0.00
17. What Can Be Expected? • Saturation of a 100Mbps NIC!!
18. MySQL Tuning • Optimizing your my.cnf for your hardware • Enable query cache in my.cnf – Disable query cache in EE • tmp_table_size / max_heap_size • InnoDB vs MyISAM?
19. InnoDB vs. MyISAM • InnoDB – Faster writes – Transactions – No full text search index – Row level locking instead of table locking – Better crash recovery than MyISAM
20. InnoDB vs. MyISAM • MyISAM – Default for Expression Engine – Less resource intensive – Full text search indexing – Simpler data model • Don't have to worry about relational constraints. – Not as resilient as InnoDB
21. Other Performance Notes • Good programming MATTERS! • Good template design MATTERS! – Watch your SQL queries! • Hardware is not always the cure • Plan ahead whenever possible – Lindsay Lohan didn't let us know :( – Clients with looming “Today Show” spotlights normally do :)
22. General Hosting Notes • ASK ASK ASK! – If you don't know for sure it's included then it probably isn't • The phone is your friend! – Secret shop sales, support, etc. • Security is important but backups are paramount! • Do you have a better way? Make sure your host does!
23. Questions?