Symfony High Availability in the Cloud

Slides of my talk at the Symfony Usergroup Zurich

Transcript

1. SYMFONY HIGH AVAILABILITY IN THE CLOUD Frank Neff - ibrows.ch
2. THE PRINCIPLES What is HA and how does it work?
3. “WE ARE NOW IN THE CLOUD”
5. FSEEKING YEAH! • Single server instance • Slow I/O • Shared Memory • Virtualised CPU • No idea where our service is hosted
7. THIS IS HA • Redundant Setup • Load is balanced • No resource conflicts • Every Service / Component can break
9. FOR THE WIN!
10. COMMON ARCHITECTURES • Micro instances • One service per host • Multi-Site distribution
11. MICRO INSTANCES Architecture • Database / Webserver / Cache 
 on same host • Lowers network traffic • Bigger impact in case of failure • Fewer machines needed / Cheap • Risk of resource conflicts • Harder to debug / identify bottlenecks
13. ONE SERVICE PER HOST Architecture • Database / Webserver / Cache 
 on own hosts • High internal network traffic • More instances needed (expensive) • Guaranteed resources • Easy to identify bottlenecks • Better performance in case of failure
15. MULTI-SITE DISTRIBUTION Architecture • Service groups are distributed over
 dedicated physical hardware • Multi-site application hosting 
 in different data centres • Synchonization traffic between 
 data centers (Internet / fibre channel) • Disaster safe • Complex and very expensive • “SysOp Porn”
18. “EVEN IF IT BLOWS UP YOUR WHOLE DATACENTER…”
19. “OKAY, WE ARE NOT GOOGLE…”
20. THE APPLICATION How to develop HA software?
21. GENERAL CODING • Always think about multiple, simultaneous instances while coding • Every application-state must be shared somehow with all other nodes • If one node fails, it must not block others • Avoid long running processes
22. COMMON PITFALLS • Session management • (Cron-) Jobs • SSL Termination • Replication lag • Asset delivery
23. “OUR USERS GET A NEW SESSION ON EVERY REQUEST”
24. SESSION MANAGEMENT • PHP cannot share sessions across servers (IMHO) • Sessions for shops, admin-backends, etc. must be handled in distributed systems • Persist sessions in Memcache / Redis • Use sticky sessions
25. “WE HAVE DATA-MESS BECAUSE JOBS RUN CONCURRENTLY ON DIFFERENT SERVERS”
26. (CRON-) JOBS • When jobs run in a distributed environment, they can disrupt each other • Data inconsistencies / Things get broken • Run jobs on dedicated job-server(s) • Use distributed locking / mutex
27. “SO WE HAVE TO ISSUE SSL CERTIFICATES FOR ALL OUR SERVERS?”
28. SSL TERMINATION • Regarding SSL, it should not make a difference on which server you are • Terminate SSL on your load balancer or caching reverse proxy • Varnish and HAProxy both support SSL termination
29. “WHY DOES USER A SEE DIFFERENT CONTENT THAN USER B?”
30. REPLICATION LAG • Can happen when you are using master/slave replication on your DB • One big problem of distributed apps, 
 even for Facebook (e.g.) • There is no perfect solution for dealing with it, but awareness • See CAP-Theorem
31. “70% OF MY SERVER IS BUSY DELIVERING ASSETS”
32. ASSET DELIVERY • Delivering CSS, JS and images can become a performance bottleneck • Avoid booting a Symfony stack for asset delivery • Cache assets (e.g. with Varnish) or use a CDN for asset delivery • Rename assets to invalidate caches
33. QUESTIONS / BREAK See you again for the practical part…
34. “SO, WHAT DO I NEED?”
35. • Time • Know-How • Money • More time
36. AN EXAMPLE SETUP …there are tons out there and no one is perfect
37. BASE BOX • Ubuntu 14 LTS • Provisioned on DigitalOcean • TODO: Box spec
38. DATABASE • Percona XTraDB Cluster (MySQL) • active/active HA database cluster • 3 nodes provisioned
39. WEBSERVER / PHP • nginx / PHP-FPM • Runs Symfony stack • 6 nodes provisioned
40. CACHE / LB • Varnish 4 • Used for asset delivery and HTTP caching • Acts as load balancer for web servers • Only instance which is publicly accessible
42. JOB SERVER Don’t do heavy lifting on the webserver
43. JOB SERVER • Outsource heavy jobs to a separate server • Cron / Supervisord / Gearman • Runs PHP / Symfony but without a webserver • Should have own DB instance
45. JOB SERVER HA • Don’t run multiple job servers • Run second / third as active standby • Distribute jobs over different active servers • Utilize distributed locking (e.g. Zookeeper)
46. MONITORING Because anything could happen anytime
47. “HA SETUPS ARE HARDER TO MONITOR, BECAUSE MULTIPLE SERVERS PLAYING TOGETHER”
48. MONITORING • Use a centralized monitoring solution • Keep watching your app instances (e.g. while deploying) • Identify performance issues / bottlenecks using visualization • Datadog, Nagios, Icinga, NewRelic, etc…
49. HANDS ON It’s like playing games, because you have to do it to become skilled
50. FORK IT • github.com/ibrows/symfony-ha-example • github.com/ibrows/symfony-ha-example-app
51. THANKS Symfony Routing Under the Hood (Liip / 21.10.) Show me Your Bundles #2 (iBROWS /18.11) Upcoming Meetups @frank_neff / frankneff.ch