Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Symfony High Availability in the Cloud

52034cfb48ca73e13e130f2711ec41f0?s=47 Frank Neff
September 16, 2015

Symfony High Availability in the Cloud

Slides of my talk at the Symfony Usergroup Zurich

52034cfb48ca73e13e130f2711ec41f0?s=128

Frank Neff

September 16, 2015
Tweet

Transcript

  1. SYMFONY HIGH AVAILABILITY IN THE CLOUD Frank Neff - ibrows.ch

  2. THE PRINCIPLES What is HA and how does it work?

  3. “WE ARE NOW IN THE CLOUD”

  4. None
  5. FSEEKING YEAH! • Single server instance • Slow I/O •

    Shared Memory • Virtualised CPU • No idea where our service is hosted
  6. None
  7. THIS IS HA • Redundant Setup • Load is balanced

    • No resource conflicts • Every Service / Component can break
  8. None
  9. FOR THE WIN!

  10. COMMON ARCHITECTURES • Micro instances • One service per host

    • Multi-Site distribution
  11. MICRO INSTANCES Architecture • Database / Webserver / Cache 


    on same host • Lowers network traffic • Bigger impact in case of failure • Fewer machines needed / Cheap • Risk of resource conflicts • Harder to debug / identify bottlenecks
  12. None
  13. ONE SERVICE PER HOST Architecture • Database / Webserver /

    Cache 
 on own hosts • High internal network traffic • More instances needed (expensive) • Guaranteed resources • Easy to identify bottlenecks • Better performance in case of failure
  14. None
  15. MULTI-SITE DISTRIBUTION Architecture • Service groups are distributed over
 dedicated

    physical hardware • Multi-site application hosting 
 in different data centres • Synchonization traffic between 
 data centers (Internet / fibre channel) • Disaster safe • Complex and very expensive • “SysOp Porn”
  16. None
  17. None
  18. “EVEN IF IT BLOWS UP YOUR WHOLE DATACENTER…”

  19. “OKAY, WE ARE NOT GOOGLE…”

  20. THE APPLICATION How to develop HA software?

  21. GENERAL CODING • Always think about multiple, simultaneous instances while

    coding • Every application-state must be shared somehow with all other nodes • If one node fails, it must not block others • Avoid long running processes
  22. COMMON PITFALLS • Session management • (Cron-) Jobs • SSL

    Termination • Replication lag • Asset delivery
  23. “OUR USERS GET A NEW SESSION ON EVERY REQUEST”

  24. SESSION MANAGEMENT • PHP cannot share sessions across servers (IMHO)

    • Sessions for shops, admin-backends, etc. must be handled in distributed systems • Persist sessions in Memcache / Redis • Use sticky sessions
  25. “WE HAVE DATA-MESS BECAUSE JOBS RUN CONCURRENTLY ON DIFFERENT SERVERS”

  26. (CRON-) JOBS • When jobs run in a distributed environment,

    they can disrupt each other • Data inconsistencies / Things get broken • Run jobs on dedicated job-server(s) • Use distributed locking / mutex
  27. “SO WE HAVE TO ISSUE SSL CERTIFICATES FOR ALL OUR

    SERVERS?”
  28. SSL TERMINATION • Regarding SSL, it should not make a

    difference on which server you are • Terminate SSL on your load balancer or caching reverse proxy • Varnish and HAProxy both support SSL termination
  29. “WHY DOES USER A SEE DIFFERENT CONTENT THAN USER B?”

  30. REPLICATION LAG • Can happen when you are using master/slave

    replication on your DB • One big problem of distributed apps, 
 even for Facebook (e.g.) • There is no perfect solution for dealing with it, but awareness • See CAP-Theorem
  31. “70% OF MY SERVER IS BUSY DELIVERING ASSETS”

  32. ASSET DELIVERY • Delivering CSS, JS and images can become

    a performance bottleneck • Avoid booting a Symfony stack for asset delivery • Cache assets (e.g. with Varnish) or use a CDN for asset delivery • Rename assets to invalidate caches
  33. QUESTIONS / BREAK See you again for the practical part…

  34. “SO, WHAT DO I NEED?”

  35. • Time • Know-How • Money • More time

  36. AN EXAMPLE SETUP …there are tons out there and no

    one is perfect
  37. BASE BOX • Ubuntu 14 LTS • Provisioned on DigitalOcean

    • TODO: Box spec
  38. DATABASE • Percona XTraDB Cluster (MySQL) • active/active HA database

    cluster • 3 nodes provisioned
  39. WEBSERVER / PHP • nginx / PHP-FPM • Runs Symfony

    stack • 6 nodes provisioned
  40. CACHE / LB • Varnish 4 • Used for asset

    delivery and HTTP caching • Acts as load balancer for web servers • Only instance which is publicly accessible
  41. None
  42. JOB SERVER Don’t do heavy lifting on the webserver

  43. JOB SERVER • Outsource heavy jobs to a separate server

    • Cron / Supervisord / Gearman • Runs PHP / Symfony but without a webserver • Should have own DB instance
  44. None
  45. JOB SERVER HA • Don’t run multiple job servers •

    Run second / third as active standby • Distribute jobs over different active servers • Utilize distributed locking (e.g. Zookeeper)
  46. MONITORING Because anything could happen anytime

  47. “HA SETUPS ARE HARDER TO MONITOR, BECAUSE MULTIPLE SERVERS PLAYING

    TOGETHER”
  48. MONITORING • Use a centralized monitoring solution • Keep watching

    your app instances (e.g. while deploying) • Identify performance issues / bottlenecks using visualization • Datadog, Nagios, Icinga, NewRelic, etc…
  49. HANDS ON It’s like playing games, because you have to

    do it to become skilled
  50. FORK IT • github.com/ibrows/symfony-ha-example • github.com/ibrows/symfony-ha-example-app

  51. THANKS Symfony Routing Under the Hood (Liip / 21.10.) Show

    me Your Bundles #2 (iBROWS /18.11) Upcoming Meetups @frank_neff / frankneff.ch