Deploying Distributed Systems

As Hailo transitions to a globally distributed SOA model, we needed to improve how we deployed software. We currently do this with an application called Conan, based on a modified version of Capistrano.

Talk given at PHP London, March 2013

1. deploying distributed systems
2. Hi, I’m Matt @MATTHEATH
3. promo code: matt (for a £5 discount ;)
9. HOW DO WE DEPLOY?
10. PICK VERSION lets try this one Genuine network graph
11. UPDATE CODE
12. UPDATE CODE
13. UPDATE CODE
14. UPDATE CODE
15. UPDATE CODE
16. DRINK BEER (and wait for alerts...)
17. WHAT ARE WE TRYING TO DO?
18. HOW DO WE DO THIS? FTP SCP HOT EDIT SERVER MAGNETIZED NEEDLE & STEADY HANDS?!
19. NON ATOMIC!
20. NON ATOMIC!
22. SERVER CODE
23. CODE
24. 1362393420 SYMLINK
25. 1362393420 1362499200 SYMLINK
26. 1362393420 1362499200 SYMLINK
27. 1362393420 1362499200 1362610174 SYMLINK
28. 1362393420 1362499200 1362610174 SYMLINK
29. WHAT ELSE? COMPRESS & MINIFY RESTART SERVERS ASSET VERSIONING DATABASE MIGRATIONS LOAD NEW CONFIGS LINT CHECKS CLEAR CACHES SPAWN WORKERS WARM CACHES
30. ENTER CAPISTRANO!
31. “Capistrano is a utility and framework for executing commands in parallel on multiple remote machines, via SSH”
32. $ cap deploy
33. • Identify version to deploy
34. • Identify version to deploy • Update remotely cached git repository
35. • Identify version to deploy • Update remotely cached git repository • Checkout specific version
36. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder
37. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks
38. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code
39. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • Restart web servers
40. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • Restart web servers • Post deploy tasks
41. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • Restart web servers • Post deploy tasks • Clean up old releases
42. GREAT SUCCESS!
43. $ capify .
44. $ capify . [add] writing './Capfile' [add] making directory './config' [add] writing './config/deploy.rb' [done] capified!
48. MULTI-STAGE
49. MULTI-STAGE cap dev deploy
50. MULTI-STAGE cap dev deploy cap staging deploy
51. MULTI-STAGE cap dev deploy cap staging deploy cap production deploy
53. MULTI-CONFIGURATION?
54. cap londonstaging deploy MULTI-CONFIGURATION?
55. cap londonstaging deploy cap dublinstaging deploy MULTI-CONFIGURATION?
56. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy MULTI-CONFIGURATION?
57. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy MULTI-CONFIGURATION?
58. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy MULTI-CONFIGURATION?
59. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy MULTI-CONFIGURATION?
60. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy MULTI-CONFIGURATION?
61. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy cap ...staging deploy MULTI-CONFIGURATION?
62. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy cap ...staging deploy ... MULTI-CONFIGURATION?
64. SOA = COMPLEX
65. INTERNAL TOOLS & APIS HAILOCAB.COM DRIVER APIS CUSTOMER SERVICES CREDIT SERVICES COMMUNICATION SERVICES AUTHENTICATION SERVICES CONFIGURATION SERVICES CRM SERVICES PAYMENT SERVICES EVENTS SERVICES STATS SERVICES ALLOCATION SERVICES BACKGROUND QUEUES CUSTOMER APIS MESSAGING SERVICES APPLICATIONS
66. 3 geographic regions
67. 9 availability zones 3 geographic regions
68. problem?
69. OVER 9000 DEPLOY TARGETS OVER 9000 DEPLOY TARGETS lol/jk lol/jk
70. EASILY MAKE MISTAKES
71. SOA, SCALING & RESILIENCE VASTLY INCREASE COMPLEXITY
72. USE ROBOTS FOR GOOD
73. UNTIL THEY BECOME SELF AWARE...
74. SO...
75. the deployer
76. Allow anyone...
77. to deploy from anywhere
78. to everywhere
79. One application to deploy all applications Deploy in parallel to n servers in distributed geographic regions with the correct configuration Dynamic server lists Add additional functionality in one central place Play a shipping horn sound when deploying
80. DEMO!
81. PHP, APC, NO STAT, REAL PATH CACHING, SYMLINKS & YOU SRSLY WTF PHP?!?!
82. "echo 'APC Cache Blitz in '`let n= $(php -i | grep realpath_cache_ttl | sed -n 's/.* \([0-9]*\).*/\1/p') +5;echo $n`' seconds... ('`date`')'; sleep `let n=$(php -i | grep realpath_cache_ttl | sed -n 's/.* \([0-9]*\).*/\1/p')+5;echo $n`; echo 'Blitzing... ('`date`')'; blitz"
83. SUCCESSFUL BUILDS AUTO DEPLOY
84. DID THE TESTS PASS? IS THE BRANCH UP TO DATE? SHIP IT!
85. OMG YOU OVERWROTE MY BUILD
86. LOCK ENVIRONMENTS
87. LOCK ENVIRONMENTS CUSTOM STAGING ENDPOINTS
88. WITH ROBOTS AUTOMATE
89. WHAT COULD GO WRONG?
90. WELL...
91. MONITOR HEALTH AFTER DEPLOY
92. ROLLBACK ON FAILURE AUTOMATICALLY
93. KEEP HUMANS UP TO DATE
94. IRC / CHAT [03:14:58] <CONAN> mobile-api/intlrelease (6957bb87) is being deployed to staging by Matt Heath [03:15:25] <CONAN> mobile-api/intlrelease successfully deployed to staging [12:29:20] <CONAN> elasticride/master (390eae4c) is being deployed to staging by Saul Howard
95. TRACK DEPLOYS GRAPHITE+STATSD=—
96. OH SHIT
97. I DON’T USE CAPISTRANO: YOUR ARGUMENT IS INVALID
98. ALTERNATIVES?
99. .deb PACKAGES PUPPET / CHEF AMIs BITTORRENT
100. fin.
101. Image credits: Conan ‘8-bit Marvels’ - Paulo Capdeville Network diagram - Elijah Meeks, dhs.stanford.edu Creative Commons images from sxc.hu thanks to Miguel Saavedra, @rbval, G. Marcus Still interested? Caphub - Multistage capistrano: http://bit.ly/Tkkn4W Deploying at Github: http://git.io/FlaFhA Asgard - Web based deployment from Netflix: http://nflx.it/NsJVtD