Deploying Distributed Systems - London Web

Transcript

1. deploying distributed systems
2. Hi, I’m Matt @MATTHEATH
5. HOW DO WE DEPLOY?
6. PICK VERSION lets try this one Genuine network graph
7. UPDATE CODE
8. DRINK BEER (and wait for alerts...)
9. WHAT ARE WE TRYING TO DO?
10. HOW DO WE DO THIS? FTP SCP EDIT LIVE SERVER
11. NON ATOMIC!
12. SERVER CODE
13. CODE
14. SYMLINK CODE
15. 1362393420 SYMLINK
16. 1362393420 1362499200 SYMLINK
17. 1362393420 1362499200 SYMLINK
18. 1362393420 1362499200 1362610174 SYMLINK
19. 1362393420 1362499200 1362610174 SYMLINK
20. WHAT ELSE? COMPRESS & MINIFY RESTART SERVERS ASSET VERSIONING DATABASE MIGRATIONS LOAD NEW CONFIGS LINT CHECKS CLEAR CACHES SPAWN WORKERS WARM CACHES
21. ENTER CAPISTRANO!
22. “Capistrano is a utility and framework for executing commands in parallel on multiple remote machines, via SSH”
23. $ cap deploy
24. • Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • ‘Restart’ web servers • Post deploy tasks • Clean up old releases
25. GREAT SUCCESS!
26. $ capify .
27. $ capify . [add] writing './Capfile' [add] making directory './config' [add] writing './config/deploy.rb' [done] capified!
31. MULTI-STAGE cap dev deploy cap staging deploy cap production deploy
33. cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy cap ...staging deploy ... MULTI-CONFIGURATION?
35. SOA = COMPLEX
36. INTERNAL TOOLS & APIS HAILOCAB.COM DRIVER APIS CUSTOMER SERVICES CREDIT SERVICES COMMUNICATION SERVICES AUTHENTICATION SERVICES CONFIGURATION SERVICES CRM SERVICES PAYMENT SERVICES EVENTS SERVICES STATS SERVICES ALLOCATION SERVICES BACKGROUND QUEUES CUSTOMER APIS MESSAGING SERVICES APPLICATIONS
37. 3 geographic regions
38. 9 availability zones 3 geographic regions
39. problem?
40. ≃1000 DEPLOY TARGETS
41. VASTLY INCREASED COMPLEXITY
42. EASILY MAKE MISTAKES
43. USE ROBOTS FOR GOOD
44. UNTIL THEY BECOME SELF AWARE...
45. SO...
46. (the deployer)
47. Allow anyone...
48. to deploy from anywhere
49. to everywhere
50. One application to deploy all applications Deploy in parallel to n servers in distributed geographic regions with the correct configuration Dynamic server lists Add additional functionality in one central place Play a shipping horn sound when deploying
51. DEMO!
52. SUCCESSFUL BUILDS AUTO DEPLOY
53. DID THE TESTS PASS? IS THE BRANCH UP TO DATE? SHIP IT!
54. OMG YOU OVERWROTE MY BUILD
55. LOCK ENVIRONMENTS
56. LOCK ENVIRONMENTS CUSTOM STAGING ENDPOINTS
57. WITH ROBOTS AUTOMATE
58. WHAT COULD GO WRONG?
59. WELL...
60. MONITOR HEALTH AFTER DEPLOY
61. ROLLBACK ON FAILURE AUTOMATICALLY
62. IRC / CHAT NOTIFICATION [03:14:58] <CONAN> mobile-api/intlrelease (6957bb87) is being deployed to staging by Matt Heath [03:15:25] <CONAN> mobile-api/intlrelease successfully deployed to staging [12:29:20] <CONAN> elasticride/master (390eae4c) is being deployed to staging by Saul Howard
63. TRACK DEPLOYS GRAPHITE+STATSD=—
64. OH SHIT
65. I DON’T USE CAPISTRANO YOUR ARGUMENT IS INVALID
66. ALTERNATIVES?
67. AUTO SCALING
68. DEB/RPM PACKAGES PUPPET / CHEF AMIs
70. fin.
71. Image credits: Conan ‘8-bit Marvels’ - Paulo Capdeville Network diagram - Elijah Meeks, dhs.stanford.edu Creative Commons images from sxc.hu thanks to Miguel Saavedra, @rbval, G. Marcus Still interested? Caphub - Multistage capistrano: http://bit.ly/Tkkn4W Deploying at Github: http://git.io/FlaFhA Asgard - Web based deployment from Netflix: http://nflx.it/NsJVtD