PyConZA 2015: "The road to continuous deploys" by Stefano Rivera

How do we get work from a developer's laptop to a production server? How do we do this repeatedly? How do we avoid releasing bad patches? How do we get changes out as fast as possible? And what happens to requests during a deploy? If you want continuous deployment, you have to solve these problems.

How do you handle configuration, so that the same code can run in multiple environment? From a developers laptop, to QA environments, to production. What about the quirks of each machine in production?

This talk will look at the path Yola took, with Jenkins, yoconfigurator, yodeploy. It's by no means a complete solution, but it's good enough that I rarely find myself working on it, these days.

We will cover the pros and cons of each path one can take. The whole platform architecture is really affected by this. We can talk about: Bare hardware, virtualization, containerization, build systems, linux distributions packages, language package ecosystems, configuration management systems, notifications, secret distribution, and much more.

Transcript

1. The road to continuous deployment Stefano Rivera Yola 2 October 2015 Pycon ZA Stefano Rivera The road to continuous deployment
2. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
3. Production server architectures What world do our apps live in? Stefano Rivera The road to continuous deployment
4. Production server architectures Typical SASS app: Redundant front-ends Central DB Stefano Rivera The road to continuous deployment
5. Production server architectures Microservices: Redundant front-ends Redundant services Central DBs Stefano Rivera The road to continuous deployment
6. Production server architectures Async: Job queues Workers Central Queues Stefano Rivera The road to continuous deployment
7. Production server architectures Client Load Balancing: haproxy / nginx / hardware Single IP / hostname for each service / DB Stefano Rivera The road to continuous deployment
8. Production server architectures Internal Service Load Balancing: the same, or intelligent fancy clients service discovery: zookeeper / something Stefano Rivera The road to continuous deployment
9. Production server architectures Development environments: Production-like environments (QA) Developer’s laptops (local dev, maybe one app only) Integration environments (all on one box, no LBs) Stefano Rivera The road to continuous deployment
10. Production server architectures Development workflows: VCS (git / whatever) Build system (?) Deployment tool ... Profit Stefano Rivera The road to continuous deployment
11. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
12. Configuration What world do our apps live in? Stefano Rivera The road to continuous deployment
13. Configuration Production: Probably most complex environment Service LBs Background workers Stefano Rivera The road to continuous deployment
14. Configuration QA: Like production, but different No redundancy? Stefano Rivera The road to continuous deployment
15. Configuration Integration environments: Everything on one machine Name-based vhosting? Save the RAM Less workers, less background workers Stefano Rivera The road to continuous deployment
16. Configuration Local dev: Against prod / QA / int-env? sqlite? Simple bootstrapping Dependencies... Stefano Rivera The road to continuous deployment
17. Configuration Things apps need to know: Where are services? Where are DBs? Secrets Other parameters Stefano Rivera The road to continuous deployment
18. Configuration Where should these live: Centrally In each app Stefano Rivera The road to continuous deployment
19. Configuration Central sucks: Adding a setting means commits to 2 repositories What uses this setting over here? Stefano Rivera The road to continuous deployment
20. Configuration In each app sucks: This setting is the same for every app in production I want to put this app in a public github repo Stefano Rivera The road to continuous deployment
21. Configuration Compromise: Both Which means probably no configuration management system involvement per-machine settings Stefano Rivera The road to continuous deployment
22. Configuration yoconfigurator smush C: hostname C: common C: common-ENV C: common-ENV-CLUSTER C: common-overrides A: APP-default A: APP-ENV (and cluster) C: APP C: APP-ENV (and cluster) C: APP-overrides Stefano Rivera The road to continuous deployment
23. Configuration Exposing config to apps: settings.py - lots of if ladders settings.py.ENVIRONMENT - lots of duplication, no shared config environment variables - local dev hurts, forks hurt config files - actually pretty good other things Stefano Rivera The road to continuous deployment
24. Configuration Tooling: Config file generator Generate against any environment Stefano Rivera The road to continuous deployment
25. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
26. Deployment tools Get the code out Stefano Rivera The road to continuous deployment
27. Deployment tools simple git pull Actually fairly good, these days What about dependencies? Configuration? Stefano Rivera The road to continuous deployment
28. Deployment tools git pull + script: What happens while this is running? What happens if it fails? Stefano Rivera The road to continuous deployment
29. Deployment tools Typical script: install OS packages manage virtualenv build non-python stuff build config migrate DBs (here be dragons) bounce workers Stefano Rivera The road to continuous deployment
30. Deployment tools Unexpected problems: PyPI is down PyPI changes Non-deterministic thing is built (esp, minification & filenames) Stefano Rivera The road to continuous deployment
31. Deployment tools Reproducibility: Goal: Same generated artifacts on each production server Stefano Rivera The road to continuous deployment
32. Deployment tools Build system: Build virtulenvs Build non-python Build config (?) Bundle Stefano Rivera The road to continuous deployment
33. Deployment tools Other benefits: Roll-backs Centralized build-effort Build-time test suites Stefano Rivera The road to continuous deployment
34. Deployment tools Downsides: Platform dependencies ABI test suite length Build scripts Stefano Rivera The road to continuous deployment
35. Deployment tools Output: git repo distro package VM other: tarball in some kind of archive Stefano Rivera The road to continuous deployment
36. Deployment tools Deployment: extract latest build generate config? run script Stefano Rivera The road to continuous deployment
37. Deployment tools Next goal: factor out scripts build scripts deploy scripts Stefano Rivera The road to continuous deployment
38. Deployment tools How to execute: ssh do stuff? configuration management system? deploy system on box: ssh run command http request Stefano Rivera The road to continuous deployment
39. Deployment tools Deployment automation: commit hook triggers build system build triggers tests build triggers deploy (?) deploy triggers validation tests? deploy triggers chat notification Stefano Rivera The road to continuous deployment
40. Deployment tools Releases: git branches? promoted builds? Stefano Rivera The road to continuous deployment
41. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
42. Configuration management Where does it fit in Stefano Rivera The road to continuous deployment
43. Configuration management Does deploys: Simple Slow (it does everything else, too) Stefano Rivera The road to continuous deployment
44. Configuration management Basic config: Standard OS packages and config Stefano Rivera The road to continuous deployment
45. Configuration management Per app: external dependencies (rarely change) service discovery initial deploy (bootstrapping) vhosts (?) Stefano Rivera The road to continuous deployment
46. Configuration management Other stuff: 3rd party apps that aren’t deployed DBs Caches Brokers Stefano Rivera The road to continuous deployment
47. Configuration management Demo envhub Stefano Rivera The road to continuous deployment
48. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
49. Commit to production latency Continuous deployment means we’re going to break things. We need to be able to fix them, quickly Stefano Rivera The road to continuous deployment
50. Commit to production latency Obvious optimizations: SSDs Cache / mirror PyPI (and npm) Cache eggs / wheels Cache virtualenvs (and node modules) pre-minify images in git / cache minification Get slow tests out of critical path Don’t build VMs Stefano Rivera The road to continuous deployment
51. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
52. yodeploy Our deployment tool. Stefano Rivera The road to continuous deployment
53. yodeploy Build time: Still shell scripts (But fairly short boilerplate) More and more gulp / grunt Spit out tarball, run spade upload Stefano Rivera The road to continuous deployment
54. yodeploy Deploy Workflow: receive deploy command extract tarball to /srv/APP/versions/VERSION/ prepare hook generate things that can’t be done at build time swing /srv/APP/live symlink deployed hook bounce things Stefano Rivera The road to continuous deployment
55. yodeploy Hooks: Knows about yoconfigurator Contains simple templating engine Knows about virtualenvs Must still be taught about npm But, actually, we keep npm to build-time Knows how to drive a Django app Knows how to drive upstart Knows how to drive tomcat Customizable in Python Stefano Rivera The road to continuous deployment
56. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
57. zero downtime What happens during the deploy Stefano Rivera The road to continuous deployment
58. zero downtime ideal: web server finishes old requests using old code uses new code for new requests Stefano Rivera The road to continuous deployment
59. zero downtime erlang: Well, duh (But we don’t actually deploy any with yodeploy) Stefano Rivera The road to continuous deployment
60. zero downtime openresty: :+1: Stefano Rivera The road to continuous deployment
61. zero downtime static HTML Apache: :+1: But maybe keep old assets around for a while / forever? Content-based hashing ftw Stefano Rivera The road to continuous deployment
62. zero downtime Python WSGI: mod wsgi mostly handles this some 500s, can be mitigated at LB or client-side retries other WSGI containers: Not so much. Alternate between a pair? Stefano Rivera The road to continuous deployment
63. zero downtime Java: tomcat7 parallel deployments PermGen Space Stefano Rivera The road to continuous deployment
64. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
65. What comes next What’s missing? Stefano Rivera The road to continuous deployment
66. What comes next Better build time: Too much class hierarchy Not enough implicit config through convention Stefano Rivera The road to continuous deployment
67. What comes next Better orchestration: Deploy button Integrated graphs Stefano Rivera The road to continuous deployment
68. What comes next Modernize: Wheels Python 3 Stefano Rivera The road to continuous deployment
69. Outline 1 Production server architectures 2 Configuration 3 Deployment tools 4 Configuration management 5 Commit to production latency 6 yodeploy 7 zero downtime 8 What comes next 9 Other topics Stefano Rivera The road to continuous deployment
70. Other topics VMs & Containers Stefano Rivera The road to continuous deployment
71. Other topics Log collection Stefano Rivera The road to continuous deployment
72. Other topics Crash collection Stefano Rivera The road to continuous deployment
73. Other topics DB migrations Stefano Rivera The road to continuous deployment
74. Other topics envhub Stefano Rivera The road to continuous deployment
75. Other topics github testing hooks Stefano Rivera The road to continuous deployment
76. Thanks! Questions? Stefano Rivera stefanor@debian.org http://tumbleweed.org.za/ Slides: Available at https://github.com/stefanor/talks Built using L ATEX Beamer Copyright c 2013–2014, Stefano Rivera License CC BY-SA 3.0 — Creative Commons Attribution-ShareAlike 3.0 Stefano Rivera The road to continuous deployment