Cyan Deployments @ Betterment w/ Ansible

Blue/Green deployments have been an important, if rarely implemented, technique in the Continuous Delivery playbook for years. Their aim is simple: provision, deploy, test — and optionally roll-back — your application before it's served to the public. Betterment's deployment architecture takes a similar, but more straightforward approach, accomplishing the important goals sought out by Blue/Green practitioners. Dubbed 'Cyan' (a mixture of Blue/Green), Betterment uses Ansible to provision new instances, push the latest artifacts to them, and ensure that they're healthy before marking them ready for production. All this ensures fast, stable, zero-downtime rollout with minimal human interaction. We'll discuss Betterment's philosophical approach to shipping new code and then dive into the nitty-gritty Ansible that powers the whole thing.

Transcript

1. Cyansible Blending Blue/Green Deployments @Betterment
3. Techcrunch Disrupt May 2010 90,000 customers, more every minute. Fastest growing automated investing service $2B+ Who are we?
6. betterment : investing :: devops : engineering
7. it’s 2012. let’s ship some code.
9. Betterment@2012
10. A Better Migration: From Snowflakes to Stormtroopers Wednesday, July 22, 6:30p - 8:00p @ AWS Pop-up Loft | NYC Shameless Plug:
11. it’s 2015. let’s ship new code… without interrupting production
13. • Predictable • Repeatable • Minimal Human Interaction • Zero User Interruption • Contained Failure Dream Delivery
14. Blue/Green Deployments http://martinfowler.com/bliki/BlueGreenDeployment.html
15. DNS ELB
16. Pre-flight Checklist ❏ Stateless Servers ❏ Server Health Check ❏ Duplicate Full Stack including RDS Replica
17. Wait. Two Databases? “There's still the issue of dealing with missed transactions while the green environment was live, but depending on your design you may be able to... • feed transactions to both environments in such a way as to keep the blue environment as a backup when the green is live. Or you may be able to... • put the application in read-only mode before cut-over, run it for a while in read-only mode, and then switch it to read-write mode.” http://martinfowler.com/bliki/BlueGreenDeployment.html
19. “Code always works on Schema +1 ” (A.K.A. old code works on new schema)
21. Publish, Migrate, Deploy
22. Jenkins’ Job 1. Build 2. Test 3. Package 4. Publish 5. Run Migrations 6. Invoke Ansible 7. Cull Zombies
23. Ansible’s Job 1. Check for S3 deliverables 2. Spin up new EC2 Instance(s) 3. Apply role(s) to instance(s) 4. Find instance(s) in ELB 5. Add new instance(s) to ELB & tag ◦ status: in-use 6. Remove & tag instances ◦ status: zombie
26. Bootstrapping Ansible ./exec/ directory contains Jenkins entry points Ansible code lives in the repo it provisions. Publish it like an app.
28. ./exec/brochure-deploy.sh
30. brochure-deploy.yml you are here. 00:00:10.888
31. provision-new-ec2-instances brochure-deploy >
32. identify-elb-AZs brochure-deploy > provision-new-ec2-instances >
33. ec2-deploy brochure-deploy > provision-new-ec2-instances >
34. AMI Hierarchy
35. brochure-deploy.yml you are here. 00:01:18.415
36. configure-instances brochure-deploy >
37. roles/deploy/brochure/tasks/main.yml brochure-deploy > configure-instances
38. brochure-deploy.yml you are here. 00:04:49.414
41. brochure-deploy.yml you are here. 00:04:50.188
42. find-instances-in-elb brochure-deploy >
43. brochure-deploy.yml you are here. 00:04:50.911
44. add-instances-to-elb brochure-deploy >
45. brochure-deploy.yml you are here. 00:05:06.295
47. decommission-instances brochure-deploy >
48. brochure-deploy.yml you are here. 00:05:21.226
54. EC2 INSTANCE
55. • Predictable • Repeatable • Minimal Human Interaction • Zero User Interruption • Contained Failure Dream Delivery Achieved
56. The Future • Long Running Instances + Docker ◦ Huge speed improvement • Post Monolith, Abandon Jenkins? ◦ Travis CI for Build/Test ◦ Tower for Deployment Orchestration • Ansible Galaxy?
57. Questions? alan@betterment.com @nonrational github.com/nonrational
58. careers@betterment.com All code snippets & diagrams contained in this presentation are property of Betterment, but please learn from them. All photographs / GIFs used in this presentation are someone else’s. Street Fighter, Back To The Future, Indiana Jones, Futurama, and Arrested Development are someone else’s property too.