AnsibleFest 2016 - Immutable Infrastructure at Scale With Ansible

In the good old days it was common practice to handcraft a server, then constantly manage it, applying updates and keeping close eye on it during its lifecycle. But in a highly dynamic cloud environment, change is constant, making this approach more difficult to maintain.

In this talk I share how the Beamly infrastructure on AWS has evolved from artisanal servers over the past two to three years. Plus, I explain how the platform team was able to improve tooling to match growth and sustain rapid scalability and agility through a succession of business pivots.

This talk was presented at AnsibleFest London on February 18th 2016

Transcript

1. Immutable Infrastructure at Scale Vik Bhatti Senior Platform Engineer @otaku_coder
2. “Immutable infrastructure is comprised of immutable components that are replaced for every deployment, rather than being updated in-place.” What is immutable infrastructure? Florian Motlik CTO @codeship blog.codeship.com/immutable-infrastructure
3. An immutable infrastructure is powerful
4. But difficult to run without the right tooling
5. Why Ansible? Masterless User Experience
6. A Brief History Zeebox • Smart TV Remote • Social Network Beamly • Content network • Ad Tech • Acquired Oct 2015
7. Live call-to-action
8. But…
9. Cue frantic debugging…
10. How Do We Scale Faster? Autoscaling
11. Configure on start-up Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
12. Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
13. Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Pull down ansible Playbook on boot (or use ansible tower)
14. Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
15. Preconfigure on image build Auto Scaling group ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
16. Preconfigure on image build Auto Scaling group ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Ansible play to bundle all requirements into the image
17. Preconfigure on image build Auto Scaling group ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance EC2 Instance EC2 Instance
18. Build Pipeline Build Artifact Provision Image Chroot Install Component Artifact Snapshot Machine Image Test Machine Image Deploy New Image SCM
19. Packer by Hashicorp Building Machine Images
20. Builders create machines and generate images { ‘builders’: [ ‘name’: ‘build-aws-hvm’, ‘type’: ‘amazon-chroot’, ‘source_ami’: ‘ami-1234abcd’, ‘ami_name’: ‘ansiblefest-2016-v1’, ‘ami_virtualization_type’: ‘hvm’, ‘tags’: { ‘component’: ‘ansiblefest’, ‘version’: ‘1.0’ } ] }
21. Provisioners define how to install dependencies { ‘provisioners’: [ ‘type’: ‘ansible-local’, ‘playbook_file’: ‘/tmp/myplay.yml’, ‘extra_arguments’: [‘--extra-vars component=ansiblefest’] ] }
22. Packer takes a template file and executes the build $ packer build –machine-readable /tmp/mytemplate.json
23. Packer build – Runs builders in parallel 12:41:13.168 1453293673,,ui,say,==> hvm: Prevalidating AMI Name... 12:41:13.871 1453293673,,ui,say,==> hvm: Gathering information about this EC2 instance... 12:41:13.922 1453293673,,ui,say,==> paravirtual: Gathering information about this EC2 instance... 12:41:13.983 1453293673,,ui,say,==> hvm: Inspecting the source AMI... 12:41:14.000 1453293673,,ui,say,==> paravirtual: Inspecting the source AMI... 12:41:14.052 1453293674,,ui,say,==> paravirtual: Checking the root device on source AMI... 12:41:14.053 1453293674,,ui,say,==> paravirtual: Creating the root volume... 12:41:14.054 1453293674,,ui,say,==> hvm: Checking the root device on source AMI... 12:41:18.399 1453293678,,ui,say,==> paravirtual: Attaching the root volume to /dev/sdf 12:41:21.182 1453293681,,ui,say,==> hvm: Creating the root volume... 12:41:21.183 1453293681,,ui,say,==> paravirtual: Mounting the root device... 12:41:21.296 1453293681,,ui,say,==> paravirtual: Mounting additional paths within the chroot... 12:41:21.497 1453293681,,ui,message, paravirtual: Mounting: /proc 12:41:21.512 1453293681,,ui,message, paravirtual: Mounting: /sys 12:41:21.570 1453293681,,ui,message, paravirtual: Mounting: /dev 12:41:21.577 1453293681,,ui,message, paravirtual: Mounting: /dev/pts 12:41:21.586 1453293681,,ui,message, paravirtual: Mounting: /proc/sys/fs/binfmt_misc 12:41:21.594 1453293681,,ui,say,==> paravirtual: Copying files from host to chroot... 12:41:21.594 1453293681,,ui,message, paravirtual: /etc/resolv.conf 12:41:21.733 1453293681,,ui,say,==> paravirtual: Provisioning with shell script: /tmp/packer-shell284329197 12:41:25.585 1453293685,,ui,say,==> hvm: Attaching the root volume to /dev/sdh 12:41:30.412 1453293690,,ui,say,==> hvm: Mounting the root device... 12:41:30.615 1453293690,,ui,say,==> hvm: Mounting additional paths within the chroot...
24. Gotcha – AWS chroot Builder Ansible tasks must not leave any processes running, or packer can’t unmount the volume. Remove handler and notify calls from galaxy tasks
25. How do we update service config? Configuration
26. v1 - Set config file to use on boot 1. Write multiple configuration files • For each environment/region 2. Inspect metadata on boot and use the matching config file
27. Use with_items to write multiple config files - name: write config files template: src: myconfig.j2 dest: /etc/{{ item.country }}-{{ item.environment }}-conf.json owner: root mode: 0644 with_items: - { country: ‘uk’, environment: ‘stage’ } - { country: ‘uk’, environment: ‘live’ } - { country: ‘us’, environment: ‘stage’ } - { country: ‘us’, environment: ‘live’ }
28. v2 – Use Service Discovery/Config KV 1. Local agent watches for changes in KV values 2. Writes new config to disk on change 3. Restarts corresponding service Note: config value changes != template changes
29. Machine images should be the same regardless of where they are deployed
30. What about system state? Sidenote
31. State is hard …but can be managed and encapsulated
32. Stateful ‘things’ • Logs • Metrics • Data Storage/Databases • Caches • Filesystem • DNS
33. Conclusions A few things to take away
34. Immutable infrastructure is complex at first
35. But wicked fast to scale up and down when all the tools are in place
36. Immutable infrastructure simplifies change management
37. Packer is great for machine image pipelines
38. Ansible can glue all the pieces together
39. Thanks for listening Questions?
40. LONDON Drury House 34-43 Russell Street London WC2B 5HA NEW YORK CITY 350 Fifth Avenue Suite 1700 New York NY 10018 © Beamly 2016. All Rights Reserved.