Immutable Infrastructure at Scale Vik Bhatti Senior Platform Engineer @otaku_coder 

“Immutable infrastructure is comprised of immutable components that are replaced for every deployment, rather than being updated in-place.” What is immutable infrastructure? Florian Motlik CTO @codeship blog.codeship.com/immutable-infrastructure 

An immutable infrastructure is powerful 

But difficult to run without the right tooling 

Why Ansible? Masterless User Experience 

A Brief History Zeebox • Smart TV Remote • Social Network Beamly • Content network • Ad Tech • Acquired Oct 2015 

Live call-to-action 

But… 

Cue frantic debugging… 

How Do We Scale Faster? Autoscaling 

Configure on start-up Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Pull down ansible Playbook on boot (or use ansible tower) 

Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Preconfigure on image build Auto Scaling group ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Preconfigure on image build Auto Scaling group ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Ansible play to bundle all requirements into the image 

Preconfigure on image build Auto Scaling group ansiblerocks.com Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance EC2 Instance EC2 Instance 

Build Pipeline Build Artifact Provision Image Chroot Install Component Artifact Snapshot Machine Image Test Machine Image Deploy New Image SCM 

Packer by Hashicorp Building Machine Images 

Builders create machines and generate images { ‘builders’: [ ‘name’: ‘build-aws-hvm’, ‘type’: ‘amazon-chroot’, ‘source_ami’: ‘ami-1234abcd’, ‘ami_name’: ‘ansiblefest-2016-v1’, ‘ami_virtualization_type’: ‘hvm’, ‘tags’: { ‘component’: ‘ansiblefest’, ‘version’: ‘1.0’ } ] } 

Provisioners define how to install dependencies { ‘provisioners’: [ ‘type’: ‘ansible-local’, ‘playbook_file’: ‘/tmp/myplay.yml’, ‘extra_arguments’: [‘--extra-vars component=ansiblefest’] ] } 

Packer takes a template file and executes the build $ packer build –machine-readable /tmp/mytemplate.json 

Packer build – Runs builders in parallel 12:41:13.168 1453293673,,ui,say,==> hvm: Prevalidating AMI Name... 12:41:13.871 1453293673,,ui,say,==> hvm: Gathering information about this EC2 instance... 12:41:13.922 1453293673,,ui,say,==> paravirtual: Gathering information about this EC2 instance... 12:41:13.983 1453293673,,ui,say,==> hvm: Inspecting the source AMI... 12:41:14.000 1453293673,,ui,say,==> paravirtual: Inspecting the source AMI... 12:41:14.052 1453293674,,ui,say,==> paravirtual: Checking the root device on source AMI... 12:41:14.053 1453293674,,ui,say,==> paravirtual: Creating the root volume... 12:41:14.054 1453293674,,ui,say,==> hvm: Checking the root device on source AMI... 12:41:18.399 1453293678,,ui,say,==> paravirtual: Attaching the root volume to /dev/sdf 12:41:21.182 1453293681,,ui,say,==> hvm: Creating the root volume... 12:41:21.183 1453293681,,ui,say,==> paravirtual: Mounting the root device... 12:41:21.296 1453293681,,ui,say,==> paravirtual: Mounting additional paths within the chroot... 12:41:21.497 1453293681,,ui,message, paravirtual: Mounting: /proc 12:41:21.512 1453293681,,ui,message, paravirtual: Mounting: /sys 12:41:21.570 1453293681,,ui,message, paravirtual: Mounting: /dev 12:41:21.577 1453293681,,ui,message, paravirtual: Mounting: /dev/pts 12:41:21.586 1453293681,,ui,message, paravirtual: Mounting: /proc/sys/fs/binfmt_misc 12:41:21.594 1453293681,,ui,say,==> paravirtual: Copying files from host to chroot... 12:41:21.594 1453293681,,ui,message, paravirtual: /etc/resolv.conf 12:41:21.733 1453293681,,ui,say,==> paravirtual: Provisioning with shell script: /tmp/packer-shell284329197 12:41:25.585 1453293685,,ui,say,==> hvm: Attaching the root volume to /dev/sdh 12:41:30.412 1453293690,,ui,say,==> hvm: Mounting the root device... 12:41:30.615 1453293690,,ui,say,==> hvm: Mounting additional paths within the chroot... 

Gotcha – AWS chroot Builder Ansible tasks must not leave any processes running, or packer can’t unmount the volume. Remove handler and notify calls from galaxy tasks 

How do we update service config? Configuration 

v1 - Set config file to use on boot 1. Write multiple configuration files • For each environment/region 2. Inspect metadata on boot and use the matching config file 

Use with_items to write multiple config files - name: write config files template: src: myconfig.j2 dest: /etc/{{ item.country }}-{{ item.environment }}-conf.json owner: root mode: 0644 with_items: - { country: ‘uk’, environment: ‘stage’ } - { country: ‘uk’, environment: ‘live’ } - { country: ‘us’, environment: ‘stage’ } - { country: ‘us’, environment: ‘live’ } 

v2 – Use Service Discovery/Config KV 1. Local agent watches for changes in KV values 2. Writes new config to disk on change 3. Restarts corresponding service Note: config value changes != template changes 

Machine images should be the same regardless of where they are deployed 

What about system state? Sidenote 

State is hard …but can be managed and encapsulated 

Stateful ‘things’ • Logs • Metrics • Data Storage/Databases • Caches • Filesystem • DNS 

Conclusions A few things to take away 

Immutable infrastructure is complex at first 

But wicked fast to scale up and down when all the tools are in place 

Immutable infrastructure simplifies change management 

Packer is great for machine image pipelines 

Ansible can glue all the pieces together 

Thanks for listening Questions? 

LONDON Drury House 34-43 Russell Street London WC2B 5HA NEW YORK CITY 350 Fifth Avenue Suite 1700 New York NY 10018 © Beamly 2016. All Rights Reserved.