Immutable Infrastructure at Scale Vik Bhatti Senior Platform Engineer @otaku_coder 

This Talk • Timeline • Build tools • Testing • Configuration management • Conclusions flickr.com/photos/toms/5678872701 

“Immutable infrastructure is comprised of immutable components that are replaced for every deployment, rather than being updated in-place.” What is immutable infrastructure? Florian Motlik CTO @codeship blog.codeship.com/immutable-infrastructure 

Immutable infrastructure is powerful 

But difficult to run without the right tooling 

A Brief History Zeebox • Smart TV Remote • Social Network Beamly • Content network • Ad Tech • Acquired Oct 2015 

Live call-to-action 

Cue frantic debugging… 

How Do We Scale Faster? Autoscaling 

Configure on start-up Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Pull down ansible Playbook on boot (or use ansible tower) 

Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Preconfigure on image build Auto Scaling group hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance 

Preconfigure on image build Auto Scaling group hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Ansible play to bundle all requirements into the image 

Preconfigure on image build Auto Scaling group hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance EC2 Instance EC2 Instance 

Build Pipeline Compile Artifact Provision Image Install Artifact Snapshot Machine Image Test Machine Image Deploy New Image SCM 

Build Pipeline 

Build Tools (gears) AMI bake • Lookup component metadata • Generate packer template • Execute packer build AMI deploy • Trigger rolling push via asgard • or run ansible deploy playbook 

Build Tools (gears) AMI test • Spin up test instance • Run smoke tests via Smolder • github.com/sky-shiny/smolder 

Automated Testing Smolder 

Standard Service Endpoints • Every service MUST implement standard endpoints • Each endpoint has consistent response type across languages • Easy to reason about and test services 

Introducting SE4 - github.com/beamly/se4 

Smolder – github.com/sky-shiny/smolder 

Smolder tests are defined in JSON or YAML --- tests: name: "foobar-service" uri: "/service/status" port: 8080 method: "GET" inputs: headers: Host: "beamly.com" content-type: "application/json” outcomes: response_body_contains: "foobar-stage" expect_status_code: 200 

Smoke testing with smolder ------------------------------------------------------------------------------------- foobar-status http://10.0.3.50:8080/service/status curl -vsH "content-type: Application/json" -X GET "http://10.0.3.50:8080/service/status" ------------------------------------------------------------------------------------- expect_status_code: Status code == 200.........................................[PASS] response_body_contains: Body contains "foobar-stage"...........................[PASS] ALL TESTS PASSED! 

Packer Building Machine Images 

Builders create machines and generate images { ‘builders’: [ ‘name’: ‘build-aws-hvm’, ‘type’: ‘amazon-chroot’, ‘source_ami’: ‘ami-1234abcd’, ‘ami_name’: ‘hashiconf_eu-2016-v1’, ‘ami_virtualization_type’: ‘hvm’, ‘tags’: { ‘component’: ‘hashiconf_eu’, ‘version’: ‘1.0.3’ } ] } 

Provisioners define how to install dependencies { ‘provisioners’: [ ‘type’: ‘ansible-local’, ‘playbook_file’: ‘/tmp/myplay.yml’, ‘extra_arguments’: [‘--extra-vars version=1.0.3’] ] } 

Packer takes a template file and executes the build $ packer build –machine-readable /tmp/mytemplate.json 

Packer build – Runs builders in parallel 12:41:13.168 1453293673,,ui,say,==> hvm: Prevalidating AMI Name... 12:41:13.871 1453293673,,ui,say,==> hvm: Gathering information about this EC2 instance... 12:41:13.922 1453293673,,ui,say,==> paravirtual: Gathering information about this EC2 instance... 12:41:13.983 1453293673,,ui,say,==> hvm: Inspecting the source AMI... 12:41:14.000 1453293673,,ui,say,==> paravirtual: Inspecting the source AMI... 12:41:14.052 1453293674,,ui,say,==> paravirtual: Checking the root device on source AMI... 12:41:14.053 1453293674,,ui,say,==> paravirtual: Creating the root volume... 12:41:14.054 1453293674,,ui,say,==> hvm: Checking the root device on source AMI... 12:41:18.399 1453293678,,ui,say,==> paravirtual: Attaching the root volume to /dev/sdf 12:41:21.182 1453293681,,ui,say,==> hvm: Creating the root volume... 12:41:21.183 1453293681,,ui,say,==> paravirtual: Mounting the root device... 12:41:21.296 1453293681,,ui,say,==> paravirtual: Mounting additional paths within the chroot... 12:41:21.497 1453293681,,ui,message, paravirtual: Mounting: /proc 12:41:21.512 1453293681,,ui,message, paravirtual: Mounting: /sys 12:41:21.570 1453293681,,ui,message, paravirtual: Mounting: /dev 12:41:21.577 1453293681,,ui,message, paravirtual: Mounting: /dev/pts 12:41:21.586 1453293681,,ui,message, paravirtual: Mounting: /proc/sys/fs/binfmt_misc 12:41:21.594 1453293681,,ui,say,==> paravirtual: Copying files from host to chroot... 12:41:21.594 1453293681,,ui,message, paravirtual: /etc/resolv.conf 12:41:21.733 1453293681,,ui,say,==> paravirtual: Provisioning with shell script: /tmp/packer-shell284329197 12:41:25.585 1453293685,,ui,say,==> hvm: Attaching the root volume to /dev/sdh 12:41:30.412 1453293690,,ui,say,==> hvm: Mounting the root device... 12:41:30.615 1453293690,,ui,say,==> hvm: Mounting additional paths within the chroot... 

Gears ‘injects’ vars into the chroot via provisioner { "type": "ansible-local", "playbook_file": "/tmp/go-agent/pipelines/cpt-1.x-bake/ansible/site.yml”, "extra_arguments": [ "--extra-vars ‘COMPONENT_NAME=cpt’", "--extra-vars ‘COMPONENT_VERSION=1.0.81’" ], "role_paths": [ "/tmp/go-agent/pipelines/cpt-1.x-bake/ansible/roles/jdauphant/ansible-role-nginx", "/tmp/go-agent/pipelines/cpt-1.x-bake/ansible/roles/beamly.cpt” ] } 

Gears looks up and generates tags for builder { "source_ami": "ami-df0607b5", "name": "hvm", "ami_regions": [ "us-east-1”, ”eu-west-1” ], "tags": { "GIT_SHA": "b675d08113ba0129fec3b6291ec7ba25e8843ad1", "AMI_VERSION": "1.0.81", "COMPONENT_REPO": "[email protected]:beamly/cpt.git", "COMPONENT_VERSION": "1.0.81", "CREATED_AT": "201606101438", "COMPONENT_NAME": ”cpt" }, "ami_name": ”cpt-1.0.81-hvm-201606101438", } 

Gotcha – AWS chroot Builder Provisioning tasks must not leave any processes running, or packer can’t unmount the volume. Remove handlers and notify calls from ansible roles/tasks 

Tracking changes Metadata 

“A manifest or customs manifest or "cargo document" is a document listing the cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials” What is a shipping manifest? Wikipedia https://en.wikipedia.org/wiki/Manifest_(transportation) 

AWS Tags Image tags • Component Name • Component Version • AMI Version • Build Time Instance Tags (via ASG config) • Component Name • Component Version • AMI Version • Environment • Country 

How do we update service config? Configuration 

v1 - Set config file to use on boot 1. Write multiple configuration files • For each environment/region 2. Inspect metadata on boot and use the matching config file 

Environment-specific config ... vars: us: stage: db_url: ‘stage-us-db.rds.aws.amazon.com’ cache_key: ‘stage-us-foobar’ live: db_url: ‘live-us-db.rds.aws.amazon.com’ uk: stage: db_url: ‘stage-uk-rds.aws.amazon.com’ cache_key: ‘stage-uk-foobar’ ... 

Use with_items to write multiple config files - name: write config files template: src: myconfig.j2 dest: “/etc/{{ item.country }}-{{ item.environment }}-conf.json” owner: root mode: 0644 with_items: - { country: ‘uk’, environment: ‘stage’ } - { country: ‘uk’, environment: ‘live’ } - { country: ‘us’, environment: ‘stage’ } - { country: ‘us’, environment: ‘live’ } 

Consul Distributed Configuration Management 

Config v2 – Use Service Discovery/Config KV 1. Local consul-template agent watches for changes in KV values 2. Writes new config to disk on change 3. Restarts corresponding service Note: config value changes != template changes 

Machine images should be the same regardless of where they are deployed 

Would a config change trigger a service outage? …yes... use consul distributed locks! 

How do you keep environmental config in sync? Marco Polo 

Structure your KV data 

Structured KV • Standard structure for KV data – Namespaced by AWS region and env level • /component/version/region/env • Easy to reason about config • Minimise cross-team config mixups 

Structured KV Component keyspace – {{ component }} / {{ version }} / {{ country }} / {{ environment }} / {{ key }} Service ports – {{ component }} / static / global / common / port Shared config (e.g. API keys) – common / {{ country }} / {{ component }} / key 

Config should be versioned just like code 

Introducing ’Marco Polo’ • Written in Scala • Hooks into CD pipelines • Synchronises config trees between environments (stage -> live) 

Marco Polo – CD Pipelines Build Agent Marcopolo Consul AMI Bake Job 

Marco Polo – CD Pipelines Build Agent Marcopolo Consul AMI Bake v23 POST components/foobar/ 23/sync 

Marco Polo – New Component Version Build Build Agent Marcopolo Consul AMI Bake v23 Read LIVE v22 config tree 

Marco Polo – New Component Version Build Build Agent Marcopolo Consul AMI Bake v23 Create STAGE v23 tree 

Marco Polo – New Component Version Build Build Agent Marcopolo Consul AMI Bake v23 200 OK 

Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 

Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 POST components/foobar/ 23/sync 

Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 Copy stagetree for v23 to live 

Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 200 OK 

How do components know what config to use? Consul Template 

Dirty Hack Alert 

Client Side Config Lookup – Consul Template AMI + Autoscaling group tagged: – Component, version, environment, country (ec2 region) Script on boot: 1. Look up tags 2. Pre-render .ctmpl file, filling in the KV lookup tree 

Config template file fb.oauth.redirect=” {{key ”foobar/1.0.23/<>/<>/facebook/redirect_url" }}” ga.tracking.code=” {{key ”foobar/1.0.23/<>/<>/ga_tracking_code" }}” 

Pre-rendered consul template file fb.oauth.redirect="{{ key ”foobar/1.0.23/us/stage/facebook/redirect_url" }}” ga.tracking.code="{{ key ”foobar/1.0.23/us/stage/ga_tracking_code" }}” 

What about system state? Sidenote 

State is hard …but can be managed and encapsulated 

Stateful ‘things’ • Logs • Metrics • Data Storage/Databases • Caches • Filesystem • DNS 

Routing • Every service runs on a unique port • Port/Component mappings stored in KV • HAProxy tier • Generate backends from instance list (based on component tag) 

Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group 

Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group EC2 Instance 

Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group EC2 Instance Script – finds all instances and generates new config 

Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group EC2 Instance 

Conclusions A few things to take away 

Immutable infrastructure is complex at first 

But wicked fast to scale up and down when all the tools are in place 

Be pragmatic. Use bootstrapping to bring instances into service 

Immutable infrastructure simplifies change management 

Trust your continuous delivery tooling 

Thanks for listening Questions? 

LONDON Drury House 34-43 Russell Street London WC2B 5HA NEW YORK CITY 350 Fifth Avenue Suite 1700 New York NY 10018 © Beamly 2016. All Rights Reserved.