Hashiconf EU 2016 - Immutable Infrastructure at Scale

In the good old days it was common practice to handcraft a server, then constantly manage it, applying updates and keeping close eye on it during its lifecycle. But in a highly dynamic cloud environment, change is constant, making this approach more difficult to maintain.
In this talk I share how the Beamly infrastructure on AWS has evolved from artisanal servers over the past two to three years. Plus, I explain how the platform team was able to improve tooling to match growth and sustain rapid scalability and agility through a succession of business pivots.

Transcript

1. Immutable Infrastructure at Scale Vik Bhatti Senior Platform Engineer @otaku_coder
2. This Talk • Timeline • Build tools • Testing • Configuration management • Conclusions flickr.com/photos/toms/5678872701
3. “Immutable infrastructure is comprised of immutable components that are replaced for every deployment, rather than being updated in-place.” What is immutable infrastructure? Florian Motlik CTO @codeship blog.codeship.com/immutable-infrastructure
4. Immutable infrastructure is powerful
5. But difficult to run without the right tooling
6. A Brief History Zeebox • Smart TV Remote • Social Network Beamly • Content network • Ad Tech • Acquired Oct 2015
7. Live call-to-action
8. Cue frantic debugging…
9. How Do We Scale Faster? Autoscaling
10. Configure on start-up Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
11. Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
12. Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Pull down ansible Playbook on boot (or use ansible tower)
13. Configure on start-up EC2 Instance Auto Scaling group Amazon S3 bucket hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
14. Preconfigure on image build Auto Scaling group hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance
15. Preconfigure on image build Auto Scaling group hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance Ansible play to bundle all requirements into the image
16. Preconfigure on image build Auto Scaling group hashiconf.eu Elastic Load Balancer Amazon Route 53 Base OS Machine Image EC2 Instance EC2 Instance EC2 Instance
17. Build Pipeline Compile Artifact Provision Image Install Artifact Snapshot Machine Image Test Machine Image Deploy New Image SCM
18. Build Pipeline
19. Build Tools (gears) AMI bake • Lookup component metadata • Generate packer template • Execute packer build AMI deploy • Trigger rolling push via asgard • or run ansible deploy playbook
20. Build Tools (gears) AMI test • Spin up test instance • Run smoke tests via Smolder • github.com/sky-shiny/smolder
21. Automated Testing Smolder
22. Standard Service Endpoints • Every service MUST implement standard endpoints • Each endpoint has consistent response type across languages • Easy to reason about and test services
23. Introducting SE4 - github.com/beamly/se4
24. Smolder – github.com/sky-shiny/smolder
25. Smolder tests are defined in JSON or YAML --- tests: name: "foobar-service" uri: "/service/status" port: 8080 method: "GET" inputs: headers: Host: "beamly.com" content-type: "application/json” outcomes: response_body_contains: "foobar-stage" expect_status_code: 200
26. Smoke testing with smolder ------------------------------------------------------------------------------------- foobar-status http://10.0.3.50:8080/service/status curl -vsH "content-type: Application/json" -X GET "http://10.0.3.50:8080/service/status" ------------------------------------------------------------------------------------- expect_status_code: Status code == 200.........................................[PASS] response_body_contains: Body contains "foobar-stage"...........................[PASS] ALL TESTS PASSED!
27. Packer Building Machine Images
28. Builders create machines and generate images { ‘builders’: [ ‘name’: ‘build-aws-hvm’, ‘type’: ‘amazon-chroot’, ‘source_ami’: ‘ami-1234abcd’, ‘ami_name’: ‘hashiconf_eu-2016-v1’, ‘ami_virtualization_type’: ‘hvm’, ‘tags’: { ‘component’: ‘hashiconf_eu’, ‘version’: ‘1.0.3’ } ] }
29. Provisioners define how to install dependencies { ‘provisioners’: [ ‘type’: ‘ansible-local’, ‘playbook_file’: ‘/tmp/myplay.yml’, ‘extra_arguments’: [‘--extra-vars version=1.0.3’] ] }
30. Packer takes a template file and executes the build $ packer build –machine-readable /tmp/mytemplate.json
31. Packer build – Runs builders in parallel 12:41:13.168 1453293673,,ui,say,==> hvm: Prevalidating AMI Name... 12:41:13.871 1453293673,,ui,say,==> hvm: Gathering information about this EC2 instance... 12:41:13.922 1453293673,,ui,say,==> paravirtual: Gathering information about this EC2 instance... 12:41:13.983 1453293673,,ui,say,==> hvm: Inspecting the source AMI... 12:41:14.000 1453293673,,ui,say,==> paravirtual: Inspecting the source AMI... 12:41:14.052 1453293674,,ui,say,==> paravirtual: Checking the root device on source AMI... 12:41:14.053 1453293674,,ui,say,==> paravirtual: Creating the root volume... 12:41:14.054 1453293674,,ui,say,==> hvm: Checking the root device on source AMI... 12:41:18.399 1453293678,,ui,say,==> paravirtual: Attaching the root volume to /dev/sdf 12:41:21.182 1453293681,,ui,say,==> hvm: Creating the root volume... 12:41:21.183 1453293681,,ui,say,==> paravirtual: Mounting the root device... 12:41:21.296 1453293681,,ui,say,==> paravirtual: Mounting additional paths within the chroot... 12:41:21.497 1453293681,,ui,message, paravirtual: Mounting: /proc 12:41:21.512 1453293681,,ui,message, paravirtual: Mounting: /sys 12:41:21.570 1453293681,,ui,message, paravirtual: Mounting: /dev 12:41:21.577 1453293681,,ui,message, paravirtual: Mounting: /dev/pts 12:41:21.586 1453293681,,ui,message, paravirtual: Mounting: /proc/sys/fs/binfmt_misc 12:41:21.594 1453293681,,ui,say,==> paravirtual: Copying files from host to chroot... 12:41:21.594 1453293681,,ui,message, paravirtual: /etc/resolv.conf 12:41:21.733 1453293681,,ui,say,==> paravirtual: Provisioning with shell script: /tmp/packer-shell284329197 12:41:25.585 1453293685,,ui,say,==> hvm: Attaching the root volume to /dev/sdh 12:41:30.412 1453293690,,ui,say,==> hvm: Mounting the root device... 12:41:30.615 1453293690,,ui,say,==> hvm: Mounting additional paths within the chroot...
32. Gears ‘injects’ vars into the chroot via provisioner { "type": "ansible-local", "playbook_file": "/tmp/go-agent/pipelines/cpt-1.x-bake/ansible/site.yml”, "extra_arguments": [ "--extra-vars ‘COMPONENT_NAME=cpt’", "--extra-vars ‘COMPONENT_VERSION=1.0.81’" ], "role_paths": [ "/tmp/go-agent/pipelines/cpt-1.x-bake/ansible/roles/jdauphant/ansible-role-nginx", "/tmp/go-agent/pipelines/cpt-1.x-bake/ansible/roles/beamly.cpt” ] }
33. Gears looks up and generates tags for builder { "source_ami": "ami-df0607b5", "name": "hvm", "ami_regions": [ "us-east-1”, ”eu-west-1” ], "tags": { "GIT_SHA": "b675d08113ba0129fec3b6291ec7ba25e8843ad1", "AMI_VERSION": "1.0.81", "COMPONENT_REPO": "git@github.com:beamly/cpt.git", "COMPONENT_VERSION": "1.0.81", "CREATED_AT": "201606101438", "COMPONENT_NAME": ”cpt" }, "ami_name": ”cpt-1.0.81-hvm-201606101438", }
34. Gotcha – AWS chroot Builder Provisioning tasks must not leave any processes running, or packer can’t unmount the volume. Remove handlers and notify calls from ansible roles/tasks
35. Tracking changes Metadata
36. “A manifest or customs manifest or "cargo document" is a document listing the cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials” What is a shipping manifest? Wikipedia https://en.wikipedia.org/wiki/Manifest_(transportation)
37. AWS Tags Image tags • Component Name • Component Version • AMI Version • Build Time Instance Tags (via ASG config) • Component Name • Component Version • AMI Version • Environment • Country
38. How do we update service config? Configuration
39. v1 - Set config file to use on boot 1. Write multiple configuration files • For each environment/region 2. Inspect metadata on boot and use the matching config file
40. Environment-specific config ... vars: us: stage: db_url: ‘stage-us-db.rds.aws.amazon.com’ cache_key: ‘stage-us-foobar’ live: db_url: ‘live-us-db.rds.aws.amazon.com’ uk: stage: db_url: ‘stage-uk-rds.aws.amazon.com’ cache_key: ‘stage-uk-foobar’ ...
41. Use with_items to write multiple config files - name: write config files template: src: myconfig.j2 dest: “/etc/{{ item.country }}-{{ item.environment }}-conf.json” owner: root mode: 0644 with_items: - { country: ‘uk’, environment: ‘stage’ } - { country: ‘uk’, environment: ‘live’ } - { country: ‘us’, environment: ‘stage’ } - { country: ‘us’, environment: ‘live’ }
42. Consul Distributed Configuration Management
43. Config v2 – Use Service Discovery/Config KV 1. Local consul-template agent watches for changes in KV values 2. Writes new config to disk on change 3. Restarts corresponding service Note: config value changes != template changes
44. Machine images should be the same regardless of where they are deployed
45. Would a config change trigger a service outage? …yes... use consul distributed locks!
46. How do you keep environmental config in sync? Marco Polo
47. Structure your KV data
48. Structured KV • Standard structure for KV data – Namespaced by AWS region and env level • /component/version/region/env • Easy to reason about config • Minimise cross-team config mixups
49. Structured KV Component keyspace – {{ component }} / {{ version }} / {{ country }} / {{ environment }} / {{ key }} Service ports – {{ component }} / static / global / common / port Shared config (e.g. API keys) – common / {{ country }} / {{ component }} / key
50. Config should be versioned just like code
51. Introducing ’Marco Polo’ • Written in Scala • Hooks into CD pipelines • Synchronises config trees between environments (stage -> live)
52. Marco Polo – CD Pipelines Build Agent Marcopolo Consul AMI Bake Job
53. Marco Polo – CD Pipelines Build Agent Marcopolo Consul AMI Bake v23 POST components/foobar/ 23/sync
54. Marco Polo – New Component Version Build Build Agent Marcopolo Consul AMI Bake v23 Read LIVE v22 config tree
55. Marco Polo – New Component Version Build Build Agent Marcopolo Consul AMI Bake v23 Create STAGE v23 tree
56. Marco Polo – New Component Version Build Build Agent Marcopolo Consul AMI Bake v23 200 OK
57. Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23
58. Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 POST components/foobar/ 23/sync
59. Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 Copy stagetree for v23 to live
60. Marco Polo – Component Version Deploy Live Build Agent Marcopolo Consul AMI Deploy v23 200 OK
61. How do components know what config to use? Consul Template
62. Dirty Hack Alert
63. Client Side Config Lookup – Consul Template AMI + Autoscaling group tagged: – Component, version, environment, country (ec2 region) Script on boot: 1. Look up tags 2. Pre-render .ctmpl file, filling in the KV lookup tree
64. Config template file fb.oauth.redirect=” {{key ”foobar/1.0.23/<<COUNTRY>>/<<ENVIRONMENT>>/facebook/redirect_url" }}” ga.tracking.code=” {{key ”foobar/1.0.23/<<COUNTRY>>/<<ENVIRONMENT>>/ga_tracking_code" }}”
65. Pre-rendered consul template file fb.oauth.redirect="{{ key ”foobar/1.0.23/us/stage/facebook/redirect_url" }}” ga.tracking.code="{{ key ”foobar/1.0.23/us/stage/ga_tracking_code" }}”
66. What about system state? Sidenote
67. State is hard …but can be managed and encapsulated
68. Stateful ‘things’ • Logs • Metrics • Data Storage/Databases • Caches • Filesystem • DNS
69. Routing • Every service runs on a unique port • Port/Component mappings stored in KV • HAProxy tier • Generate backends from instance list (based on component tag)
70. Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group
71. Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group EC2 Instance
72. Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group EC2 Instance Script – finds all instances and generates new config
73. Routing HAProxy Instance Zone a Auto Scaling group EC2 Instance EC2 Instance haproxy.beamly.com Component Ports KV HAProxy Instance Zone b Auto Scaling group EC2 Instance
74. Conclusions A few things to take away
75. Immutable infrastructure is complex at first
76. But wicked fast to scale up and down when all the tools are in place
77. Be pragmatic. Use bootstrapping to bring instances into service
78. Immutable infrastructure simplifies change management
79. Trust your continuous delivery tooling
80. Thanks for listening Questions?
81. LONDON Drury House 34-43 Russell Street London WC2B 5HA NEW YORK CITY 350 Fifth Avenue Suite 1700 New York NY 10018 © Beamly 2016. All Rights Reserved.