Build Immutable Servers with Packer & Ansible @ Microsoft OS Camp PT

Slide 1

Slide 1 text

Build Immutable Servers with Packer & Ansible Daniel Gomes #MSOSCAMP @_dcsg

Slide 2

Slide 2 text

About me • Senior Software Engineer at Talkdesk • Co-founder of phplx • DevOps Enthusiast • @_dcsg • dcsg.me

Slide 3

Slide 3 text

Once upon a time… @_dcsg #MSOSCAMP

Slide 4

Slide 4 text

A happy team @_dcsg #MSOSCAMP

Slide 5

Slide 5 text

Delivers new features in a high pace @_dcsg #MSOSCAMP 0 35 70 105 140 April May June July

Slide 6

Slide 6 text

Build/Deploy Processes are manual and time expensive @_dcsg #MSOSCAMP

Slide 7

Slide 7 text

Deploy 1 per week @_dcsg #MSOSCAMP

Slide 8

Slide 8 text

The cluster 4 instances @_dcsg #MSOSCAMP

Slide 9

Slide 9 text

But… problems happen! @_dcsg #MSOSCAMP https://http.cat/500

Slide 10

Slide 10 text

Instance degraded Out of memory issues @_dcsg #MSOSCAMP

Slide 11

Slide 11 text

SSH’s in production machine and live ﬁxes are done @_dcsg #MSOSCAMP

Slide 12

Slide 12 text

Problem Solved @_dcsg #MSOSCAMP

Slide 13

Slide 13 text

The new cluster state Fixed Instance Instances  not in same state @_dcsg #MSOSCAMP

Slide 14

Slide 14 text

Conﬁguration Drift • Manual ad-hoc changes and updates to servers that are not recorded • Servers in your infrastructure became more and more different from each others @_dcsg #MSOSCAMP

Slide 15

Slide 15 text

Snowflake Server • Long running servers • Difficult to reproduce • No consistency between servers • Lack of confidence in your systems • Hard to spin up another instance in the same state @_dcsg https://martinfowler.com/bliki/SnowflakeServer.html

Slide 16

Slide 16 text

The cluster state after a few weeks State A @_dcsg #MSOSCAMP State C State B State B

Slide 17

Slide 17 text

How can we solve this? @_dcsg #MSOSCAMP

Slide 18

Slide 18 text

Conﬁguration Management with Automation Tools @_dcsg #MSOSCAMP

Slide 19

Slide 19 text

Conﬁguration Management The process of systematically handling changes to a system in a way that it maintains integrity over time. @_dcsg #MSOSCAMP

Slide 20

Slide 20 text

Configuration Management Spin up Base Image Run Config Management Server in desire state Changes Edit config file Upgrade/install package … Running Server

Slide 21

Slide 21 text

Applying to the cluster state State A @_dcsg #MSOSCAMP State C State B State B Conﬁg Management Apply State D

Slide 22

Slide 22 text

The cluster state after CM ran State D @_dcsg #MSOSCAMP State D State D State D

Slide 23

Slide 23 text

Automation Tools for CM

Slide 24

Slide 24 text

Ansible @_dcsg https://www.ansible.com/

Slide 25

Slide 25 text

How Ansible works? • Agent-less by operating over SSH • Ad-hoc commands execution • Playbooks @_dcsg #MSOSCAMP

Slide 26

Slide 26 text

Ansible “components” • Inventory of hosts - your raw materials (servers) • Modules - your tools (apt, yum, etc) • Playbooks - your instructions manual (executes tasks) @_dcsg #MSOSCAMP

Slide 27

Slide 27 text

Ansible Inventory - Hosts & Groups # /my_project/hosts mail.example.com [webservers] foo.example.com bar.example.com [dbservers] one.example.com two.example.com

Slide 28

Slide 28 text

Ansible Modules - — name: Install vim  apt: pkg=vim state=latest - — name: reboot the server  command: /sbin/reboot -t now

Slide 29

Slide 29 text

Ansible Playbook “components” • Hosts • Variables, Conditions, Loops • Templating (Jinja2) • Tasks • Handlers • Roles @_dcsg http://docs.ansible.com/ansible/playbooks.html

Slide 30

Slide 30 text

--- - hosts: webservers vars: http_port: 80 remote_user: root tasks: - name: write the apache config file template: src=/srv/httpd.j2 dest=/etc/httpd.conf notify: - restart apache - name: ensure apache is running (and enable it at boot) service: name=httpd state=started enabled=yes handlers: - name: restart apache service: name=httpd state=restarted Ansible Variables, Tasks, Handlers

Slide 31

Slide 31 text

Ansible Conditions & Loops - name: add several users user: name: "{{ item.name }}" state: present groups: "{{ item.groups }}" with_items: - { name: 'testuser1', groups: 'wheel' } - { name: 'testuser2', groups: 'root' } when: - ansible_distribution == "CentOS"

Slide 32

Slide 32 text

Ansible Roles Example # Folder structure roles/ common/ files/ templates/ tasks/ main.yml handlers/ vars/ defaults/ meta/ # roles/common/tasks/main.yml - name: install common packages yum: name: "{{ item }}" state: present with_items: - htop - vim

Slide 33

Slide 33 text

Recap • Configuration Drift • Problems of the Snowflake Servers • Configuration Management • Ansible @_dcsg #MSOSCAMP

Slide 34

Slide 34 text

Phoenix Servers @_dcsg https://martinfowler.com/bliki/PhoenixServer.html

Slide 35

Slide 35 text

“A server should be like a phoenix, regularly rising from the ashes.” - Martin Fowler in PhoenixServer @_dcsg #MSOSCAMP

Slide 36

Slide 36 text

Phoenix Servers • Avoid conﬁguration drifts • Disposable servers • Servers can be built from scratch @_dcsg #MSOSCAMP

Slide 37

Slide 37 text

The cluster Re-launch an instance State D @_dcsg #MSOSCAMP State D State D State D Terminate   Instance Conﬁg Management Apply State D State D

Slide 38

Slide 38 text

Spinning up new servers are not a problem anymore! @_dcsg #MSOSCAMP

Slide 39

Slide 39 text

But… @_dcsg #MSOSCAMP

Slide 40

Slide 40 text

Is Idempotence guaranteed? @_dcsg #MSOSCAMP

Slide 41

Slide 41 text

What if the packages repositories are down? @_dcsg #MSOSCAMP

Slide 42

Slide 42 text

Built Process Spin up Base Image Run Config Management Server in desire state Install packages Create folders Create user Upload app etc Run Config Management Repositories unavailable Package not locked to specific version

Slide 43

Slide 43 text

How can we ﬁx this? @_dcsg #MSOSCAMP

Slide 44

Slide 44 text

Immutable Servers @_dcsg https://martinfowler.com/bliki/ImmutableServer.html

Slide 45

Slide 45 text

“An Immutable Server is a server, that once deployed, is never modiﬁed, merely replaced with a new updated instance.” - Kief Morris in ImmutableServer @_dcsg #MSOSCAMP

Slide 46

Slide 46 text

Immutable Servers • Final state image with everything baked in. • No changes after it’s built. • Include scripts to start the application at boot. • Easy to scale out, deploy and rollback • Trustable and testable • Easy to adopt A/B testing, Canary releases or Blue/Green deployments @_dcsg #MSOSCAMP

Slide 47

Slide 47 text

Immutable Server Build Process Run Conﬁg Management (puppet, chef, ansible) Bake In the App App Final Image Server in desire state conﬁgure application  environment Spin up Base Image

Slide 48

Slide 48 text

Build Image Stages Flow Example Base Image OS Hardening Common tools  (vim, htop, etc) etc Application Base Image Install necessary software to run the App Create user/folders Application Final Image Upload App Script to run App at boot System upgrades & Security updates Application security, package, conﬁguration updates

Slide 49

Slide 49 text

• SnowFlake Servers • Inconsistent states between machine (Config Drifts) • Phoenix Serves • Avoids Config Drifts using CM Automation Tools • Can be built from scratch • Immutable Serves • Final image with everything baked in • After built cannot be modified • Can only be replaced with an updated instance Recap

Slide 50

Slide 50 text

Hello Packer @_dcsg https://www.packer.io/

Slide 51

Slide 51 text

“Packer is an open source tool for creating identical machine images for multiple platforms from a single source conﬁguration.” What’s Packer? @_dcsg #MSOSCAMP

Slide 52

Slide 52 text

Packer components Builders Provisioners Post-Processors Build machine images from conﬁguration management compress, upload, etc

Slide 53

Slide 53 text

Packer Template // base-image.json { "description": "optional", "min_packer_version": "optional", "variables": {}, "builders": [], "provisioners": [], "post-processors": [] }

Slide 54

Slide 54 text

Validating the template syntax > packer validate -syntax-only base-image.json Syntax-only check passed. Everything looks okay.

Slide 55

Slide 55 text

User Variables Can be deﬁned from: • The command line • Environment Variables • From a ﬁle @_dcsg https://www.packer.io/docs/templates/user-variables.html

Slide 56

Slide 56 text

User Variables { "variables": { "client_id": "{{env `CLIENT_ID`}}", "my_var": "" }, "builders": [ { "type": "azure-arm", "client_id": "{{ user `client_id` }}", "…": "…" } ] }

Slide 57

Slide 57 text

Building using variables and Env vars > CLIENT_ID=ID packer build -var ‘my_var=foo' \  base-image.json

Slide 58

Slide 58 text

Building using variables and Env vars > packer build -var-file=vars.json base-image.json # vars.json { “client_id": "foo", "my_var": "bar" }

Slide 59

Slide 59 text

Builders @_dcsg https://www.packer.io/docs/builders/index.html

Slide 60

Slide 60 text

"builders": [ { "type": "docker", "image": "ansible/ubuntu14.04-ansible", "export_path": "out/base-image.tar" }, { "type": "virtualbox-ovf", "source_path": "vagrant-base-box/box.ovf", "ssh_username": "vagrant", "ssh_password": "vagrant" } ] Builders

Slide 61

Slide 61 text

Run packer using only docker Builder > packer build -only=docker base-image.json > packer build -parallel=true base-image.json Run packer using both Builders in Parallel

Slide 62

Slide 62 text

Provisioners @_dcsg https://www.packer.io/docs/provisioners/index.html

Slide 63

Slide 63 text

Provisioners "provisioners": [ { "type": “shell-local“, "command": “tar -zcvf ./toupload/app.tar.gz ../app” }, { "type": “file", "source": “./toupload", "destination": "/tmp/" }, { "type": "ansible", "playbook_file": "ansible/site.yml" } ]

Slide 64

Slide 64 text

Post-Processors @_dcsg https://www.packer.io/docs/post-processors/index.html

Slide 65

Slide 65 text

"post-processors": [ { "type": "vagrant", "output": "out/my-app.box", "only": ["virtualbox-ovf"] }, [ { "type": "docker-import", "repository": "dcsg/my-app", "tag": "0.1", "only": ["docker"] }, { "type": "docker-push", "login": true, "login_username": "{{user `docker_hub_user`}}", "login_password": "{{user `docker_hub_password`}}", "only": ["docker"] } ] ]

Slide 66

Slide 66 text

Demo @_dcsg https://github.com/dcsg/build-immutable-servers-packer-ansible

Slide 67

Slide 67 text

Questions   Twitter: @_dcsg Blog: dcsg.me github.com/dcsg/build-immutable-servers-packer-ansible   ? @_dcsg #MSOSCAMP we are hiring!

Slide 68

Slide 68 text

Thanks! @_dcsg #MSOSCAMP