Build Immutable Servers with Packer & Ansible @ Microsoft OS Camp PT

Slides of my talk about "Build Immutable Servers with Packer & Ansible" at Microsoft Open-Source Camp in Lisbon, Portugal.

The code samples can be found here: https://github.com/dcsg/build-immutable-servers-packer-ansible

Transcript

1. Build
   Immutable Servers
   with Packer & Ansible
   Daniel Gomes
   #MSOSCAMP @_dcsg
   

   View Slide

2. About me
   • Senior Software Engineer at Talkdesk
   • Co-founder of phplx
   • DevOps Enthusiast
   • @_dcsg
   • dcsg.me
   

   View Slide

3. Once upon a time…
   @_dcsg
   #MSOSCAMP
   

   View Slide

4. A happy team
   @_dcsg
   #MSOSCAMP
   

   View Slide

5. Delivers new features in a high pace
   @_dcsg
   #MSOSCAMP
   0
   35
   70
   105
   140
   April May June July
   

   View Slide

6. Build/Deploy Processes
   are manual and time expensive
   @_dcsg
   #MSOSCAMP
   

   View Slide

7. Deploy
   1 per week
   @_dcsg
   #MSOSCAMP
   

   View Slide

8. The cluster
   4 instances
   @_dcsg
   #MSOSCAMP
   

   View Slide

9. But… problems happen!
   @_dcsg
   #MSOSCAMP
   https://http.cat/500
   

   View Slide

10. Instance degraded
    Out of memory issues
    @_dcsg
    #MSOSCAMP
    

    View Slide

11. SSH’s in production machine and live fixes are done
    @_dcsg
    #MSOSCAMP
    

    View Slide

12. Problem Solved
    @_dcsg
    #MSOSCAMP
    

    View Slide

13. The new cluster state
    Fixed Instance
    Instances

    not in same state
    @_dcsg
    #MSOSCAMP
    

    View Slide

14. Configuration Drift
    • Manual ad-hoc changes and updates to servers that
    are not recorded
    • Servers in your infrastructure became more and more
    different from each others
    @_dcsg
    #MSOSCAMP
    

    View Slide

15. Snowflake Server
    • Long running servers
    • Difficult to reproduce
    • No consistency between servers
    • Lack of confidence in your systems
    • Hard to spin up another instance in the same state
    @_dcsg
    https://martinfowler.com/bliki/SnowflakeServer.html
    

    View Slide

16. The cluster state
    after a few weeks
    State A
    @_dcsg
    #MSOSCAMP
    State C
    State B
    State B
    

    View Slide

17. How can we solve this?
    @_dcsg
    #MSOSCAMP
    

    View Slide

18. Configuration Management
    with Automation Tools
    @_dcsg
    #MSOSCAMP
    

    View Slide

19. Configuration Management
    The process of systematically
    handling changes to a system in a
    way that it maintains integrity over
    time.
    @_dcsg
    #MSOSCAMP
    

    View Slide

20. Configuration Management
    Spin up
    Base Image
    Run
    Config Management
    Server
    in desire state
    Changes
    Edit config file
    Upgrade/install package
    …
    Running Server
    

    View Slide

21. Applying
    to the cluster state
    State A
    @_dcsg
    #MSOSCAMP
    State C
    State B
    State B
    Config
    Management
    Apply
    State D
    

    View Slide

22. The cluster state
    after CM ran
    State D
    @_dcsg
    #MSOSCAMP
    State D
    State D
    State D
    

    View Slide

23. Automation Tools for CM
    

    View Slide

24. Ansible
    @_dcsg
    https://www.ansible.com/
    

    View Slide

25. How Ansible works?
    • Agent-less by operating over SSH
    • Ad-hoc commands execution
    • Playbooks
    @_dcsg
    #MSOSCAMP
    

    View Slide

26. Ansible “components”
    • Inventory of hosts - your raw materials (servers)
    • Modules - your tools (apt, yum, etc)
    • Playbooks - your instructions manual (executes tasks)
    @_dcsg
    #MSOSCAMP
    

    View Slide

27. Ansible Inventory - Hosts & Groups
    # /my_project/hosts
    mail.example.com
    [webservers]
    foo.example.com
    bar.example.com
    [dbservers]
    one.example.com
    two.example.com
    

    View Slide

28. Ansible Modules
    - — name: Install vim

    apt: pkg=vim state=latest
    - — name: reboot the server

    command: /sbin/reboot -t now
    

    View Slide

29. Ansible Playbook “components”
    • Hosts
    • Variables, Conditions, Loops
    • Templating (Jinja2)
    • Tasks
    • Handlers
    • Roles
    @_dcsg
    http://docs.ansible.com/ansible/playbooks.html
    

    View Slide

30. ---
    - hosts: webservers
    vars:
    http_port: 80
    remote_user: root
    tasks:
    - name: write the apache config file
    template: src=/srv/httpd.j2 dest=/etc/httpd.conf
    notify:
    - restart apache
    - name: ensure apache is running (and enable it at boot)
    service: name=httpd state=started enabled=yes
    handlers:
    - name: restart apache
    service: name=httpd state=restarted
    Ansible Variables, Tasks, Handlers
    

    View Slide

31. Ansible Conditions & Loops
    - name: add several users
    user:
    name: "{{ item.name }}"
    state: present
    groups: "{{ item.groups }}"
    with_items:
    - { name: 'testuser1', groups: 'wheel' }
    - { name: 'testuser2', groups: 'root' }
    when:
    - ansible_distribution == "CentOS"
    

    View Slide

32. Ansible Roles Example
    # Folder structure
    roles/
    common/
    files/
    templates/
    tasks/
    main.yml
    handlers/
    vars/
    defaults/
    meta/
    # roles/common/tasks/main.yml
    - name: install common packages
    yum:
    name: "{{ item }}"
    state: present
    with_items:
    - htop
    - vim
    

    View Slide

33. Recap
    • Configuration Drift
    • Problems of the Snowflake Servers
    • Configuration Management
    • Ansible
    @_dcsg
    #MSOSCAMP
    

    View Slide

34. Phoenix Servers
    @_dcsg
    https://martinfowler.com/bliki/PhoenixServer.html
    

    View Slide

35. “A server should be like a
    phoenix, regularly rising
    from the ashes.”
    - Martin Fowler in PhoenixServer
    @_dcsg
    #MSOSCAMP
    

    View Slide

36. Phoenix Servers
    • Avoid configuration drifts
    • Disposable servers
    • Servers can be built from scratch
    @_dcsg
    #MSOSCAMP
    

    View Slide

37. The cluster
    Re-launch an instance
    State D
    @_dcsg
    #MSOSCAMP
    State D State D
    State D
    Terminate 

    Instance
    Config
    Management
    Apply
    State D
    State D
    

    View Slide

38. Spinning up new servers
    are not a problem anymore!
    @_dcsg
    #MSOSCAMP
    

    View Slide

39. But…
    @_dcsg
    #MSOSCAMP
    

    View Slide

40. Is Idempotence guaranteed?
    @_dcsg
    #MSOSCAMP
    

    View Slide

41. What if the packages
    repositories are down?
    @_dcsg
    #MSOSCAMP
    

    View Slide

42. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    Repositories
    unavailable
    Package not locked
    to specific version
    

    View Slide

43. How can we fix this?
    @_dcsg
    #MSOSCAMP
    

    View Slide

44. Immutable Servers
    @_dcsg
    https://martinfowler.com/bliki/ImmutableServer.html
    

    View Slide

45. “An Immutable Server is a
    server, that once deployed, is
    never modified, merely
    replaced with a new updated
    instance.”
    - Kief Morris in ImmutableServer
    @_dcsg
    #MSOSCAMP
    

    View Slide

46. Immutable Servers
    • Final state image with everything baked in.
    • No changes after it’s built.
    • Include scripts to start the application at boot.
    • Easy to scale out, deploy and rollback
    • Trustable and testable
    • Easy to adopt A/B testing, Canary releases or Blue/Green
    deployments
    @_dcsg
    #MSOSCAMP
    

    View Slide

47. Immutable Server Build Process
    Run
    Config Management
    (puppet, chef, ansible)
    Bake In
    the App
    App
    Final Image
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    

    View Slide

48. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    Application
    Base Image
    Install necessary
    software to run the App
    Create user/folders
    Application
    Final Image
    Upload App
    Script to run App at boot
    System upgrades &
    Security updates
    Application security,
    package, configuration
    updates
    

    View Slide

49. • SnowFlake Servers
    • Inconsistent states between machine (Config Drifts)
    • Phoenix Serves
    • Avoids Config Drifts using CM Automation Tools
    • Can be built from scratch
    • Immutable Serves
    • Final image with everything baked in
    • After built cannot be modified
    • Can only be replaced with an updated instance
    Recap
    

    View Slide

50. Hello Packer
    @_dcsg
    https://www.packer.io/
    

    View Slide

51. “Packer is an open source tool for creating
    identical machine images for multiple
    platforms from a single source configuration.”
    What’s Packer?
    @_dcsg
    #MSOSCAMP
    

    View Slide

52. Packer components
    Builders Provisioners Post-Processors
    Build machine images from configuration management compress, upload, etc
    

    View Slide

53. Packer Template
    // base-image.json
    {
    "description": "optional",
    "min_packer_version": "optional",
    "variables": {},
    "builders": [],
    "provisioners": [],
    "post-processors": []
    }
    

    View Slide

54. Validating the template syntax
    > packer validate -syntax-only base-image.json
    Syntax-only check passed. Everything looks okay.
    

    View Slide

55. User Variables
    Can be defined from:
    • The command line
    • Environment Variables
    • From a file
    @_dcsg
    https://www.packer.io/docs/templates/user-variables.html
    

    View Slide

56. User Variables
    {
    "variables": {
    "client_id": "{{env `CLIENT_ID`}}",
    "my_var": ""
    },
    "builders": [
    {
    "type": "azure-arm",
    "client_id": "{{ user `client_id` }}",
    "…": "…"
    }
    ]
    }
    

    View Slide

57. Building using variables and Env vars
    > CLIENT_ID=ID packer build -var ‘my_var=foo' \

    base-image.json
    

    View Slide

58. Building using variables and Env vars
    > packer build -var-file=vars.json base-image.json
    # vars.json
    {
    “client_id": "foo",
    "my_var": "bar"
    }
    

    View Slide

59. Builders
    @_dcsg
    https://www.packer.io/docs/builders/index.html
    

    View Slide

60. "builders": [
    {
    "type": "docker",
    "image": "ansible/ubuntu14.04-ansible",
    "export_path": "out/base-image.tar"
    },
    {
    "type": "virtualbox-ovf",
    "source_path": "vagrant-base-box/box.ovf",
    "ssh_username": "vagrant",
    "ssh_password": "vagrant"
    }
    ]
    Builders
    

    View Slide

61. Run packer using only docker Builder
    > packer build -only=docker base-image.json
    > packer build -parallel=true base-image.json
    Run packer using both Builders in Parallel
    

    View Slide

62. Provisioners
    @_dcsg
    https://www.packer.io/docs/provisioners/index.html
    

    View Slide

63. Provisioners
    "provisioners": [
    {
    "type": “shell-local“,
    "command": “tar -zcvf ./toupload/app.tar.gz ../app”
    },
    {
    "type": “file",
    "source": “./toupload",
    "destination": "/tmp/"
    },
    {
    "type": "ansible",
    "playbook_file": "ansible/site.yml"
    }
    ]
    

    View Slide

64. Post-Processors
    @_dcsg
    https://www.packer.io/docs/post-processors/index.html
    

    View Slide

65. "post-processors": [
    {
    "type": "vagrant",
    "output": "out/my-app.box",
    "only": ["virtualbox-ovf"]
    },
    [
    {
    "type": "docker-import",
    "repository": "dcsg/my-app",
    "tag": "0.1",
    "only": ["docker"]
    },
    {
    "type": "docker-push",
    "login": true,
    "login_username": "{{user `docker_hub_user`}}",
    "login_password": "{{user `docker_hub_password`}}",
    "only": ["docker"]
    }
    ]
    ]
    

    View Slide

66. Demo
    @_dcsg
    https://github.com/dcsg/build-immutable-servers-packer-ansible
    

    View Slide

67. Questions
    

    Twitter: @_dcsg
    Blog: dcsg.me
    github.com/dcsg/build-immutable-servers-packer-ansible
    

    ?
    @_dcsg
    #MSOSCAMP
    we are hiring!
    

    View Slide

68. Thanks!
    @_dcsg
    #MSOSCAMP
    

    View Slide