Build Immutable Servers with Packer & Ansible @ PixelsCamp 2017

Slides of my talk about "Build Immutable Servers with Packer & Ansible" at Pixels Camp in Lisbon, Portugal.

The code samples can be found here: https://github.com/dcsg/build-immutable-servers-packer-ansible

Transcript

1. Build
   Immutable Servers
   with Packer & Ansible
   Daniel Gomes
   #PixelsCamp @_dcsg
   

   View Slide

2. About me
   • Team Lead at Talkdesk
   • @phplx co-founder
   • @_dcsg
   • dcsg.me
   Daniel Gomes
   

   View Slide

3. Once upon a time…
   @_dcsg
   #PixelsCamp
   

   View Slide

4. A happy team
   @_dcsg
   #PixelsCamp
   

   View Slide

5. Delivers new features in a high pace
   @_dcsg
   #PixelsCamp
   0
   35
   70
   105
   140
   April May June July
   

   View Slide

6. Build/Deploy Processes
   are manual and time expensive
   @_dcsg
   #PixelsCamp
   

   View Slide

7. Deploy
   1 per week
   @_dcsg
   #PixelsCamp
   

   View Slide

8. The cluster
   4 instances
   @_dcsg
   #PixelsCamp
   

   View Slide

9. But… problems happen!
   @_dcsg
   #PixelsCamp
   https://http.cat/500
   

   View Slide

10. Instance degraded
    Out of memory issues
    @_dcsg
    #PixelsCamp
    

    View Slide

11. SSH’s in production machine and live fixes are done
    @_dcsg
    #PixelsCamp
    

    View Slide

12. Problem Solved
    @_dcsg
    #PixelsCamp
    

    View Slide

13. The new cluster state
    Fixed Instance
    Instances

    not in same state
    @_dcsg
    #PixelsCamp
    

    View Slide

14. Configuration Drift
    @_dcsg
    #PixelsCamp
    

    View Slide

15. Configuration Drift
    • Manual ad-hoc changes and updates to servers that
    are not recorded
    • Servers in your infrastructure became more and more
    different from each others
    @_dcsg
    #PixelsCamp
    

    View Slide

16. The cluster state
    after a few weeks
    State A
    @_dcsg
    #PixelsCamp
    State C
    State B
    State B
    

    View Slide

17. Snowflake Server
    @_dcsg
    #PixelsCamp
    

    View Slide

18. Snowflake Server
    • Long running servers
    • Difficult to reproduce
    • No consistency between servers
    • Lack of confidence in your systems
    • Hard to spin up another instance in the same state
    @_dcsg
    https://martinfowler.com/bliki/SnowflakeServer.html
    

    View Slide

19. How can we solve this?
    @_dcsg
    #PixelsCamp
    

    View Slide

20. Configuration Management
    with Automation Tools
    @_dcsg
    #PixelsCamp
    

    View Slide

21. Configuration Management
    The process of systematically handling
    changes to a system in a way that it
    maintains integrity over time.
    @_dcsg
    #PixelsCamp
    

    View Slide

22. Configuration Management
    Spin up
    Base Image
    Run
    Config Management
    Server
    in desired state
    Changes
    Edit config file
    Upgrade/install package
    …
    Running Server
    

    View Slide

23. Applying
    to the cluster state
    State A
    @_dcsg
    #PixelsCamp
    State C
    State B
    State B
    Config
    Management
    Apply
    State D
    

    View Slide

24. The cluster state
    after CM ran
    State D
    @_dcsg
    #PixelsCamp
    State D
    State D
    State D
    

    View Slide

25. Automation Tools for CM
    

    View Slide

26. Ansible
    @_dcsg
    https://www.ansible.com/
    

    View Slide

27. How does Ansible work?
    • Agent-less by operating over SSH
    • Ad-hoc commands execution
    • Playbooks
    @_dcsg
    #PixelsCamp
    

    View Slide

28. Ansible terminology
    • Inventory of hosts - your raw materials (servers)
    • Modules - your tools (apt, yum, etc)
    • Playbooks - your instructions manual (executes tasks)
    @_dcsg
    #PixelsCamp
    

    View Slide

29. Let’s see the code
    @_dcsg
    https://github.com/dcsg/build-immutable-servers-packer-ansible
    

    View Slide

30. Quick recap
    • Configuration Drift
    • Problems of the Snowflake Servers
    • Configuration Management
    • Ansible
    @_dcsg
    #PixelsCamp
    

    View Slide

31. Phoenix Servers
    @_dcsg
    https://martinfowler.com/bliki/PhoenixServer.html
    

    View Slide

32. “A server should be like a
    phoenix, regularly rising
    from the ashes.”
    - Martin Fowler in PhoenixServer
    @_dcsg
    #PixelsCamp
    

    View Slide

33. Phoenix Servers
    • Avoid configuration drifts
    • Disposable servers
    • Servers can be built from scratch
    @_dcsg
    #PixelsCamp
    

    View Slide

34. The cluster
    Re-launch an instance
    State D
    @_dcsg
    #PixelsCamp
    State D State D
    State D
    Terminate 

    Instance
    Config
    Management
    Apply
    State D
    State D
    

    View Slide

35. Spinning up new servers
    is not a problem anymore!
    @_dcsg
    #PixelsCamp
    

    View Slide

36. But…
    @_dcsg
    #PixelsCamp
    

    View Slide

37. Is Idempotence guaranteed?
    @_dcsg
    #PixelsCamp
    

    View Slide

38. What if the packages
    repositories are down?
    @_dcsg
    #PixelsCamp
    

    View Slide

39. Built Process
    Spin up
    Base Image Run
    Config Management
    Server
    in desire state
    Install packages
    Create folders
    Create user
    Upload app
    etc
    Run
    Config Management
    Repositories
    unavailable
    

    View Slide

40. How can we fix this?
    @_dcsg
    #PixelsCamp
    

    View Slide

41. Immutable Servers
    @_dcsg
    https://martinfowler.com/bliki/ImmutableServer.html
    

    View Slide

42. "An Immutable Server is a
    server, that once deployed, is
    never modified, merely
    replaced with a new updated
    instance."
    - Kief Morris in ImmutableServer
    @_dcsg
    #PixelsCamp
    

    View Slide

43. Immutable Servers
    • Final state image with everything baked in.
    • No changes after it’s built.
    • Include scripts to start the application at boot.
    • Easy to scale out, deploy and rollback.
    • Trustable and testable.
    • Easy to adopt A/B testing, Canary releases or Blue/Green
    deployments.
    @_dcsg
    #PixelsCamp
    

    View Slide

44. Immutable Server Build Process
    Bake In
    the App
    App
    Final Image
    Server
    in desire state
    configure application

    environment
    Spin up
    Base Image
    Run
    Config Management
    (puppet, chef, ansible)
    

    View Slide

45. Build Image Stages Flow Example
    Base Image
    OS Hardening
    Common tools

    (vim, htop, etc)
    etc
    Application
    Base Image
    Install necessary
    software to run the App
    Create user/folders
    Application
    Final Image
    Upload App
    Script to run App at boot
    System upgrades &
    Security updates
    Application security,
    package, configuration
    updates
    

    View Slide

46. • SnowFlake Servers
    • Inconsistent states between machine (Config Drifts)
    • Phoenix Servers
    • Avoids Config Drifts using CM Automation Tools
    • Can be built from scratch
    • Immutable Servers
    • Final image with everything baked in
    • After built cannot be modified
    • Can only be replaced with an updated instance
    Recap
    

    View Slide

47. How can we build those machines?
    @_dcsg
    #PixelsCamp
    

    View Slide

48. Hello Packer
    @_dcsg
    https://www.packer.io/
    

    View Slide

49. What’s Packer?
    • Tool for creating identical machine images
    • For multiple platforms and in Parallel
    @_dcsg
    https://www.packer.io/
    

    View Slide

50. Packer terminology
    • Artifact
    • Builders
    • Builds
    • Commands
    • Post-processors
    • Provisioners
    • Templates
    @_dcsg
    https://www.packer.io/
    

    View Slide

51. Packer flow
    Builders Provisioners Post-Processors
    Build machine images from configuration management compress, upload, etc
    

    View Slide

52. Demo
    @_dcsg
    https://github.com/dcsg/build-immutable-servers-packer-ansible
    

    View Slide

53. Questions
    

    Twitter: @_dcsg
    Blog: dcsg.me
    github.com/dcsg/build-immutable-servers-packer-ansible
    

    ?
    @_dcsg
    #PixelsCamp
    we are hiring!
    

    View Slide

54. Thanks!
    @_dcsg
    #PixelsCamp
    

    View Slide