Build Immutable Servers with Packer & Ansible @ Microsoft OS Camp PT

Slides of my talk about "Build Immutable Servers with Packer & Ansible" at Microsoft Open-Source Camp in Lisbon, Portugal.

The code samples can be found here: https://github.com/dcsg/build-immutable-servers-packer-ansible

Transcript

1. Build Immutable Servers with Packer & Ansible Daniel Gomes #MSOSCAMP @_dcsg
2. About me • Senior Software Engineer at Talkdesk • Co-founder of phplx • DevOps Enthusiast • @_dcsg • dcsg.me
3. Once upon a time… @_dcsg #MSOSCAMP
4. A happy team @_dcsg #MSOSCAMP
5. Delivers new features in a high pace @_dcsg #MSOSCAMP 0 35 70 105 140 April May June July
6. Build/Deploy Processes are manual and time expensive @_dcsg #MSOSCAMP
7. Deploy 1 per week @_dcsg #MSOSCAMP
8. The cluster 4 instances @_dcsg #MSOSCAMP
9. But… problems happen! @_dcsg #MSOSCAMP https://http.cat/500
10. Instance degraded Out of memory issues @_dcsg #MSOSCAMP
11. SSH’s in production machine and live fixes are done @_dcsg #MSOSCAMP
12. Problem Solved @_dcsg #MSOSCAMP
13. The new cluster state Fixed Instance Instances
 not in same state @_dcsg #MSOSCAMP
14. Configuration Drift • Manual ad-hoc changes and updates to servers that are not recorded • Servers in your infrastructure became more and more different from each others @_dcsg #MSOSCAMP
15. Snowflake Server • Long running servers • Difficult to reproduce • No consistency between servers • Lack of confidence in your systems • Hard to spin up another instance in the same state @_dcsg https://martinfowler.com/bliki/SnowflakeServer.html
16. The cluster state after a few weeks State A @_dcsg #MSOSCAMP State C State B State B
17. How can we solve this? @_dcsg #MSOSCAMP
18. Configuration Management with Automation Tools @_dcsg #MSOSCAMP
19. Configuration Management The process of systematically handling changes to a system in a way that it maintains integrity over time. @_dcsg #MSOSCAMP
20. Configuration Management Spin up Base Image Run Config Management Server in desire state Changes Edit config file Upgrade/install package … Running Server
21. Applying to the cluster state State A @_dcsg #MSOSCAMP State C State B State B Config Management Apply State D
22. The cluster state after CM ran State D @_dcsg #MSOSCAMP State D State D State D
23. Automation Tools for CM
24. Ansible @_dcsg https://www.ansible.com/
25. How Ansible works? • Agent-less by operating over SSH • Ad-hoc commands execution • Playbooks @_dcsg #MSOSCAMP
26. Ansible “components” • Inventory of hosts - your raw materials (servers) • Modules - your tools (apt, yum, etc) • Playbooks - your instructions manual (executes tasks) @_dcsg #MSOSCAMP
27. Ansible Inventory - Hosts & Groups # /my_project/hosts mail.example.com [webservers] foo.example.com bar.example.com [dbservers] one.example.com two.example.com
28. Ansible Modules - — name: Install vim
 apt: pkg=vim state=latest - — name: reboot the server
 command: /sbin/reboot -t now
29. Ansible Playbook “components” • Hosts • Variables, Conditions, Loops • Templating (Jinja2) • Tasks • Handlers • Roles @_dcsg http://docs.ansible.com/ansible/playbooks.html
30. --- - hosts: webservers vars: http_port: 80 remote_user: root tasks: - name: write the apache config file template: src=/srv/httpd.j2 dest=/etc/httpd.conf notify: - restart apache - name: ensure apache is running (and enable it at boot) service: name=httpd state=started enabled=yes handlers: - name: restart apache service: name=httpd state=restarted Ansible Variables, Tasks, Handlers
31. Ansible Conditions & Loops - name: add several users user: name: "{{ item.name }}" state: present groups: "{{ item.groups }}" with_items: - { name: 'testuser1', groups: 'wheel' } - { name: 'testuser2', groups: 'root' } when: - ansible_distribution == "CentOS"
32. Ansible Roles Example # Folder structure roles/ common/ files/ templates/ tasks/ main.yml handlers/ vars/ defaults/ meta/ # roles/common/tasks/main.yml - name: install common packages yum: name: "{{ item }}" state: present with_items: - htop - vim
33. Recap • Configuration Drift • Problems of the Snowflake Servers • Configuration Management • Ansible @_dcsg #MSOSCAMP
34. Phoenix Servers @_dcsg https://martinfowler.com/bliki/PhoenixServer.html
35. “A server should be like a phoenix, regularly rising from the ashes.” - Martin Fowler in PhoenixServer @_dcsg #MSOSCAMP
36. Phoenix Servers • Avoid configuration drifts • Disposable servers • Servers can be built from scratch @_dcsg #MSOSCAMP
37. The cluster Re-launch an instance State D @_dcsg #MSOSCAMP State D State D State D Terminate 
 Instance Config Management Apply State D State D
38. Spinning up new servers are not a problem anymore! @_dcsg #MSOSCAMP
39. But… @_dcsg #MSOSCAMP
40. Is Idempotence guaranteed? @_dcsg #MSOSCAMP
41. What if the packages repositories are down? @_dcsg #MSOSCAMP
42. Built Process Spin up Base Image Run Config Management Server in desire state Install packages Create folders Create user Upload app etc Run Config Management Repositories unavailable Package not locked to specific version
43. How can we fix this? @_dcsg #MSOSCAMP
44. Immutable Servers @_dcsg https://martinfowler.com/bliki/ImmutableServer.html
45. “An Immutable Server is a server, that once deployed, is never modified, merely replaced with a new updated instance.” - Kief Morris in ImmutableServer @_dcsg #MSOSCAMP
46. Immutable Servers • Final state image with everything baked in. • No changes after it’s built. • Include scripts to start the application at boot. • Easy to scale out, deploy and rollback • Trustable and testable • Easy to adopt A/B testing, Canary releases or Blue/Green deployments @_dcsg #MSOSCAMP
47. Immutable Server Build Process Run Config Management (puppet, chef, ansible) Bake In the App App Final Image Server in desire state configure application
 environment Spin up Base Image
48. Build Image Stages Flow Example Base Image OS Hardening Common tools
 (vim, htop, etc) etc Application Base Image Install necessary software to run the App Create user/folders Application Final Image Upload App Script to run App at boot System upgrades & Security updates Application security, package, configuration updates
49. • SnowFlake Servers • Inconsistent states between machine (Config Drifts) • Phoenix Serves • Avoids Config Drifts using CM Automation Tools • Can be built from scratch • Immutable Serves • Final image with everything baked in • After built cannot be modified • Can only be replaced with an updated instance Recap
50. Hello Packer @_dcsg https://www.packer.io/
51. “Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration.” What’s Packer? @_dcsg #MSOSCAMP
52. Packer components Builders Provisioners Post-Processors Build machine images from configuration management compress, upload, etc
53. Packer Template // base-image.json { "description": "optional", "min_packer_version": "optional", "variables": {}, "builders": [], "provisioners": [], "post-processors": [] }
54. Validating the template syntax > packer validate -syntax-only base-image.json Syntax-only check passed. Everything looks okay.
55. User Variables Can be defined from: • The command line • Environment Variables • From a file @_dcsg https://www.packer.io/docs/templates/user-variables.html
56. User Variables { "variables": { "client_id": "{{env `CLIENT_ID`}}", "my_var": "" }, "builders": [ { "type": "azure-arm", "client_id": "{{ user `client_id` }}", "…": "…" } ] }
57. Building using variables and Env vars > CLIENT_ID=ID packer build -var ‘my_var=foo' \
 base-image.json
58. Building using variables and Env vars > packer build -var-file=vars.json base-image.json # vars.json { “client_id": "foo", "my_var": "bar" }
59. Builders @_dcsg https://www.packer.io/docs/builders/index.html
60. "builders": [ { "type": "docker", "image": "ansible/ubuntu14.04-ansible", "export_path": "out/base-image.tar" }, { "type": "virtualbox-ovf", "source_path": "vagrant-base-box/box.ovf", "ssh_username": "vagrant", "ssh_password": "vagrant" } ] Builders
61. Run packer using only docker Builder > packer build -only=docker base-image.json > packer build -parallel=true base-image.json Run packer using both Builders in Parallel
62. Provisioners @_dcsg https://www.packer.io/docs/provisioners/index.html
63. Provisioners "provisioners": [ { "type": “shell-local“, "command": “tar -zcvf ./toupload/app.tar.gz ../app” }, { "type": “file", "source": “./toupload", "destination": "/tmp/" }, { "type": "ansible", "playbook_file": "ansible/site.yml" } ]
64. Post-Processors @_dcsg https://www.packer.io/docs/post-processors/index.html
65. "post-processors": [ { "type": "vagrant", "output": "out/my-app.box", "only": ["virtualbox-ovf"] }, [ { "type": "docker-import", "repository": "dcsg/my-app", "tag": "0.1", "only": ["docker"] }, { "type": "docker-push", "login": true, "login_username": "{{user `docker_hub_user`}}", "login_password": "{{user `docker_hub_password`}}", "only": ["docker"] } ] ]
66. Demo @_dcsg https://github.com/dcsg/build-immutable-servers-packer-ansible
67. Questions 
 Twitter: @_dcsg Blog: dcsg.me github.com/dcsg/build-immutable-servers-packer-ansible 
 ? @_dcsg #MSOSCAMP we are hiring!
68. Thanks! @_dcsg #MSOSCAMP