Using Vagrant, Docker, & Terraform for environments

Talk given at HashiConf 2015 on 9/28/2015 in Portland, Oregon.

This was a 50 minute talk centered around how Electric Cloud identified certain problems, came up with requirements and found solutions.

There was a focus on how to gather consensus within an organization and how I would explain these technologies to others.

Transcript

1. © Electric Cloud | electric-cloud.com Nikhil Vaze | @therealnikhil | HashiConf 2015 USING VAGRANT, DOCKER AND TERRAFORM TO STREAMLINE YOUR DEV AND DEMO ENVIRONMENTS
2. © Electric Cloud | electric-cloud.com Introductions Nikhil Vaze https://github.com/nikhilv @therealnikhil Senior Software Engineer at Electric Cloud Full stack engineer and loves to hack on things. Johns Hopkins University Alum
3. © Electric Cloud | electric-cloud.com I love tools that make my life easier! Vagrant Docker Terraform
4. © Electric Cloud | electric-cloud.com DEV OPS Electric Cloud: Powering Continuous Delivery ElectricFlow Deploy Pipeline Management and Visibility ElectricAccelerator Build & Test Acceleration DevOps Automation Platform (ElectricFlow) Build DELIVERY Plugs right in to your existing tools CONTINUOUS Build Test
5. © Electric Cloud | electric-cloud.com I spend my time on integrations Source Control, Defect Tracking, Databases, Clouds, Testing tools , Application Servers….
6. © Electric Cloud | electric-cloud.com The dark days before Vagrant... 2010-2013
7. © Electric Cloud | electric-cloud.com Our situation in 2010 • One engineering site (Sunnyvale, CA) • All engineers were in one room, on the same schedule. • Code reviews were done informally by calling someone over to your desk. • Engineers documented setup on wiki. Other employees expected to follow.
8. © Electric Cloud | electric-cloud.com Show of hands Who has been burned following a wiki document?
9. © Electric Cloud | electric-cloud.com How to start a code review in 2010?
10. © Electric Cloud | electric-cloud.com The company grows - 2012 • We were lucky enough to grow. • Added three new sites in three different time zones. • New employees? Read this wiki!
11. © Electric Cloud | electric-cloud.com “I tried reading the wiki…” • Information out of date • Incomplete information • Couldn’t find it • Person who can answer the question to unblock you is asleep.
12. © Electric Cloud | electric-cloud.com No Problem! I’ll just update the wiki
13. © Electric Cloud | electric-cloud.com How to install our product on AWS EC2?
14. © Electric Cloud | electric-cloud.com “I tried reading the wiki…” • Information out of date • Incomplete information • Couldn’t find it • Person who can answer the question to unblock you is asleep.
15. © Electric Cloud | electric-cloud.com What were the problems? • Wiki was out of date. • Our software was getting harder to setup.  For example: we introduced support for clustering • Found ourselves spending more time troubleshooting individual environments instead of product issues.
16. © Electric Cloud | electric-cloud.com After spending a day on a configuration issue
17. © Electric Cloud | electric-cloud.com There has to be a better way ~ 2013 How do we create development environments such that they can be: • reconstructed • shared • local to each tester • deployed into an automated test environment
18. © Electric Cloud | electric-cloud.com Side note: Cultural Observation • What worked well and got you this far may not be the thing to get you the next level.  It’s hard because the people suffering may be new to the organization  It’s very natural to think “But it has worked for us so well!”
19. © Electric Cloud | electric-cloud.com Partly Cloudy Days: Adopting better tooling ~2013
20. © Electric Cloud | electric-cloud.com “If it hurts, do it more frequently, and bring the pain forward.” Jez Humble
21. © Electric Cloud | electric-cloud.com Bringing the pain forward: • It hurts to construct these environments so….let’s do it more often! • How can we run through the process regularly? • How can we have documentation that is always up to date?
22. © Electric Cloud | electric-cloud.com Introducing Vagrant  https://www.vagrantup.com/  Open Source  Made by HashiCorp $ vagrant up (start or create the VM) $ vagrant ssh (ssh to the VM) $ vagrant halt (stop the VM) $ vagrant destroy (destroy the VM) $ vagrant package (snapshot the VM)
23. © Electric Cloud | electric-cloud.com Vagrant up
24. © Electric Cloud | electric-cloud.com Vagrant ssh
25. © Electric Cloud | electric-cloud.com Vagrant status
26. © Electric Cloud | electric-cloud.com Vagrant destroy
27. © Electric Cloud | electric-cloud.com So…. I know what you’re all thinking “Big deal. You just described VMs”
28. © Electric Cloud | electric-cloud.com Step 1: Vanilla Vagrantfile $ vagrant up
29. © Electric Cloud | electric-cloud.com Step 1.5: Vanilla Vagrantfile $ vagrant up
30. © Electric Cloud | electric-cloud.com Step 2: Install software within Vagrantfile
31. © Electric Cloud | electric-cloud.com Step 2: Install software within Vagrantfile
32. © Electric Cloud | electric-cloud.com What problems did Vagrant solve? • Simple commands to instantiate and destroy • Documentation via code inside Vagrantfile (or shell script invoked by Vagrantfile) • Check in Vagrantfile. Now available to all developers • Each developer can run Vagrant on their laptop • Same process can be used in automated testing
33. © Electric Cloud | electric-cloud.com How do I get Vagrant and what do I need to use it? • Install Vagrant on each machine where vagrant will be used. • Install virtualization provider (VirtualBox). • Supported Host Operating Systems:  Mac OSX  Windows  Linux (Debian, RedHat) • Vagrant  http://downloads.vagrantup.com/ • Oracle VirtualBox  https://www.virtualbox.org/wiki/Downloads
34. © Electric Cloud | electric-cloud.com Vagrant Tips • Your relationship with VirtualBox will be love/hate • At a high level, Vagrant uses Ruby to wrap VBoxManage. So if you are running into any issues you can open up the VirtualBox UI or instrument the ruby code (add puts/print statements) • As with all Open Source: read the source, check out the open issues. https://github.com/mitchellh/vagrant/ • Stackoverflow also helpful
35. © Electric Cloud | electric-cloud.com Side note: Cultural Observation • When trying to gather consensus or persuade an organization to adopt change:  Identify a problem, formulate a hypothesis and then look for a tool/solution. • Avoid  “Oooh shiny technology!”  “All the cool kids are doing it” • Even if you have followed the problem solving steps people may still feel like you’re just chasing technology.
36. © Electric Cloud | electric-cloud.com Real emails showing Vagrant adoption
37. © Electric Cloud | electric-cloud.com Next use case: Multi App Server, Multi Host Deployments
38. © Electric Cloud | electric-cloud.com Use case : Creating a demo environment  Jboss/Wildfly  Domain mode, 3 machines  Business on the line! Fast fast fast!
39. © Electric Cloud | electric-cloud.com Diagram
40. © Electric Cloud | electric-cloud.com Requirements for environment creation  Definition based  Reproducible  Shareable  Each invocation must be unique to an individual
41. © Electric Cloud | electric-cloud.com Cue Terraform
42. © Electric Cloud | electric-cloud.com Introducing Terraform  https://terraform.io/  Open Source  Made by HashiCorp $ terraform plan $ terraform apply $ terraform destroy
43. © Electric Cloud | electric-cloud.com Let’s read some code! https://gist.github.com/nikhilv/74a107af7e44866e3f14 Note: these days terraform’s doc have improved. A lot of this is available in the aws-two-tier example https://github.com/hashicorp/terraf orm/tree/master/examples/aws-two- tier
44. © Electric Cloud | electric-cloud.com Plan $ terraform plan -var 'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
45. © Electric Cloud | electric-cloud.com Terraform Graph $ terraform graph | dot –Tpng > graph.png
46. © Electric Cloud | electric-cloud.com Apply $ terraform apply -var 'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
47. © Electric Cloud | electric-cloud.com Apply $ terraform apply -var 'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
48. © Electric Cloud | electric-cloud.com Destroy $ terraform destroy -var 'name=nikhilDemo' -var 'key_name=<keyName>' –var 'key_path=<path>'
49. © Electric Cloud | electric-cloud.com Review: Requirements for environment creation  Definition based .tf files  Reproducible Plan -> Apply – Destroy.  Shareable Check into source control  Each invocation must be unique to an individual Parametrized scripts
50. © Electric Cloud | electric-cloud.com What problems did Terraform solve? • Simple commands to instantiate and destroy cloud VMs • Documentation via code inside Terraform files • Check in Terraform files. Now available to all developers • Each developer can run Terraform on their laptop • Save those UI clicks! Write it once in Terraform and build on it later
51. © Electric Cloud | electric-cloud.com Terraform Tips • Super easy to install and upgrade (directory of executables) • Caution: Using Terraform will cost you money with your cloud provider! • Some syntax is not intuitive. But the documentation has improved. • Parameterize and use the graph • IMO: Good for demo and short term work. But I wouldn’t use it alone to deploy to production
52. © Electric Cloud | electric-cloud.com Next use case: Integrated Software Delivery Scenario
53. © Electric Cloud | electric-cloud.com Integrated Software Delivery Scenario
54. © Electric Cloud | electric-cloud.com Delivery Pipeline Commit Pipelines
55. © Electric Cloud | electric-cloud.com Interlude: Docker overview
56. © Electric Cloud | electric-cloud.com Docker • https://docker.io/ • Open Source • Used to wrap LXC, now has its own container implementation • Really, really fast. Start measuring in milliseconds
57. © Electric Cloud | electric-cloud.com Docker 102 – Different drivers
58. © Electric Cloud | electric-cloud.com Docker 102 containers vs. VMs
59. © Electric Cloud | electric-cloud.com Back to the use case
60. © Electric Cloud | electric-cloud.com Use case : Simulating ‘Big Data Software Team’  Pieces  Front End  Dashing Dashboard  Database  Redis  Backend  Ratpack Server
61. © Electric Cloud | electric-cloud.com Use case : Simulating ‘Big Data Software Team’ Communication paths  ELB-> instance(s)  Backend -> DB  Frontend -> Backend  IoT -> ELB / Backend  Mobile Device -> ELB / Backend
62. © Electric Cloud | electric-cloud.com Use case : Simulating ‘Big Data Software Team’  How does the Front end know the IP the backend gets spawned at?  How does the Back end know the IP the database gets spawned at?  How does the ELB know about the two availability zones? What is the criteria for the health check?
63. © Electric Cloud | electric-cloud.com Technologies used  Didn’t use Terraform  Regretted it. All my setup is now bespoke and won’t be easy to re-use  Lost: ELB (Health Check, Port forward and instances), VPC, Subnets, EC2 instance data  Decided Docker would be the way to go from “dev” to “production”  Easy to simulate pieces locally on dev laptop and then run same thing in AWS.  Was easy to share with others working on the same demo  Feel confident that I could go back to the source for a future demo
64. © Electric Cloud | electric-cloud.com “Demo”
65. © Electric Cloud | electric-cloud.com Pictures because I didn’t want to bring this on a plane
66. © Electric Cloud | electric-cloud.com Docker Tips • Where do you save, version and test your containers? • Your disk usage will skyrocket. One reason Docker is fast is because it caches intermediate layers. • As with all Open Source: read the source, check out the open issues. https://github.com/docker/docker • Check out Greg Osuri’s guide on Guide to automating a multi- tiered application securely on AWS with Docker and Terraform https://www.airpair.com/aws/posts/ntiered-aws-docker-terraform-guide
67. © Electric Cloud | electric-cloud.com Wrap up
68. © Electric Cloud | electric-cloud.com Vagrant, Terraform, Docker FTW! • It’s an exciting time to be in software • Warning bells should go off if you keep clicking on a UI. • Organization and structure increase velocity and confidence • Drift is minimized if the ‘documentation’ is exercised regularly • You will find bugs or areas where documentation could be improved
69. © Electric Cloud | electric-cloud.com Thank you Networking Other Systems ISV, Internet & Entertainment Heavy Industry Financial Services Automotive Medical Devices Semiconductor Aerospace / Defense Mobile Devices Q&A