Using Vagrant, Docker, & Terraform for environments

© Electric Cloud | electric-cloud.com
Nikhil Vaze | @therealnikhil | HashiConf 2015
USING VAGRANT, DOCKER AND TERRAFORM
TO STREAMLINE YOUR DEV AND DEMO
ENVIRONMENTS

View Slide

Introductions
Nikhil Vaze
https://github.com/nikhilv
@therealnikhil
Senior Software Engineer at Electric Cloud
Full stack engineer and loves to hack on things.
Johns Hopkins University Alum

View Slide

I love tools that make my life easier!
Vagrant
Docker
Terraform

View Slide

DEV
OPS
Electric Cloud: Powering Continuous Delivery
ElectricFlow
Deploy
Pipeline Management and Visibility
ElectricAccelerator
Build & Test Acceleration
DevOps Automation Platform (ElectricFlow)
Build
DELIVERY
Plugs right in to
your existing tools
CONTINUOUS
Build Test

View Slide

I spend my time on integrations
Source Control, Defect Tracking, Databases,
Clouds, Testing tools , Application Servers….

View Slide

The dark days before
Vagrant...
2010-2013

View Slide

Our situation in 2010
• One engineering site (Sunnyvale, CA)
• All engineers were in one room, on the
same schedule.
• Code reviews were done informally by
calling someone over to your desk.
• Engineers documented setup on wiki.
Other employees expected to follow.

View Slide

Show of hands
Who has been
burned following a
wiki document?

View Slide

How to start a code review in 2010?

View Slide

The company grows - 2012
• We were lucky enough
to grow.
• Added three new sites in
three different time
zones.
• New employees? Read
this wiki!

View Slide

“I tried reading the wiki…”
• Information out of date
• Incomplete information
• Couldn’t find it
• Person who can answer
the question to unblock
you is asleep.

View Slide

No Problem! I’ll just update the wiki

View Slide

How to install our product on AWS EC2?

View Slide

“I tried reading the wiki…”
• Information out of date
• Incomplete information
• Couldn’t find it
• Person who can answer
the question to unblock
you is asleep.

View Slide

What were the problems?
• Wiki was out of date.
• Our software was getting harder to
setup.
 For example: we introduced support
for clustering
• Found ourselves spending more time
troubleshooting individual environments
instead of product issues.

View Slide

After spending a day on a configuration issue

View Slide

There has to be a better way ~ 2013
How do we create development
environments such that they can
be:
• reconstructed
• shared
• local to each tester
• deployed into an automated test
environment

View Slide

Side note: Cultural Observation
• What worked well and got you this far
may not be the thing to get you the
next level.
 It’s hard because the people suffering
may be new to the organization
 It’s very natural to think “But it has
worked for us so well!”

View Slide

Partly Cloudy Days: Adopting better
tooling ~2013

View Slide

“If it hurts, do it more
frequently, and bring
the pain forward.”
Jez Humble

View Slide

Bringing the pain forward:
• It hurts to construct these
environments so….let’s do it more
often!
• How can we run through the
process regularly?
• How can we have documentation
that is always up to date?

View Slide

Introducing Vagrant
 https://www.vagrantup.com/
 Open Source
 Made by HashiCorp
$ vagrant up (start or create the VM)
$ vagrant ssh (ssh to the VM)
$ vagrant halt (stop the VM)
$ vagrant destroy (destroy the VM)
$ vagrant package (snapshot the VM)

View Slide

Vagrant up

View Slide

Vagrant ssh

View Slide

Vagrant status

View Slide

Vagrant destroy

View Slide

So…. I know what you’re all thinking
“Big deal. You just
described VMs”

View Slide

Step 1: Vanilla Vagrantfile
$ vagrant up

View Slide

Step 1.5: Vanilla Vagrantfile
$ vagrant up

View Slide

Step 2: Install software within Vagrantfile

View Slide

What problems did Vagrant solve?
• Simple commands to instantiate and destroy
• Documentation via code inside Vagrantfile (or shell script invoked
by Vagrantfile)
• Check in Vagrantfile. Now available to all developers
• Each developer can run Vagrant on their laptop
• Same process can be used in automated testing

View Slide

How do I get Vagrant and what do I need to use it?
• Install Vagrant on each machine where vagrant will be used.
• Install virtualization provider (VirtualBox).
• Supported Host Operating Systems:
 Mac OSX
 Windows
 Linux (Debian, RedHat)
• Vagrant
 http://downloads.vagrantup.com/
• Oracle VirtualBox
 https://www.virtualbox.org/wiki/Downloads

View Slide

Vagrant Tips
• Your relationship with VirtualBox will be love/hate
• At a high level, Vagrant uses Ruby to wrap VBoxManage. So if you
are running into any issues you can open up the VirtualBox UI or
instrument the ruby code (add puts/print statements)
• As with all Open Source: read the source, check out the open
issues. https://github.com/mitchellh/vagrant/
• Stackoverflow also helpful

View Slide

Side note: Cultural Observation
• When trying to gather consensus or persuade an
organization to adopt change:
 Identify a problem, formulate a hypothesis and then
look for a tool/solution.
• Avoid
 “Oooh shiny technology!”
 “All the cool kids are doing it”
• Even if you have followed the problem solving steps
people may still feel like you’re just chasing technology.

View Slide

Real emails showing Vagrant adoption

View Slide

Next use case: Multi App Server,
Multi Host Deployments

View Slide

Use case : Creating a demo environment
 Jboss/Wildfly
 Domain mode, 3 machines
 Business on the line! Fast fast
fast!

View Slide

Diagram

View Slide

Requirements for environment creation
 Definition based
 Reproducible
 Shareable
 Each invocation must be unique to
an individual

View Slide

Cue Terraform

View Slide

Introducing Terraform
 https://terraform.io/
 Open Source
 Made by HashiCorp
$ terraform plan
$ terraform apply
$ terraform destroy

View Slide

Let’s read some code!
https://gist.github.com/nikhilv/74a107af7e44866e3f14
Note: these days terraform’s doc
have improved. A lot of this is
available in the aws-two-tier
example
https://github.com/hashicorp/terraf
orm/tree/master/examples/aws-two-
tier

View Slide

Plan
$ terraform plan -var 'name=nikhilDemo' -var 'key_name=' –var 'key_path='

View Slide

Terraform Graph
$ terraform graph | dot –Tpng > graph.png

View Slide

Apply
$ terraform apply -var 'name=nikhilDemo' -var 'key_name=' –var 'key_path='

View Slide

Destroy
$ terraform destroy -var 'name=nikhilDemo' -var 'key_name=' –var 'key_path='

View Slide

Review: Requirements for environment creation
 Definition based
.tf files
 Reproducible
Plan -> Apply – Destroy.
 Shareable
Check into source control
 Each invocation must be unique to
an individual
Parametrized scripts

View Slide

What problems did Terraform solve?
• Simple commands to instantiate and destroy cloud VMs
• Documentation via code inside Terraform files
• Check in Terraform files. Now available to all developers
• Each developer can run Terraform on their laptop
• Save those UI clicks! Write it once in Terraform and build on it
later

View Slide

Terraform Tips
• Super easy to install and upgrade (directory of executables)
• Caution: Using Terraform will cost you money with your cloud
provider!
• Some syntax is not intuitive. But the documentation has improved.
• Parameterize and use the graph
• IMO: Good for demo and short term work. But I wouldn’t use it
alone to deploy to production

View Slide

Next use case: Integrated Software
Delivery Scenario

View Slide

Integrated Software Delivery Scenario

View Slide

Delivery Pipeline
Commit Pipelines

View Slide

Interlude: Docker overview

View Slide

Docker
• https://docker.io/
• Open Source
• Used to wrap LXC, now has its own container implementation
• Really, really fast. Start measuring in milliseconds

View Slide

Docker 102 – Different drivers

View Slide

Docker 102 containers vs. VMs

View Slide

Back to the use case

View Slide

Use case : Simulating ‘Big Data Software Team’
 Pieces
 Front End
 Dashing Dashboard
 Database
 Redis
 Backend
 Ratpack Server

View Slide

Communication paths
 ELB-> instance(s)
 Backend -> DB
 Frontend -> Backend
 IoT -> ELB / Backend
 Mobile Device -> ELB /
Backend

View Slide

 How does the Front end know
the IP the backend gets
spawned at?
 How does the Back end know
the IP the database gets
spawned at?
 How does the ELB know about
the two availability zones?
What is the criteria for the
health check?

View Slide

Technologies used
 Didn’t use Terraform
 Regretted it. All my setup is now bespoke and won’t be easy to re-use
 Lost: ELB (Health Check, Port forward and instances), VPC, Subnets, EC2
instance data
 Decided Docker would be the way to go from “dev” to “production”
 Easy to simulate pieces locally on dev laptop and then run same thing in
AWS.
 Was easy to share with others working on the same demo
 Feel confident that I could go back to the source for a future demo

View Slide

“Demo”

View Slide

Pictures because I didn’t want to bring this on a plane

View Slide

Docker Tips
• Where do you save, version and test your containers?
• Your disk usage will skyrocket. One reason Docker is fast is
because it caches intermediate layers.
• As with all Open Source: read the source, check out the open
issues. https://github.com/docker/docker
• Check out Greg Osuri’s guide on Guide to automating a multi-
tiered application securely on AWS with Docker and Terraform
https://www.airpair.com/aws/posts/ntiered-aws-docker-terraform-guide

View Slide

Wrap up

View Slide

Vagrant, Terraform, Docker FTW!
• It’s an exciting time to be in software
• Warning bells should go off if you keep
clicking on a UI.
• Organization and structure increase
velocity and confidence
• Drift is minimized if the ‘documentation’
is exercised regularly
• You will find bugs or areas where
documentation could be improved

View Slide

Thank you
Networking
Other Systems
ISV, Internet
& Entertainment
Heavy Industry
Financial
Services
Automotive Medical Devices
Semiconductor
Aerospace /
Defense
Mobile Devices
Q&A

View Slide

Using Vagrant, Docker, & Terraform for environments

Using Vagrant, Docker, & Terraform for environments

Nikhil Vaze

More Decks by Nikhil Vaze

Other Decks in Technology

Featured

Transcript