Using Vagrant, Docker, & Terraform for environments

Slide 1

Slide 1 text

Slide 2

Slide 2 text

© Electric Cloud | electric-cloud.com Introductions Nikhil Vaze https://github.com/nikhilv @therealnikhil Senior Software Engineer at Electric Cloud Full stack engineer and loves to hack on things. Johns Hopkins University Alum

Slide 3

Slide 3 text

Slide 4

Slide 4 text

© Electric Cloud | electric-cloud.com DEV OPS Electric Cloud: Powering Continuous Delivery ElectricFlow Deploy Pipeline Management and Visibility ElectricAccelerator Build & Test Acceleration DevOps Automation Platform (ElectricFlow) Build DELIVERY Plugs right in to your existing tools CONTINUOUS Build Test

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

© Electric Cloud | electric-cloud.com Our situation in 2010 • One engineering site (Sunnyvale, CA) • All engineers were in one room, on the same schedule. • Code reviews were done informally by calling someone over to your desk. • Engineers documented setup on wiki. Other employees expected to follow.

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

© Electric Cloud | electric-cloud.com What were the problems? • Wiki was out of date. • Our software was getting harder to setup.  For example: we introduced support for clustering • Found ourselves spending more time troubleshooting individual environments instead of product issues.

Slide 16

Slide 16 text

Slide 17

Slide 17 text

© Electric Cloud | electric-cloud.com There has to be a better way ~ 2013 How do we create development environments such that they can be: • reconstructed • shared • local to each tester • deployed into an automated test environment

Slide 18

Slide 18 text

© Electric Cloud | electric-cloud.com Side note: Cultural Observation • What worked well and got you this far may not be the thing to get you the next level.  It’s hard because the people suffering may be new to the organization  It’s very natural to think “But it has worked for us so well!”

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

© Electric Cloud | electric-cloud.com Bringing the pain forward: • It hurts to construct these environments so….let’s do it more often! • How can we run through the process regularly? • How can we have documentation that is always up to date?

Slide 22

Slide 22 text

© Electric Cloud | electric-cloud.com Introducing Vagrant  https://www.vagrantup.com/  Open Source  Made by HashiCorp $ vagrant up (start or create the VM) $ vagrant ssh (ssh to the VM) $ vagrant halt (stop the VM) $ vagrant destroy (destroy the VM) $ vagrant package (snapshot the VM)

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© Electric Cloud | electric-cloud.com What problems did Vagrant solve? • Simple commands to instantiate and destroy • Documentation via code inside Vagrantfile (or shell script invoked by Vagrantfile) • Check in Vagrantfile. Now available to all developers • Each developer can run Vagrant on their laptop • Same process can be used in automated testing

Slide 33

Slide 33 text

© Electric Cloud | electric-cloud.com How do I get Vagrant and what do I need to use it? • Install Vagrant on each machine where vagrant will be used. • Install virtualization provider (VirtualBox). • Supported Host Operating Systems:  Mac OSX  Windows  Linux (Debian, RedHat) • Vagrant  http://downloads.vagrantup.com/ • Oracle VirtualBox  https://www.virtualbox.org/wiki/Downloads

Slide 34

Slide 34 text

© Electric Cloud | electric-cloud.com Vagrant Tips • Your relationship with VirtualBox will be love/hate • At a high level, Vagrant uses Ruby to wrap VBoxManage. So if you are running into any issues you can open up the VirtualBox UI or instrument the ruby code (add puts/print statements) • As with all Open Source: read the source, check out the open issues. https://github.com/mitchellh/vagrant/ • Stackoverflow also helpful

Slide 35

Slide 35 text

© Electric Cloud | electric-cloud.com Side note: Cultural Observation • When trying to gather consensus or persuade an organization to adopt change:  Identify a problem, formulate a hypothesis and then look for a tool/solution. • Avoid  “Oooh shiny technology!”  “All the cool kids are doing it” • Even if you have followed the problem solving steps people may still feel like you’re just chasing technology.

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

© Electric Cloud | electric-cloud.com Let’s read some code! https://gist.github.com/nikhilv/74a107af7e44866e3f14 Note: these days terraform’s doc have improved. A lot of this is available in the aws-two-tier example https://github.com/hashicorp/terraf orm/tree/master/examples/aws-two- tier

Slide 44

Slide 44 text

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

© Electric Cloud | electric-cloud.com Review: Requirements for environment creation  Definition based .tf files  Reproducible Plan -> Apply – Destroy.  Shareable Check into source control  Each invocation must be unique to an individual Parametrized scripts

Slide 50

Slide 50 text

© Electric Cloud | electric-cloud.com What problems did Terraform solve? • Simple commands to instantiate and destroy cloud VMs • Documentation via code inside Terraform files • Check in Terraform files. Now available to all developers • Each developer can run Terraform on their laptop • Save those UI clicks! Write it once in Terraform and build on it later

Slide 51

Slide 51 text

© Electric Cloud | electric-cloud.com Terraform Tips • Super easy to install and upgrade (directory of executables) • Caution: Using Terraform will cost you money with your cloud provider! • Some syntax is not intuitive. But the documentation has improved. • Parameterize and use the graph • IMO: Good for demo and short term work. But I wouldn’t use it alone to deploy to production

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

© Electric Cloud | electric-cloud.com Use case : Simulating ‘Big Data Software Team’ Communication paths  ELB-> instance(s)  Backend -> DB  Frontend -> Backend  IoT -> ELB / Backend  Mobile Device -> ELB / Backend

Slide 62

Slide 62 text

© Electric Cloud | electric-cloud.com Use case : Simulating ‘Big Data Software Team’  How does the Front end know the IP the backend gets spawned at?  How does the Back end know the IP the database gets spawned at?  How does the ELB know about the two availability zones? What is the criteria for the health check?

Slide 63

Slide 63 text

© Electric Cloud | electric-cloud.com Technologies used  Didn’t use Terraform  Regretted it. All my setup is now bespoke and won’t be easy to re-use  Lost: ELB (Health Check, Port forward and instances), VPC, Subnets, EC2 instance data  Decided Docker would be the way to go from “dev” to “production”  Easy to simulate pieces locally on dev laptop and then run same thing in AWS.  Was easy to share with others working on the same demo  Feel confident that I could go back to the source for a future demo

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

© Electric Cloud | electric-cloud.com Docker Tips • Where do you save, version and test your containers? • Your disk usage will skyrocket. One reason Docker is fast is because it caches intermediate layers. • As with all Open Source: read the source, check out the open issues. https://github.com/docker/docker • Check out Greg Osuri’s guide on Guide to automating a multi- tiered application securely on AWS with Docker and Terraform https://www.airpair.com/aws/posts/ntiered-aws-docker-terraform-guide

Slide 67

Slide 67 text

Slide 68

Slide 68 text

© Electric Cloud | electric-cloud.com Vagrant, Terraform, Docker FTW! • It’s an exciting time to be in software • Warning bells should go off if you keep clicking on a UI. • Organization and structure increase velocity and confidence • Drift is minimized if the ‘documentation’ is exercised regularly • You will find bugs or areas where documentation could be improved

Slide 69

Slide 69 text

© Electric Cloud | electric-cloud.com Thank you Networking Other Systems ISV, Internet & Entertainment Heavy Industry Financial Services Automotive Medical Devices Semiconductor Aerospace / Defense Mobile Devices Q&A