Terraform, Consul & Nomad

High-level overview of some popular HashiCorp's tools

Transcript

1. Terraform, Consul & Nomad HashiCorp’s suite to deploy and run your infrastructure Jonathan Ballet Lausanne Cloud Meetup – 2017/11/09 Build Configure Deploy
2. 2 Introduction Lead Infrastructure Engineer @ EdgeLab ~ 25 people company, developing risk analytics for banks & insurances Dedicated web applications + REST/HTTP API We have a services-oriented architecture • ~ 20 in-house services (HTTP, background workers) • ~ 40 deployed services in total • Java, C++, JavaScript, Go, Python, Ruby • Long term goals: more data, bigger batches, load spikes, worldwide domination
3. 3 Program for tonight! High-level overview of HashiCorp’s tools Nomad How to run the services Consul How to provide confguration for the services Terraform How to manage the lifecycle of the infrastructure
4. 4 Step 1 - Infrastructure
5. 5 Terraform : Infrastructure As Code
6. 6 Terraform : Infrastructure As Code « Compiler » for « infrastructure » .tf source files - Plan - Apply - Destroy
7. 7 Terraform : demo
8. 8 Terraform : Beside the « core » features Multiple providers available AWS, GCP, CloudStack, … MySQL, PostgreSQL, RabbitMQ, … Github, DNSimple, Kubernetes, Consul, Nomad, etc. Provisioning Shell scripts, Ansible, Salt, Chef, Puppet, etc. Collaborative work Modules, shared states, locking
9. 9 Step 2 – Discovery / Confguration
10. 10 Consul – Service Discovery & Distributed Confguration
11. 11 Consul – Core features … a service catalog To register and query informations about services … a key/value datastore To distribute confguration to services … a distributed system To provide resiliency, scalability and failures detection
12. 13 Consul – Demo Deployment: (see next slide) Clusters of 3 / 5 servers One client-per-node model Consul UI Catalog Key / Value datastore
13. 14 Consul – Deployment model Consul Servers DNS HTTP Instance Local Consul client
14. 15 Consul - Bonus ACLs Fine grained access via a token + policies Consul companions Consul-Template : generate confg fles from Consul values Fabio / Traefk : zero-conf HTTP routers Envconsul : inject environment variables from Consul values Distributed locks Leader election, distributed semaphore (Google Chubby-style)
15. 16 Step 3 – Applications Scheduler
16. 17 Nomad – Cluster Scheduler
17. 18 Nomad – Core features As a « cluster scheduler », Nomad provides: A job abstraction Describe what you want to run, Nomad decides where and how to run it Pool the resources together Create many instances as « Nomad clients », and let Nomad run your applications on
18. 19 Nomad - workfow Users Nomad Servers Nomad Clients Write job Plan / Run Evaluate Allocate
19. 20 Nomad - demonstration Let’s break things
20. 21 Nomad - Bonus Stateful / active monitoring Checks if allocations / clients are alive, checks new / removed clients Advanced upgrade policies Rolling upgrade, rollbacks, canary deployments ACLs Since version 0.7.0! Distributed, available, scalable (hopefully)! (Citadel : ~18000 nodes, 2000 jobs/second, 100000 containers) Single binary Hello Kubernetes!
21. 22 Conclusion Terraform Allows you to reliably build infrastructure Consul Helps you to distribute your confguration across all your nodes Nomad Deploys and monitor our services seemlessly in the infrastructure