Bridging Past, Present, and Future infrastructure with Consul

Today’s data centers aren’t the same anymore. We are constantly moving from one set of technologies to the other. We often find it hard to connect newly deployed applications to rest of our infrastructure. See how to use Consul to help connect applications across your infrastructure. Consul is an open source tool for service discovery, monitoring, and globally distributed key-value store. In this talk we will cover how Consul can help bridge the gap between various types of applications that might exist in a datacenter at any given time. As we move towards micro service oriented architectures we find ourselves using cluster schedulers like Kubernetes and Nomad. We will discuss how Consul’s service discovery features help connect applications running in such environments. We will also explore how we can connect applications that are running in traditional environments to ones running in these cluster schedulers using Consul’s API and built in DNS support. We will do a live demo of connecting two applications running in separate environments using Consul and showcase important features of Consul.

Transcript

1. Copyright © 2017 HashiCorp Any Infrastructure. Any Application. Anywhere. DevOps Barcelona 2018
2. Bridging Past, Present, and Future Infrastructure with Consul https://unsplash.com/photos/-a4tzI2fNW8 Photo by Rita Morais
3. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp
4. @anubhavm Anubhav Mishra Developer Advocate, HashiCorp has stickers
5. PROVISION, SECURE AND RUN ANY INFRASTRUCTURE Nomad Consul Vault Vagrant Packer Terraform Consul Enterprise Terraform Enterprise Vault Enterprise PRODUCT SUITE OSS TOOL SUITE RUN Applications SECURE Application Infrastructure PROVISION Infrastructure FOR INDIVIDUALS FOR TEAMS Nomad Enterprise
7. Why?
8. Copyright © 2017 HashiCorp @anubhavm  History !8 1995-2000 Physical Machines, Static Provisioning 2017-2018 Virtual Machines, Dynamic Provisioning
9. -/+ Change
10. SERVER MONOLITHIC APP DATABASE CACHE
11. SERVER MONOLITHIC APP DATABASE CACHE www.webscale.com yoda.webscale.com cache1.webscale.com randomtest1.webscale.com
12. SERVER MONOLITHIC APP DATABASE CACHE CACHE CACHE SERVER SERVER MONOLITHIC APP MONOLITHIC APP SERVER MONOLITHIC APP
13. SERVER MONOLITHIC APP DATABASE CACHE
14. SERVER MICROSERVICE DATABASE MONOLITHIC APP
15. SERVER MICROSERVICES DATABASES CLUSTER MANAGER MONOLITHIC APP
16. SERVER MICROSERVICES DATABASES CLUSTER MANAGER www.webscale.com yoda.webscale.com app-xww123.webscale.com db-analyticsx123.webscale.com MONOLITHIC APP
17. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !17 Abstractions Datacenters Physical/Virtual machines Servers IP addresses, DNS Monolithic applications
18. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !18 Abstractions Datacenters Physical/Virtual machines Servers IP addresses, DNS Monolithic applications
19. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !19 Abstractions Datacenters Physical/Virtual machines Servers IP addresses, DNS Monolithic applications Platform Containers Services Service discovery Microservices
20. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !20 Overview Cloud/Infrastructure Providers
21. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !21 Overview Cloud/Infrastructure Providers Network Protocols
22. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !22 Overview Cloud/Infrastructure Providers Network Protocols Runtime, Platform, CI/CD, Configuration Management, Developer Tools…..
23. Copyright © 2017 HashiCorp @anubhavm  -/+ Change !23 Overview Cloud/Infrastructure Providers Network Protocols “Stuff” You Need to Make Software Work
25. s Copyright © 2017 HashiCorp @anubhavm  !25 Challenges
26. Copyright © 2017 HashiCorp @anubhavm  Networking !26 How do we connect applications?
27. Copyright © 2017 HashiCorp @anubhavm  Discovery !27 How do we discover applications?
28. Copyright © 2017 HashiCorp @anubhavm  Configuring !28 How do we dynamically configure applications?
29. Challenges Service Discovery Common Interface Runtime Configuration
31. Copyright © 2017 HashiCorp @anubhavm  • Service Discovery • DNS or HTTP Interface • Health Checks • Dynamic Configuration via Consistent Key-Value Store Consul !31
32. Copyright © 2017 HashiCorp @anubhavm  • Service Discovery • DNS or HTTP Interface • Health Checks • Dynamic Configuration via Consistent Key-Value Store Consul !32
33. s Copyright © 2017 HashiCorp @anubhavm  How do things find each other? !33 Discovery
34. Copyright © 2017 HashiCorp @anubhavm  Consul Cluster !34 CLIENT CLIENT CLIENT CLIENT CLIENT CLIENT SERVER SERVER SERVER REPLICATION REPLICATION RPC RPC LAN GOSSIP SERVER SERVER SERVER REPLICATION REPLICATION WAN GOSSIP
35. Copyright © 2017 HashiCorp @anubhavm  Discovery !35 NODE PROXY FRONTEND
36. Copyright © 2017 HashiCorp @anubhavm  Discovery !36 NODE PROXY FRONTEND
37. Copyright © 2017 HashiCorp @anubhavm  Discovery !37 NODE PROXY FRONTEND NODE BACKEND
38. Copyright © 2017 HashiCorp @anubhavm  Discovery !38 NODE PROXY FRONTEND NODE BACKEND
39. Copyright © 2017 HashiCorp @anubhavm  Discovery !39 NODE PROXY FRONTEND NODE BACKEND
40. Copyright © 2017 HashiCorp Terminal !40 Via DNS Discovery
41. Copyright © 2017 HashiCorp Terminal !41 Via DNS Discovery $ dig backend.service.consul
42. Copyright © 2017 HashiCorp Terminal !42 Via DNS Discovery $ dig backend.service.consul
43. Copyright © 2017 HashiCorp Terminal !43 Via DNS Discovery $ dig backend.service.consul
44. Copyright © 2017 HashiCorp Terminal !44 Via DNS Discovery $ dig backend.service.consul
45. Copyright © 2017 HashiCorp Terminal !45 Via DNS Discovery $ dig backend.service.consul # ... ;; QUESTION SECTION: ;backend.service.consul. IN A ;; ANSWER SECTION: backend.service.consul.0 IN A 10.0.5.69 backend.service.consul.0 IN A 10.0.5.34 backend.service.consul.0 IN A 10.0.5.138
46. Copyright © 2017 HashiCorp Terminal !46 Via DNS Load Balancing $ dig backend.service.consul # ... ;; QUESTION SECTION: ;backend.service.consul. IN A ;; ANSWER SECTION: backend.service.consul.0 IN A 10.0.5.69 backend.service.consul.0 IN A 10.0.5.34 backend.service.consul.0 IN A 10.0.5.138
47. Copyright © 2017 HashiCorp Terminal !47 Via DNS Load Balancing $ dig backend.service.consul # ... ;; QUESTION SECTION: ;backend.service.consul. IN A ;; ANSWER SECTION: backend.service.consul.0 IN A 10.0.5.69 backend.service.consul.0 IN A 10.0.5.138 backend.service.consul.0 IN A 10.0.5.34
48. Copyright © 2017 HashiCorp Terminal !48 Via HTTP Discovery $ curl http://localhost:8500/v1/health/service/backend [ { "Node": { "ID": "071d5055-6afa-c457-1194-1a070d15e35f", "Node": "abd28dac", "Address": "10.0.5.52", "Datacenter": "dc1", }, "Service": { "ID": "backend-01", "Service": "backend", "Port": 80, }, # ... } ]
49. Copyright © 2017 HashiCorp Terminal !49 Via CLI Key-Value Store $ consul kv put hello world
50. Copyright © 2017 HashiCorp Terminal !50 Via CLI Key-Value Store $ consul kv put hello world Success! Data written to: hello
51. Copyright © 2017 HashiCorp Terminal !51 Via CLI Key-Value Store $ consul kv get hello world
52. Copyright © 2017 HashiCorp Terminal !52 Via CLI Key-Value Store $ consul kv get hello world
53. Copyright © 2017 HashiCorp Terminal !53 Via HTTP Key-Value Store $ curl http://localhost:8500/v1/kv/hello [ { "LockIndex": 0, "Key": "hello", "Flags": 0, "Value": "d29ybGQ=", "CreateIndex": 7, "ModifyIndex": 7 } ]
54. Copyright © 2017 HashiCorp Terminal !54 Via HTTP Key-Value Store $ curl -s http://localhost:8500/v1/kv/hello | jq -r .[].Value | base64 --decode world
55. s Copyright © 2017 HashiCorp @anubhavm  !55 DEMO http://giphylink.hashicorp.live/
56. Consul @anubhavm  !56 Massive Clusters (10k to 100k nodes)
57. Copyright © 2017 HashiCorp @anubhavm  Trusted By !57
58. s Copyright © 2017 HashiCorp @anubhavm  Discover and Connect Applications at Scale. !58 Conclusion
59. Thank You! I have stickers! Ask me anything. @anubhavm www.hashicorp.com