HASHICORP

HASHICORP Armon Dadgar @armon

HASHICORP

HASHICORP

CONSUL HASHICORP Enable SOA / Microservices Datacenter Runtime

HASHICORP SOA PRIMER Autonomous Limited Scope Loose Coupling

HASHICORP ORDER PROCESSING WEB APP ORDER HISTORY FORECASTING

HASHICORP ORDER PROCESSING WEB APP DISCOVERY Which nodes are part of "order processing"?

HASHICORP ORDER PROCESSING WEB APP LOAD BALANCING How to ensure request leveling across providers? NODE 1 NODE 2 NODE N

HASHICORP ORDER PROCESSING WEB APP ANTI-PATTERN Load Balancer is a Single Point of Failure (SPOF) NODE 1 NODE 2 NODE N LOAD BALANCER

HASHICORP ORDER PROCESSING WEB APP HEALTH CHECKING How to avoid routing to unhealthy hosts? NODE 1 NODE 2 NODE 3 LOAD BALANCER

HASHICORP WEB APP CONFIGURATION How to efficiently push dynamic configuration? WEB 1 WEB 2 WEB N maintenance: false feature_a: true role: "web"

HASHICORP SERVICE DISCOVERY LOAD BALANCING HEALTH CHECKING KEY-VALUE CONFIGURATION 4 BASIC PROBLEMS

CONSUL HASHICORP Consul 0.6 Network Tomography Prepared Queries MemDB Enhanced ACLs TCP & Docker Health Checks

HASHICORP Network Tomography Model Underlying Network EsFmate RTT Nearest Neighbor RouFng

HASHICORP Network Tomography

HASHICORP Network Tomography

HASHICORP Network Tomography

Terminal HASHICORP $ consul rtt node-10-0-1-8 Estimated node-10-0-1-8 <-> node-10-0-1-6 rtt: 0.781 ms (using LAN coordinates)$ $ sleep 30 $ consul rtt node-10-0-1-8 Estimated node-10-0-1-8 <-> node-10-0-1-6 rtt: 0.719 ms (using LAN coordinates)

Terminal HASHICORP $ curl localhost:8500/v1/catalog/nodes?near=node-78r16zb3q | jq '.[].Node' "node-78r16zb3q" "node-10-0-4-190" "node-10-0-1-7" "node-10-0-4-240" $ curl localhost:8500/v1/catalog/service/vault? near=node-78r16zb3q | jq '.[].Node' "node-10-0-1-71" "node-10-0-3-119" "node-10-0-3-249"

CONSUL HASHICORP Service Discovery DNS Interface HTTP API Smart Clients Consul-Template

Terminal HASHICORP $ dig vault.service.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> vault.service.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9406 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;vault.service.consul. IN A ;; ANSWER SECTION: vault.service.consul. 0 IN A 10.0.1.71 vault.service.consul. 0 IN A 10.0.3.119 vault.service.consul. 0 IN A 10.0.3.249 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:16 UTC 2015 ;; MSG SIZE rcvd: 146

Terminal HASHICORP $ dig beta.vault.service.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> beta.vault.service.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2613 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;beta.vault.service.consul. IN A ;; ANSWER SECTION: beta.vault.service.consul. 0 IN A 10.0.3.119 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:20 UTC 2015 ;; MSG SIZE rcvd: 126

HASHICORP { "Name": "vault-with-failover", "Service": { "Service": "vault", "Failover": { "NearestN": 3 }, "Tags": ["!beta"] }, "DNS": { "TTL": "5s" } }

Terminal HASHICORP $ dig vault-with-failover.query.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> vault-with- failover.query.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1039 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;vault-with-failover.query.consul. IN A ;; ANSWER SECTION: vault-with-failover.query.consul. 0 IN A 10.0.1.71 vault-with-failover.query.consul. 0 IN A 10.0.3.249 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:30 UTC 2015 ;; MSG SIZE rcvd: 150

HASHICORP Prepared Queries Complex Queries Datacenter Failover ACL IntegraFon Change Behavior via API

HASHICORP Read Path Optimization LMDB (C) B-Tree API (Go) Go <= 1.4, Mark & Sweep LMDB Off-Heap Serialization on boundary

HASHICORP Read Path Optimization MemDB (Go) Immutable Radix API (Go) Go >= 1.5, Soft Realtime MemDB On-Heap Pure Go

HASHICORP MemDB MulF-Version Concurrency Control (MVCC) TransacFon Support Rich Indexing Avoids SerializaFon Cost

HASHICORP Enhanced ACLs (0.4) Key / Value Store (0.5) Service RegistraFon (0.6) Service Discovery (0.6) User Events (0.6) Keyring Updates

HASHICORP Enhanced ACLs User API WEB APP Database Catalog API X

CONSUL HASHICORP HTTP Check TTL Check Script Check +TCP Check +Docker Check

HASHICORP { "check": { "id": "ssh", "name": "SSH TCP on port 22", "tcp": "localhost:22", "interval": "10s", "timeout": "1s" } }

HASHICORP { "check": { "id": "mem-util", "name": "Memory utilization", "docker_container_id": "f972c95ebf0e", "shell": "/bin/bash", "script": "/usr/local/bin/check_mem.py", "interval": "10s" } }

CONSUL HASHICORP Consul 0.7 Performance Stability Security

HASHICORP Consul Eco-System Consul Template Consul Replicate envconsul consul-cli fabio …

HASHICORP Consul Template Dynamically render templates Reload on Change Integrate with Everything* Real Time

HASHICORP backend frontend maxconn {{ key "frontend/maxconn" }} balance roundrobin{{range service "app.frontend"}} service {{.ID}} {{.Address}}:{{.Port}}{{end}} backend frontend maxconn 256 balance roundrobin server web1 10.0.1.100:80 server web2 10.0.2.200:80

HASHICORP {{ with $service := range services }} backend {{$service.Name}} maxconn {{ key (printf “service/%s/maxconn”, $service.Name) }} balance roundrobin{{range service $service.Name}} service {{.ID}} {{.Address}}:{{.Port}}{{end}} 1) Query for all services (1 Query) 2) Query maxconn for each service (N Queries) 3) Query the instances of each service (N Queries) 4) Total of 2*N+1 Queries

HASHICORP Consul 2*N+1 Queries M(2*N+1) Queries N = Num Services M = Instances of CT

HASHICORP Number of Services Number of Instances Total Queries 10 5 105 25 10 510 50 20 2020 100 30 6030

HASHICORP Query De-Duplication Reduce Load Improve QoS

HASHICORP Consul 2*N+1 Queries Leader

HASHICORP Consul 2*N+1 Queries Leader 1 Write

HASHICORP Consul 2*N+1 Queries Leader 1 Write 1 Query Followers

HASHICORP Consul 2*N+1 Queries Leader 1 Write 1 Query Followers M+2*N Queries 1 Write N = Num Services M = Instances of CT

HASHICORP Number of Services Number of Instances Previous Queries New Queries 10 5 105 25 25 10 510 60 50 20 2020 120 100 30 6030 230

HASHICORP 1-2 Order of Magnitude Reduction

CONSUL HASHICORP Client Access Patterns Distributed Database Best Practices

CONSUL HASHICORP TLS & Encryption ACLs Stale Queries Query De-Duplication CPU & IO Capacity Cluster Health

CONSUL HASHICORP Static Configuration Client Usage Dynamic State

CONSUL HASHICORP Telemetry Statsd Statsite

HASHICORP

HASHICORP Best practice for all systems

CONSUL HASHICORP Consul 0.6 Consul Template Best Practices & Telemetry

HASHICORP Thanks! Q/A