Slide 1

Slide 1 text

HASHICORP

Slide 2

Slide 2 text

HASHICORP Armon Dadgar @armon

Slide 3

Slide 3 text

HASHICORP

Slide 4

Slide 4 text

HASHICORP

Slide 5

Slide 5 text

CONSUL HASHICORP Enable SOA / Microservices Datacenter Runtime

Slide 6

Slide 6 text

HASHICORP SOA PRIMER Autonomous Limited Scope Loose Coupling

Slide 7

Slide 7 text

HASHICORP ORDER PROCESSING WEB APP ORDER HISTORY FORECASTING

Slide 8

Slide 8 text

HASHICORP ORDER PROCESSING WEB APP DISCOVERY Which nodes are part of "order processing"?

Slide 9

Slide 9 text

HASHICORP ORDER PROCESSING WEB APP LOAD BALANCING How to ensure request leveling across providers? NODE 1 NODE 2 NODE N

Slide 10

Slide 10 text

HASHICORP ORDER PROCESSING WEB APP ANTI-PATTERN Load Balancer is a Single Point of Failure (SPOF) NODE 1 NODE 2 NODE N LOAD BALANCER

Slide 11

Slide 11 text

HASHICORP ORDER PROCESSING WEB APP HEALTH CHECKING How to avoid routing to unhealthy hosts? NODE 1 NODE 2 NODE 3 LOAD BALANCER

Slide 12

Slide 12 text

HASHICORP WEB APP CONFIGURATION How to efficiently push dynamic configuration? WEB 1 WEB 2 WEB N maintenance: false feature_a: true role: "web"

Slide 13

Slide 13 text

HASHICORP SERVICE DISCOVERY LOAD BALANCING HEALTH CHECKING KEY-VALUE CONFIGURATION 4 BASIC PROBLEMS

Slide 14

Slide 14 text

CONSUL HASHICORP Consul 0.6 Network Tomography Prepared Queries MemDB Enhanced ACLs TCP & Docker Health Checks

Slide 15

Slide 15 text

HASHICORP Network Tomography Model Underlying Network EsFmate RTT Nearest Neighbor RouFng

Slide 16

Slide 16 text

HASHICORP Network Tomography

Slide 17

Slide 17 text

HASHICORP Network Tomography

Slide 18

Slide 18 text

HASHICORP Network Tomography

Slide 19

Slide 19 text

Terminal HASHICORP $ consul rtt node-10-0-1-8 Estimated node-10-0-1-8 <-> node-10-0-1-6 rtt: 0.781 ms (using LAN coordinates)$ $ sleep 30 $ consul rtt node-10-0-1-8 Estimated node-10-0-1-8 <-> node-10-0-1-6 rtt: 0.719 ms (using LAN coordinates)

Slide 20

Slide 20 text

Terminal HASHICORP $ curl localhost:8500/v1/catalog/nodes?near=node-78r16zb3q | jq '.[].Node' "node-78r16zb3q" "node-10-0-4-190" "node-10-0-1-7" "node-10-0-4-240" $ curl localhost:8500/v1/catalog/service/vault? near=node-78r16zb3q | jq '.[].Node' "node-10-0-1-71" "node-10-0-3-119" "node-10-0-3-249"

Slide 21

Slide 21 text

CONSUL HASHICORP Service Discovery DNS Interface HTTP API Smart Clients Consul-Template

Slide 22

Slide 22 text

Terminal HASHICORP $ dig vault.service.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> vault.service.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9406 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;vault.service.consul. IN A ;; ANSWER SECTION: vault.service.consul. 0 IN A 10.0.1.71 vault.service.consul. 0 IN A 10.0.3.119 vault.service.consul. 0 IN A 10.0.3.249 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:16 UTC 2015 ;; MSG SIZE rcvd: 146

Slide 23

Slide 23 text

Terminal HASHICORP $ dig beta.vault.service.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> beta.vault.service.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2613 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;beta.vault.service.consul. IN A ;; ANSWER SECTION: beta.vault.service.consul. 0 IN A 10.0.3.119 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:20 UTC 2015 ;; MSG SIZE rcvd: 126

Slide 24

Slide 24 text

HASHICORP { "Name": "vault-with-failover", "Service": { "Service": "vault", "Failover": { "NearestN": 3 }, "Tags": ["!beta"] }, "DNS": { "TTL": "5s" } }

Slide 25

Slide 25 text

Terminal HASHICORP $ dig vault-with-failover.query.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> vault-with- failover.query.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1039 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;vault-with-failover.query.consul. IN A ;; ANSWER SECTION: vault-with-failover.query.consul. 0 IN A 10.0.1.71 vault-with-failover.query.consul. 0 IN A 10.0.3.249 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:30 UTC 2015 ;; MSG SIZE rcvd: 150

Slide 26

Slide 26 text

HASHICORP Prepared Queries Complex Queries Datacenter Failover ACL IntegraFon Change Behavior via API

Slide 27

Slide 27 text

HASHICORP Read Path Optimization LMDB (C) B-Tree API (Go) Go <= 1.4, Mark & Sweep LMDB Off-Heap Serialization on boundary

Slide 28

Slide 28 text

HASHICORP Read Path Optimization MemDB (Go) Immutable Radix API (Go) Go >= 1.5, Soft Realtime MemDB On-Heap Pure Go

Slide 29

Slide 29 text

HASHICORP MemDB MulF-Version Concurrency Control (MVCC) TransacFon Support Rich Indexing Avoids SerializaFon Cost

Slide 30

Slide 30 text

HASHICORP Enhanced ACLs (0.4) Key / Value Store (0.5) Service RegistraFon (0.6) Service Discovery (0.6) User Events (0.6) Keyring Updates

Slide 31

Slide 31 text

HASHICORP Enhanced ACLs User API WEB APP Database Catalog API X

Slide 32

Slide 32 text

CONSUL HASHICORP HTTP Check TTL Check Script Check +TCP Check +Docker Check

Slide 33

Slide 33 text

HASHICORP { "check": { "id": "ssh", "name": "SSH TCP on port 22", "tcp": "localhost:22", "interval": "10s", "timeout": "1s" } }

Slide 34

Slide 34 text

HASHICORP { "check": { "id": "mem-util", "name": "Memory utilization", "docker_container_id": "f972c95ebf0e", "shell": "/bin/bash", "script": "/usr/local/bin/check_mem.py", "interval": "10s" } }

Slide 35

Slide 35 text

CONSUL HASHICORP Consul 0.7 Performance Stability Security

Slide 36

Slide 36 text

HASHICORP Consul Eco-System Consul Template Consul Replicate envconsul consul-cli fabio …

Slide 37

Slide 37 text

HASHICORP Consul Template Dynamically render templates Reload on Change Integrate with Everything* Real Time

Slide 38

Slide 38 text

HASHICORP backend frontend maxconn {{ key "frontend/maxconn" }} balance roundrobin{{range service "app.frontend"}} service {{.ID}} {{.Address}}:{{.Port}}{{end}} backend frontend maxconn 256 balance roundrobin server web1 10.0.1.100:80 server web2 10.0.2.200:80

Slide 39

Slide 39 text

HASHICORP {{ with $service := range services }} backend {{$service.Name}} maxconn {{ key (printf “service/%s/maxconn”, $service.Name) }} balance roundrobin{{range service $service.Name}} service {{.ID}} {{.Address}}:{{.Port}}{{end}} 1) Query for all services (1 Query) 2) Query maxconn for each service (N Queries) 3) Query the instances of each service (N Queries) 4) Total of 2*N+1 Queries

Slide 40

Slide 40 text

HASHICORP Consul 2*N+1 Queries M(2*N+1) Queries N = Num Services M = Instances of CT

Slide 41

Slide 41 text

HASHICORP Number of Services Number of Instances Total Queries 10 5 105 25 10 510 50 20 2020 100 30 6030

Slide 42

Slide 42 text

HASHICORP Query De-Duplication Reduce Load Improve QoS

Slide 43

Slide 43 text

HASHICORP Consul 2*N+1 Queries Leader

Slide 44

Slide 44 text

HASHICORP Consul 2*N+1 Queries Leader 1 Write

Slide 45

Slide 45 text

HASHICORP Consul 2*N+1 Queries Leader 1 Write 1 Query Followers

Slide 46

Slide 46 text

HASHICORP Consul 2*N+1 Queries Leader 1 Write 1 Query Followers M+2*N Queries 1 Write N = Num Services M = Instances of CT

Slide 47

Slide 47 text

HASHICORP Number of Services Number of Instances Previous Queries New Queries 10 5 105 25 25 10 510 60 50 20 2020 120 100 30 6030 230

Slide 48

Slide 48 text

HASHICORP 1-2 Order of Magnitude Reduction

Slide 49

Slide 49 text

CONSUL HASHICORP Client Access Patterns Distributed Database Best Practices

Slide 50

Slide 50 text

CONSUL HASHICORP TLS & Encryption ACLs Stale Queries Query De-Duplication CPU & IO Capacity Cluster Health

Slide 51

Slide 51 text

CONSUL HASHICORP Static Configuration Client Usage Dynamic State

Slide 52

Slide 52 text

CONSUL HASHICORP Telemetry Statsd Statsite

Slide 53

Slide 53 text

HASHICORP

Slide 54

Slide 54 text

HASHICORP Best practice for all systems

Slide 55

Slide 55 text

CONSUL HASHICORP Consul 0.6 Consul Template Best Practices & Telemetry

Slide 56

Slide 56 text

HASHICORP Thanks! Q/A