About Consul - Speaker Deck

Slide 1

Slide 1 text

ABOUT CONSUL 6/22/2014 @ijin July Tech Festa 2014

Slide 2

Slide 2 text

ABOUT ME (@IJIN) • Michael H. Oshita • Japanese American • 10+ Years in Japan • Software/Infra/Cloud Engineer • http://ijin.github.io

Slide 3

Slide 3 text

ABOUT ME (@IJIN) τϥϒϧˑ͠Ύʔͨʔͣ

Slide 4

Slide 4 text

τϥϒϧˑ͠Ύʔͨʔͣ #08 http://bit.ly/torash08

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

CONSUL

Slide 7

Slide 7 text

CONSUL HTTP APIͱDNSͰૢ࡞͕Ͱ͖Δ෼ࢄܕΫϥελ

Slide 8

Slide 8 text

SERVICE DISCOVERY • ؆୯ʹserviceΛొ࿥͠ɺࣗಈͰΫϥελ্Ͱݕग़Մೳ • DNS·ͨ͸HTTPʹΑͬͯࢀর

Slide 9

Slide 9 text

FAILURE DETECTION • ϔϧενΣοΫػೳ • ࢮΜͩϊʔυ΁ͷϦΫΤετᷖճ !

Slide 10

Slide 10 text

MULTI DATACENTER • ෳ਺ͷσʔληϯλʔΛ·͍ͨͰ௨৴ • ଞͷDC্Ͱͷserviceͷࢀর

Slide 11

Slide 11 text

KEY/VALUE STORE • ಈతͳઃఆ৘ใͷஔ͖৔ॴͱͯ͠ • Long PollingͰઃఆมߋͷଈ࣌௨஌

Slide 12

Slide 12 text

TECHNOLOGY • Consensus Protocol (Raft, FSM) In Search of an Understandable Consensus Algorithm - Stanford University • Gossip Protocol (Serf) SWIM: Scalable Weakly-consistent Infection-style Process Group Membership - Cornel University • Sessions (Chuby) The Chubby Lock Service for Loosely-Coupled Distributed Systems - Google • TLS, Jespen Testing !

Slide 13

Slide 13 text

ARCHITECTURE

Slide 14

Slide 14 text

CONSISTENCY

Slide 15

Slide 15 text

CONSISTENCY default ! ! consistent: ! ! stale: "?state" or "?consistent" parameter ɾେମStrongly Consistent ɾ৽Leaderঢ֨࣌ʹstale readͷՄೳੑ͋Γɻ ɾ଎͍ ! ɾStrongly Consistent ɾLeader͸QuromͰ֬ೝ ɾLatency͕ൃੜ͢ΔͷͰएׯ஗Ί ! ɾStale Readɺޡࠩ͸֓Ͷ50ms͙Β͍ ɾLeaderҎ֎ͷServer͕Ԡ౴Մ ɾߴ଎

Slide 16

Slide 16 text

CONSUL AGENT Nodes & Services foo bar baz redis 01.web 02.web node service

Slide 17

Slide 17 text

CONSUL AGENT • DNS Interface • HTTP API • Health Check !

Slide 18

Slide 18 text

CONSUL AGENT DNS Interface .node.. Node Lookup:

Slide 19

Slide 19 text

CONSUL AGENT DNS Interface

Slide 20

Slide 20 text

CONSUL AGENT DNS Interface ..service. Service Lookup:

Slide 21

Slide 21 text

CONSUL AGENT DNS Interface

Slide 22

Slide 22 text

CONSUL AGENT DNS Interface

Slide 23

Slide 23 text

CONSUL AGENT DNS CACHING Stale Reads ɾࢀর͸σϑΥϧτͰLeader NodeͷΈ ɾ༗ޮԽ͢ΔͱଞͷServer Node͔ΒࢀরՄೳ ɾεϧʔϓοτ޲্ ɾmax_staleͰ࣌ؒΛઃఆ ! TTL ɾDNS TTL͸σϑΥϧτͰ0 ɾdnsmasq౳ͰΩϟογϡՄೳʹ { "dns_conﬁg": { "service_ttl": { "*": "5s", "web": "30s" } } new! (0.3.0)

Slide 24

Slide 24 text

CONSUL AGENT HTTP API /v1/kv/ key/value endpoint: RESTful: GET, PUT, DELETE

Slide 25

Slide 25 text

CONSUL AGENT HTTP API key/value: base64 raw

Slide 26

Slide 26 text

CONSUL AGENT HTTP API /v1/agent/checks /v1/agent/services /v1/agent/members /v1/agent/self /v1/agent/join/

/v1/agent/force-leave/ /v1/agent/service/deregister/ agent endpoint: /v1/agent/register /v1/agent/deregister/ /v1/agent/pass/ /v1/agent/warn/ /v1/agent/fail/ /v1/agent/service/register

Slide 27

Slide 27 text

CONSUL AGENT HTTP API /v1/catalog/register /v1/catalog/deregister /v1/catalog/datacenters /v1/catalog/nodes catalog endpoint: /v1/catalog/services /v1/catalog/service/ /v1/catalog/node/

Slide 28

Slide 28 text

CONSUL AGENT HTTP API /v1/health/node/ /v1/health/checks/ /v1/health/service/ /v1/state/ health endpoint:

Slide 29

Slide 29 text

CONSUL AGENT HEALTH CHECK { "service": { "check": { "script": "curl localhost > /dev/null 2>&1", "interval": "10s" } } check deﬁnition:

Slide 30

Slide 30 text

CONSUL AGENT HEALTH CHECK Exit Code 0 - Check is passing ! Exit Code 1 - Check is warning ! Any Other - Check is failing check script:

Slide 31

Slide 31 text

CONSUL AGENT HEALTH CHECK

Slide 32

Slide 32 text

BLOCKING QUERIES ɾLong PollingΛ࢖ͬͯঢ়ଶมߋΛଈ࣌ݕ஌ ! ɾԠ౴͸λΠϜΞ΢τ͔࣌ҟৗൃੜ࣌ ! ?wait=&index=

Slide 33

Slide 33 text

USE CASES ɾInternal DNS ! ɾInventory Management ! ɾFailover

Slide 34

Slide 34 text

DEMO

Slide 35

Slide 35 text

BENCHMARK GET 8 CPU Cores, 2Ghz 16GB RAM 160GB SSD disk 1Gbps NIC Requests/sec rps 0 2750 5500 8250 11000 10,469.54 1,054.031 v0.2.0 v0.3.0

Slide 36

Slide 36 text

BENCHMARK GET response time distribution # of responses 0.00 3000.00 6000.00 9000.00 12000.00 response time (s) 0.001 0.007 0.014 0.017 0.023 0.030 0.033 0.062 0.094 0.125 0.156 v0.2.0 v0.3.0 8 CPU Cores, 2Ghz 16GB RAM 160GB SSD disk 1Gbps NIC

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Slide 39

Slide 39 text