Upgrade to Pro — share decks privately, control downloads, hide ads and more …

About Consul

About Consul

July Tech Festa 2014 slides #techfesta #consul

Michael H. Oshita

June 22, 2014
Tweet

More Decks by Michael H. Oshita

Other Decks in Technology

Transcript

  1. ABOUT CONSUL
    6/22/2014
    @ijin

    July Tech Festa 2014

    View full-size slide

  2. ABOUT ME (@IJIN)
    • Michael H. Oshita

    • Japanese American

    • 10+ Years in Japan

    • Software/Infra/Cloud Engineer

    • http://ijin.github.io

    View full-size slide

  3. ABOUT ME (@IJIN)
    τϥϒϧˑ͠Ύʔͨʔͣ

    View full-size slide

  4. τϥϒϧˑ͠Ύʔͨʔͣ #08
    http://bit.ly/torash08

    View full-size slide

  5. CONSUL
    HTTP APIͱDNSͰૢ࡞͕Ͱ͖Δ෼ࢄܕΫϥελ

    View full-size slide

  6. SERVICE DISCOVERY
    • ؆୯ʹserviceΛొ࿥͠ɺࣗಈͰΫϥελ্Ͱݕग़Մೳ

    • DNS·ͨ͸HTTPʹΑͬͯࢀর

    View full-size slide

  7. FAILURE DETECTION
    • ϔϧενΣοΫػೳ

    • ࢮΜͩϊʔυ΁ͷϦΫΤετᷖճ

    !

    View full-size slide

  8. MULTI DATACENTER
    • ෳ਺ͷσʔληϯλʔΛ·͍ͨͰ௨৴

    • ଞͷDC্Ͱͷserviceͷࢀর

    View full-size slide

  9. KEY/VALUE STORE
    • ಈతͳઃఆ৘ใͷஔ͖৔ॴͱͯ͠

    • Long PollingͰઃఆมߋͷଈ࣌௨஌

    View full-size slide

  10. TECHNOLOGY
    • Consensus Protocol (Raft, FSM)


    In Search of an Understandable Consensus Algorithm - Stanford University

    • Gossip Protocol (Serf)


    SWIM: Scalable Weakly-consistent Infection-style Process Group Membership - Cornel University

    • Sessions (Chuby)


    The Chubby Lock Service for Loosely-Coupled Distributed Systems - Google

    • TLS, Jespen Testing

    !

    View full-size slide

  11. ARCHITECTURE

    View full-size slide

  12. CONSISTENCY
    default

    !
    !
    consistent:

    !
    !
    stale:
    "?state" or "?consistent" parameter
    ɾେମStrongly Consistent

    ɾ৽Leaderঢ֨࣌ʹstale readͷՄೳੑ͋Γɻ

    ɾ଎͍

    !
    ɾStrongly Consistent

    ɾLeader͸QuromͰ֬ೝ

    ɾLatency͕ൃੜ͢ΔͷͰएׯ஗Ί

    !
    ɾStale Readɺޡࠩ͸֓Ͷ50ms͙Β͍

    ɾLeaderҎ֎ͷServer͕Ԡ౴Մ

    ɾߴ଎

    View full-size slide

  13. CONSUL AGENT
    Nodes & Services
    foo bar baz
    redis 01.web 02.web
    node
    service

    View full-size slide

  14. CONSUL AGENT
    • DNS Interface

    • HTTP API

    • Health Check

    !

    View full-size slide

  15. CONSUL AGENT
    DNS Interface
    .node..
    Node Lookup:

    View full-size slide

  16. CONSUL AGENT
    DNS Interface

    View full-size slide

  17. CONSUL AGENT
    DNS Interface
    ..service.
    Service Lookup:

    View full-size slide

  18. CONSUL AGENT
    DNS Interface

    View full-size slide

  19. CONSUL AGENT
    DNS Interface

    View full-size slide

  20. CONSUL AGENT
    DNS CACHING
    Stale Reads

    ɾࢀর͸σϑΥϧτͰLeader NodeͷΈ

    ɾ༗ޮԽ͢ΔͱଞͷServer Node͔ΒࢀরՄೳ

    ɾεϧʔϓοτ޲্

    ɾmax_staleͰ࣌ؒΛઃఆ

    !
    TTL

    ɾDNS TTL͸σϑΥϧτͰ0

    ɾdnsmasq౳ͰΩϟογϡՄೳʹ

    {
    "dns_config": {
    "service_ttl": {
    "*": "5s",
    "web": "30s"
    }
    }
    new!

    (0.3.0)

    View full-size slide

  21. CONSUL AGENT
    HTTP API
    /v1/kv/
    key/value endpoint:
    RESTful:
    GET, PUT, DELETE

    View full-size slide

  22. CONSUL AGENT
    HTTP API
    key/value:
    base64
    raw

    View full-size slide

  23. CONSUL AGENT
    HTTP API
    /v1/agent/checks
    /v1/agent/services
    /v1/agent/members
    /v1/agent/self
    /v1/agent/join/
    /v1/agent/force-leave/
    /v1/agent/service/deregister/
    agent endpoint:
    /v1/agent/register
    /v1/agent/deregister/
    /v1/agent/pass/
    /v1/agent/warn/
    /v1/agent/fail/
    /v1/agent/service/register

    View full-size slide

  24. CONSUL AGENT
    HTTP API
    /v1/catalog/register
    /v1/catalog/deregister
    /v1/catalog/datacenters
    /v1/catalog/nodes
    catalog endpoint:
    /v1/catalog/services
    /v1/catalog/service/
    /v1/catalog/node/

    View full-size slide

  25. CONSUL AGENT
    HTTP API
    /v1/health/node/
    /v1/health/checks/
    /v1/health/service/
    /v1/state/
    health endpoint:

    View full-size slide

  26. CONSUL AGENT
    HEALTH CHECK
    {
    "service": {
    "check": {
    "script": "curl localhost > /dev/null 2>&1",
    "interval": "10s"
    }
    }
    check definition:

    View full-size slide

  27. CONSUL AGENT
    HEALTH CHECK
    Exit Code 0 - Check is passing
    !
    Exit Code 1 - Check is warning
    !
    Any Other - Check is failing
    check script:

    View full-size slide

  28. CONSUL AGENT
    HEALTH CHECK

    View full-size slide

  29. BLOCKING QUERIES
    ɾLong PollingΛ࢖ͬͯঢ়ଶมߋΛଈ࣌ݕ஌

    !
    ɾԠ౴͸λΠϜΞ΢τ͔࣌ҟৗൃੜ࣌

    !
    ?wait=&index=

    View full-size slide

  30. USE CASES
    ɾInternal DNS

    !
    ɾInventory Management

    !
    ɾFailover

    View full-size slide

  31. BENCHMARK
    GET
    8 CPU Cores, 2Ghz

    16GB RAM

    160GB SSD disk

    1Gbps NIC
    Requests/sec
    rps
    0 2750 5500 8250 11000
    10,469.54
    1,054.031
    v0.2.0 v0.3.0

    View full-size slide

  32. BENCHMARK
    GET
    response time distribution
    # of responses
    0.00
    3000.00
    6000.00
    9000.00
    12000.00
    response time (s)
    0.001 0.007 0.014 0.017 0.023 0.030 0.033 0.062 0.094 0.125 0.156
    v0.2.0 v0.3.0
    8 CPU Cores, 2Ghz

    16GB RAM

    160GB SSD disk

    1Gbps NIC

    View full-size slide

  33. CONSUL-API
    Library in GO
    https://github.com/armon/consul-api

    View full-size slide

  34. IN CLOSING
    consulΠΠΑʂ

    View full-size slide