About Consul

July Tech Festa 2014 slides #techfesta #consul

Transcript

1. ABOUT CONSUL
   6/22/2014
   @ijin
   
   July Tech Festa 2014
   

   View Slide

2. ABOUT ME (@IJIN)
   • Michael H. Oshita
   
   • Japanese American
   
   • 10+ Years in Japan
   
   • Software/Infra/Cloud Engineer
   
   • http://ijin.github.io
   

   View Slide

3. ABOUT ME (@IJIN)
   τϥϒϧˑ͠Ύʔͨʔͣ
   

   View Slide

4. τϥϒϧˑ͠Ύʔͨʔͣ #08
   http://bit.ly/torash08
   

   View Slide

5. View Slide

6. CONSUL
   

   View Slide

7. CONSUL
   HTTP APIͱDNSͰૢ࡞͕Ͱ͖Δ෼ࢄܕΫϥελ
   

   View Slide

8. SERVICE DISCOVERY
   • ؆୯ʹserviceΛొ࿥͠ɺࣗಈͰΫϥελ্Ͱݕग़Մೳ
   
   • DNS·ͨ͸HTTPʹΑͬͯࢀর
   
   

   View Slide

9. FAILURE DETECTION
   • ϔϧενΣοΫػೳ
   
   • ࢮΜͩϊʔυ΁ͷϦΫΤετᷖճ
   
   !
   

   View Slide

10. MULTI DATACENTER
    • ෳ਺ͷσʔληϯλʔΛ·͍ͨͰ௨৴
    
    • ଞͷDC্Ͱͷserviceͷࢀর
    
    

    View Slide

11. KEY/VALUE STORE
    • ಈతͳઃఆ৘ใͷஔ͖৔ॴͱͯ͠
    
    • Long PollingͰઃఆมߋͷଈ࣌௨஌
    
    

    View Slide

12. TECHNOLOGY
    • Consensus Protocol (Raft, FSM)
    
    
    In Search of an Understandable Consensus Algorithm - Stanford University
    
    • Gossip Protocol (Serf)
    
    
    SWIM: Scalable Weakly-consistent Infection-style Process Group Membership - Cornel University
    
    • Sessions (Chuby)
    
    
    The Chubby Lock Service for Loosely-Coupled Distributed Systems - Google
    
    • TLS, Jespen Testing
    
    !
    

    View Slide

13. ARCHITECTURE
    

    View Slide

14. CONSISTENCY
    

    View Slide

15. CONSISTENCY
    default
    
    !
    !
    consistent:
    
    !
    !
    stale:
    "?state" or "?consistent" parameter
    ɾେମStrongly Consistent
    
    ɾ৽Leaderঢ֨࣌ʹstale readͷՄೳੑ͋Γɻ
    
    ɾ଎͍
    
    !
    ɾStrongly Consistent
    
    ɾLeader͸QuromͰ֬ೝ
    
    ɾLatency͕ൃੜ͢ΔͷͰएׯ஗Ί
    
    !
    ɾStale Readɺޡࠩ͸֓Ͷ50ms͙Β͍
    
    ɾLeaderҎ֎ͷServer͕Ԡ౴Մ
    
    ɾߴ଎
    
    

    View Slide

16. CONSUL AGENT
    Nodes & Services
    foo bar baz
    redis 01.web 02.web
    node
    service
    

    View Slide

17. CONSUL AGENT
    • DNS Interface
    
    • HTTP API
    
    • Health Check
    
    !
    

    View Slide

18. CONSUL AGENT
    DNS Interface
    .node..
    Node Lookup:
    

    View Slide

19. CONSUL AGENT
    DNS Interface
    

    View Slide

20. CONSUL AGENT
    DNS Interface
    ..service.
    Service Lookup:
    

    View Slide

21. CONSUL AGENT
    DNS Interface
    

    View Slide

22. CONSUL AGENT
    DNS Interface
    

    View Slide

23. CONSUL AGENT
    DNS CACHING
    Stale Reads
    
    ɾࢀর͸σϑΥϧτͰLeader NodeͷΈ
    
    ɾ༗ޮԽ͢ΔͱଞͷServer Node͔ΒࢀরՄೳ
    
    ɾεϧʔϓοτ޲্
    
    ɾmax_staleͰ࣌ؒΛઃఆ
    
    !
    TTL
    
    ɾDNS TTL͸σϑΥϧτͰ0
    
    ɾdnsmasq౳ͰΩϟογϡՄೳʹ
    
    {
    "dns_config": {
    "service_ttl": {
    "*": "5s",
    "web": "30s"
    }
    }
    new!
    
    (0.3.0)
    

    View Slide

24. CONSUL AGENT
    HTTP API
    /v1/kv/
    key/value endpoint:
    RESTful:
    GET, PUT, DELETE
    

    View Slide

25. CONSUL AGENT
    HTTP API
    key/value:
    base64
    raw
    

    View Slide

26. CONSUL AGENT
    HTTP API
    /v1/agent/checks
    /v1/agent/services
    /v1/agent/members
    /v1/agent/self
    /v1/agent/join/
    /v1/agent/force-leave/
    /v1/agent/service/deregister/
    agent endpoint:
    /v1/agent/register
    /v1/agent/deregister/
    /v1/agent/pass/
    /v1/agent/warn/
    /v1/agent/fail/
    /v1/agent/service/register
    

    View Slide

27. CONSUL AGENT
    HTTP API
    /v1/catalog/register
    /v1/catalog/deregister
    /v1/catalog/datacenters
    /v1/catalog/nodes
    catalog endpoint:
    /v1/catalog/services
    /v1/catalog/service/
    /v1/catalog/node/
    

    View Slide

28. CONSUL AGENT
    HTTP API
    /v1/health/node/
    /v1/health/checks/
    /v1/health/service/
    /v1/state/
    health endpoint:
    

    View Slide

29. CONSUL AGENT
    HEALTH CHECK
    {
    "service": {
    "check": {
    "script": "curl localhost > /dev/null 2>&1",
    "interval": "10s"
    }
    }
    check definition:
    

    View Slide

30. CONSUL AGENT
    HEALTH CHECK
    Exit Code 0 - Check is passing
    !
    Exit Code 1 - Check is warning
    !
    Any Other - Check is failing
    check script:
    

    View Slide

31. CONSUL AGENT
    HEALTH CHECK
    

    View Slide

32. BLOCKING QUERIES
    ɾLong PollingΛ࢖ͬͯঢ়ଶมߋΛଈ࣌ݕ஌
    
    !
    ɾԠ౴͸λΠϜΞ΢τ͔࣌ҟৗൃੜ࣌
    
    !
    ?wait=&index=
    

    View Slide

33. USE CASES
    ɾInternal DNS
    
    !
    ɾInventory Management
    
    !
    ɾFailover
    

    View Slide

34. DEMO
    

    View Slide

35. BENCHMARK
    GET
    8 CPU Cores, 2Ghz
    
    16GB RAM
    
    160GB SSD disk
    
    1Gbps NIC
    Requests/sec
    rps
    0 2750 5500 8250 11000
    10,469.54
    1,054.031
    v0.2.0 v0.3.0
    

    View Slide

36. BENCHMARK
    GET
    response time distribution
    # of responses
    0.00
    3000.00
    6000.00
    9000.00
    12000.00
    response time (s)
    0.001 0.007 0.014 0.017 0.023 0.030 0.033 0.062 0.094 0.125 0.156
    v0.2.0 v0.3.0
    8 CPU Cores, 2Ghz
    
    16GB RAM
    
    160GB SSD disk
    
    1Gbps NIC
    

    View Slide

37. WEB UI
    

    View Slide

38. CONSUL-API
    Library in GO
    https://github.com/armon/consul-api
    

    View Slide

39. IN CLOSING
    consulΠΠΑʂ
    
    

    View Slide