Consulと自作OSSを活用した100台規模のWebサービス運用

YAPC::Asia 2015
http://yapcasia.org/2015/talk/edit/b335dee0-09ad-11e5-8d7a-67dc7d574c3a

Transcript

1. Consulͱࣗ࡞OSS
   Λ׆༻ͨ͠
   100୆ن໛ͷWeb
   αʔϏεӡ༻
   YAPC::Asia 2015
   @fujiwara
   

   View Slide

2. ౻ݪ ढ़Ұ࿠
   @fujiwara
   github.com/fujiwara
   sfujiwara.hatenablog.com
   ٕज़෦
   

   View Slide

3. Agenda
   Lobi
   Consul
   Consul & My OSS in Lobi
   

   View Slide

4. Game & Community
   

   View Slide

5. Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ
   

   View Slide

6. Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ
   

   View Slide

7. Lobiͷαʔό
   

   View Slide

8. Lobiͷαʔόมભ
   1. 2010~2011 AWS (US) : 4୆(?)
   2. 2011~2013 ࣗࣾαʔό : 4 ~ 20୆
   3. 2013.11~ AWS (Tokyo) : 20 ~ 100୆
   

   View Slide

9. View Slide

10. Lobiͷαʔό
    EC2ͰՔಇ͍ͯ͠Δϗετͷछྨ͕ଟ͍
    • app, sdk, stream, sdk-stream, db(3shard),
    transcode, log(aggregate,analyze),batch,
    deploy...
    ࣮૷ݴޠ Perl, Node.js, Go
    ϛυϧ΢ΣΞ͍Ζ͍Ζ
    • Nginx, MySQL, Starlet, Fluentd, Norikra,
    memcached, HAProxy, gearman, twemproxy,
    MHA, dnsmasq...
    

    View Slide

11. AWS(Tokyo)Ҡߦޙͷ೰Έ(2014)
    ಺෦DNS͕ͳ͍
    • /etc/hostsΛChefͰੜ੒
    • DNSαʔόࣗલ͸໘౗…(౰࣌Internal Route53
    ͳ͠)
    ϗετ௥Ճɼ࡟আͷස౓͕ߴ͍
    • Φʔτεέʔϧ΋͍ͨ͠
    ConsulΛ࢖͓͏ʂ
    

    View Slide

12. Consul?
    

    View Slide

13. What's Consul
    www.consul.io - HashiCorp product
    • Service Discovery
    • Health Checking
    • Key/Value Store
    • Multi Datacenter
    

    View Slide

14. Architecture
    

    View Slide

15. View Slide

16. View Slide

17. Agent
    Ϋϥελ಺ͷnodeશ୆Ͱಈ࡞͢Δdaemon
    Client mode or Server mode ͷͲͪΒ͔Ͱಈ࡞
    ϢʔβʹDNS, HTTP interfaceΛఏڙ͢Δ
    • Ϣʔβ͸جຊతʹlocalhostͷagentͱ௨৴
    • Agentಉ͕࢜RPCͰ௨৴(͋·Γҙࣝ͢Δඞཁ͸
    ͳ͍)
    GoͰॻ͔Ε͍ͯͯ1όΠφϦͰಈ࡞ (CLI΋ಉҰ)
    

    View Slide

18. Server
    Ϋϥελ಺ͷগ਺nodeͰಈ࡞(3Ҏ্ͷح਺ਪ঑)
    ৗʹͲΕ͔1୆ͷΈLeaderʹͳΔ
    • RaftΞϧΰϦζϜʹΑͬͯLeader͕બग़͞ΕΔ
    σʔλͷॻ͖ࠐΈ͸ৗʹLeaderʹ
    σʔλ͸Leader͔ΒଞͷServer΁ϨϓϦϨʔγϣ
    ϯ
    

    View Slide

19. Raft
    Raft is a protocol for implementing distributed
    consensus.
    ෼ࢄ؀ڥͰͷ߹ҙΞϧΰϦζϜ(ϓϩτίϧ)
    Leaderબग़ʹserver nodeͷա൒਺ͷಉҙ͕ඞཁ
    ͳͷͰ࠷௿3 node͕ඞཁ
    

    View Slide

20. Raft ࢀߟࢿྉ
    thesecretlivesofdata.com/raft/
    • ΞχϝʔγϣϯͰΘ͔Γ΍͍͢
    www.slideshare.net/pfi/raft-36155398
    • ೔ຊޠͰͷղઆࢿྉ
    github.com/hashicorp/raft
    • Consul͕࢖༻͍ͯ͠ΔGo࣮૷
    

    View Slide

21. Service / Node Discovery
    

    View Slide

22. Service / Node Discovery
    αʔϏε಺ͷnode͢΂ͯͰΫϥελΛߏ੒͢Δ
    • nodeɺnode্Ͱఏڙ͢ΔαʔϏεͷൃݟ
    • DNS, HTTP API
    ࢮ׆؂ࢹࣦഊͨ͠node͸Ԡ౴͔ΒࣗಈͰ֎ΕΔ
    

    View Slide

23. View Slide

24. Node Discovery
    consul members ͰΫϥελ಺ͷnodeΛҰཡ
    $ consul members
    Node Address Status Type Build Protocol DC
    my-app-i-123456 192.168.1.12:8301 alive server 0.5.2 2 dc1
    my-app-i-234567 192.168.1.23:8301 alive server 0.5.2 2 dc1
    my-app-i-345678 192.168.1.34:8301 alive server 0.5.2 2 dc1
    my-db-i-456789 192.168.1.45:8301 alive client 0.5.2 2 dc1
    my-db-i-567890 192.168.1.56:8301 alive client 0.5.2 2 dc1
    my-db-i-678901 192.168.1.67:8301 alive client 0.5.2 2 dc1
    my-app-i-987654 192.168.1.99:8301 failed client 0.5.2 2 dc1
    my-app-i-876543 192.168.1.87:8301 left client 0.5.2 2 dc1
    (࣮ࡍ͸݁Ռͷॱ൪͸ෆఆ)
    

    View Slide

25. Node Discovery
    Status=failed : agentͷࢮ׆؂ࢹʹࣦഊͨ͠node
    $ consul members -status failed
    Node Address Status Type Build Protocol DC
    my-app-i-987654 192.168.1.99:8301 failed server 0.5.2 2 dc1
    Status=left : ਖ਼ৗʹΫϥελ͔Β཭୤ͨ͠node
    $ consul members -status left
    Node Address Status Type Build Protocol DC
    my-app-i-876543 192.168.1.87:8301 left server 0.5.2 2 dc1
    

    View Slide

26. Node Discovery via DNS interface
    consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ
    $ dig @127.0.0.1 -p 8600 my-app-i-123456.node.consul
    ;; QUESTION SECTION:
    ;my-app-i-123456.node.consul. IN A
    ;; ANSWER SECTION:
    my-app-i-123456.node.consul. 0 IN A 192.168.1.12
    

    View Slide

27. Node Discovery via DNS interface
    Status=failed : DNSͰΞυϨε͕Ҿ͚Δ
    • Ұ࣌తʹࣄނͰ཭୤͍ͯ͠ΔՄೳੑ͕͋Δ
    Status=left : DNSͰΞυϨε͕Ҿ͚ͳ͘ͳΔ
    • .consul υϝΠϯͷ໊લղܾΛ consul agent ʹ
    ౤͛Δ͜ͱͰ಺෦DNSͱͯ͠ར༻Ͱ͖Δ
    • υϝΠϯ໊͸ઃఆͰมߋՄೳ
    

    View Slide

28. Service Difinition of App
    my-app-i-*:/etc/consul.d/app.json
    {
    "service": {
    "name": "app",
    "port": 3000
    }
    }
    

    View Slide

29. Service Discovery of App via DNS interface
    consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ
    $ dig @127.0.0.1 -p 8600 app.service.consul
    ;; QUESTION SECTION:
    ;app.service.consul. IN A
    ;; ANSWER SECTION:
    app.service.consul. 0 IN A 192.168.1.12
    app.service.consul. 0 IN A 192.168.1.23
    app.service.consul. 0 IN A 192.168.1.34
    

    View Slide

30. Service Discovery via DNS interface
    Answerͷॱ൪͸ϥϯμϜ (≒ Round Robin)
    UDPͰͷ໰͍߹ΘͤͰ͸ର৅͕4ΞυϨεҎ্
    ͋Δ৔߹ɺ3ΞυϨεͷΈฦΔ
    • TCPͰ͸͢΂ͯฦΔ
    TTL ઃఆՄೳ (default 0)
    

    View Slide

31. Service Discovery of App via HTTP API
    http://127.0.0.1:8500 ʹΞΫηε
    $ curl http://127.0.0.1:8500/v1/catalog/service/app
    [
    {
    "Node": "my-app-i-123456", "Address": "192.168.1.12",
    "ServiceID": "app", "ServiceName": "app", "ServicePort": 3000, ...
    },
    {
    "Node": "my-app-i-234567", "Address": "192.168.1.23", ...
    }
    {
    "Node": "my-app-i-345678", "Address": "192.168.1.34", ...
    }
    ]
    

    View Slide

32. Service Difinision of DB (master)
    my-db-i-456789:/etc/consul.d/db.json
    {
    "service": {
    "name": "db",
    "port": 3306,
    "tags": ["master"]
    }
    }
    

    View Slide

33. Service Difinision of DB (slave)
    my-db-i-567890,678901:/etc/consul.d/db.json
    {
    "service": {
    "name": "db",
    "port": 3306,
    "tags": ["slave"]
    }
    }
    

    View Slide

34. Service Discovery of DB (master/slave)
    {tag}.{service}.service.consul Ͱ໊લղܾ
    $ dig @127.0.0.1 -p 8600 master.db.service.consul
    master.db.service.consul. 0 IN A 192.168.1.45
    $ dig @127.0.0.1 -p 8600 slave.db.service.consul
    slave.db.service.consul. 0 IN A 192.168.1.67
    slave.db.service.consul. 0 IN A 192.168.1.56
    $ dig @127.0.0.1 -p 8600 db.service.consul
    db.service.consul. 0 IN A 192.168.1.56
    db.service.consul. 0 IN A 192.168.1.45
    db.service.consul. 0 IN A 192.168.1.67
    

    View Slide

35. External Service
    nodeʹؔ࿈͠ͳ͍ɺ֎෦DNSͰఆٛ͞Ε͍ͯΔ
    ໊લ΍IPΞυϨε΋αʔϏεͱͯ͠ఆٛͰ͖Δ
    $ curl -X PUT -d '{
    "Node":"rds",
    "Address":"my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com",
    "Service":{"Service": "rds"}
    }'
    http://127.0.0.1:8500/v1/catalog/register
    $ dig @127.0.0.1 -p 8600 rds.service.consul.
    ;; ANSWER SECTION:
    rds.service.consul. 0 IN CNAME my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com.
    my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. 10 IN A 192.168.1.100
    

    View Slide

36. External Service
    DNS໊Λొ࿥͢Δ৔߹ɺconsulͷઃఆʹ
    recursors (֎෦ͷ໊લղܾΛ͢ΔDNSαʔόͷ
    ΞυϨε) Λ͓ͯ͘͠
    • ઃఆ͠ͳ͍ͱ CNAME ͷΈ͕ฦΔ
    consul agent͸֎෦ͷ໊લղܾ݁ՌΛcache͠ͳ
    ͍
    

    View Slide

37. Health Checking
    

    View Slide

38. Health Checking
    ֤αʔϏεʹ͍ͭͯɺ3λΠϓͷϔϧενΣοΫ
    ΛఆٛͰ͖Δ
    • Script
    • HTTP
    • TTL
    

    View Slide

39. Health Checking by script
    ϢʔβఆٛͷϔϧενΣοΫίϚϯυΛ࣮ߦ
    exit codeͰঢ়ଶΛ௨஌ (Nagios/Sensuޓ׵)
    • 0 : success
    • 1 : warning
    • 2 : fail
    failͷ৔߹͸DNS, HTTPͷԠ౴͔Β֎ΕΔ
    

    View Slide

40. Health Checking by HTTP
    consul agent͕HTTPͰΞΫηε
    • HTTP 2xx : success
    • HTTP 429 : warning
    • ͦΕҎ֎ : fail
    

    View Slide

41. Health Checking by TTL
    ఆظతʹagentʹHTTP PUTͯ͠ੜଘΛ఻͑Δ
    TTL͕੾ΕΔ·ͰʹPUT͕ͳ͚Ε͹failѻ͍
    

    View Slide

42. Key/Value Store
    

    View Slide

43. Key/Value Store
    ೚ҙͷ஋Λग़͠ೖΕͰ͖ΔKVS
    $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1'
    true
    $ curl http://127.0.0.1:8500/v1/kv/web/key1
    [{
    "CreateIndex": 112,
    "ModifyIndex": 112,
    "LockIndex": 0,
    "Key": "web/key1",
    "Flags": 0,
    "Value": "dGVzdA=="
    }]
    

    View Slide

44. Key/Value Store
    URLҾ਺Ͱϝλσʔλ (flags) Λอ࣋Ͱ͖Δ
    64bit int, ༻్͸Ϣʔβͷ೚ҙ
    $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1?flags=123'
    true
    $ curl 'http://127.0.0.1:8500/v1/kv/web/key1'
    [{
    ...
    "Key": "web/key1",
    "Flags": 123, // <------- ͜Ε
    "Value": "dGVzdA=="
    }]
    

    View Slide

45. Key/Value Store
    jsonϨεϙϯεͷ஋͸Base64 encode͞Ε͍ͯΔ
    jsonͰ͸ͳ͘஋͚ͩੜͰऔΓ͍ͨ৔߹͸Ҿ਺ raw
    $ curl "http://127.0.0.1:8500/v1/kv/web/key1?raw"
    test
    

    View Slide

46. Key/Value Store
    ͋Δ֊૚ͷԼͷ஋Λ࠶ؼతʹऔΓ͍ͨ৔߹͸
    Ҿ਺ recurse
    $ curl -s "http://127.0.0.1:8500/v1/kv/web/?recurse"
    [
    {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0,
    "Key":"web/key1","Flags":123,"Value":"dGVzdA=="},
    {"CreateIndex":122,"ModifyIndex":122,"LockIndex":0,
    "Key":"web/key2","Flags":0,"Value":"dGVzdDI="},
    {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0,
    "Key":"web/test/1","Flags":0,"Value":"dGVzdDM="}
    ]
    όοΫΞοϓʹ΋ར༻Մೳ
    

    View Slide

47. Key/Value Store Benchmark GET
    $ wrk -c 10 -d 10 -t 2 http://127.0.0.1:8500/v1/kv/web/key1
    Server(Leader): 41,832 Requests/sec
    Server/Client(Follower): 17,281
    Server(Follower) stale mode: 37,013
    Client(Follower) stale mode: 16,938
    Consul v0.5.2 on EC2 c4.2xlarge, GOMAXPROCS=4
    

    View Slide

48. Key/Value Store Benchmark PUT
    $ wrk -c 10 -d 10 -t 2 -s put.lua http://127.0.0.1:8500/v1/kv/web/key1
    -- put.lua
    wrk.method = "PUT"
    wrk.body = "test"
    wrk.headers["Content-Type"] = "application/x-www-form-urlencoded"
    Server/Client(Follower): 427.56 Requests/sec
    

    View Slide

49. Key/Value Store
    ৗʹ 127.0.0.1:8500 Λ࢖͑͹Α͍ͷͰָ
    ͔ͳΓߴ଎ͳͷͰGET͸ԕྀͳ͘࢖͑Δ
    σϑΥϧτͰ͸͢΂ͯͷ໰͍߹ΘͤΛ
    Leader node͕ॲཧ͢Δ
    • stale modeʹ͢ΔͱLeaderҎ֎ͷServer΋Ԡ
    ౴Ͱ͖Δ
    (Ұ؏ੑʹ͍ͭͯ͸ޙड़)
    

    View Slide

50. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    

    View Slide

51. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    node,serviceͷ໊લղܾΛConsul Agent΁޲͚Δ
    ! resolv.confͰ͸ϙʔτࢦఆͰ͖ͳ͍
    ! Port 53͸ಛݖ͕ඞཁ
    → ͦͷͨΊ͚ͩʹAgentΛrootͰಈ͔͢ͷ͸ !
    Agent͸࠶ؼ໰͍߹Θͤ͸(ҰԠ)Ͱ͖Δ͕cacheػ
    ೳ͕ͳ͍ !
    → dnsmasq, bindͳͲ͔Β.consulυϝΠϯͷΈ
    forward !
    

    View Slide

52. View Slide

53. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    dnsmasqΛશ୆Ͱىಈ
    .consul υϝΠϯͷ໊લղܾ͸consul agent΁
    127.0.0.1:53 Λdnsmasq͕Listen͢Δ
    # dnsmasq.conf
    server=/consul/127.0.0.1#8600
    bind-interfaces
    listen-address=127.0.0.1
    

    View Slide

54. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    resolv.conf Ͱ (node|service).consul Λݕࡧυ
    ϝΠϯʹࢦఆ
    → node໊ɺservice໊͚ͩͰ઀ଓͰ͖Δ
    # /etc/resolv.conf
    search node.consul service.consul
    nameserver 127.0.0.1 # dnsmasq
    nameserver 172.16.0.2 # VPC resolver
    nameserver 172.16.0.254 # Unbound on EC2
    

    View Slide

55. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    consul.io/docs/guides/forwarding.html
    BINDΛ࢖͏ྫ
    

    View Slide

56. ConsulΛຊ൪؀ڥͰӡ༻͢Δ
    ͨΊʹ
    

    View Slide

57. Deployment Table
    Server node: ຊ൪Ͱ͸࠷௿3୆, 3 or 5͕ਪ঑
    consul.io/docs/internals/consensus.html
    

    View Slide

58. ServerʹඞཁͳϦιʔε
    • CPU: 2CPUͰे෼
    • GOMAXPROCS=2 Ҏ্Ͱಈ࡞ͤ͞ΔͷΛڧ͘ਪ
    ঑
    • Memory: 20MBʙ
    • Disk: 2MBʙ
    Memory, Disk͸KVͷར༻ঢ়گ࣍ୈ
    

    View Slide

59. Serverʹ͸ઐ༻ϗετ͕ඞཁʁ
    consul agentࣗମ͸ͦΕ΄ͲϦιʔεΛ
    ࢖༻͠ͳ͍ͨΊɺಉډՄೳ͕ͩ…
    Disk IO͕ߴෛՙͳ৔߹ʹRaftͷHeartbeat͕
    ࣦഊ͠΍͍͢
    • Timeout 500ms
    • Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ
    • ௨ৗ2,3ඵͰબग़͸׬ྃ͢Δ
    • ͦͷؒॻ͖ࠐΈॲཧ͕Ͱ͖ͳ͍
    

    View Slide

60. Daemonize
    consul agentࣗ਎͸Deamonಈ࡞Ϟʔυ͕ͳ͍
    • Daemontools
    • RPM & init script
    • github.com/tomhillable/consul-rpm
    • Systemd
    ͲΕͰ΋͓޷ΈͰ
    

    View Slide

61. Bootstrapping
    consul agent͸ىಈޙɺΫϥελʹjoin͢Δඞཁ
    ͕͋Δ
    1. ίϚϯυͰ consul joinΛ࣮ߦ
    2. ઃఆϑΝΠϧͰ start_join Λࢦఆ
    3. Atlas࿈ܞ
    

    View Slide

62. 1. consul join
    αʔόͷΞυϨεΛࢦఆͯ͠join
    $ consul join 192.168.1.11 192.168.1.12 192.168.1.13
    ࣗಈԽͮ͠Β͍ͷͰςετҎ֎Ͱ͸࢖Θͳ͍
    

    View Slide

63. 2. start_join
    consul.confʹαʔόͷΞυϨεΛࢦఆ͓ͯ͘͠
    {
    "start_join": [ "192.168.1.11", "192.168.1.12", "192.168.1.13"]
    }
    consul serverʹ͸ݻఆIPΞυϨεΛৼΔ͔ɺผͷ
    DNSͰղܾͰ͖ΔΑ͏ʹ͓ͯ͘͠
    

    View Slide

64. 3. Atlas࿈ܞ
    Atlas - atlas.hashicorp.com
    Vagrant Packer
    Terraform ConsulΛ౷߹͢ΔαʔϏε
    $ consul agent ... \
    -atlas=ATLAS_USERNAME/infrastructure \
    -atlas-join \
    -atlas-token="YOUR_ATLAS_TOKEN" \
    ! ServerͷΞυϨεΛ؅ཧ͢Δඞཁ͕ͳ͍ͷͰָ
    " 11nodeҎ্͸$40/node
    

    View Slide

65. ߴՄ༻ੑͷͨΊʹ
    

    View Slide

66. ߴՄ༻ੑͷͨΊʹ
    Server୆਺ʹΑΓಉ࣌ʹো֐
    Λىͯ͜͠΋໰୊ͳ͍node਺
    ͕มΘΔ
    • 3 node → 1
    • 5 node → 2
    3 nodeߏ੒࣌ɺ2୆མͪͯ࢒
    Γ1୆ʹͳͬͯ͠·͏ͱLeader
    ͕બग़Ͱ͖ͳ͍
    ௕࣌ؒ੾Γ཭͢ϝϯςφϯε
    ࣌ʹ͸Ұ࣌తʹServer nodeΛ
    ૿΍͢ख΋
    

    View Slide

67. ߴՄ༻ੑͷͨΊʹ
    Server nodeͷfailover͸ࣗಈ
    ! Ϣʔβ͸ৗʹlocalhostͷagent͚ͩΛΈ͍ͯΕ
    ͹Α͍
    

    View Slide

68. nodeো֐࣌ͷӨڹ
    ! LeaderͰ͸ͳ͍ → " ଞnodeʹ͸Өڹͳ͠
    ! Leader → " Leader࠶બग़
    σϑΥϧτͰ͸͢΂ͯͷಡΈॻ͖ΛLeader͕ॲཧ
    (ڧҰ؏ੑ)
    Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
    

    View Slide

69. Stale mode (DNS)
    Leader࠶બग़͸௨ৗ2ʙ3ඵͰ׬ྃ
    ͦͷؒ΋DNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ
    → Stale mode : Leaderະબग़Ͱ΋Ԡ౴Մೳ
    "dns_config":{
    "allow_stale": true, // default false
    "max_stale": "10s" // default 5s
    }
    ݁Ռ͸ݹ͍Մೳੑ͕͋Δ(݁Ռ੔߹ੑ)
    

    View Slide

70. DNS TTL
    default͸TTL 0 → cache͞Εͳ͍
    node, serviceผʹTTLΛઃఆՄೳ
    DNS cacheΛલஈʹ഑ஔͯ͠cacheͰ͖Δ
    "dns_config":{
    "node_ttl": "60s",
    "service_ttl": {
    "*": "15s"
    }
    }
    

    View Slide

71. Stale mode (HTTP API)
    HTTP APIͰstale modeʹ͢Δ৔߹͸Ҿ਺ stale
    $ curl "http://127.0.0.1:8500/v1/kv/web/key1?stale"
    staleҾ਺ͳ͠ͰLeaderબग़தʹΞΫηε
    → 500 Internal Server Error
    

    View Slide

72. ӡ༻தͷUpgrade
    consul.io/docs/upgrading.html
    consul.io/docs/upgrade-specific.html
    όʔδϣϯผʹ஫ҙ఺͕͋ΔͷͰυΩϡϝϯτΛ
    ॱ൪ʹAgentΛೖΕସ͑Δ͜ͱͰ
    Rolling upgradeՄೳ
    (Leader nodeೖΕସ͑Ͱ࠶બग़͸ى͖Δ)
    

    View Slide

73. ҆ఆੑ
    v0.2࣌୅͔Β1೥Ҏ্ӡ༻
    Agentϓϩηε͕མͪͨ͜ͱ͸ͳ͍
    ! ΦϖϛεͰશServerΛಉ࣌ʹམͱ͢ͱճ෮ෆೳ
    → KV͸ఆظόοΫΞοϓΛ
    (Ϣʔβσʔλ͸ೖΕͪΌμϝ ! )
    

    View Slide

74. Agentࣗମͷ؂ࢹ
    Agent processࣗମͷ؂ࢹ͸ผ్
    • process؂ࢹ(consul agent)
    • TCP/UDP 8600 (DNS)
    • TCP 8500 (HTTP)
    • http://127.0.0.1:8500/v1/status/leader
    ಺༰มߋݕ஌
    • Leader Lost→࠶બग़ͰมΘΔ
    

    View Slide

75. Φʔτεέʔϧ؀ڥͰ
    ૿ݮ͢ΔαʔόΛѻ͏
    

    View Slide

76. 2014.02ʙ Lobi Rec SDKϦϦʔε
    εϚϑΥΞϓϦʹSDKΛ૊ΈࠐΜͰPlayಈը࿥ը
    αʔόʹuploadͯ͠ม׵ɺӾཡ
    ม׵ػೳ͸ElasticTranscoderͰ࣮૷
    

    View Slide

77. ElasticTranscoder
    ! Managed ServiceͳͷͰ؅ཧָ͕
    ! ม׵ೳྗ͸উखʹεέʔϧ
    ! ͪΐͬͱ͓ߴ͍…
    • SD $0.017/min
    • HD $0.034/min
    ౤ߘ͋ͨΓ4ύλʔϯ ฏۉ2෼ = $0.204 ≒ 25ԁ
    

    View Slide

78. 2014.10
    ϞϯελʔετϥΠΫ
    Lobi Rec SDKಋೖʂʂʂ
    

    View Slide

79. ϞϯελʔετϥΠΫ
    Lobi Rec SDKಋೖ
    ௒ώοτήʔϜʹ͖ͭ
    େྔͷಈը͕…
    !""""""!
    

    View Slide

80. EC2 Spot InstanceͰಈըม׵
    ! ؅ཧ͕໘౗
    ! উखʹεέʔϧ͸ͯ͘͠Εͳ͍
    ! ElasticTranscoderΑΓѹ౗తʹ҆Ձ
    • ElasticTranscoder = $0.204/౤ߘ
    • EC2 Spot cc2.8xlarge(32core) = $0.45/hour
    

    View Slide

81. Spot InstanceͰΦʔτεέʔϧ
    

    View Slide

82. Spot InstanceͰΦʔτεέʔϧ
    • εέʔϧΞ΢τ: CPUෛՙͰ͸ͳ͘JobͷྔͰ
    • Job͕ཷ·Βͳ͍ݶΓCPU 100%͸ !
    • εέʔϧΠϯ: CPU idle͕શମͷ25%Ҏ্
    

    View Slide

83. Φʔτεέʔϧ؀ڥͰͷΠϯελϯεىಈ
    1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ
    2. consul join
    3. ChefʹΑΔϓϩϏδϣχϯά
    4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy
    5. Zabbix΁ࣗಈొ࿥ͯ͠؂ࢹର৅ʹ௥Ճ
    

    View Slide

84. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ
    ConsulͷͨΊʹҰҙͳhostname͕ඞཁ
    Cloud-InitͰىಈ͢ΔϗετͷλΠϓΛઃఆ
    #cloud-config
    runcmd:
    - [sh, -c, 'echo "HOSTNAME_PREFIX=transcode" > /etc/sysconfig/hostname-prefix']
    rc.localͰಡΈࠐΉ
    # /etc/rc.local
    if [ -f /etc/sysconfig/hostname-prefix ]; then
    . /etc/sysconfig/hostname-prefix
    fi
    

    View Slide

85. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ
    hostnameΛ prefix + InstanceID
    EC2 Name tag ෇༩
    # /etc/rc.local
    instance_id=$(curl -s 169.254.169.254/latest/meta-data/instance-id)
    new_hostname="${HOSTNAME_PREFIX}-$instance_id"
    hostname $new_hostname
    aws ec2 create-tags \
    --resources $instance_id \
    --tags "Key=Name,Value=$new_hostname"
    

    View Slide

86. 2. ConsulΫϥελʹjoin
    consul agent -node $(hostname)
    ࣗಈతʹ಺෦DNSͰ໊લղܾՄೳʹ
    

    View Slide

87. 3. ChefʹΑΔϓϩϏδϣχϯά
    Consul KV͔ΒϗετλΠϓ͝ͱͷJSONΛऔಘ͠
    ࣮ͯߦ
    JSON=$(curl -s "localhost:8500/v1/kv/nodes/bootup/${HOSTNAME_PREFIX}.json?raw")
    echo "$JSON" > /tmp/chef-bootup.json
    chef-client -j /tmp/chef-bootup.json
    ॳճChef࣮ߦ࣌ʹChef-Serverʹొ࿥͞ΕΔ
    

    View Slide

88. 4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy
    Stretcher(ޙड़)Ͱdeploy
    5. Zabbix΁ࣗಈొ࿥ͯ͠؂ࢹର৅ʹ௥Ճ
    deploy࣮ߦޙɺzabbix-agentΛىಈˠࣗಈొ࿥
    deploy׬ྃલʹىಈ͢ΔͱΞϥʔτ͕ൃ๒ͯ͠͠
    ·͏
    

    View Slide

89. StretcherΛར༻ͨ͠σϓϩΠ
    Dan Zen https://www.flickr.com/photos/danzen/2288626158/
    

    View Slide

90. What's Stretcher
    github.com/fujiwara/stretcher
    Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ
    

    View Slide

91. Why Stretcher?
    Archer(rsync) ʹΑΔதԝϗετ͔Βͷdeploy
    ! pushͰ͸ΦʔτεέʔϧʹରԠͰ͖ͳ͍
    ! ֤ϗετ͔ΒrsyncͰpull?
    → buildதʹrsync͞ΕͨΒ…
    ! ֤ϗετ͔Βgit pull?
    → grunt, GoͳͲͷbuildੜ੒෺ΛೖΕͨ͘ͳ͍
    ! ୆਺͕ଟ͍ͱssh+rsync΋git pull΋πϥ͍
    ! AMI࡞Γ௚͠&ೖΕସ͑͸଴ͯͳ͍
    

    View Slide

92. Inspired by
    github.com/sorah/mamiya
    &
    AWS CodeDeploy
    

    View Slide

93. ઃܭํ਑
    AWS͡Όͳͯ͘΋ಈ͘
    rsync & ίϚϯυ࣮ߦɺͱ͍͏ϑϩʔ͸౿ऻ
    Consul eventͰΠϕϯτ௨஌
    Consul ͱ͸ૄ݁߹ (consul watchͰىಈ)
    GoͰॻ͘
    

    View Slide

94. Architecture
    

    View Slide

95. Consul event
    ֤nodeʹGossip ProtocolͰΠϕϯτΛ
    ૹ৴͢Δ࢓૊Έ
    $ consul event -name EVENT_NAME [-node REGEX] PAYLOAD
    Event ID: 3b1f3199-6e69-4b82-4812-b35058864fdd
    ࢦఆͨ͠Πϕϯτ໊Ͱ
    (ਖ਼نදݱʹϚον͢Δnodeʹ)
    payloadΛૹ৴
    

    View Slide

96. Consul watch
    ࢦఆͨ͠ΠϕϯτΛड৴ͨ͠ΒίϚϯυΛ
    ࣮ߦ͢Δ࢓૊Έ
    $ consul watch -type event -name EVENT_NAME COMMAND
    payload͸ඪ४ೖྗ͔ΒJSONͰ౉͞ΕΔ
    [{
    "ID": "3b1f3199-6e69-4b82-4812-b35058864fdd",
    "Name": "test",
    "Payload": "TXkgcGF5bG9hZA==",
    ...
    }]
    

    View Slide

97. Deployment process
    1. ΞϓϦέʔγϣϯΛbuildͯ͠tar.gzʹ͢Δ
    ґଘcpan moduleͳͲ͢΂ͯݻΊΔ
    2. खॱॻ(manifest)Λॻ͘
    3. tar.gz, manifestΛS3(or httpd)ʹ্͛Δ
    4. consul event Ͱ manifest URLΛ௨஌
    consul event -name deploy s3://...
    ✄---------- ͜͜·ͰstretcherͰ͸ͳ͍ ---------✄
    

    View Slide

98. ✄------------ stretcher͔͜͜Β ------------✄
    consul watch -type event -name deploy stretcher
    1. event͔Βmanifest URLΛऔಘ
    2. tar.gzΛऔಘͯ͠TMPDIRʹల։
    3. rsync -av --deleteͰߋ৽
    4. command࣮ߦ
    • ΞϓϦέʔγϣϯ࠶ىಈͳͲ
    

    View Slide

99. Manifest
    src: s3://example.com/app.tar.gz
    checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93
    dest: /home/stretcher/app
    commands:
    pre:
    - echo 'staring deploy'
    post:
    - echo 'deploy done'
    success:
    - cat >> /path/to/success.log
    failure:
    - cat >> /path/to/failure.log
    excludes:
    - "*.pid"
    - "*.socket"
    

    View Slide

100. LobiͰͷdeploy
     tar.gz ໿200MB
     ల։͢Δͱ໿400MB
     • CPAN modules 110MB
     • node_modules 10MB × 5
     • Go app binaries 8MB × 5
     • Static files (S3ʹஔ͖͍ͨ)
     

     View Slide

101. LobiͰͷdeploy
     1. ‐ Push to production branch
     2. ! Build (1 min~)
     carton install, grunt, npm install, go
     build ...
     3. " Pack tar.gz & Upload (1 min)
     4. # Deploy by Startecher (10~30 sec)
     

     View Slide

102. LobiͰͷdeploy
     consul event ૹ৴͔Β10ʙ20ඵͰ׬ྃ !
     2015/08/05 14:58:30 Starting up stretcher agent
     2015/08/05 14:58:30 Waiting for consul events from STDIN...
     2015/08/05 14:58:30 Executing manifest: s3://...
     2015/08/05 14:58:33 Extract archive: /dev/shm/stretcher539962129 to /dev/shm/stretcher_src648982332
     2015/08/05 14:58:36 rsync [-av --delete --exclude-from
     /dev/shm/stretcher_src648982332/conf/rsync_exclude.web
     /dev/shm/stretcher_src648982332/ /home/xxx/web/]
     2015/08/05 14:58:36 sending incremental file list
     ...
     sent 787780 bytes received 5230 bytes 1586020.00 bytes/sec
     total size is 359702435 speedup is 453.59
     2015/08/05 14:58:36 invoking command: /home/xxx/web/refresh_services.sh
     2015/08/05 14:58:41 success.
     2015/08/05 14:58:41 Deploy manifest succeeded.
     Rollback͍ͨ͠ˠ௚લͷmanifestΛeventૹ৴
     →10secͰ໭Δ
     

     View Slide

103. Deployͷ׬ྃΛ଴ͭ
     Πϕϯτૹ৴ݩͰ͸࣮ߦ׬͕ྃ෼͔Βͳ͍
     • ࣦഊ͢Δϗετ͕͋Δ͔΋͠Εͳ͍
     • Քಇதͷશ୆Ͱdeploy׬ྃͨ͠ͷΛ଴͍ͪͨ
     ……ϩά΋ݟ͍͚ͨͲ100୆෼Ͳ͏΍ͬͯʁ
     

     View Slide

104. Consul KV Dashboard
     

     View Slide

105. Chef, Serverspec, Deploy, etc...
     ֤ϗετͰ࣮ߦͨ݁͠ՌΛ֬ೝ͍ͨ͠΋ͷ
     • Chef
     • Serverspec
     • Stretcher
     • etc
     

     View Slide

106. IRC / Slackʹ௨஌ʁ
     ࣾ಺༻ nopaste-cli command
     $ nopaste-cli -channel "lobi" -summary "Deploy done!" < deploy.log
     nopasteʹPOSTͱಉ࣌ʹURL͕௨஌͞ΕΔ
     100୆͋Δͱ௨஌͕……!!!!!!!!×100
     Ͳ͔͜ͰҰཡͯ͠ݟ͍ͨʂ
     

     View Slide

107. Consul KV Dashboard
     github.com/fujiwara/consul-kv-dashboard
     Consul KVΛσʔλετΞʹͨ͠
     μογϡϘʔυWebΞϓϦ
     

     View Slide

108. σʔλొ࿥
     Consul HTTP APIͰ௚઀ૹΔ
     $ curl -X PUT -d "message" \
     '127.0.0.1:8500/v1/kv/dashboard/example/myhostname?flags=1422607461000'
     

     View Slide

109. keyߏ଄
     /v1/kv/dashboard/{category}/{nodename}?
     flags=({unixtime} * 1000 + {status})
     • category: chef, serverspec, deploy...
     • nodename: Consulͷnode໊
     • flags: unixtime * 1000 + status
     • status: 0=Success 1=Warning
     2=Danger 3=Info
     

     View Slide

110. ը໘ͷଈ࣌ߋ৽
     ϒϩοΩϯάΫΤϦΛ࢖͏
     consul.io/docs/agent/http.html
     $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?recurese
     HTTP/1.1 200 OK
     Content-Type: application/json
     X-Consul-Index: 261975
     [{"CreateIndex":261891,"ModifyIndex":261975,"LockIndex":0,
     "Key":"dashboard/chef/myhost","Flags":1422602855000,"Value":".....
     

     View Slide

111. Blocking query
     Ϩεϙϯεϔομͷ X-Consul-Index Λ
     ࣍ͷϦΫΤετͷҾ਺ʹࢦఆ
     $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost
     HTTP/1.1 200 OK
     X-Consul-Index: 261975
     ...
     $ curl 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?index=261975
     ৽͍͠σʔλ͕ൃੜ͢Δ·ͰϨεϙϯε͕஗Ԇ
     ͍ΘΏΔ Long pooling
     

     View Slide

112. Blocking queryͷ׆༻
     Consul Template
     github.com/hashicorp/consul-template
     KV, node, service౳ͷঢ়ଶมԽΛଈ൓ө
     Template→fileߋ৽ˠcommand࣮ߦ
     $ consul-template \
     -consul 127.0.0.1:8500 \
     -template "/tmp/template.ctmpl:/var/www/nginx.conf:service nginx restart"
     

     View Slide

113. consul-kv-dashboard͕΍Δ͜ͱ(1)
     GoͰॻ͔ΕͨWebΞϓϦέʔγϣϯ
     HTML, CSS, JavaScriptΛ഑෍
     • go-bindataͰ·ͱΊͨϑΝΠϧΛ
     http.FileServerͰ഑৴
     • όΠφϦҰݸΛஔ͍ͯىಈ͢Δ͚ͩͰ
     ੩తϑΝΠϧ΋഑৴Ͱ͖Δ
     

     View Slide

114. consul-kv-dashboard͕΍Δ͜ͱ(2)
     Consul HTTP API ΁ͷ reverse proxy
     • /api/... → 127.0.0.1:8500/v1/kv/dashboard/...
     • ϨεϙϯεͷJSONΛ੔ܗ
     {"Flags":1422608524001}
     !
     {"timestamp":"2015-01-30 18:02:04 +0900","status":"warning"}
     • /v1/catalog/nodes Λblocking queryͰ؂ࢹ
     ଘࡏ͢ΔnodeͷσʔλͷΈϑΟϧλ
     

     View Slide

115. consul-kv-dashboard trigger
     $ consul-kv-dashboard -trigger COMMAND
     ΧςΰϦຖʹঢ়ଶ(success→warningͳͲ)͕
     มԽͨ͠ΒίϚϯυ࣮ߦՄೳ
     JSON͕ඪ४ೖྗʹ౉͞ΕΔ
     {
     "category":"testing",
     "node":"web01",
     "address":"192.168.1.10",
     "timestamp":"2015-01-21 11:22:33 +0900",
     "status":"danger",
     "key":"","data":"failure!!"
     }
     

     View Slide

116. ΦʔτεέʔϧͰͷ
     Deploy Tips
     

     View Slide

117. ىಈޙʹ࠷৽൛Λdeploy
     deploy࣌ɺKVʹ࠷৽ͷmanifest URLΛอଘ
     1. consul join
     2. stretcherىಈ
     3. ࠷৽ͷmanifest URLΛKV͔Βऔಘ
     4. ࣗ෼ࣗ਎ʹ event (manifest URL) Λૹ৴
     5. deploy !
     

     View Slide

118. ΦʔτεέʔϧͰͷ஫ҙ఺
     ! AMIʹ࢒͍ͬͯΔݹ͍ΞϓϦ͕ىಈ
     " ࠷৽ͷ deploy IDͰͳ͍৔߹͸ىಈ͠ͳ͍
     deploy࣌: unique ͳ ID Λൃߦ
     • ϑΝΠϧʹॻ͍ͯ tar ʹೖΕΔ
     • KV ʹ΋ೖΕΔ
     ىಈ࣌:ϩʔΧϧϑΝΠϧͷ ID ͱ KV Λൺֱ
     • ҟͳ͍ͬͯͨΒ sleep 10 && exit → restart
     

     View Slide

119. bash-completionͰ
     sshͷϗετ໊ิ׬
     

     View Slide

120. bash-completionͰsshͷϗετ໊ิ׬
     σϑΥϧτͰ͸ ~/.ssh/known_hosts Λݩʹิ׬
     ! աڈʹsshͨ͜͠ͱ͕͋Δϗετ͕ग़ͯ͘Δ
     " ݱࡏ͸طʹଘࡏ͠ͳ͍Մೳੑ͕͋Δ
     " Ұ౓΋sshͨ͜͠ͱ͕ͳ͍৽نϗετ͸ग़ͯ͜
     ͳ͍
     

     View Slide

121. bash-completionͰsshͷϗετ໊ิ׬
     ~/.bash_profile
     _known_hosts_real()
     {
     local members=$(consul members -status=alive | awk '!/Node/{printf("%s ", $1)}')
     COMPREPLY=( $( \
     compgen -W "$members" \
     ${COMP_WORDS[COMP_CWORD]} \
     ) )
     return 0
     }
     ! ݱࡏaliveͳϗετͷΈ͕ग़ͯ͘Δʂ
     

     View Slide

122. ·ͱΊ
     

     View Slide

123. ·ͱΊ
     Consul͸ػೳ๛෋…͕ͩ
     ࢖͍͍ͨػೳ͚ͩ࢖͑͹Α͍
     • KVɺEventɺHealth Check...
     DNSɺHTTPΛ࢖͍͜ͳ͢ͱߴ౓ͳࣗಈԽ͕Մೳ
     • consul-templateͰಈతLBઃఆ
     

     View Slide

124. Questions?
     • Architecture, Service Discovery
     Health Checking, Key/Value Store
     • ಺෦DNS
     • ຊ൪؀ڥӡ༻ / ߴՄ༻ੑ
     • Φʔτεέʔϧ
     • Stretcher
     • Consul KV Dashboard
     

     View Slide