Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consulと自作OSSを活用した100台規模のWebサービス運用

FUJIWARA Shunichiro
August 21, 2015
22k

 Consulと自作OSSを活用した100台規模のWebサービス運用

FUJIWARA Shunichiro

August 21, 2015
Tweet

More Decks by FUJIWARA Shunichiro

Transcript

  1. Consulͱࣗ࡞OSS
    Λ׆༻ͨ͠
    100୆ن໛ͷWeb
    αʔϏεӡ༻
    YAPC::Asia 2015
    @fujiwara

    View Slide

  2. ౻ݪ ढ़Ұ࿠
    @fujiwara
    github.com/fujiwara
    sfujiwara.hatenablog.com
    ٕज़෦

    View Slide

  3. Agenda
    Lobi
    Consul
    Consul & My OSS in Lobi

    View Slide

  4. Game & Community

    View Slide

  5. Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ

    View Slide

  6. Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ

    View Slide

  7. Lobiͷαʔό

    View Slide

  8. Lobiͷαʔόมભ
    1. 2010~2011 AWS (US) : 4୆(?)
    2. 2011~2013 ࣗࣾαʔό : 4 ~ 20୆
    3. 2013.11~ AWS (Tokyo) : 20 ~ 100୆

    View Slide

  9. View Slide

  10. Lobiͷαʔό
    EC2ͰՔಇ͍ͯ͠Δϗετͷछྨ͕ଟ͍
    • app, sdk, stream, sdk-stream, db(3shard),
    transcode, log(aggregate,analyze),batch,
    deploy...
    ࣮૷ݴޠ Perl, Node.js, Go
    ϛυϧ΢ΣΞ͍Ζ͍Ζ
    • Nginx, MySQL, Starlet, Fluentd, Norikra,
    memcached, HAProxy, gearman, twemproxy,
    MHA, dnsmasq...

    View Slide

  11. AWS(Tokyo)Ҡߦޙͷ೰Έ(2014)
    ಺෦DNS͕ͳ͍
    • /etc/hostsΛChefͰੜ੒
    • DNSαʔόࣗલ͸໘౗…(౰࣌Internal Route53
    ͳ͠)
    ϗετ௥Ճɼ࡟আͷස౓͕ߴ͍
    • Φʔτεέʔϧ΋͍ͨ͠
    ConsulΛ࢖͓͏ʂ

    View Slide

  12. Consul?

    View Slide

  13. What's Consul
    www.consul.io - HashiCorp product
    • Service Discovery
    • Health Checking
    • Key/Value Store
    • Multi Datacenter

    View Slide

  14. Architecture

    View Slide

  15. View Slide

  16. View Slide

  17. Agent
    Ϋϥελ಺ͷnodeશ୆Ͱಈ࡞͢Δdaemon
    Client mode or Server mode ͷͲͪΒ͔Ͱಈ࡞
    ϢʔβʹDNS, HTTP interfaceΛఏڙ͢Δ
    • Ϣʔβ͸جຊతʹlocalhostͷagentͱ௨৴
    • Agentಉ͕࢜RPCͰ௨৴(͋·Γҙࣝ͢Δඞཁ͸
    ͳ͍)
    GoͰॻ͔Ε͍ͯͯ1όΠφϦͰಈ࡞ (CLI΋ಉҰ)

    View Slide

  18. Server
    Ϋϥελ಺ͷগ਺nodeͰಈ࡞(3Ҏ্ͷح਺ਪ঑)
    ৗʹͲΕ͔1୆ͷΈLeaderʹͳΔ
    • RaftΞϧΰϦζϜʹΑͬͯLeader͕બग़͞ΕΔ
    σʔλͷॻ͖ࠐΈ͸ৗʹLeaderʹ
    σʔλ͸Leader͔ΒଞͷServer΁ϨϓϦϨʔγϣ
    ϯ

    View Slide

  19. Raft
    Raft is a protocol for implementing distributed
    consensus.
    ෼ࢄ؀ڥͰͷ߹ҙΞϧΰϦζϜ(ϓϩτίϧ)
    Leaderબग़ʹserver nodeͷա൒਺ͷಉҙ͕ඞཁ
    ͳͷͰ࠷௿3 node͕ඞཁ

    View Slide

  20. Raft ࢀߟࢿྉ
    thesecretlivesofdata.com/raft/
    • ΞχϝʔγϣϯͰΘ͔Γ΍͍͢
    www.slideshare.net/pfi/raft-36155398
    • ೔ຊޠͰͷղઆࢿྉ
    github.com/hashicorp/raft
    • Consul͕࢖༻͍ͯ͠ΔGo࣮૷

    View Slide

  21. Service / Node Discovery

    View Slide

  22. Service / Node Discovery
    αʔϏε಺ͷnode͢΂ͯͰΫϥελΛߏ੒͢Δ
    • nodeɺnode্Ͱఏڙ͢ΔαʔϏεͷൃݟ
    • DNS, HTTP API
    ࢮ׆؂ࢹࣦഊͨ͠node͸Ԡ౴͔ΒࣗಈͰ֎ΕΔ

    View Slide

  23. View Slide

  24. Node Discovery
    consul members ͰΫϥελ಺ͷnodeΛҰཡ
    $ consul members
    Node Address Status Type Build Protocol DC
    my-app-i-123456 192.168.1.12:8301 alive server 0.5.2 2 dc1
    my-app-i-234567 192.168.1.23:8301 alive server 0.5.2 2 dc1
    my-app-i-345678 192.168.1.34:8301 alive server 0.5.2 2 dc1
    my-db-i-456789 192.168.1.45:8301 alive client 0.5.2 2 dc1
    my-db-i-567890 192.168.1.56:8301 alive client 0.5.2 2 dc1
    my-db-i-678901 192.168.1.67:8301 alive client 0.5.2 2 dc1
    my-app-i-987654 192.168.1.99:8301 failed client 0.5.2 2 dc1
    my-app-i-876543 192.168.1.87:8301 left client 0.5.2 2 dc1
    (࣮ࡍ͸݁Ռͷॱ൪͸ෆఆ)

    View Slide

  25. Node Discovery
    Status=failed : agentͷࢮ׆؂ࢹʹࣦഊͨ͠node
    $ consul members -status failed
    Node Address Status Type Build Protocol DC
    my-app-i-987654 192.168.1.99:8301 failed server 0.5.2 2 dc1
    Status=left : ਖ਼ৗʹΫϥελ͔Β཭୤ͨ͠node
    $ consul members -status left
    Node Address Status Type Build Protocol DC
    my-app-i-876543 192.168.1.87:8301 left server 0.5.2 2 dc1

    View Slide

  26. Node Discovery via DNS interface
    consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ
    $ dig @127.0.0.1 -p 8600 my-app-i-123456.node.consul
    ;; QUESTION SECTION:
    ;my-app-i-123456.node.consul. IN A
    ;; ANSWER SECTION:
    my-app-i-123456.node.consul. 0 IN A 192.168.1.12

    View Slide

  27. Node Discovery via DNS interface
    Status=failed : DNSͰΞυϨε͕Ҿ͚Δ
    • Ұ࣌తʹࣄނͰ཭୤͍ͯ͠ΔՄೳੑ͕͋Δ
    Status=left : DNSͰΞυϨε͕Ҿ͚ͳ͘ͳΔ
    • .consul υϝΠϯͷ໊લղܾΛ consul agent ʹ
    ౤͛Δ͜ͱͰ಺෦DNSͱͯ͠ར༻Ͱ͖Δ
    • υϝΠϯ໊͸ઃఆͰมߋՄೳ

    View Slide

  28. Service Difinition of App
    my-app-i-*:/etc/consul.d/app.json
    {
    "service": {
    "name": "app",
    "port": 3000
    }
    }

    View Slide

  29. Service Discovery of App via DNS interface
    consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ
    $ dig @127.0.0.1 -p 8600 app.service.consul
    ;; QUESTION SECTION:
    ;app.service.consul. IN A
    ;; ANSWER SECTION:
    app.service.consul. 0 IN A 192.168.1.12
    app.service.consul. 0 IN A 192.168.1.23
    app.service.consul. 0 IN A 192.168.1.34

    View Slide

  30. Service Discovery via DNS interface
    Answerͷॱ൪͸ϥϯμϜ (≒ Round Robin)
    UDPͰͷ໰͍߹ΘͤͰ͸ର৅͕4ΞυϨεҎ্
    ͋Δ৔߹ɺ3ΞυϨεͷΈฦΔ
    • TCPͰ͸͢΂ͯฦΔ
    TTL ઃఆՄೳ (default 0)

    View Slide

  31. Service Discovery of App via HTTP API
    http://127.0.0.1:8500 ʹΞΫηε
    $ curl http://127.0.0.1:8500/v1/catalog/service/app
    [
    {
    "Node": "my-app-i-123456", "Address": "192.168.1.12",
    "ServiceID": "app", "ServiceName": "app", "ServicePort": 3000, ...
    },
    {
    "Node": "my-app-i-234567", "Address": "192.168.1.23", ...
    }
    {
    "Node": "my-app-i-345678", "Address": "192.168.1.34", ...
    }
    ]

    View Slide

  32. Service Difinision of DB (master)
    my-db-i-456789:/etc/consul.d/db.json
    {
    "service": {
    "name": "db",
    "port": 3306,
    "tags": ["master"]
    }
    }

    View Slide

  33. Service Difinision of DB (slave)
    my-db-i-567890,678901:/etc/consul.d/db.json
    {
    "service": {
    "name": "db",
    "port": 3306,
    "tags": ["slave"]
    }
    }

    View Slide

  34. Service Discovery of DB (master/slave)
    {tag}.{service}.service.consul Ͱ໊લղܾ
    $ dig @127.0.0.1 -p 8600 master.db.service.consul
    master.db.service.consul. 0 IN A 192.168.1.45
    $ dig @127.0.0.1 -p 8600 slave.db.service.consul
    slave.db.service.consul. 0 IN A 192.168.1.67
    slave.db.service.consul. 0 IN A 192.168.1.56
    $ dig @127.0.0.1 -p 8600 db.service.consul
    db.service.consul. 0 IN A 192.168.1.56
    db.service.consul. 0 IN A 192.168.1.45
    db.service.consul. 0 IN A 192.168.1.67

    View Slide

  35. External Service
    nodeʹؔ࿈͠ͳ͍ɺ֎෦DNSͰఆٛ͞Ε͍ͯΔ
    ໊લ΍IPΞυϨε΋αʔϏεͱͯ͠ఆٛͰ͖Δ
    $ curl -X PUT -d '{
    "Node":"rds",
    "Address":"my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com",
    "Service":{"Service": "rds"}
    }'
    http://127.0.0.1:8500/v1/catalog/register
    $ dig @127.0.0.1 -p 8600 rds.service.consul.
    ;; ANSWER SECTION:
    rds.service.consul. 0 IN CNAME my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com.
    my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. 10 IN A 192.168.1.100

    View Slide

  36. External Service
    DNS໊Λొ࿥͢Δ৔߹ɺconsulͷઃఆʹ
    recursors (֎෦ͷ໊લղܾΛ͢ΔDNSαʔόͷ
    ΞυϨε) Λ͓ͯ͘͠
    • ઃఆ͠ͳ͍ͱ CNAME ͷΈ͕ฦΔ
    consul agent͸֎෦ͷ໊લղܾ݁ՌΛcache͠ͳ
    ͍

    View Slide

  37. Health Checking

    View Slide

  38. Health Checking
    ֤αʔϏεʹ͍ͭͯɺ3λΠϓͷϔϧενΣοΫ
    ΛఆٛͰ͖Δ
    • Script
    • HTTP
    • TTL

    View Slide

  39. Health Checking by script
    ϢʔβఆٛͷϔϧενΣοΫίϚϯυΛ࣮ߦ
    exit codeͰঢ়ଶΛ௨஌ (Nagios/Sensuޓ׵)
    • 0 : success
    • 1 : warning
    • 2 : fail
    failͷ৔߹͸DNS, HTTPͷԠ౴͔Β֎ΕΔ

    View Slide

  40. Health Checking by HTTP
    consul agent͕HTTPͰΞΫηε
    • HTTP 2xx : success
    • HTTP 429 : warning
    • ͦΕҎ֎ : fail

    View Slide

  41. Health Checking by TTL
    ఆظతʹagentʹHTTP PUTͯ͠ੜଘΛ఻͑Δ
    TTL͕੾ΕΔ·ͰʹPUT͕ͳ͚Ε͹failѻ͍

    View Slide

  42. Key/Value Store

    View Slide

  43. Key/Value Store
    ೚ҙͷ஋Λग़͠ೖΕͰ͖ΔKVS
    $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1'
    true
    $ curl http://127.0.0.1:8500/v1/kv/web/key1
    [{
    "CreateIndex": 112,
    "ModifyIndex": 112,
    "LockIndex": 0,
    "Key": "web/key1",
    "Flags": 0,
    "Value": "dGVzdA=="
    }]

    View Slide

  44. Key/Value Store
    URLҾ਺Ͱϝλσʔλ (flags) Λอ࣋Ͱ͖Δ
    64bit int, ༻్͸Ϣʔβͷ೚ҙ
    $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1?flags=123'
    true
    $ curl 'http://127.0.0.1:8500/v1/kv/web/key1'
    [{
    ...
    "Key": "web/key1",
    "Flags": 123, // <------- ͜Ε
    "Value": "dGVzdA=="
    }]

    View Slide

  45. Key/Value Store
    jsonϨεϙϯεͷ஋͸Base64 encode͞Ε͍ͯΔ
    jsonͰ͸ͳ͘஋͚ͩੜͰऔΓ͍ͨ৔߹͸Ҿ਺ raw
    $ curl "http://127.0.0.1:8500/v1/kv/web/key1?raw"
    test

    View Slide

  46. Key/Value Store
    ͋Δ֊૚ͷԼͷ஋Λ࠶ؼతʹऔΓ͍ͨ৔߹͸
    Ҿ਺ recurse
    $ curl -s "http://127.0.0.1:8500/v1/kv/web/?recurse"
    [
    {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0,
    "Key":"web/key1","Flags":123,"Value":"dGVzdA=="},
    {"CreateIndex":122,"ModifyIndex":122,"LockIndex":0,
    "Key":"web/key2","Flags":0,"Value":"dGVzdDI="},
    {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0,
    "Key":"web/test/1","Flags":0,"Value":"dGVzdDM="}
    ]
    όοΫΞοϓʹ΋ར༻Մೳ

    View Slide

  47. Key/Value Store Benchmark GET
    $ wrk -c 10 -d 10 -t 2 http://127.0.0.1:8500/v1/kv/web/key1
    Server(Leader): 41,832 Requests/sec
    Server/Client(Follower): 17,281
    Server(Follower) stale mode: 37,013
    Client(Follower) stale mode: 16,938
    Consul v0.5.2 on EC2 c4.2xlarge, GOMAXPROCS=4

    View Slide

  48. Key/Value Store Benchmark PUT
    $ wrk -c 10 -d 10 -t 2 -s put.lua http://127.0.0.1:8500/v1/kv/web/key1
    -- put.lua
    wrk.method = "PUT"
    wrk.body = "test"
    wrk.headers["Content-Type"] = "application/x-www-form-urlencoded"
    Server/Client(Follower): 427.56 Requests/sec

    View Slide

  49. Key/Value Store
    ৗʹ 127.0.0.1:8500 Λ࢖͑͹Α͍ͷͰָ
    ͔ͳΓߴ଎ͳͷͰGET͸ԕྀͳ͘࢖͑Δ
    σϑΥϧτͰ͸͢΂ͯͷ໰͍߹ΘͤΛ
    Leader node͕ॲཧ͢Δ
    • stale modeʹ͢ΔͱLeaderҎ֎ͷServer΋Ԡ
    ౴Ͱ͖Δ
    (Ұ؏ੑʹ͍ͭͯ͸ޙड़)

    View Slide

  50. ಺෦DNSͱͯ͠ConsulΛ࢖͏

    View Slide

  51. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    node,serviceͷ໊લղܾΛConsul Agent΁޲͚Δ
    ! resolv.confͰ͸ϙʔτࢦఆͰ͖ͳ͍
    ! Port 53͸ಛݖ͕ඞཁ
    → ͦͷͨΊ͚ͩʹAgentΛrootͰಈ͔͢ͷ͸ !
    Agent͸࠶ؼ໰͍߹Θͤ͸(ҰԠ)Ͱ͖Δ͕cacheػ
    ೳ͕ͳ͍ !
    → dnsmasq, bindͳͲ͔Β.consulυϝΠϯͷΈ
    forward !

    View Slide

  52. View Slide

  53. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    dnsmasqΛશ୆Ͱىಈ
    .consul υϝΠϯͷ໊લղܾ͸consul agent΁
    127.0.0.1:53 Λdnsmasq͕Listen͢Δ
    # dnsmasq.conf
    server=/consul/127.0.0.1#8600
    bind-interfaces
    listen-address=127.0.0.1

    View Slide

  54. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    resolv.conf Ͱ (node|service).consul Λݕࡧυ
    ϝΠϯʹࢦఆ
    → node໊ɺservice໊͚ͩͰ઀ଓͰ͖Δ
    # /etc/resolv.conf
    search node.consul service.consul
    nameserver 127.0.0.1 # dnsmasq
    nameserver 172.16.0.2 # VPC resolver
    nameserver 172.16.0.254 # Unbound on EC2

    View Slide

  55. ಺෦DNSͱͯ͠ConsulΛ࢖͏
    consul.io/docs/guides/forwarding.html
    BINDΛ࢖͏ྫ

    View Slide

  56. ConsulΛຊ൪؀ڥͰӡ༻͢Δ
    ͨΊʹ

    View Slide

  57. Deployment Table
    Server node: ຊ൪Ͱ͸࠷௿3୆, 3 or 5͕ਪ঑
    consul.io/docs/internals/consensus.html

    View Slide

  58. ServerʹඞཁͳϦιʔε
    • CPU: 2CPUͰे෼
    • GOMAXPROCS=2 Ҏ্Ͱಈ࡞ͤ͞ΔͷΛڧ͘ਪ

    • Memory: 20MBʙ
    • Disk: 2MBʙ
    Memory, Disk͸KVͷར༻ঢ়گ࣍ୈ

    View Slide

  59. Serverʹ͸ઐ༻ϗετ͕ඞཁʁ
    consul agentࣗମ͸ͦΕ΄ͲϦιʔεΛ
    ࢖༻͠ͳ͍ͨΊɺಉډՄೳ͕ͩ…
    Disk IO͕ߴෛՙͳ৔߹ʹRaftͷHeartbeat͕
    ࣦഊ͠΍͍͢
    • Timeout 500ms
    • Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ
    • ௨ৗ2,3ඵͰબग़͸׬ྃ͢Δ
    • ͦͷؒॻ͖ࠐΈॲཧ͕Ͱ͖ͳ͍

    View Slide

  60. Daemonize
    consul agentࣗ਎͸Deamonಈ࡞Ϟʔυ͕ͳ͍
    • Daemontools
    • RPM & init script
    • github.com/tomhillable/consul-rpm
    • Systemd
    ͲΕͰ΋͓޷ΈͰ

    View Slide

  61. Bootstrapping
    consul agent͸ىಈޙɺΫϥελʹjoin͢Δඞཁ
    ͕͋Δ
    1. ίϚϯυͰ consul joinΛ࣮ߦ
    2. ઃఆϑΝΠϧͰ start_join Λࢦఆ
    3. Atlas࿈ܞ

    View Slide

  62. 1. consul join
    αʔόͷΞυϨεΛࢦఆͯ͠join
    $ consul join 192.168.1.11 192.168.1.12 192.168.1.13
    ࣗಈԽͮ͠Β͍ͷͰςετҎ֎Ͱ͸࢖Θͳ͍

    View Slide

  63. 2. start_join
    consul.confʹαʔόͷΞυϨεΛࢦఆ͓ͯ͘͠
    {
    "start_join": [ "192.168.1.11", "192.168.1.12", "192.168.1.13"]
    }
    consul serverʹ͸ݻఆIPΞυϨεΛৼΔ͔ɺผͷ
    DNSͰղܾͰ͖ΔΑ͏ʹ͓ͯ͘͠

    View Slide

  64. 3. Atlas࿈ܞ
    Atlas - atlas.hashicorp.com
    Vagrant Packer
    Terraform ConsulΛ౷߹͢ΔαʔϏε
    $ consul agent ... \
    -atlas=ATLAS_USERNAME/infrastructure \
    -atlas-join \
    -atlas-token="YOUR_ATLAS_TOKEN" \
    ! ServerͷΞυϨεΛ؅ཧ͢Δඞཁ͕ͳ͍ͷͰָ
    " 11nodeҎ্͸$40/node

    View Slide

  65. ߴՄ༻ੑͷͨΊʹ

    View Slide

  66. ߴՄ༻ੑͷͨΊʹ
    Server୆਺ʹΑΓಉ࣌ʹো֐
    Λىͯ͜͠΋໰୊ͳ͍node਺
    ͕มΘΔ
    • 3 node → 1
    • 5 node → 2
    3 nodeߏ੒࣌ɺ2୆མͪͯ࢒
    Γ1୆ʹͳͬͯ͠·͏ͱLeader
    ͕બग़Ͱ͖ͳ͍
    ௕࣌ؒ੾Γ཭͢ϝϯςφϯε
    ࣌ʹ͸Ұ࣌తʹServer nodeΛ
    ૿΍͢ख΋

    View Slide

  67. ߴՄ༻ੑͷͨΊʹ
    Server nodeͷfailover͸ࣗಈ
    ! Ϣʔβ͸ৗʹlocalhostͷagent͚ͩΛΈ͍ͯΕ
    ͹Α͍

    View Slide

  68. nodeো֐࣌ͷӨڹ
    ! LeaderͰ͸ͳ͍ → " ଞnodeʹ͸Өڹͳ͠
    ! Leader → " Leader࠶બग़
    σϑΥϧτͰ͸͢΂ͯͷಡΈॻ͖ΛLeader͕ॲཧ
    (ڧҰ؏ੑ)
    Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)

    View Slide

  69. Stale mode (DNS)
    Leader࠶બग़͸௨ৗ2ʙ3ඵͰ׬ྃ
    ͦͷؒ΋DNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ
    → Stale mode : Leaderະબग़Ͱ΋Ԡ౴Մೳ
    "dns_config":{
    "allow_stale": true, // default false
    "max_stale": "10s" // default 5s
    }
    ݁Ռ͸ݹ͍Մೳੑ͕͋Δ(݁Ռ੔߹ੑ)

    View Slide

  70. DNS TTL
    default͸TTL 0 → cache͞Εͳ͍
    node, serviceผʹTTLΛઃఆՄೳ
    DNS cacheΛલஈʹ഑ஔͯ͠cacheͰ͖Δ
    "dns_config":{
    "node_ttl": "60s",
    "service_ttl": {
    "*": "15s"
    }
    }

    View Slide

  71. Stale mode (HTTP API)
    HTTP APIͰstale modeʹ͢Δ৔߹͸Ҿ਺ stale
    $ curl "http://127.0.0.1:8500/v1/kv/web/key1?stale"
    staleҾ਺ͳ͠ͰLeaderબग़தʹΞΫηε
    → 500 Internal Server Error

    View Slide

  72. ӡ༻தͷUpgrade
    consul.io/docs/upgrading.html
    consul.io/docs/upgrade-specific.html
    όʔδϣϯผʹ஫ҙ఺͕͋ΔͷͰυΩϡϝϯτΛ
    ॱ൪ʹAgentΛೖΕସ͑Δ͜ͱͰ
    Rolling upgradeՄೳ
    (Leader nodeೖΕସ͑Ͱ࠶બग़͸ى͖Δ)

    View Slide

  73. ҆ఆੑ
    v0.2࣌୅͔Β1೥Ҏ্ӡ༻
    Agentϓϩηε͕མͪͨ͜ͱ͸ͳ͍
    ! ΦϖϛεͰશServerΛಉ࣌ʹམͱ͢ͱճ෮ෆೳ
    → KV͸ఆظόοΫΞοϓΛ
    (Ϣʔβσʔλ͸ೖΕͪΌμϝ ! )

    View Slide

  74. Agentࣗମͷ؂ࢹ
    Agent processࣗମͷ؂ࢹ͸ผ్
    • process؂ࢹ(consul agent)
    • TCP/UDP 8600 (DNS)
    • TCP 8500 (HTTP)
    • http://127.0.0.1:8500/v1/status/leader
    ಺༰มߋݕ஌
    • Leader Lost→࠶બग़ͰมΘΔ

    View Slide

  75. Φʔτεέʔϧ؀ڥͰ
    ૿ݮ͢ΔαʔόΛѻ͏

    View Slide

  76. 2014.02ʙ Lobi Rec SDKϦϦʔε
    εϚϑΥΞϓϦʹSDKΛ૊ΈࠐΜͰPlayಈը࿥ը
    αʔόʹuploadͯ͠ม׵ɺӾཡ
    ม׵ػೳ͸ElasticTranscoderͰ࣮૷

    View Slide

  77. ElasticTranscoder
    ! Managed ServiceͳͷͰ؅ཧָ͕
    ! ม׵ೳྗ͸উखʹεέʔϧ
    ! ͪΐͬͱ͓ߴ͍…
    • SD $0.017/min
    • HD $0.034/min
    ౤ߘ͋ͨΓ4ύλʔϯ ฏۉ2෼ = $0.204 ≒ 25ԁ

    View Slide

  78. 2014.10
    ϞϯελʔετϥΠΫ
    Lobi Rec SDKಋೖʂʂʂ

    View Slide

  79. ϞϯελʔετϥΠΫ
    Lobi Rec SDKಋೖ
    ௒ώοτήʔϜʹ͖ͭ
    େྔͷಈը͕…
    !""""""!

    View Slide

  80. EC2 Spot InstanceͰಈըม׵
    ! ؅ཧ͕໘౗
    ! উखʹεέʔϧ͸ͯ͘͠Εͳ͍
    ! ElasticTranscoderΑΓѹ౗తʹ҆Ձ
    • ElasticTranscoder = $0.204/౤ߘ
    • EC2 Spot cc2.8xlarge(32core) = $0.45/hour

    View Slide

  81. Spot InstanceͰΦʔτεέʔϧ

    View Slide

  82. Spot InstanceͰΦʔτεέʔϧ
    • εέʔϧΞ΢τ: CPUෛՙͰ͸ͳ͘JobͷྔͰ
    • Job͕ཷ·Βͳ͍ݶΓCPU 100%͸ !
    • εέʔϧΠϯ: CPU idle͕શମͷ25%Ҏ্

    View Slide

  83. Φʔτεέʔϧ؀ڥͰͷΠϯελϯεىಈ
    1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ
    2. consul join
    3. ChefʹΑΔϓϩϏδϣχϯά
    4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy
    5. Zabbix΁ࣗಈొ࿥ͯ͠؂ࢹର৅ʹ௥Ճ

    View Slide

  84. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ
    ConsulͷͨΊʹҰҙͳhostname͕ඞཁ
    Cloud-InitͰىಈ͢ΔϗετͷλΠϓΛઃఆ
    #cloud-config
    runcmd:
    - [sh, -c, 'echo "HOSTNAME_PREFIX=transcode" > /etc/sysconfig/hostname-prefix']
    rc.localͰಡΈࠐΉ
    # /etc/rc.local
    if [ -f /etc/sysconfig/hostname-prefix ]; then
    . /etc/sysconfig/hostname-prefix
    fi

    View Slide

  85. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ
    hostnameΛ prefix + InstanceID
    EC2 Name tag ෇༩
    # /etc/rc.local
    instance_id=$(curl -s 169.254.169.254/latest/meta-data/instance-id)
    new_hostname="${HOSTNAME_PREFIX}-$instance_id"
    hostname $new_hostname
    aws ec2 create-tags \
    --resources $instance_id \
    --tags "Key=Name,Value=$new_hostname"

    View Slide

  86. 2. ConsulΫϥελʹjoin
    consul agent -node $(hostname)
    ࣗಈతʹ಺෦DNSͰ໊લղܾՄೳʹ

    View Slide

  87. 3. ChefʹΑΔϓϩϏδϣχϯά
    Consul KV͔ΒϗετλΠϓ͝ͱͷJSONΛऔಘ͠
    ࣮ͯߦ
    JSON=$(curl -s "localhost:8500/v1/kv/nodes/bootup/${HOSTNAME_PREFIX}.json?raw")
    echo "$JSON" > /tmp/chef-bootup.json
    chef-client -j /tmp/chef-bootup.json
    ॳճChef࣮ߦ࣌ʹChef-Serverʹొ࿥͞ΕΔ

    View Slide

  88. 4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy
    Stretcher(ޙड़)Ͱdeploy
    5. Zabbix΁ࣗಈొ࿥ͯ͠؂ࢹର৅ʹ௥Ճ
    deploy࣮ߦޙɺzabbix-agentΛىಈˠࣗಈొ࿥
    deploy׬ྃલʹىಈ͢ΔͱΞϥʔτ͕ൃ๒ͯ͠͠
    ·͏

    View Slide

  89. StretcherΛར༻ͨ͠σϓϩΠ
    Dan Zen https://www.flickr.com/photos/danzen/2288626158/

    View Slide

  90. What's Stretcher
    github.com/fujiwara/stretcher
    Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ

    View Slide

  91. Why Stretcher?
    Archer(rsync) ʹΑΔதԝϗετ͔Βͷdeploy
    ! pushͰ͸ΦʔτεέʔϧʹରԠͰ͖ͳ͍
    ! ֤ϗετ͔ΒrsyncͰpull?
    → buildதʹrsync͞ΕͨΒ…
    ! ֤ϗετ͔Βgit pull?
    → grunt, GoͳͲͷbuildੜ੒෺ΛೖΕͨ͘ͳ͍
    ! ୆਺͕ଟ͍ͱssh+rsync΋git pull΋πϥ͍
    ! AMI࡞Γ௚͠&ೖΕସ͑͸଴ͯͳ͍

    View Slide

  92. Inspired by
    github.com/sorah/mamiya
    &
    AWS CodeDeploy

    View Slide

  93. ઃܭํ਑
    AWS͡Όͳͯ͘΋ಈ͘
    rsync & ίϚϯυ࣮ߦɺͱ͍͏ϑϩʔ͸౿ऻ
    Consul eventͰΠϕϯτ௨஌
    Consul ͱ͸ૄ݁߹ (consul watchͰىಈ)
    GoͰॻ͘

    View Slide

  94. Architecture

    View Slide

  95. Consul event
    ֤nodeʹGossip ProtocolͰΠϕϯτΛ
    ૹ৴͢Δ࢓૊Έ
    $ consul event -name EVENT_NAME [-node REGEX] PAYLOAD
    Event ID: 3b1f3199-6e69-4b82-4812-b35058864fdd
    ࢦఆͨ͠Πϕϯτ໊Ͱ
    (ਖ਼نදݱʹϚον͢Δnodeʹ)
    payloadΛૹ৴

    View Slide

  96. Consul watch
    ࢦఆͨ͠ΠϕϯτΛड৴ͨ͠ΒίϚϯυΛ
    ࣮ߦ͢Δ࢓૊Έ
    $ consul watch -type event -name EVENT_NAME COMMAND
    payload͸ඪ४ೖྗ͔ΒJSONͰ౉͞ΕΔ
    [{
    "ID": "3b1f3199-6e69-4b82-4812-b35058864fdd",
    "Name": "test",
    "Payload": "TXkgcGF5bG9hZA==",
    ...
    }]

    View Slide

  97. Deployment process
    1. ΞϓϦέʔγϣϯΛbuildͯ͠tar.gzʹ͢Δ
    ґଘcpan moduleͳͲ͢΂ͯݻΊΔ
    2. खॱॻ(manifest)Λॻ͘
    3. tar.gz, manifestΛS3(or httpd)ʹ্͛Δ
    4. consul event Ͱ manifest URLΛ௨஌
    consul event -name deploy s3://...
    ✄---------- ͜͜·ͰstretcherͰ͸ͳ͍ ---------✄

    View Slide

  98. ✄------------ stretcher͔͜͜Β ------------✄
    consul watch -type event -name deploy stretcher
    1. event͔Βmanifest URLΛऔಘ
    2. tar.gzΛऔಘͯ͠TMPDIRʹల։
    3. rsync -av --deleteͰߋ৽
    4. command࣮ߦ
    • ΞϓϦέʔγϣϯ࠶ىಈͳͲ

    View Slide

  99. Manifest
    src: s3://example.com/app.tar.gz
    checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93
    dest: /home/stretcher/app
    commands:
    pre:
    - echo 'staring deploy'
    post:
    - echo 'deploy done'
    success:
    - cat >> /path/to/success.log
    failure:
    - cat >> /path/to/failure.log
    excludes:
    - "*.pid"
    - "*.socket"

    View Slide

  100. LobiͰͷdeploy
    tar.gz ໿200MB
    ల։͢Δͱ໿400MB
    • CPAN modules 110MB
    • node_modules 10MB × 5
    • Go app binaries 8MB × 5
    • Static files (S3ʹஔ͖͍ͨ)

    View Slide

  101. LobiͰͷdeploy
    1. ‐ Push to production branch
    2. ! Build (1 min~)
    carton install, grunt, npm install, go
    build ...
    3. " Pack tar.gz & Upload (1 min)
    4. # Deploy by Startecher (10~30 sec)

    View Slide

  102. LobiͰͷdeploy
    consul event ૹ৴͔Β10ʙ20ඵͰ׬ྃ !
    2015/08/05 14:58:30 Starting up stretcher agent
    2015/08/05 14:58:30 Waiting for consul events from STDIN...
    2015/08/05 14:58:30 Executing manifest: s3://...
    2015/08/05 14:58:33 Extract archive: /dev/shm/stretcher539962129 to /dev/shm/stretcher_src648982332
    2015/08/05 14:58:36 rsync [-av --delete --exclude-from
    /dev/shm/stretcher_src648982332/conf/rsync_exclude.web
    /dev/shm/stretcher_src648982332/ /home/xxx/web/]
    2015/08/05 14:58:36 sending incremental file list
    ...
    sent 787780 bytes received 5230 bytes 1586020.00 bytes/sec
    total size is 359702435 speedup is 453.59
    2015/08/05 14:58:36 invoking command: /home/xxx/web/refresh_services.sh
    2015/08/05 14:58:41 success.
    2015/08/05 14:58:41 Deploy manifest succeeded.
    Rollback͍ͨ͠ˠ௚લͷmanifestΛeventૹ৴
    →10secͰ໭Δ

    View Slide

  103. Deployͷ׬ྃΛ଴ͭ
    Πϕϯτૹ৴ݩͰ͸࣮ߦ׬͕ྃ෼͔Βͳ͍
    • ࣦഊ͢Δϗετ͕͋Δ͔΋͠Εͳ͍
    • Քಇதͷશ୆Ͱdeploy׬ྃͨ͠ͷΛ଴͍ͪͨ
    ……ϩά΋ݟ͍͚ͨͲ100୆෼Ͳ͏΍ͬͯʁ

    View Slide

  104. Consul KV Dashboard

    View Slide

  105. Chef, Serverspec, Deploy, etc...
    ֤ϗετͰ࣮ߦͨ݁͠ՌΛ֬ೝ͍ͨ͠΋ͷ
    • Chef
    • Serverspec
    • Stretcher
    • etc

    View Slide

  106. IRC / Slackʹ௨஌ʁ
    ࣾ಺༻ nopaste-cli command
    $ nopaste-cli -channel "lobi" -summary "Deploy done!" < deploy.log
    nopasteʹPOSTͱಉ࣌ʹURL͕௨஌͞ΕΔ
    100୆͋Δͱ௨஌͕……!!!!!!!!×100
    Ͳ͔͜ͰҰཡͯ͠ݟ͍ͨʂ

    View Slide

  107. Consul KV Dashboard
    github.com/fujiwara/consul-kv-dashboard
    Consul KVΛσʔλετΞʹͨ͠
    μογϡϘʔυWebΞϓϦ

    View Slide

  108. σʔλొ࿥
    Consul HTTP APIͰ௚઀ૹΔ
    $ curl -X PUT -d "message" \
    '127.0.0.1:8500/v1/kv/dashboard/example/myhostname?flags=1422607461000'

    View Slide

  109. keyߏ଄
    /v1/kv/dashboard/{category}/{nodename}?
    flags=({unixtime} * 1000 + {status})
    • category: chef, serverspec, deploy...
    • nodename: Consulͷnode໊
    • flags: unixtime * 1000 + status
    • status: 0=Success 1=Warning
    2=Danger 3=Info

    View Slide

  110. ը໘ͷଈ࣌ߋ৽
    ϒϩοΩϯάΫΤϦΛ࢖͏
    consul.io/docs/agent/http.html
    $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?recurese
    HTTP/1.1 200 OK
    Content-Type: application/json
    X-Consul-Index: 261975
    [{"CreateIndex":261891,"ModifyIndex":261975,"LockIndex":0,
    "Key":"dashboard/chef/myhost","Flags":1422602855000,"Value":".....

    View Slide

  111. Blocking query
    Ϩεϙϯεϔομͷ X-Consul-Index Λ
    ࣍ͷϦΫΤετͷҾ਺ʹࢦఆ
    $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost
    HTTP/1.1 200 OK
    X-Consul-Index: 261975
    ...
    $ curl 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?index=261975
    ৽͍͠σʔλ͕ൃੜ͢Δ·ͰϨεϙϯε͕஗Ԇ
    ͍ΘΏΔ Long pooling

    View Slide

  112. Blocking queryͷ׆༻
    Consul Template
    github.com/hashicorp/consul-template
    KV, node, service౳ͷঢ়ଶมԽΛଈ൓ө
    Template→fileߋ৽ˠcommand࣮ߦ
    $ consul-template \
    -consul 127.0.0.1:8500 \
    -template "/tmp/template.ctmpl:/var/www/nginx.conf:service nginx restart"

    View Slide

  113. consul-kv-dashboard͕΍Δ͜ͱ(1)
    GoͰॻ͔ΕͨWebΞϓϦέʔγϣϯ
    HTML, CSS, JavaScriptΛ഑෍
    • go-bindataͰ·ͱΊͨϑΝΠϧΛ
    http.FileServerͰ഑৴
    • όΠφϦҰݸΛஔ͍ͯىಈ͢Δ͚ͩͰ
    ੩తϑΝΠϧ΋഑৴Ͱ͖Δ

    View Slide

  114. consul-kv-dashboard͕΍Δ͜ͱ(2)
    Consul HTTP API ΁ͷ reverse proxy
    • /api/... → 127.0.0.1:8500/v1/kv/dashboard/...
    • ϨεϙϯεͷJSONΛ੔ܗ
    {"Flags":1422608524001}
    !
    {"timestamp":"2015-01-30 18:02:04 +0900","status":"warning"}
    • /v1/catalog/nodes Λblocking queryͰ؂ࢹ
    ଘࡏ͢ΔnodeͷσʔλͷΈϑΟϧλ

    View Slide

  115. consul-kv-dashboard trigger
    $ consul-kv-dashboard -trigger COMMAND
    ΧςΰϦຖʹঢ়ଶ(success→warningͳͲ)͕
    มԽͨ͠ΒίϚϯυ࣮ߦՄೳ
    JSON͕ඪ४ೖྗʹ౉͞ΕΔ
    {
    "category":"testing",
    "node":"web01",
    "address":"192.168.1.10",
    "timestamp":"2015-01-21 11:22:33 +0900",
    "status":"danger",
    "key":"","data":"failure!!"
    }

    View Slide

  116. ΦʔτεέʔϧͰͷ
    Deploy Tips

    View Slide

  117. ىಈޙʹ࠷৽൛Λdeploy
    deploy࣌ɺKVʹ࠷৽ͷmanifest URLΛอଘ
    1. consul join
    2. stretcherىಈ
    3. ࠷৽ͷmanifest URLΛKV͔Βऔಘ
    4. ࣗ෼ࣗ਎ʹ event (manifest URL) Λૹ৴
    5. deploy !

    View Slide

  118. ΦʔτεέʔϧͰͷ஫ҙ఺
    ! AMIʹ࢒͍ͬͯΔݹ͍ΞϓϦ͕ىಈ
    " ࠷৽ͷ deploy IDͰͳ͍৔߹͸ىಈ͠ͳ͍
    deploy࣌: unique ͳ ID Λൃߦ
    • ϑΝΠϧʹॻ͍ͯ tar ʹೖΕΔ
    • KV ʹ΋ೖΕΔ
    ىಈ࣌:ϩʔΧϧϑΝΠϧͷ ID ͱ KV Λൺֱ
    • ҟͳ͍ͬͯͨΒ sleep 10 && exit → restart

    View Slide

  119. bash-completionͰ
    sshͷϗετ໊ิ׬

    View Slide

  120. bash-completionͰsshͷϗετ໊ิ׬
    σϑΥϧτͰ͸ ~/.ssh/known_hosts Λݩʹิ׬
    ! աڈʹsshͨ͜͠ͱ͕͋Δϗετ͕ग़ͯ͘Δ
    " ݱࡏ͸طʹଘࡏ͠ͳ͍Մೳੑ͕͋Δ
    " Ұ౓΋sshͨ͜͠ͱ͕ͳ͍৽نϗετ͸ग़ͯ͜
    ͳ͍

    View Slide

  121. bash-completionͰsshͷϗετ໊ิ׬
    ~/.bash_profile
    _known_hosts_real()
    {
    local members=$(consul members -status=alive | awk '!/Node/{printf("%s ", $1)}')
    COMPREPLY=( $( \
    compgen -W "$members" \
    ${COMP_WORDS[COMP_CWORD]} \
    ) )
    return 0
    }
    ! ݱࡏaliveͳϗετͷΈ͕ग़ͯ͘Δʂ

    View Slide

  122. ·ͱΊ

    View Slide

  123. ·ͱΊ
    Consul͸ػೳ๛෋…͕ͩ
    ࢖͍͍ͨػೳ͚ͩ࢖͑͹Α͍
    • KVɺEventɺHealth Check...
    DNSɺHTTPΛ࢖͍͜ͳ͢ͱߴ౓ͳࣗಈԽ͕Մೳ
    • consul-templateͰಈతLBઃఆ

    View Slide

  124. Questions?
    • Architecture, Service Discovery
    Health Checking, Key/Value Store
    • ಺෦DNS
    • ຊ൪؀ڥӡ༻ / ߴՄ༻ੑ
    • Φʔτεέʔϧ
    • Stretcher
    • Consul KV Dashboard

    View Slide