Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consulと自作OSSを活用した100台規模のWebサービス運用

FUJIWARA Shunichiro
August 21, 2015
23k

 Consulと自作OSSを活用した100台規模のWebサービス運用

FUJIWARA Shunichiro

August 21, 2015
Tweet

More Decks by FUJIWARA Shunichiro

Transcript

  1. Lobiͷαʔόมભ 1. 2010~2011 AWS (US) : 4୆(?) 2. 2011~2013 ࣗࣾαʔό

    : 4 ~ 20୆ 3. 2013.11~ AWS (Tokyo) : 20 ~ 100୆
  2. Lobiͷαʔό EC2ͰՔಇ͍ͯ͠Δϗετͷछྨ͕ଟ͍ • app, sdk, stream, sdk-stream, db(3shard), transcode, log(aggregate,analyze),batch,

    deploy... ࣮૷ݴޠ Perl, Node.js, Go ϛυϧ΢ΣΞ͍Ζ͍Ζ • Nginx, MySQL, Starlet, Fluentd, Norikra, memcached, HAProxy, gearman, twemproxy, MHA, dnsmasq...
  3. What's Consul www.consul.io - HashiCorp product • Service Discovery •

    Health Checking • Key/Value Store • Multi Datacenter
  4. Agent Ϋϥελ಺ͷnodeશ୆Ͱಈ࡞͢Δdaemon Client mode or Server mode ͷͲͪΒ͔Ͱಈ࡞ ϢʔβʹDNS, HTTP

    interfaceΛఏڙ͢Δ • Ϣʔβ͸جຊతʹlocalhostͷagentͱ௨৴ • Agentಉ͕࢜RPCͰ௨৴(͋·Γҙࣝ͢Δඞཁ͸ ͳ͍) GoͰॻ͔Ε͍ͯͯ1όΠφϦͰಈ࡞ (CLI΋ಉҰ)
  5. Raft Raft is a protocol for implementing distributed consensus. ෼ࢄ؀ڥͰͷ߹ҙΞϧΰϦζϜ(ϓϩτίϧ)

    Leaderબग़ʹserver nodeͷա൒਺ͷಉҙ͕ඞཁ ͳͷͰ࠷௿3 node͕ඞཁ
  6. Node Discovery consul members ͰΫϥελ಺ͷnodeΛҰཡ $ consul members Node Address

    Status Type Build Protocol DC my-app-i-123456 192.168.1.12:8301 alive server 0.5.2 2 dc1 my-app-i-234567 192.168.1.23:8301 alive server 0.5.2 2 dc1 my-app-i-345678 192.168.1.34:8301 alive server 0.5.2 2 dc1 my-db-i-456789 192.168.1.45:8301 alive client 0.5.2 2 dc1 my-db-i-567890 192.168.1.56:8301 alive client 0.5.2 2 dc1 my-db-i-678901 192.168.1.67:8301 alive client 0.5.2 2 dc1 my-app-i-987654 192.168.1.99:8301 failed client 0.5.2 2 dc1 my-app-i-876543 192.168.1.87:8301 left client 0.5.2 2 dc1 (࣮ࡍ͸݁Ռͷॱ൪͸ෆఆ)
  7. Node Discovery Status=failed : agentͷࢮ׆؂ࢹʹࣦഊͨ͠node $ consul members -status failed

    Node Address Status Type Build Protocol DC my-app-i-987654 192.168.1.99:8301 failed server 0.5.2 2 dc1 Status=left : ਖ਼ৗʹΫϥελ͔Β཭୤ͨ͠node $ consul members -status left Node Address Status Type Build Protocol DC my-app-i-876543 192.168.1.87:8301 left server 0.5.2 2 dc1
  8. Node Discovery via DNS interface consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ $

    dig @127.0.0.1 -p 8600 my-app-i-123456.node.consul ;; QUESTION SECTION: ;my-app-i-123456.node.consul. IN A ;; ANSWER SECTION: my-app-i-123456.node.consul. 0 IN A 192.168.1.12
  9. Node Discovery via DNS interface Status=failed : DNSͰΞυϨε͕Ҿ͚Δ • Ұ࣌తʹࣄނͰ཭୤͍ͯ͠ΔՄೳੑ͕͋Δ

    Status=left : DNSͰΞυϨε͕Ҿ͚ͳ͘ͳΔ • .consul υϝΠϯͷ໊લղܾΛ consul agent ʹ ౤͛Δ͜ͱͰ಺෦DNSͱͯ͠ར༻Ͱ͖Δ • υϝΠϯ໊͸ઃఆͰมߋՄೳ
  10. Service Discovery of App via DNS interface consul agent (127.0.0.1:8600)

    ʹ໰͍߹ΘͤΔ $ dig @127.0.0.1 -p 8600 app.service.consul ;; QUESTION SECTION: ;app.service.consul. IN A ;; ANSWER SECTION: app.service.consul. 0 IN A 192.168.1.12 app.service.consul. 0 IN A 192.168.1.23 app.service.consul. 0 IN A 192.168.1.34
  11. Service Discovery via DNS interface Answerͷॱ൪͸ϥϯμϜ (≒ Round Robin) UDPͰͷ໰͍߹ΘͤͰ͸ର৅͕4ΞυϨεҎ্

    ͋Δ৔߹ɺ3ΞυϨεͷΈฦΔ • TCPͰ͸͢΂ͯฦΔ TTL ઃఆՄೳ (default 0)
  12. Service Discovery of App via HTTP API http://127.0.0.1:8500 ʹΞΫηε $

    curl http://127.0.0.1:8500/v1/catalog/service/app [ { "Node": "my-app-i-123456", "Address": "192.168.1.12", "ServiceID": "app", "ServiceName": "app", "ServicePort": 3000, ... }, { "Node": "my-app-i-234567", "Address": "192.168.1.23", ... } { "Node": "my-app-i-345678", "Address": "192.168.1.34", ... } ]
  13. Service Discovery of DB (master/slave) {tag}.{service}.service.consul Ͱ໊લղܾ $ dig @127.0.0.1

    -p 8600 master.db.service.consul master.db.service.consul. 0 IN A 192.168.1.45 $ dig @127.0.0.1 -p 8600 slave.db.service.consul slave.db.service.consul. 0 IN A 192.168.1.67 slave.db.service.consul. 0 IN A 192.168.1.56 $ dig @127.0.0.1 -p 8600 db.service.consul db.service.consul. 0 IN A 192.168.1.56 db.service.consul. 0 IN A 192.168.1.45 db.service.consul. 0 IN A 192.168.1.67
  14. External Service nodeʹؔ࿈͠ͳ͍ɺ֎෦DNSͰఆٛ͞Ε͍ͯΔ ໊લ΍IPΞυϨε΋αʔϏεͱͯ͠ఆٛͰ͖Δ $ curl -X PUT -d '{

    "Node":"rds", "Address":"my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com", "Service":{"Service": "rds"} }' http://127.0.0.1:8500/v1/catalog/register $ dig @127.0.0.1 -p 8600 rds.service.consul. ;; ANSWER SECTION: rds.service.consul. 0 IN CNAME my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. 10 IN A 192.168.1.100
  15. Health Checking by script ϢʔβఆٛͷϔϧενΣοΫίϚϯυΛ࣮ߦ exit codeͰঢ়ଶΛ௨஌ (Nagios/Sensuޓ׵) • 0

    : success • 1 : warning • 2 : fail failͷ৔߹͸DNS, HTTPͷԠ౴͔Β֎ΕΔ
  16. Health Checking by HTTP consul agent͕HTTPͰΞΫηε • HTTP 2xx :

    success • HTTP 429 : warning • ͦΕҎ֎ : fail
  17. Key/Value Store ೚ҙͷ஋Λग़͠ೖΕͰ͖ΔKVS $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1' true

    $ curl http://127.0.0.1:8500/v1/kv/web/key1 [{ "CreateIndex": 112, "ModifyIndex": 112, "LockIndex": 0, "Key": "web/key1", "Flags": 0, "Value": "dGVzdA==" }]
  18. Key/Value Store URLҾ਺Ͱϝλσʔλ (flags) Λอ࣋Ͱ͖Δ 64bit int, ༻్͸Ϣʔβͷ೚ҙ $ curl

    -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1?flags=123' true $ curl 'http://127.0.0.1:8500/v1/kv/web/key1' [{ ... "Key": "web/key1", "Flags": 123, // <------- ͜Ε "Value": "dGVzdA==" }]
  19. Key/Value Store ͋Δ֊૚ͷԼͷ஋Λ࠶ؼతʹऔΓ͍ͨ৔߹͸ Ҿ਺ recurse $ curl -s "http://127.0.0.1:8500/v1/kv/web/?recurse" [

    {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0, "Key":"web/key1","Flags":123,"Value":"dGVzdA=="}, {"CreateIndex":122,"ModifyIndex":122,"LockIndex":0, "Key":"web/key2","Flags":0,"Value":"dGVzdDI="}, {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0, "Key":"web/test/1","Flags":0,"Value":"dGVzdDM="} ] όοΫΞοϓʹ΋ར༻Մೳ
  20. Key/Value Store Benchmark GET $ wrk -c 10 -d 10

    -t 2 http://127.0.0.1:8500/v1/kv/web/key1 Server(Leader): 41,832 Requests/sec Server/Client(Follower): 17,281 Server(Follower) stale mode: 37,013 Client(Follower) stale mode: 16,938 Consul v0.5.2 on EC2 c4.2xlarge, GOMAXPROCS=4
  21. Key/Value Store Benchmark PUT $ wrk -c 10 -d 10

    -t 2 -s put.lua http://127.0.0.1:8500/v1/kv/web/key1 -- put.lua wrk.method = "PUT" wrk.body = "test" wrk.headers["Content-Type"] = "application/x-www-form-urlencoded" Server/Client(Follower): 427.56 Requests/sec
  22. ಺෦DNSͱͯ͠ConsulΛ࢖͏ resolv.conf Ͱ (node|service).consul Λݕࡧυ ϝΠϯʹࢦఆ → node໊ɺservice໊͚ͩͰ઀ଓͰ͖Δ # /etc/resolv.conf

    search node.consul service.consul nameserver 127.0.0.1 # dnsmasq nameserver 172.16.0.2 # VPC resolver nameserver 172.16.0.254 # Unbound on EC2
  23. Serverʹ͸ઐ༻ϗετ͕ඞཁʁ consul agentࣗମ͸ͦΕ΄ͲϦιʔεΛ ࢖༻͠ͳ͍ͨΊɺಉډՄೳ͕ͩ… Disk IO͕ߴෛՙͳ৔߹ʹRaftͷHeartbeat͕ ࣦഊ͠΍͍͢ • Timeout 500ms

    • Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ • ௨ৗ2,3ඵͰબग़͸׬ྃ͢Δ • ͦͷؒॻ͖ࠐΈॲཧ͕Ͱ͖ͳ͍
  24. Daemonize consul agentࣗ਎͸Deamonಈ࡞Ϟʔυ͕ͳ͍ • Daemontools • RPM & init script

    • github.com/tomhillable/consul-rpm • Systemd ͲΕͰ΋͓޷ΈͰ
  25. 3. Atlas࿈ܞ Atlas - atlas.hashicorp.com Vagrant Packer Terraform ConsulΛ౷߹͢ΔαʔϏε $

    consul agent ... \ -atlas=ATLAS_USERNAME/infrastructure \ -atlas-join \ -atlas-token="YOUR_ATLAS_TOKEN" \ ! ServerͷΞυϨεΛ؅ཧ͢Δඞཁ͕ͳ͍ͷͰָ " 11nodeҎ্͸$40/node
  26. ߴՄ༻ੑͷͨΊʹ Server୆਺ʹΑΓಉ࣌ʹো֐ Λىͯ͜͠΋໰୊ͳ͍node਺ ͕มΘΔ • 3 node → 1 •

    5 node → 2 3 nodeߏ੒࣌ɺ2୆མͪͯ࢒ Γ1୆ʹͳͬͯ͠·͏ͱLeader ͕બग़Ͱ͖ͳ͍ ௕࣌ؒ੾Γ཭͢ϝϯςφϯε ࣌ʹ͸Ұ࣌తʹServer nodeΛ ૿΍͢ख΋
  27. nodeো֐࣌ͷӨڹ ! LeaderͰ͸ͳ͍ → " ଞnodeʹ͸Өڹͳ͠ ! Leader → "

    Leader࠶બग़ σϑΥϧτͰ͸͢΂ͯͷಡΈॻ͖ΛLeader͕ॲཧ (ڧҰ؏ੑ) Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
  28. Stale mode (DNS) Leader࠶બग़͸௨ৗ2ʙ3ඵͰ׬ྃ ͦͷؒ΋DNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ → Stale mode :

    Leaderະબग़Ͱ΋Ԡ౴Մೳ "dns_config":{ "allow_stale": true, // default false "max_stale": "10s" // default 5s } ݁Ռ͸ݹ͍Մೳੑ͕͋Δ(݁Ռ੔߹ੑ)
  29. Stale mode (HTTP API) HTTP APIͰstale modeʹ͢Δ৔߹͸Ҿ਺ stale $ curl

    "http://127.0.0.1:8500/v1/kv/web/key1?stale" staleҾ਺ͳ͠ͰLeaderબग़தʹΞΫηε → 500 Internal Server Error
  30. Agentࣗମͷ؂ࢹ Agent processࣗମͷ؂ࢹ͸ผ్ • process؂ࢹ(consul agent) • TCP/UDP 8600 (DNS)

    • TCP 8500 (HTTP) • http://127.0.0.1:8500/v1/status/leader ಺༰มߋݕ஌ • Leader Lost→࠶બग़ͰมΘΔ
  31. ElasticTranscoder ! Managed ServiceͳͷͰ؅ཧָ͕ ! ม׵ೳྗ͸উखʹεέʔϧ ! ͪΐͬͱ͓ߴ͍… • SD

    $0.017/min • HD $0.034/min ౤ߘ͋ͨΓ4ύλʔϯ ฏۉ2෼ = $0.204 ≒ 25ԁ
  32. EC2 Spot InstanceͰಈըม׵ ! ؅ཧ͕໘౗ ! উखʹεέʔϧ͸ͯ͘͠Εͳ͍ ! ElasticTranscoderΑΓѹ౗తʹ҆Ձ •

    ElasticTranscoder = $0.204/౤ߘ • EC2 Spot cc2.8xlarge(32core) = $0.45/hour
  33. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ ConsulͷͨΊʹҰҙͳhostname͕ඞཁ Cloud-InitͰىಈ͢ΔϗετͷλΠϓΛઃఆ #cloud-config runcmd: - [sh, -c, 'echo

    "HOSTNAME_PREFIX=transcode" > /etc/sysconfig/hostname-prefix'] rc.localͰಡΈࠐΉ # /etc/rc.local if [ -f /etc/sysconfig/hostname-prefix ]; then . /etc/sysconfig/hostname-prefix fi
  34. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ hostnameΛ prefix + InstanceID EC2 Name tag ෇༩

    # /etc/rc.local instance_id=$(curl -s 169.254.169.254/latest/meta-data/instance-id) new_hostname="${HOSTNAME_PREFIX}-$instance_id" hostname $new_hostname aws ec2 create-tags \ --resources $instance_id \ --tags "Key=Name,Value=$new_hostname"
  35. Why Stretcher? Archer(rsync) ʹΑΔதԝϗετ͔Βͷdeploy ! pushͰ͸ΦʔτεέʔϧʹରԠͰ͖ͳ͍ ! ֤ϗετ͔ΒrsyncͰpull? → buildதʹrsync͞ΕͨΒ…

    ! ֤ϗετ͔Βgit pull? → grunt, GoͳͲͷbuildੜ੒෺ΛೖΕͨ͘ͳ͍ ! ୆਺͕ଟ͍ͱssh+rsync΋git pull΋πϥ͍ ! AMI࡞Γ௚͠&ೖΕସ͑͸଴ͯͳ͍
  36. Consul event ֤nodeʹGossip ProtocolͰΠϕϯτΛ ૹ৴͢Δ࢓૊Έ $ consul event -name EVENT_NAME

    [-node REGEX] PAYLOAD Event ID: 3b1f3199-6e69-4b82-4812-b35058864fdd ࢦఆͨ͠Πϕϯτ໊Ͱ (ਖ਼نදݱʹϚον͢Δnodeʹ) payloadΛૹ৴
  37. Consul watch ࢦఆͨ͠ΠϕϯτΛड৴ͨ͠ΒίϚϯυΛ ࣮ߦ͢Δ࢓૊Έ $ consul watch -type event -name

    EVENT_NAME COMMAND payload͸ඪ४ೖྗ͔ΒJSONͰ౉͞ΕΔ [{ "ID": "3b1f3199-6e69-4b82-4812-b35058864fdd", "Name": "test", "Payload": "TXkgcGF5bG9hZA==", ... }]
  38. Deployment process 1. ΞϓϦέʔγϣϯΛbuildͯ͠tar.gzʹ͢Δ ґଘcpan moduleͳͲ͢΂ͯݻΊΔ 2. खॱॻ(manifest)Λॻ͘ 3. tar.gz,

    manifestΛS3(or httpd)ʹ্͛Δ 4. consul event Ͱ manifest URLΛ௨஌ consul event -name deploy s3://... ✄---------- ͜͜·ͰstretcherͰ͸ͳ͍ ---------✄
  39. ✄------------ stretcher͔͜͜Β ------------✄ consul watch -type event -name deploy stretcher

    1. event͔Βmanifest URLΛऔಘ 2. tar.gzΛऔಘͯ͠TMPDIRʹల։ 3. rsync -av --deleteͰߋ৽ 4. command࣮ߦ • ΞϓϦέʔγϣϯ࠶ىಈͳͲ
  40. Manifest src: s3://example.com/app.tar.gz checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93 dest: /home/stretcher/app commands: pre: -

    echo 'staring deploy' post: - echo 'deploy done' success: - cat >> /path/to/success.log failure: - cat >> /path/to/failure.log excludes: - "*.pid" - "*.socket"
  41. LobiͰͷdeploy tar.gz ໿200MB ల։͢Δͱ໿400MB • CPAN modules 110MB • node_modules

    10MB × 5 • Go app binaries 8MB × 5 • Static files (S3ʹஔ͖͍ͨ)
  42. LobiͰͷdeploy 1. ‐ Push to production branch 2. ! Build

    (1 min~) carton install, grunt, npm install, go build ... 3. " Pack tar.gz & Upload (1 min) 4. # Deploy by Startecher (10~30 sec)
  43. LobiͰͷdeploy consul event ૹ৴͔Β10ʙ20ඵͰ׬ྃ ! 2015/08/05 14:58:30 Starting up stretcher

    agent 2015/08/05 14:58:30 Waiting for consul events from STDIN... 2015/08/05 14:58:30 Executing manifest: s3://... 2015/08/05 14:58:33 Extract archive: /dev/shm/stretcher539962129 to /dev/shm/stretcher_src648982332 2015/08/05 14:58:36 rsync [-av --delete --exclude-from /dev/shm/stretcher_src648982332/conf/rsync_exclude.web /dev/shm/stretcher_src648982332/ /home/xxx/web/] 2015/08/05 14:58:36 sending incremental file list ... sent 787780 bytes received 5230 bytes 1586020.00 bytes/sec total size is 359702435 speedup is 453.59 2015/08/05 14:58:36 invoking command: /home/xxx/web/refresh_services.sh 2015/08/05 14:58:41 success. 2015/08/05 14:58:41 Deploy manifest succeeded. Rollback͍ͨ͠ˠ௚લͷmanifestΛeventૹ৴ →10secͰ໭Δ
  44. IRC / Slackʹ௨஌ʁ ࣾ಺༻ nopaste-cli command $ nopaste-cli -channel "lobi"

    -summary "Deploy done!" < deploy.log nopasteʹPOSTͱಉ࣌ʹURL͕௨஌͞ΕΔ 100୆͋Δͱ௨஌͕……!!!!!!!!×100 Ͳ͔͜ͰҰཡͯ͠ݟ͍ͨʂ
  45. σʔλొ࿥ Consul HTTP APIͰ௚઀ૹΔ $ curl -X PUT -d "message"

    \ '127.0.0.1:8500/v1/kv/dashboard/example/myhostname?flags=1422607461000'
  46. keyߏ଄ /v1/kv/dashboard/{category}/{nodename}? flags=({unixtime} * 1000 + {status}) • category: chef,

    serverspec, deploy... • nodename: Consulͷnode໊ • flags: unixtime * 1000 + status • status: 0=Success 1=Warning 2=Danger 3=Info
  47. ը໘ͷଈ࣌ߋ৽ ϒϩοΩϯάΫΤϦΛ࢖͏ consul.io/docs/agent/http.html $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?recurese HTTP/1.1 200 OK

    Content-Type: application/json X-Consul-Index: 261975 [{"CreateIndex":261891,"ModifyIndex":261975,"LockIndex":0, "Key":"dashboard/chef/myhost","Flags":1422602855000,"Value":".....
  48. Blocking query Ϩεϙϯεϔομͷ X-Consul-Index Λ ࣍ͷϦΫΤετͷҾ਺ʹࢦఆ $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost

    HTTP/1.1 200 OK X-Consul-Index: 261975 ... $ curl 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?index=261975 ৽͍͠σʔλ͕ൃੜ͢Δ·ͰϨεϙϯε͕஗Ԇ ͍ΘΏΔ Long pooling
  49. Blocking queryͷ׆༻ Consul Template github.com/hashicorp/consul-template KV, node, service౳ͷঢ়ଶมԽΛଈ൓ө Template→fileߋ৽ˠcommand࣮ߦ $

    consul-template \ -consul 127.0.0.1:8500 \ -template "/tmp/template.ctmpl:/var/www/nginx.conf:service nginx restart"
  50. consul-kv-dashboard͕΍Δ͜ͱ(2) Consul HTTP API ΁ͷ reverse proxy • /api/... →

    127.0.0.1:8500/v1/kv/dashboard/... • ϨεϙϯεͷJSONΛ੔ܗ {"Flags":1422608524001} ! {"timestamp":"2015-01-30 18:02:04 +0900","status":"warning"} • /v1/catalog/nodes Λblocking queryͰ؂ࢹ ଘࡏ͢ΔnodeͷσʔλͷΈϑΟϧλ
  51. consul-kv-dashboard trigger $ consul-kv-dashboard -trigger COMMAND ΧςΰϦຖʹঢ়ଶ(success→warningͳͲ)͕ มԽͨ͠ΒίϚϯυ࣮ߦՄೳ JSON͕ඪ४ೖྗʹ౉͞ΕΔ {

    "category":"testing", "node":"web01", "address":"192.168.1.10", "timestamp":"2015-01-21 11:22:33 +0900", "status":"danger", "key":"","data":"failure!!" }
  52. ΦʔτεέʔϧͰͷ஫ҙ఺ ! AMIʹ࢒͍ͬͯΔݹ͍ΞϓϦ͕ىಈ " ࠷৽ͷ deploy IDͰͳ͍৔߹͸ىಈ͠ͳ͍ deploy࣌: unique ͳ

    ID Λൃߦ • ϑΝΠϧʹॻ͍ͯ tar ʹೖΕΔ • KV ʹ΋ೖΕΔ ىಈ࣌:ϩʔΧϧϑΝΠϧͷ ID ͱ KV Λൺֱ • ҟͳ͍ͬͯͨΒ sleep 10 && exit → restart
  53. bash-completionͰsshͷϗετ໊ิ׬ ~/.bash_profile _known_hosts_real() { local members=$(consul members -status=alive | awk

    '!/Node/{printf("%s ", $1)}') COMPREPLY=( $( \ compgen -W "$members" \ ${COMP_WORDS[COMP_CWORD]} \ ) ) return 0 } ! ݱࡏaliveͳϗετͷΈ͕ग़ͯ͘Δʂ
  54. Questions? • Architecture, Service Discovery Health Checking, Key/Value Store •

    ಺෦DNS • ຊ൪؀ڥӡ༻ / ߴՄ༻ੑ • Φʔτεέʔϧ • Stretcher • Consul KV Dashboard