Slide 1
Slide 1 text
Consulͱࣗ࡞OSS
Λ׆༻ͨ͠
100نͷWeb
αʔϏεӡ༻
YAPC::Asia 2015
@fujiwara
Slide 2
Slide 2 text
౻ݪ ढ़Ұ
@fujiwara
github.com/fujiwara
sfujiwara.hatenablog.com
ٕज़෦
Slide 3
Slide 3 text
Agenda
Lobi
Consul
Consul & My OSS in Lobi
Slide 4
Slide 4 text
Game & Community
Slide 5
Slide 5 text
Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ
Slide 6
Slide 6 text
Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ
Slide 7
Slide 7 text
Lobiͷαʔό
Slide 8
Slide 8 text
Lobiͷαʔόมભ
1. 2010~2011 AWS (US) : 4(?)
2. 2011~2013 ࣗࣾαʔό : 4 ~ 20
3. 2013.11~ AWS (Tokyo) : 20 ~ 100
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Lobiͷαʔό
EC2ͰՔಇ͍ͯ͠Δϗετͷछྨ͕ଟ͍
• app, sdk, stream, sdk-stream, db(3shard),
transcode, log(aggregate,analyze),batch,
deploy...
࣮ݴޠ Perl, Node.js, Go
ϛυϧΣΞ͍Ζ͍Ζ
• Nginx, MySQL, Starlet, Fluentd, Norikra,
memcached, HAProxy, gearman, twemproxy,
MHA, dnsmasq...
Slide 11
Slide 11 text
AWS(Tokyo)ҠߦޙͷΈ(2014)
෦DNS͕ͳ͍
• /etc/hostsΛChefͰੜ
• DNSαʔόࣗલ໘…(࣌Internal Route53
ͳ͠)
ϗετՃɼআͷස͕ߴ͍
• Φʔτεέʔϧ͍ͨ͠
ConsulΛ͓͏ʂ
Slide 12
Slide 12 text
Consul?
Slide 13
Slide 13 text
What's Consul
www.consul.io - HashiCorp product
• Service Discovery
• Health Checking
• Key/Value Store
• Multi Datacenter
Slide 14
Slide 14 text
Architecture
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
Agent
ΫϥελͷnodeશͰಈ࡞͢Δdaemon
Client mode or Server mode ͷͲͪΒ͔Ͱಈ࡞
ϢʔβʹDNS, HTTP interfaceΛఏڙ͢Δ
• Ϣʔβجຊతʹlocalhostͷagentͱ௨৴
• Agentಉ͕࢜RPCͰ௨৴(͋·Γҙࣝ͢Δඞཁ
ͳ͍)
GoͰॻ͔Ε͍ͯͯ1όΠφϦͰಈ࡞ (CLIಉҰ)
Slide 18
Slide 18 text
Server
ΫϥελͷগnodeͰಈ࡞(3Ҏ্ͷحਪ)
ৗʹͲΕ͔1ͷΈLeaderʹͳΔ
• RaftΞϧΰϦζϜʹΑͬͯLeader͕બग़͞ΕΔ
σʔλͷॻ͖ࠐΈৗʹLeaderʹ
σʔλLeader͔ΒଞͷServerϨϓϦϨʔγϣ
ϯ
Slide 19
Slide 19 text
Raft
Raft is a protocol for implementing distributed
consensus.
ࢄڥͰͷ߹ҙΞϧΰϦζϜ(ϓϩτίϧ)
Leaderબग़ʹserver nodeͷաͷಉҙ͕ඞཁ
ͳͷͰ࠷3 node͕ඞཁ
Slide 20
Slide 20 text
Raft ࢀߟࢿྉ
thesecretlivesofdata.com/raft/
• ΞχϝʔγϣϯͰΘ͔Γ͍͢
www.slideshare.net/pfi/raft-36155398
• ຊޠͰͷղઆࢿྉ
github.com/hashicorp/raft
• Consul͕༻͍ͯ͠ΔGo࣮
Slide 21
Slide 21 text
Service / Node Discovery
Slide 22
Slide 22 text
Service / Node Discovery
αʔϏεͷnodeͯ͢ͰΫϥελΛߏ͢Δ
• nodeɺnode্Ͱఏڙ͢ΔαʔϏεͷൃݟ
• DNS, HTTP API
ࢮ׆ࢹࣦഊͨ͠nodeԠ͔ΒࣗಈͰ֎ΕΔ
Slide 23
Slide 23 text
No content
Slide 24
Slide 24 text
Node Discovery
consul members ͰΫϥελͷnodeΛҰཡ
$ consul members
Node Address Status Type Build Protocol DC
my-app-i-123456 192.168.1.12:8301 alive server 0.5.2 2 dc1
my-app-i-234567 192.168.1.23:8301 alive server 0.5.2 2 dc1
my-app-i-345678 192.168.1.34:8301 alive server 0.5.2 2 dc1
my-db-i-456789 192.168.1.45:8301 alive client 0.5.2 2 dc1
my-db-i-567890 192.168.1.56:8301 alive client 0.5.2 2 dc1
my-db-i-678901 192.168.1.67:8301 alive client 0.5.2 2 dc1
my-app-i-987654 192.168.1.99:8301 failed client 0.5.2 2 dc1
my-app-i-876543 192.168.1.87:8301 left client 0.5.2 2 dc1
(࣮ࡍ݁Ռͷॱ൪ෆఆ)
Slide 25
Slide 25 text
Node Discovery
Status=failed : agentͷࢮ׆ࢹʹࣦഊͨ͠node
$ consul members -status failed
Node Address Status Type Build Protocol DC
my-app-i-987654 192.168.1.99:8301 failed server 0.5.2 2 dc1
Status=left : ਖ਼ৗʹΫϥελ͔Βͨ͠node
$ consul members -status left
Node Address Status Type Build Protocol DC
my-app-i-876543 192.168.1.87:8301 left server 0.5.2 2 dc1
Slide 26
Slide 26 text
Node Discovery via DNS interface
consul agent (127.0.0.1:8600) ʹ͍߹ΘͤΔ
$ dig @127.0.0.1 -p 8600 my-app-i-123456.node.consul
;; QUESTION SECTION:
;my-app-i-123456.node.consul. IN A
;; ANSWER SECTION:
my-app-i-123456.node.consul. 0 IN A 192.168.1.12
Slide 27
Slide 27 text
Node Discovery via DNS interface
Status=failed : DNSͰΞυϨε͕Ҿ͚Δ
• Ұ࣌తʹࣄނͰ͍ͯ͠ΔՄೳੑ͕͋Δ
Status=left : DNSͰΞυϨε͕Ҿ͚ͳ͘ͳΔ
• .consul υϝΠϯͷ໊લղܾΛ consul agent ʹ
͛Δ͜ͱͰ෦DNSͱͯ͠ར༻Ͱ͖Δ
• υϝΠϯ໊ઃఆͰมߋՄೳ
Slide 28
Slide 28 text
Service Difinition of App
my-app-i-*:/etc/consul.d/app.json
{
"service": {
"name": "app",
"port": 3000
}
}
Slide 29
Slide 29 text
Service Discovery of App via DNS interface
consul agent (127.0.0.1:8600) ʹ͍߹ΘͤΔ
$ dig @127.0.0.1 -p 8600 app.service.consul
;; QUESTION SECTION:
;app.service.consul. IN A
;; ANSWER SECTION:
app.service.consul. 0 IN A 192.168.1.12
app.service.consul. 0 IN A 192.168.1.23
app.service.consul. 0 IN A 192.168.1.34
Slide 30
Slide 30 text
Service Discovery via DNS interface
Answerͷॱ൪ϥϯμϜ (≒ Round Robin)
UDPͰͷ͍߹ΘͤͰର͕4ΞυϨεҎ্
͋Δ߹ɺ3ΞυϨεͷΈฦΔ
• TCPͰͯ͢ฦΔ
TTL ઃఆՄೳ (default 0)
Slide 31
Slide 31 text
Service Discovery of App via HTTP API
http://127.0.0.1:8500 ʹΞΫηε
$ curl http://127.0.0.1:8500/v1/catalog/service/app
[
{
"Node": "my-app-i-123456", "Address": "192.168.1.12",
"ServiceID": "app", "ServiceName": "app", "ServicePort": 3000, ...
},
{
"Node": "my-app-i-234567", "Address": "192.168.1.23", ...
}
{
"Node": "my-app-i-345678", "Address": "192.168.1.34", ...
}
]
Slide 32
Slide 32 text
Service Difinision of DB (master)
my-db-i-456789:/etc/consul.d/db.json
{
"service": {
"name": "db",
"port": 3306,
"tags": ["master"]
}
}
Slide 33
Slide 33 text
Service Difinision of DB (slave)
my-db-i-567890,678901:/etc/consul.d/db.json
{
"service": {
"name": "db",
"port": 3306,
"tags": ["slave"]
}
}
Slide 34
Slide 34 text
Service Discovery of DB (master/slave)
{tag}.{service}.service.consul Ͱ໊લղܾ
$ dig @127.0.0.1 -p 8600 master.db.service.consul
master.db.service.consul. 0 IN A 192.168.1.45
$ dig @127.0.0.1 -p 8600 slave.db.service.consul
slave.db.service.consul. 0 IN A 192.168.1.67
slave.db.service.consul. 0 IN A 192.168.1.56
$ dig @127.0.0.1 -p 8600 db.service.consul
db.service.consul. 0 IN A 192.168.1.56
db.service.consul. 0 IN A 192.168.1.45
db.service.consul. 0 IN A 192.168.1.67
Slide 35
Slide 35 text
External Service
nodeʹؔ࿈͠ͳ͍ɺ֎෦DNSͰఆٛ͞Ε͍ͯΔ
໊લIPΞυϨεαʔϏεͱͯ͠ఆٛͰ͖Δ
$ curl -X PUT -d '{
"Node":"rds",
"Address":"my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com",
"Service":{"Service": "rds"}
}'
http://127.0.0.1:8500/v1/catalog/register
$ dig @127.0.0.1 -p 8600 rds.service.consul.
;; ANSWER SECTION:
rds.service.consul. 0 IN CNAME my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com.
my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. 10 IN A 192.168.1.100
Slide 36
Slide 36 text
External Service
DNS໊Λొ͢Δ߹ɺconsulͷઃఆʹ
recursors (֎෦ͷ໊લղܾΛ͢ΔDNSαʔόͷ
ΞυϨε) Λ͓ͯ͘͠
• ઃఆ͠ͳ͍ͱ CNAME ͷΈ͕ฦΔ
consul agent֎෦ͷ໊લղܾ݁ՌΛcache͠ͳ
͍
Slide 37
Slide 37 text
Health Checking
Slide 38
Slide 38 text
Health Checking
֤αʔϏεʹ͍ͭͯɺ3λΠϓͷϔϧενΣοΫ
ΛఆٛͰ͖Δ
• Script
• HTTP
• TTL
Slide 39
Slide 39 text
Health Checking by script
ϢʔβఆٛͷϔϧενΣοΫίϚϯυΛ࣮ߦ
exit codeͰঢ়ଶΛ௨ (Nagios/Sensuޓ)
• 0 : success
• 1 : warning
• 2 : fail
failͷ߹DNS, HTTPͷԠ͔Β֎ΕΔ
Slide 40
Slide 40 text
Health Checking by HTTP
consul agent͕HTTPͰΞΫηε
• HTTP 2xx : success
• HTTP 429 : warning
• ͦΕҎ֎ : fail
Slide 41
Slide 41 text
Health Checking by TTL
ఆظతʹagentʹHTTP PUTͯ͠ੜଘΛ͑Δ
TTL͕ΕΔ·ͰʹPUT͕ͳ͚Εfailѻ͍
Slide 42
Slide 42 text
Key/Value Store
Slide 43
Slide 43 text
Key/Value Store
ҙͷΛग़͠ೖΕͰ͖ΔKVS
$ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1'
true
$ curl http://127.0.0.1:8500/v1/kv/web/key1
[{
"CreateIndex": 112,
"ModifyIndex": 112,
"LockIndex": 0,
"Key": "web/key1",
"Flags": 0,
"Value": "dGVzdA=="
}]
Slide 44
Slide 44 text
Key/Value Store
URLҾͰϝλσʔλ (flags) Λอ࣋Ͱ͖Δ
64bit int, ༻్Ϣʔβͷҙ
$ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1?flags=123'
true
$ curl 'http://127.0.0.1:8500/v1/kv/web/key1'
[{
...
"Key": "web/key1",
"Flags": 123, // <------- ͜Ε
"Value": "dGVzdA=="
}]
Slide 45
Slide 45 text
Key/Value Store
jsonϨεϙϯεͷBase64 encode͞Ε͍ͯΔ
jsonͰͳ͚ͩ͘ੜͰऔΓ͍ͨ߹Ҿ raw
$ curl "http://127.0.0.1:8500/v1/kv/web/key1?raw"
test
Slide 46
Slide 46 text
Key/Value Store
͋Δ֊ͷԼͷΛ࠶ؼతʹऔΓ͍ͨ߹
Ҿ recurse
$ curl -s "http://127.0.0.1:8500/v1/kv/web/?recurse"
[
{"CreateIndex":112,"ModifyIndex":115,"LockIndex":0,
"Key":"web/key1","Flags":123,"Value":"dGVzdA=="},
{"CreateIndex":122,"ModifyIndex":122,"LockIndex":0,
"Key":"web/key2","Flags":0,"Value":"dGVzdDI="},
{"CreateIndex":124,"ModifyIndex":124,"LockIndex":0,
"Key":"web/test/1","Flags":0,"Value":"dGVzdDM="}
]
όοΫΞοϓʹར༻Մೳ
Slide 47
Slide 47 text
Key/Value Store Benchmark GET
$ wrk -c 10 -d 10 -t 2 http://127.0.0.1:8500/v1/kv/web/key1
Server(Leader): 41,832 Requests/sec
Server/Client(Follower): 17,281
Server(Follower) stale mode: 37,013
Client(Follower) stale mode: 16,938
Consul v0.5.2 on EC2 c4.2xlarge, GOMAXPROCS=4
Slide 48
Slide 48 text
Key/Value Store Benchmark PUT
$ wrk -c 10 -d 10 -t 2 -s put.lua http://127.0.0.1:8500/v1/kv/web/key1
-- put.lua
wrk.method = "PUT"
wrk.body = "test"
wrk.headers["Content-Type"] = "application/x-www-form-urlencoded"
Server/Client(Follower): 427.56 Requests/sec
Slide 49
Slide 49 text
Key/Value Store
ৗʹ 127.0.0.1:8500 Λ͑Α͍ͷͰָ
͔ͳΓߴͳͷͰGETԕྀͳ͑͘Δ
σϑΥϧτͰͯ͢ͷ͍߹ΘͤΛ
Leader node͕ॲཧ͢Δ
• stale modeʹ͢ΔͱLeaderҎ֎ͷServerԠ
Ͱ͖Δ
(Ұ؏ੑʹ͍ͭͯޙड़)
Slide 50
Slide 50 text
෦DNSͱͯ͠ConsulΛ͏
Slide 51
Slide 51 text
෦DNSͱͯ͠ConsulΛ͏
node,serviceͷ໊લղܾΛConsul Agent͚Δ
! resolv.confͰϙʔτࢦఆͰ͖ͳ͍
! Port 53ಛݖ͕ඞཁ
→ ͦͷͨΊ͚ͩʹAgentΛrootͰಈ͔͢ͷ !
Agent࠶ؼ͍߹Θͤ(ҰԠ)Ͱ͖Δ͕cacheػ
ೳ͕ͳ͍ !
→ dnsmasq, bindͳͲ͔Β.consulυϝΠϯͷΈ
forward !
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
෦DNSͱͯ͠ConsulΛ͏
dnsmasqΛશͰىಈ
.consul υϝΠϯͷ໊લղܾconsul agent
127.0.0.1:53 Λdnsmasq͕Listen͢Δ
# dnsmasq.conf
server=/consul/127.0.0.1#8600
bind-interfaces
listen-address=127.0.0.1
Slide 54
Slide 54 text
෦DNSͱͯ͠ConsulΛ͏
resolv.conf Ͱ (node|service).consul Λݕࡧυ
ϝΠϯʹࢦఆ
→ node໊ɺservice໊͚ͩͰଓͰ͖Δ
# /etc/resolv.conf
search node.consul service.consul
nameserver 127.0.0.1 # dnsmasq
nameserver 172.16.0.2 # VPC resolver
nameserver 172.16.0.254 # Unbound on EC2
Slide 55
Slide 55 text
෦DNSͱͯ͠ConsulΛ͏
consul.io/docs/guides/forwarding.html
BINDΛ͏ྫ
Slide 56
Slide 56 text
ConsulΛຊ൪ڥͰӡ༻͢Δ
ͨΊʹ
Slide 57
Slide 57 text
Deployment Table
Server node: ຊ൪Ͱ࠷3, 3 or 5͕ਪ
consul.io/docs/internals/consensus.html
Slide 58
Slide 58 text
ServerʹඞཁͳϦιʔε
• CPU: 2CPUͰे
• GOMAXPROCS=2 Ҏ্Ͱಈ࡞ͤ͞ΔͷΛڧ͘ਪ
• Memory: 20MBʙ
• Disk: 2MBʙ
Memory, DiskKVͷར༻ঢ়گ࣍ୈ
Slide 59
Slide 59 text
Serverʹઐ༻ϗετ͕ඞཁʁ
consul agentࣗମͦΕ΄ͲϦιʔεΛ
༻͠ͳ͍ͨΊɺಉډՄೳ͕ͩ…
Disk IO͕ߴෛՙͳ߹ʹRaftͷHeartbeat͕
ࣦഊ͍͢͠
• Timeout 500ms
• Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ
• ௨ৗ2,3ඵͰબग़ྃ͢Δ
• ͦͷؒॻ͖ࠐΈॲཧ͕Ͱ͖ͳ͍
Slide 60
Slide 60 text
Daemonize
consul agentࣗDeamonಈ࡞Ϟʔυ͕ͳ͍
• Daemontools
• RPM & init script
• github.com/tomhillable/consul-rpm
• Systemd
ͲΕͰ͓ΈͰ
Slide 61
Slide 61 text
Bootstrapping
consul agentىಈޙɺΫϥελʹjoin͢Δඞཁ
͕͋Δ
1. ίϚϯυͰ consul joinΛ࣮ߦ
2. ઃఆϑΝΠϧͰ start_join Λࢦఆ
3. Atlas࿈ܞ
Slide 62
Slide 62 text
1. consul join
αʔόͷΞυϨεΛࢦఆͯ͠join
$ consul join 192.168.1.11 192.168.1.12 192.168.1.13
ࣗಈԽͮ͠Β͍ͷͰςετҎ֎ͰΘͳ͍
Slide 63
Slide 63 text
2. start_join
consul.confʹαʔόͷΞυϨεΛࢦఆ͓ͯ͘͠
{
"start_join": [ "192.168.1.11", "192.168.1.12", "192.168.1.13"]
}
consul serverʹݻఆIPΞυϨεΛৼΔ͔ɺผͷ
DNSͰղܾͰ͖ΔΑ͏ʹ͓ͯ͘͠
Slide 64
Slide 64 text
3. Atlas࿈ܞ
Atlas - atlas.hashicorp.com
Vagrant Packer
Terraform ConsulΛ౷߹͢ΔαʔϏε
$ consul agent ... \
-atlas=ATLAS_USERNAME/infrastructure \
-atlas-join \
-atlas-token="YOUR_ATLAS_TOKEN" \
! ServerͷΞυϨεΛཧ͢Δඞཁ͕ͳ͍ͷͰָ
" 11nodeҎ্$40/node
Slide 65
Slide 65 text
ߴՄ༻ੑͷͨΊʹ
Slide 66
Slide 66 text
ߴՄ༻ੑͷͨΊʹ
ServerʹΑΓಉ࣌ʹো
Λىͯ͜͠ͳ͍node
͕มΘΔ
• 3 node → 1
• 5 node → 2
3 nodeߏ࣌ɺ2མͪͯ
Γ1ʹͳͬͯ͠·͏ͱLeader
͕બग़Ͱ͖ͳ͍
࣌ؒΓ͢ϝϯςφϯε
࣌ʹҰ࣌తʹServer nodeΛ
૿͢ख
Slide 67
Slide 67 text
ߴՄ༻ੑͷͨΊʹ
Server nodeͷfailoverࣗಈ
! Ϣʔβৗʹlocalhostͷagent͚ͩΛΈ͍ͯΕ
Α͍
Slide 68
Slide 68 text
nodeো࣌ͷӨڹ
! LeaderͰͳ͍ → " ଞnodeʹӨڹͳ͠
! Leader → " Leader࠶બग़
σϑΥϧτͰͯ͢ͷಡΈॻ͖ΛLeader͕ॲཧ
(ڧҰ؏ੑ)
Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
Slide 69
Slide 69 text
Stale mode (DNS)
Leader࠶બग़௨ৗ2ʙ3ඵͰྃ
ͦͷؒDNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ
→ Stale mode : Leaderະબग़ͰԠՄೳ
"dns_config":{
"allow_stale": true, // default false
"max_stale": "10s" // default 5s
}
݁Ռݹ͍Մೳੑ͕͋Δ(݁Ռ߹ੑ)
Slide 70
Slide 70 text
DNS TTL
defaultTTL 0 → cache͞Εͳ͍
node, serviceผʹTTLΛઃఆՄೳ
DNS cacheΛલஈʹஔͯ͠cacheͰ͖Δ
"dns_config":{
"node_ttl": "60s",
"service_ttl": {
"*": "15s"
}
}
Slide 71
Slide 71 text
Stale mode (HTTP API)
HTTP APIͰstale modeʹ͢Δ߹Ҿ stale
$ curl "http://127.0.0.1:8500/v1/kv/web/key1?stale"
staleҾͳ͠ͰLeaderબग़தʹΞΫηε
→ 500 Internal Server Error
Slide 72
Slide 72 text
ӡ༻தͷUpgrade
consul.io/docs/upgrading.html
consul.io/docs/upgrade-specific.html
όʔδϣϯผʹҙ͕͋ΔͷͰυΩϡϝϯτΛ
ॱ൪ʹAgentΛೖΕସ͑Δ͜ͱͰ
Rolling upgradeՄೳ
(Leader nodeೖΕସ͑Ͱ࠶બग़ى͖Δ)
Slide 73
Slide 73 text
҆ఆੑ
v0.2͔࣌Β1Ҏ্ӡ༻
Agentϓϩηε͕མͪͨ͜ͱͳ͍
! ΦϖϛεͰશServerΛಉ࣌ʹམͱ͢ͱճ෮ෆೳ
→ KVఆظόοΫΞοϓΛ
(ϢʔβσʔλೖΕͪΌμϝ ! )
Slide 74
Slide 74 text
Agentࣗମͷࢹ
Agent processࣗମͷࢹผ్
• processࢹ(consul agent)
• TCP/UDP 8600 (DNS)
• TCP 8500 (HTTP)
• http://127.0.0.1:8500/v1/status/leader
༰มߋݕ
• Leader Lost→࠶બग़ͰมΘΔ
Slide 75
Slide 75 text
ΦʔτεέʔϧڥͰ
૿ݮ͢ΔαʔόΛѻ͏
Slide 76
Slide 76 text
2014.02ʙ Lobi Rec SDKϦϦʔε
εϚϑΥΞϓϦʹSDKΛΈࠐΜͰPlayಈըը
αʔόʹuploadͯ͠มɺӾཡ
มػೳElasticTranscoderͰ࣮
Slide 77
Slide 77 text
ElasticTranscoder
! Managed ServiceͳͷͰཧָ͕
! มೳྗউखʹεέʔϧ
! ͪΐͬͱ͓ߴ͍…
• SD $0.017/min
• HD $0.034/min
ߘ͋ͨΓ4ύλʔϯ ฏۉ2 = $0.204 ≒ 25ԁ
Slide 78
Slide 78 text
2014.10
ϞϯελʔετϥΠΫ
Lobi Rec SDKಋೖʂʂʂ
Slide 79
Slide 79 text
ϞϯελʔετϥΠΫ
Lobi Rec SDKಋೖ
ώοτήʔϜʹ͖ͭ
େྔͷಈը͕…
!""""""!
Slide 80
Slide 80 text
EC2 Spot InstanceͰಈըม
! ཧ͕໘
! উखʹεέʔϧͯ͘͠Εͳ͍
! ElasticTranscoderΑΓѹతʹ҆Ձ
• ElasticTranscoder = $0.204/ߘ
• EC2 Spot cc2.8xlarge(32core) = $0.45/hour
Slide 81
Slide 81 text
Spot InstanceͰΦʔτεέʔϧ
Slide 82
Slide 82 text
Spot InstanceͰΦʔτεέʔϧ
• εέʔϧΞτ: CPUෛՙͰͳ͘JobͷྔͰ
• Job͕ཷ·Βͳ͍ݶΓCPU 100% !
• εέʔϧΠϯ: CPU idle͕શମͷ25%Ҏ্
Slide 83
Slide 83 text
ΦʔτεέʔϧڥͰͷΠϯελϯεىಈ
1. ࣗಈͰuniqueͳϗετ໊Λ͚Δ
2. consul join
3. ChefʹΑΔϓϩϏδϣχϯά
4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy
5. Zabbixࣗಈొͯ͠ࢹରʹՃ
Slide 84
Slide 84 text
1. ࣗಈͰuniqueͳϗετ໊Λ͚Δ
ConsulͷͨΊʹҰҙͳhostname͕ඞཁ
Cloud-InitͰىಈ͢ΔϗετͷλΠϓΛઃఆ
#cloud-config
runcmd:
- [sh, -c, 'echo "HOSTNAME_PREFIX=transcode" > /etc/sysconfig/hostname-prefix']
rc.localͰಡΈࠐΉ
# /etc/rc.local
if [ -f /etc/sysconfig/hostname-prefix ]; then
. /etc/sysconfig/hostname-prefix
fi
Slide 85
Slide 85 text
1. ࣗಈͰuniqueͳϗετ໊Λ͚Δ
hostnameΛ prefix + InstanceID
EC2 Name tag ༩
# /etc/rc.local
instance_id=$(curl -s 169.254.169.254/latest/meta-data/instance-id)
new_hostname="${HOSTNAME_PREFIX}-$instance_id"
hostname $new_hostname
aws ec2 create-tags \
--resources $instance_id \
--tags "Key=Name,Value=$new_hostname"
Slide 86
Slide 86 text
2. ConsulΫϥελʹjoin
consul agent -node $(hostname)
ࣗಈతʹ෦DNSͰ໊લղܾՄೳʹ
Slide 87
Slide 87 text
3. ChefʹΑΔϓϩϏδϣχϯά
Consul KV͔ΒϗετλΠϓ͝ͱͷJSONΛऔಘ͠
࣮ͯߦ
JSON=$(curl -s "localhost:8500/v1/kv/nodes/bootup/${HOSTNAME_PREFIX}.json?raw")
echo "$JSON" > /tmp/chef-bootup.json
chef-client -j /tmp/chef-bootup.json
ॳճChef࣮ߦ࣌ʹChef-Serverʹొ͞ΕΔ
Slide 88
Slide 88 text
4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy
Stretcher(ޙड़)Ͱdeploy
5. Zabbixࣗಈొͯ͠ࢹରʹՃ
deploy࣮ߦޙɺzabbix-agentΛىಈˠࣗಈొ
deployྃલʹىಈ͢ΔͱΞϥʔτ͕ൃ๒ͯ͠͠
·͏
Slide 89
Slide 89 text
StretcherΛར༻ͨ͠σϓϩΠ
Dan Zen https://www.flickr.com/photos/danzen/2288626158/
Slide 90
Slide 90 text
What's Stretcher
github.com/fujiwara/stretcher
Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ
Slide 91
Slide 91 text
Why Stretcher?
Archer(rsync) ʹΑΔதԝϗετ͔Βͷdeploy
! pushͰΦʔτεέʔϧʹରԠͰ͖ͳ͍
! ֤ϗετ͔ΒrsyncͰpull?
→ buildதʹrsync͞ΕͨΒ…
! ֤ϗετ͔Βgit pull?
→ grunt, GoͳͲͷbuildੜΛೖΕͨ͘ͳ͍
! ͕ଟ͍ͱssh+rsyncgit pullπϥ͍
! AMI࡞Γ͠&ೖΕସ͑ͯͳ͍
Slide 92
Slide 92 text
Inspired by
github.com/sorah/mamiya
&
AWS CodeDeploy
Slide 93
Slide 93 text
ઃܭํ
AWS͡Όͳͯ͘ಈ͘
rsync & ίϚϯυ࣮ߦɺͱ͍͏ϑϩʔ౿ऻ
Consul eventͰΠϕϯτ௨
Consul ͱૄ݁߹ (consul watchͰىಈ)
GoͰॻ͘
Slide 94
Slide 94 text
Architecture
Slide 95
Slide 95 text
Consul event
֤nodeʹGossip ProtocolͰΠϕϯτΛ
ૹ৴͢ΔΈ
$ consul event -name EVENT_NAME [-node REGEX] PAYLOAD
Event ID: 3b1f3199-6e69-4b82-4812-b35058864fdd
ࢦఆͨ͠Πϕϯτ໊Ͱ
(ਖ਼نදݱʹϚον͢Δnodeʹ)
payloadΛૹ৴
Slide 96
Slide 96 text
Consul watch
ࢦఆͨ͠ΠϕϯτΛड৴ͨ͠ΒίϚϯυΛ
࣮ߦ͢ΔΈ
$ consul watch -type event -name EVENT_NAME COMMAND
payloadඪ४ೖྗ͔ΒJSONͰ͞ΕΔ
[{
"ID": "3b1f3199-6e69-4b82-4812-b35058864fdd",
"Name": "test",
"Payload": "TXkgcGF5bG9hZA==",
...
}]
Slide 97
Slide 97 text
Deployment process
1. ΞϓϦέʔγϣϯΛbuildͯ͠tar.gzʹ͢Δ
ґଘcpan moduleͳͲͯ͢ݻΊΔ
2. खॱॻ(manifest)Λॻ͘
3. tar.gz, manifestΛS3(or httpd)ʹ্͛Δ
4. consul event Ͱ manifest URLΛ௨
consul event -name deploy s3://...
✄---------- ͜͜·ͰstretcherͰͳ͍ ---------✄
Slide 98
Slide 98 text
✄------------ stretcher͔͜͜Β ------------✄
consul watch -type event -name deploy stretcher
1. event͔Βmanifest URLΛऔಘ
2. tar.gzΛऔಘͯ͠TMPDIRʹల։
3. rsync -av --deleteͰߋ৽
4. command࣮ߦ
• ΞϓϦέʔγϣϯ࠶ىಈͳͲ
Slide 99
Slide 99 text
Manifest
src: s3://example.com/app.tar.gz
checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93
dest: /home/stretcher/app
commands:
pre:
- echo 'staring deploy'
post:
- echo 'deploy done'
success:
- cat >> /path/to/success.log
failure:
- cat >> /path/to/failure.log
excludes:
- "*.pid"
- "*.socket"
Slide 100
Slide 100 text
LobiͰͷdeploy
tar.gz 200MB
ల։͢Δͱ400MB
• CPAN modules 110MB
• node_modules 10MB × 5
• Go app binaries 8MB × 5
• Static files (S3ʹஔ͖͍ͨ)
Slide 101
Slide 101 text
LobiͰͷdeploy
1. ‐ Push to production branch
2. ! Build (1 min~)
carton install, grunt, npm install, go
build ...
3. " Pack tar.gz & Upload (1 min)
4. # Deploy by Startecher (10~30 sec)
Slide 102
Slide 102 text
LobiͰͷdeploy
consul event ૹ৴͔Β10ʙ20ඵͰྃ !
2015/08/05 14:58:30 Starting up stretcher agent
2015/08/05 14:58:30 Waiting for consul events from STDIN...
2015/08/05 14:58:30 Executing manifest: s3://...
2015/08/05 14:58:33 Extract archive: /dev/shm/stretcher539962129 to /dev/shm/stretcher_src648982332
2015/08/05 14:58:36 rsync [-av --delete --exclude-from
/dev/shm/stretcher_src648982332/conf/rsync_exclude.web
/dev/shm/stretcher_src648982332/ /home/xxx/web/]
2015/08/05 14:58:36 sending incremental file list
...
sent 787780 bytes received 5230 bytes 1586020.00 bytes/sec
total size is 359702435 speedup is 453.59
2015/08/05 14:58:36 invoking command: /home/xxx/web/refresh_services.sh
2015/08/05 14:58:41 success.
2015/08/05 14:58:41 Deploy manifest succeeded.
Rollback͍ͨ͠ˠલͷmanifestΛeventૹ৴
→10secͰΔ
Slide 103
Slide 103 text
DeployͷྃΛͭ
Πϕϯτૹ৴ݩͰ࣮ߦ͕͔ྃΒͳ͍
• ࣦഊ͢Δϗετ͕͋Δ͔͠Εͳ͍
• ՔಇதͷશͰdeployྃͨ͠ͷΛ͍ͪͨ
……ϩάݟ͍͚ͨͲ100Ͳ͏ͬͯʁ
Slide 104
Slide 104 text
Consul KV Dashboard
Slide 105
Slide 105 text
Chef, Serverspec, Deploy, etc...
֤ϗετͰ࣮ߦͨ݁͠ՌΛ֬ೝ͍ͨ͠ͷ
• Chef
• Serverspec
• Stretcher
• etc
Slide 106
Slide 106 text
IRC / Slackʹ௨ʁ
ࣾ༻ nopaste-cli command
$ nopaste-cli -channel "lobi" -summary "Deploy done!" < deploy.log
nopasteʹPOSTͱಉ࣌ʹURL͕௨͞ΕΔ
100͋Δͱ௨͕……!!!!!!!!×100
Ͳ͔͜ͰҰཡͯ͠ݟ͍ͨʂ
Slide 107
Slide 107 text
Consul KV Dashboard
github.com/fujiwara/consul-kv-dashboard
Consul KVΛσʔλετΞʹͨ͠
μογϡϘʔυWebΞϓϦ
Slide 108
Slide 108 text
σʔλొ
Consul HTTP APIͰૹΔ
$ curl -X PUT -d "message" \
'127.0.0.1:8500/v1/kv/dashboard/example/myhostname?flags=1422607461000'
Slide 109
Slide 109 text
keyߏ
/v1/kv/dashboard/{category}/{nodename}?
flags=({unixtime} * 1000 + {status})
• category: chef, serverspec, deploy...
• nodename: Consulͷnode໊
• flags: unixtime * 1000 + status
• status: 0=Success 1=Warning
2=Danger 3=Info
Slide 110
Slide 110 text
ը໘ͷଈ࣌ߋ৽
ϒϩοΩϯάΫΤϦΛ͏
consul.io/docs/agent/http.html
$ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?recurese
HTTP/1.1 200 OK
Content-Type: application/json
X-Consul-Index: 261975
[{"CreateIndex":261891,"ModifyIndex":261975,"LockIndex":0,
"Key":"dashboard/chef/myhost","Flags":1422602855000,"Value":".....
Slide 111
Slide 111 text
Blocking query
Ϩεϙϯεϔομͷ X-Consul-Index Λ
࣍ͷϦΫΤετͷҾʹࢦఆ
$ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost
HTTP/1.1 200 OK
X-Consul-Index: 261975
...
$ curl 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?index=261975
৽͍͠σʔλ͕ൃੜ͢Δ·ͰϨεϙϯε͕Ԇ
͍ΘΏΔ Long pooling
Slide 112
Slide 112 text
Blocking queryͷ׆༻
Consul Template
github.com/hashicorp/consul-template
KV, node, serviceͷঢ়ଶมԽΛଈө
Template→fileߋ৽ˠcommand࣮ߦ
$ consul-template \
-consul 127.0.0.1:8500 \
-template "/tmp/template.ctmpl:/var/www/nginx.conf:service nginx restart"
Slide 113
Slide 113 text
consul-kv-dashboard͕Δ͜ͱ(1)
GoͰॻ͔ΕͨWebΞϓϦέʔγϣϯ
HTML, CSS, JavaScriptΛ
• go-bindataͰ·ͱΊͨϑΝΠϧΛ
http.FileServerͰ৴
• όΠφϦҰݸΛஔ͍ͯىಈ͢Δ͚ͩͰ
੩తϑΝΠϧ৴Ͱ͖Δ
Slide 114
Slide 114 text
consul-kv-dashboard͕Δ͜ͱ(2)
Consul HTTP API ͷ reverse proxy
• /api/... → 127.0.0.1:8500/v1/kv/dashboard/...
• ϨεϙϯεͷJSONΛܗ
{"Flags":1422608524001}
!
{"timestamp":"2015-01-30 18:02:04 +0900","status":"warning"}
• /v1/catalog/nodes Λblocking queryͰࢹ
ଘࡏ͢ΔnodeͷσʔλͷΈϑΟϧλ
Slide 115
Slide 115 text
consul-kv-dashboard trigger
$ consul-kv-dashboard -trigger COMMAND
ΧςΰϦຖʹঢ়ଶ(success→warningͳͲ)͕
มԽͨ͠ΒίϚϯυ࣮ߦՄೳ
JSON͕ඪ४ೖྗʹ͞ΕΔ
{
"category":"testing",
"node":"web01",
"address":"192.168.1.10",
"timestamp":"2015-01-21 11:22:33 +0900",
"status":"danger",
"key":"","data":"failure!!"
}
Slide 116
Slide 116 text
ΦʔτεέʔϧͰͷ
Deploy Tips
Slide 117
Slide 117 text
ىಈޙʹ࠷৽൛Λdeploy
deploy࣌ɺKVʹ࠷৽ͷmanifest URLΛอଘ
1. consul join
2. stretcherىಈ
3. ࠷৽ͷmanifest URLΛKV͔Βऔಘ
4. ࣗࣗʹ event (manifest URL) Λૹ৴
5. deploy !
Slide 118
Slide 118 text
ΦʔτεέʔϧͰͷҙ
! AMIʹ͍ͬͯΔݹ͍ΞϓϦ͕ىಈ
" ࠷৽ͷ deploy IDͰͳ͍߹ىಈ͠ͳ͍
deploy࣌: unique ͳ ID Λൃߦ
• ϑΝΠϧʹॻ͍ͯ tar ʹೖΕΔ
• KV ʹೖΕΔ
ىಈ࣌:ϩʔΧϧϑΝΠϧͷ ID ͱ KV Λൺֱ
• ҟͳ͍ͬͯͨΒ sleep 10 && exit → restart
Slide 119
Slide 119 text
bash-completionͰ
sshͷϗετ໊ิ
Slide 120
Slide 120 text
bash-completionͰsshͷϗετ໊ิ
σϑΥϧτͰ ~/.ssh/known_hosts Λݩʹิ
! աڈʹsshͨ͜͠ͱ͕͋Δϗετ͕ग़ͯ͘Δ
" ݱࡏطʹଘࡏ͠ͳ͍Մೳੑ͕͋Δ
" Ұsshͨ͜͠ͱ͕ͳ͍৽نϗετग़ͯ͜
ͳ͍
Slide 121
Slide 121 text
bash-completionͰsshͷϗετ໊ิ
~/.bash_profile
_known_hosts_real()
{
local members=$(consul members -status=alive | awk '!/Node/{printf("%s ", $1)}')
COMPREPLY=( $( \
compgen -W "$members" \
${COMP_WORDS[COMP_CWORD]} \
) )
return 0
}
! ݱࡏaliveͳϗετͷΈ͕ग़ͯ͘Δʂ
Slide 122
Slide 122 text
·ͱΊ
Slide 123
Slide 123 text
·ͱΊ
Consulػೳ๛…͕ͩ
͍͍ͨػೳ͚ͩ͑Α͍
• KVɺEventɺHealth Check...
DNSɺHTTPΛ͍͜ͳ͢ͱߴͳࣗಈԽ͕Մೳ
• consul-templateͰಈతLBઃఆ
Slide 124
Slide 124 text
Questions?
• Architecture, Service Discovery
Health Checking, Key/Value Store
• ෦DNS
• ຊ൪ڥӡ༻ / ߴՄ༻ੑ
• Φʔτεέʔϧ
• Stretcher
• Consul KV Dashboard