ADC2016: Axion meets HashiCorp

Axion meets HashiCorp
@atty303 / AdTech Core Development Group
© 2016 CyberAgent, Inc. 1

View Slide

ࣗݾ঺հ
Ѩ઒ ߞ࢘
4 2016-02 ʙ ج൫։ൃάϧʔϓ

View Slide

Agenda
4 Consul : Service Discovery
4 Nomad : Container Scheduling
4 Terraform : Infrastructure as a Code

View Slide

Axion Overview

View Slide

What's Axion ?
4 ࣾ಺޲͚(ϚΠΫϩ)αʔϏε
4 σʔλϕʔεͷΑ͏ͳ΋ͷ
4 ൺֱతߴ͍ࢀরස౓ (10k ops/s ~)
4 εϥΠυ಺ͷ IP ΍ݻ༗໊ࢺͳͲ͸μϛʔͰ͢

View Slide

Axion ։ൃʹ͓͚Δ
ٕज़త௅ઓ

View Slide

Docker ͱ
޲͖߹͏!
(AWS/GCP ʹཔΒͣ)

View Slide

Docker - Technology Radar
4 Technology Radar
2016-04 1 Ͱ ADOPT(࠾
༻ͤΑ)
4 ADOPT = ͜ͷτϨϯυ
ʹࠓ͙͢৐Δ΂͖
1 https://www.thoughtworks.com/radar/tools/
docker

View Slide

Docker ͱ޲͖߹͏ͨΊʹ…
Nomad ΫϥελʹΞϓϦΛσϓϩΠͯ͠ɺ
σϓϩΠͨ͠ΞϓϦͷΤϯυϙΠϯτ؅ཧʹ
Consul ͕ඞཁͰɺ͜ΕΒͷ؀ڥߏஙΛ
Terraform ʹ΍ΒͤΑ͏ɻ

View Slide

Consul
Service Discovery

View Slide

What's Consul ?
4 Service ͷ؅ཧͱݕࡧ
4 Key-Value Storage
4 ߴ͍Մ༻ੑΛඋ͑ͨ෼ࢄγεςϜ

View Slide

Technology Radar
4 Technology Radar
2016-04 2 Ͱ ADOPT(࠾
༻ͤΑ)
4 ADOPT = ͜ͷτϨϯυ
ʹࠓ͙͢৐Δ΂͖
2 https://www.thoughtworks.com/radar/tools/
consul

View Slide

Service Discovery
ωοτϫʔΫΛ௨ͯ͠ػೳΛఏڙ͢Δ Service Λ
ൃݟ͢Δػߏ
4 Service ͷྫ: api, mysql, kafka
4 Service ͷϝϯόʔ͸ IP:PORT ͷϦετ
4 ϔϧενΣοΫͰϝϯόʔͷࢮ׆؂ࢹ
4 HTTP ͱ DNS Ͱ໰͍߹Θ͕ͤͰ͖Δ

View Slide

DNS Ͱ A Ϩίʔυ໰͍߹Θͤ
$ dig @127.0.0.1 -p 8600 kafka.service.consul
;; QUESTION SECTION:
;kafka.service.consul. IN A
;; ANSWER SECTION:
kafka.service.consul. 0 IN A 192.168.0.21

View Slide

DNS Ͱ A Ϩίʔυ໰͍߹Θͤ
4 Consul agent Λ DNS αʔόʹࢦఆͯ͠໰͍߹
Θͤ
4 *.service.consul Λղܾ͢ΔͱαʔϏεͷϝ
ϯόʔ͕ฦͬͯ͘Δ
4 Ϧετͷฒͼ͸ϥϯμϜ (DNS ϥ΢ϯυϩϏϯ)
4 ϔϧενΣοΫΛ௨ͬͨੜ͖͍ͯΔϝϯόʔͩ
ؚ͚͕·ΕΔ

View Slide

DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ
$ dig @127.0.0.1 -p 8600 axion-api-admin.service.consul SRV
;; QUESTION SECTION:
;axion-api-admin.service.consul. IN SRV
;; ANSWER SECTION:
axion-api-admin.service.consul. 0 IN SRV 1 1 42319
acd-stg-axion-dock-y002.node.openstack-stg.consul.
axion-api-admin.service.consul. 0 IN SRV 1 1 38475
acd-stg-axion-dock-y001.node.openstack-stg.consul.
;; ADDITIONAL SECTION:
acd-stg-axion-dock-y002.node.openstack-stg.consul. 0 IN A 192.168.0.26
acd-stg-axion-dock-y001.node.openstack-stg.consul. 0 IN A 192.168.0.27

View Slide

DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ
4 SRV ϨίʔυͰ͋Ε͹ϙʔτ·Ͱ෼͔Δ
4 ϙʔτΛಈతׂΓ౰ͯ͢Δͱ͖ʹ༗༻
4 ͨͩ͠ར༻ଆͰ SRV ͷϨκϧό͕ඞཁ
4 طʹ SRV ΛಡΉ࣮૷͕ͳ͚Ε͹ HTTP API
ΛಡΉ࣮૷Λॻ͍ͨ΄͏ָ͕͔ͱ

View Slide

HTTP Ͱͷ໰͍߹Θͤ
$ curl -s http://localhost:8500/v1/catalog/service/kafka | jq .
[
{
"Node": "acd-stg-axion-kafka-y001.local",
"Address": "192.168.0.19",
"ServiceID": "kafka",
"ServiceName": "kafka",
"ServiceTags": [],
"ServiceAddress": "",
"ServicePort": 9092,
"ServiceEnableTagOverride": false,
"CreateIndex": 55508,
"ModifyIndex": 151479
},
...
]

View Slide

HTTP Ͱͷ໰͍߹Θͤ
4 ී௨ʹ JSON Ͱฦͬͯ͘Δ
4 DNS ͱҧͬͯࢮΜͰΔϝϯόʔΛ஌Δ͜ͱ͕
Ͱ͖Δ

View Slide

HTTP Blocking Query
4 HTTP API Ͱఏڙ͞Ε͍ͯΔσʔλͷมߋΛ؂
ࢹͰ͖Δ
4 DNS ΍ HTTP ΛϙʔϦϯάͤͣʹଈ࠲ʹϝϯ
όʔมߋΛݕ஌Ͱ͖Δ
4 ͍ΘΏΔ Server Push (HTTP Long Polling)
4 Non-Blocking ͳ HTTP Client Ͱ࢖͏͜ͱ

View Slide

αʔϏεͷొ࿥
4 ઃఆϑΝΠϧͰ੩తʹొ࿥
4 ϗετʹΠϯετʔϧ͞Ε͍ͯΔϛυϧ΢
ΣΞͳͲ
4 HTTP API Ͱಈతʹొ࿥
4 ಈతʹσϓϩΠ͢ΔΞϓϦέʔγϣϯͳͲ
4 Docker registrator ΍ Nomad ͱ౷߹

View Slide

ྫ͑͹ Kafka Λొ࿥͢Δ
{ "service": {
"name": "kafka-broker",
"port": 9092,
"checks": [ {
"id": "kakfa-broker",
"tcp": "localhost:9092",
"interval": "5s",
"timeout": "1s"
} ]
} }
4 Kafka ͕ಈ͍͍ͯΔϗετͰ Consul ͷઃఆϑΝΠϧʹهड़
4 tcp/9200 ΛϔϧενΣοΫ

View Slide

Docker ͱͷ࿈ܞ (registrator 3)
docker run --name=registrator -d --net=host
--restart=unless-stopped --log-driver=journald \
-v /var/run/docker.sock:/tmp/docker.sock \
gliderlabs/registrator:v6 \
-tags registrator -resync 3600 \
-ip $(curl -s http://169.254.169.254/2009-04-04/meta-data/local-ipv4) \
consul://localhost:8500
4 Docker ίϯςφΛࣗಈతʹ Consul ΁
Service ͱͯ͠ొ࿥ͯ͘͠ΕΔπʔϧ
3 https://github.com/gliderlabs/registrator

View Slide

Docker registrator
docker run --name=cadvisor -d \
--restart=unless-stopped --log-driver=journald \
--dns=192.168.168.1 --privileged=true \
-p 9101:8080 \
-e "SERVICE_NAME=cadvisor" \
-e "SERVICE_CHECK_HTTP=/" \
-e "SERVICE_CHECK_INTERVAL=30s" \
google/cadvisor:v0.22.0
4 ؀ڥม਺ SERVICE_* ʹ Service ͷఆٛΛॻ͍ͯίϯςφΛىಈ
4 registrator ͕ίϯςφىಈΛݕग़ͯ͠ Consul ΁ొ࿥
4 ίϯςφ͕ࢭ·ͬͨͱ͖͸ Consul ͔Β࡟আ

View Slide

Key-Value Storage
$ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1
true
$ curl http://localhost:8500/v1/kv/web/key1
[ {
"CreateIndex":97,
"ModifyIndex":97,
"Key":"web/key1",
"Flags":0,
"Value":"dGVzdA=="
} ]

View Slide

Key-Value Storage
4 ֊૚Խ͞ΕͨΩʔͷ GET/PUT/DELETE
4 Blocking Query Ͱ watch Ͱ͖Δ͠ɺ CAS ΋
͋Δ
4 εέʔϥϏϦςΟ͸ແ͍ͷͰ஫ҙ
4 ϗετ΍ΞϓϦͷઃఆͳͲࢀরස౓ͷ௿͍
σʔλͷΈ֨ೲ

View Slide

consul-template
4 Service ΍ KVS ͷมߋΛݕ஌ͯ͠ςϯϓϨʔ
τϑΝΠϧΛߋ৽ɺίϚϯυΛ࣮ߦͯ͘͠ΕΔ
πʔϧ
4 Consul ʹରԠ͍ͯ͠ͳ͍πʔϧ/ϛυϧ΢ΣΞ
ͱ Consul ͷΪϟοϓΛຒΊΔ

View Slide

consul-template ͱ HAProxy ͷྫ
global
daemon
maxconn {{key "service/haproxy/maxconn"}}
listen mysql-slave
bind *:3306{{range service "mysql-slave"}}
server {{.Node}} {{.Address}}:{{.Port}}{{end}}
4 haproxy.conf ͷςϯϓϨʔτΛ४උ
$ consul-template -consul localhost:8500 -template \
haproxy.ctmpl:/etc/haproxy/haproxy.conf:systemctl reload haproxy
4 consul-template Λىಈ͓ͯ͘͠

View Slide

consul-template ͱ HAProxy ͷྫ
global
daemon
maxconn 128
listen mysql-slave
bind *:3306
server dbslave01 10.0.0.1:3306
server dbslave02 10.0.0.2:3306
4 Service ͷมߋΛݕ஌ͯ͠ HAProxy ͷઃఆϑΝΠ
ϧΛߋ৽ͯ͠Ϧϩʔυͯ͘͠ΕΔ

View Slide

Consul ͷΞʔΩςΫνϟ
4 Server(Master) ઐ༻ϗετΛ 3 or 5 ୆
4 ଞͷશͯͷϗετͰ Agent(Client) Λ࣮ߦ

View Slide

Axion ͱ Consul

View Slide

Consul ͱ dnsmasq
# /etc/resolv.conf
nameserver 127.0.0.1
# /etc/dnsmasq.conf
server=/consul/127.0.0.1#8600
4 શͯͷϗετͰ Consul ͱ dnsmasq Λ࣮ߦ
4 Consul ͷଘࡏΛҙࣝ͠ͳͯ͘΋ *.consul Λ
໊લղܾͰ͖Δ

View Slide

Service
4 શͯͷωοτϫʔΫαʔϏεΛ Consul ʹొ࿥

View Slide

ϛυϧ΢ΣΞ΁ͷ઀ଓ
4 ΞϓϦ͔Βͷϛυϧ΢ΣΞ΁ͷ઀ଓͰ Consul
ͷ DNS Λࢀর
// Aerospike
new com.aerospike.client.async.AsyncClient(
policy, "aerospike.service.consul", 3000)
// Kafka Consumer
kafka.consumer.bootstrap.servers = "kafka.service.consul:9092"
// ࣮ࡍ͸ Configurable Ͱ͢

View Slide

ΫϥελϦϯάͱ Consul
4 Kafka, Aerospike, Cassandra, Redis Cluster ͷ
Α͏ͳΫϥελΛ૊Ήϛυϧ΢ΣΞͰ઀ଓઌͱͯ͠
ࢦఆ͢Δͷ͸ Ϋϥελʔ΁ͷίϯλΫτϙΠϯτ
4 ΫϥΠΞϯτ͸ੜ͖͍ͯΔϊʔυʹ઀ଓͰ͖Ε͹ɺ
͔ͦ͜ΒΫϥελʔϝϯόʔΛऔಘ͢Δ
4 Consul ͷ DNS ͸ੜ͖͍ͯΔϝϯόʔ͚ͩฦ͢ͷ
ͰɺՄ༻ੑͷͨΊʹෳ਺ϗετΛྻڍ͢Δඞཁ͕ແ
͘ͳΔ

View Slide

axion-client
4 Axion ͷ API αʔόʹ઀ଓ͢ΔΫϥΠΞϯτϥΠϒϥϦ
4 τϥϯεϙʔτʹ Scala ͷ RPC ϥΠϒϥϦͰ͋Δ finagle Λ
ར༻
4 finagle ͕ΫϥΠΞϯταΠυͷϩʔυόϥϯαʔΛ͍࣋ͬͯ
ΔͨΊɺͦͷϝϯόʔΛ Consul ͷαʔϏεͱಉظ͢Δίʔυ
Λ࣮૷ͯ͠ར༻
4 Finagle Resolver ͷॻ͖ํ͸ finagle-consul 4 Λࢀߟʹ
Blocking Query Λར༻࣮ͯ͠૷
4 https://github.com/kachayev/finagle-consul

View Slide

axcsync
4 Consul ͷ Service ͱ BIG-IP ͷ Pool
Member Λಉظ
4 consul-template + axc 5 + python εΫϦϓ
τͰ࣮૷
5 https://adtech.cyberagent.io/techblog/archives/59

View Slide

Prometheus
4 Pull ܕͷ؂ࢹγεςϜ Prometheus Λ࢖͍ͬͯΔ
4 ؂ࢹର৅Λऔಘ͢Δͷʹ Consul Service Λར༻ (૬ੑൈ܈ !)
scrape_configs:
- job_name: 'consul'
consul_sd_configs:
- server: '192.168.168.1:8500'
services:
- prometheus-pushgateway
- prometheus-node-exporter
- prometheus-consul-exporter
- prometheus-burrow
- cadvisor
- axion-api-admin

View Slide

Consul Λ࠾༻ͨ͠ཧ༝
Docker ίϯςφͷಈత഑ஔ͢Δ৔߹ɺαʔϏε
ͷ഑ஔΛܾΊଧͪͰ͖ͳ͍ͷͰ Service
Discovery ػߏ͕ඞਢʹͳΔ
4 ZooKeeper ʹ͸ͳ͍ DNS ͱ HTTP ʹΑΔ؆
୯ͳΞΫηε
4 etcd ʹ͸ͳ͍ Service ͷϔϧενΣοΫ
4 HashiCorp ͱ͍͏ϒϥϯυ

View Slide

Consul ·ͱΊ
4 Service Discovery ͕͋Δ͜ͱΛલఏʹγες
ϜΛ૊Ή͜ͱͰίϯϙʔωϯτͷґଘؔ܎Λ៉
ྷʹදݱͰ͖Δ
4 αʔϏεͷಈతͳมԽ΁ͷରԠ͕ඇৗʹߦ͍΍
͍͢
4 Docker ʹؔ܎ͳ͘༗༻

View Slide

Consul ·ͱΊ
4 Consul ͸ϋϚΓͲ͜Ζ͕ͳ͘ૉ௚ʹ࢖͑ͨ !
4 ඞཁ࠷খݶͷγϯϓϧͳػೳηοτͰ͋Γͳ͕
ΒԠ༻ੑ͕ߴ͍
4 ಋೖͷϦεΫ͕ແ͍Α͏ʹࢥ͑Δ
4 Kubernetes ͳͲ SD Λ͍࣋ͬͯΔ΋ͷΛ࢖͏
৔߹͸ඞཁͳ͍͔΋

View Slide

Nomad
Container Scheduling

View Slide

What's Nomad ?
4 Docker Ϋϥελͷ؅ཧπʔϧ
4 AWS ECS, Kubernetes, Docker Swarm,
Mesos ͱಉ͡໾ׂ
4 Ϋϥελʹίϯςφ(ΞϓϦ)Λεέδϡʔϧ
(σϓϩΠ)͢Δ
4 Consul ͱͷΠϯςάϨʔγϣϯ

View Slide

Technology Radar
4 Technology Radar 2016-04 6 Ͱ
ASSESS(ௐࠪͤΑ)
4 ASSESS = ࠓ͙͢࠾༻͢΂͖ͱ
͸ݴΘͳ͍͕ɺ஫໨͓ͯ͘͠΂
͖
4 ͪͳΈʹ Kubernetes ͸
TRIAL(ࢼߦͤΑ)
4 TRIAL = ϦεΫͷ௿͍ϓϩδΣ
ΫτͰ࠾༻ͯ͠ΈΔ͜ͱΛਪ঑
͢Δ
6 https://assets.thoughtworks.com/assets/
technology-radar-apr-2016-en.pdf

View Slide

Nomad ͷΞʔΩςΫνϟ
4 Server ઐ༻ϗετΛ 3 or 5 ୆
4 Docker Ϋϥελʹ͢ΔϗετͰ Client Λ࣮ߦ

View Slide

Job ఆٛ
job "axion-api" {
type = "service" # or "batch" or "system"
group "api" {
count = 2
task "api" {
driver = "docker"
config {
image = "internal-repository/axion-api:$TAG"
port_map {
thrift = 8000
}
}
}
}
}

View Slide

ϦιʔεׂΓ౰ͯ
task "api" {
resources {
cpu = 8000 # MHz
memory = 4096 # MB
disk = 20000 # MB
network {
mbits = 100 # MBits
port "thrift" {
static = 8888
}
}
}
}

View Slide

Consul Service ొ࿥
task "api" {
service {
name = "axion-api-thrift"
port = "thrift"
check {
type = "tcp"
interval = "2s"
timeout = "1s"
}
}
}

View Slide

Job ͷ࣮ߦ
$ nomad run axion-api.nomad
==> Monitoring evaluation "50f80881"
Evaluation triggered by job "axion-api"
Allocation "73526b78" created: node "0cc2a12f", group "api"
Evaluation status changed: "pending" -> "complete"
==> Evaluation "50f80881" finished with status "complete"

View Slide

Job ͷঢ়ଶ
$ nomad status axion-api
ID = axion-api
Name = axion-api
Type = service
Priority = 50
Datacenters = openstack-stg
Status = running
Periodic = false
==> Evaluations
ID Priority Triggered By Status
3bc903ef 50 rolling-update complete
3b772796 50 job-register complete
==> Allocations
ID Eval ID Node ID Task Group Desired Status
45f2e143 3bc903ef 7a6b1272 api run running
546a9c55 3b772796 8959a37c api run running

View Slide

഑ஔ(Alloc)ͷঢ়ଶ
$ nomad alloc-status 45f2e143
ID = 45f2e143
Eval ID = 3bc903ef
Name = axion-api.api[1]
Node ID = 7a6b1272
Job ID = axion-api
Client Status = running
==> Task Resources
Task: "api"
CPU Memory MB Disk MB IOPS Addresses
8000 4096 20000 0 admin: 10.4.85.27:40660
thrift: 10.4.85.27:9999
==> Task "api" is "running"
Recent Events:
Time Type Description
07/07/16 16:38:49 JST Started Task started by client
07/07/16 16:38:45 JST Received Task received by client

View Slide

ϩάͷ֬ೝ
$ nomad fs ls 45f2e143 alloc/logs
Mode Size Modfied Time Name
-rw-r--r-- 3.5 kB 07/07/16 16:38:52 JST api.stderr.0
-rw-r--r-- 4.6 kB 08/07/16 09:14:31 JST api.stdout.0
# nomad fs cat 45f2e143 alloc/logs/api.stdout.0

View Slide

શ Job ͷ֬ೝ
$ nomad status
ID Type Priority Status
axion-api service 50 running
axion-another-service service 50 running
axion-foo-batch batch 50 running
axion-foo-batch/periodic-1467924600 batch 50 dead

View Slide

Axion ͱ Nomad

View Slide

λεΫͷ഑ஔ
4 Nomad Ϋϥελ͸ 1 ͚ͭͩ
4 ͲͷϊʔυʹλεΫ͕഑ஔ͞ΕΔ͔͸෼͔Βͳ
͍
4 ύϑΥʔϚϯεʹහײͳ api ͸ϊʔυΛ઎༗͠
͍ͨ
4 ֤छ worker ΍όον͸ద౰ʹۭ͍͍ͯΔϊ
ʔυͰಈ͍ͯ͘ΕΕ͹͍͍

View Slide

Job constraint
constraint {
attribute = "${meta.role}"
value = "api"
}
4 Nomad ͷϊʔυʹઃఆͰ͖Δ meta ม਺Λ΋
ͱʹλεΫ഑ஔΛ੍ݶ

View Slide

Nomad ͷ meta ઃఆ
4 Consul ͷ KVS ͰϊʔυʹରԠ͢ΔΩʔʹ஋Λઃఆ͢Δͱ consul-template ͕൓Ԡ
ͯ͠ Nomad ͷઃఆΛม͑Δ

View Slide

Nomad Λ࢖ͬͨ
Axion ͷσϓϩΠϑϩʔ

View Slide

1). Bot ʹϦϦʔε४උΛґཔ

View Slide

2). ϦϦʔε PR ͕࡞੒͞ΕΔ
4 master -> deployment/release ΁ͷϚʔδ

View Slide

master ʹϚʔδ͞Εͨ PR Λऩूͯ͠Ϧετ
Խ

View Slide

֤ PR ͷ ## Release Notes ηΫγϣϯΛ
ू໿

View Slide

3). PR ϚʔδͰ CircleCI ͕Ϗϧυ
deployment:
release:
branch: deployment/release
commands:
- tar zcf deploy.tar.gz deploy/*
- slack-post.sh @beckyy: deploy acd-axion #$CIRCLE_BUILD_NUM
using deploy.tar.gz with deploy/deploy.sh stg
general:
artifacts:
- deploy.tar.gz
4 ϦϙδτϦ಺ͷ deploy εΫϦϓτͳͲΛ artifact
ͱͯ͠อଘ
4 Bot ΁ͷσϓϩΠίϚϯυΛ Slack ΁ϙετ

View Slide

4). Bot ͕ stg ΁σϓϩΠΛ࣮ߦ
4 CircleCI ͔Β deploy artifact Λऔಘͯ͠ίϚϯυΛ࣮ߦ

View Slide

5). ୲౰͕ prd σϓϩΠΛࢦࣔ
4 stg Ͱͷ֬ೝͱϦϦʔεͷ߹ҙΛͱ͔ͬͯΒ࣮
ߦ

View Slide

deploy.sh
# Copy job spec to deploy server
scp -pq *.nomad centos@deploy:/home/centos/axion/$ENV/nomad
# Run new job spec on deploy server
cat <cd /home/centos/axion/$ENV
nomad run nomad/axion-api.nomad
EOF

View Slide

Nomad Λ࠾༻ͨ͠ཧ༝
4 Kubernetes ͱ໎ͬͨ
4 Nomad / Kubernetes ͱ΋ʹ΄΅஌͕ࣝແ͍
ঢ়ଶ
4 طʹ Consul Λ࢖͏͜ͱ͸ܾΊ͍ͯͨ
4 ߏ੒͕γϯϓϧͰػೳ΋গͳ͍͜ͱ͔Βֶशί
ετͷ௿ͦ͏ͳ Nomad Λબ୒

View Slide

Nomad Ͱࠔͬͨͱ͜Ζ
4 Web UI (μογϡϘʔυ)͕ͳ͍
4 Docker ͷ Volume Λαϙʔτ͍ͯ͠ͳ͍
4 Rolling Upadte ͱϔϧενΣοΫ͕౷߹͞Ε
͍ͯͳ͍
2015-09 ͷϦϦʔε͔Β·ͩ 1 ೥ܦ͍ͬͯͳ͍ͷ
Ͱ࢓ํͳ͍໘΋…

View Slide

Nomad ·ͱΊ
4 খ͞ͳ࢓ࣄΛ࣮֬ʹ͜ͳͯ͘͠ΕΔϠπͰ͸͋
Δ
4 ΞϓϦͷ഑ஔ͚ͩɺͳͲͱখ͘͞ར༻͢Δʹ͸
ྑ͍
4 ͓΋ͯͳ͠ײ͸ͳ͍
4 ے͸͍͍ͷͰࠓޙͷ੒ख़ʹظ଴

View Slide

Terraform
Infrastructure as a Code

View Slide

What's Terraform ?
4 Πϯϑϥͷ࡞੒ɾมߋɾόʔδϣϯ؅ཧͷͨΊ
ͷπʔϧ
4 Πϯϑϥͷߏ੒ཁૉΛશͯίʔυͰهड़
4 ༷ʑͳϦιʔεͷ؅ཧʹରԠ
4 AWS CloudFormation ΍ OpenStack Heat
ʹ૬౰͢Δπʔϧ

View Slide

Technology Radar
4 Technology Rader 2015-01 7 Ͱ ASSESS
(ௐࠪͤΑ)
7 https://www.thoughtworks.com/radar/tools/terraform

View Slide

Resource Providers
Atlas / AWS / Azure (Service Management) / Azure (Resource
Manager)
Chef / CenturyLinkCloud / CloudFlare / CloudStack / Cobbler /
Consul
Datadog / DigitalOcean / DNSMadeEasy / DNSimple / Docker /
Dyn
GitHub / Fastly / Google Cloud / Heroku / InfluxDB / Librato
Mailgun / MySQL / OpenStack / Packet / PostgreSQL /
PowerDNS
Rundeck / StatusCake / SoftLayer / Template / Terraform / TLS
Triton / UltraDNS / VMware vCloud Director / VMware vSphere

View Slide

ఆٛϑΝΠϧ
resource "openstack_compute_instance_v2" "dock" {
region = "${var.region}"
name = "${format("acd-${var.env}-axion-dock-y%03d", count.index + 1)}"
count = "10"
image_name = "centos-7.2.1511"
flavor_name = "s2.medium"
security_groups = ["default"]
network {
name = "${var.network_name}"
access_network = true
}
user_data = "${template_cloudinit_config.dock_cloudinit.rendered}"
lifecycle {
ignore_changes = ["user_data"]
}
}

View Slide

terraform plan
$ terraform plan
+ module.axion.openstack_compute_instance_v2.dock.1
access_ip_v4: "" => ""
flavor_name: "" => "s2.medium"
image_name: "" => "centos-7.2.1511"
name: "" => "acd-stg-axion-dock-y001"
security_groups.#: "" => "1"
security_groups.3814588639: "" => "default"
user_data: "" => "4a068b7a5a43f31d9bd280a20a04823bbf4082d8"

View Slide

terraform apply
$ terraform apply -parallelism=1
module.axion.openstack_compute_instance_v2.dock.1: Creating...
access_ip_v4: "" => ""
flavor_name: "" => "s2.medium"
image_name: "" => "centos-7.2.1511"
name: "" => "acd-stg-axion-dock-y001"
security_groups.#: "" => "1"
security_groups.3814588639: "" => "default"
user_data: "" => "b0f6c60d289b151ef07f89fe543ee6edb46f0f1a"
module.axion.openstack_compute_instance_v2.dock.1: Still creating... (10s elapsed)
module.axion.openstack_compute_instance_v2.dock.1: Creation complete
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

View Slide

Axion ͱ Terraform

View Slide

؅ཧର৅ʹ͍ͯ͠ΔϦιʔε
4 openstack_compute_instance_v2
4 openstack_compute_servergroup_v2
ηΩϡϦςΟάϧʔϓ΍ωοτϫʔΫ͸
Terraform ͷ؅ཧର৅֎ͱ͠ɺCIA ͕༻ҙͨ͠
΋ͷΛ ID ΍໊લͰࢀর͢ΔΑ͏ʹ͍ͯ͠ΔɻΑͬ
ͯɺεΫϥον͔Β Terraform ͚ͩͰߏஙͰ͖
ΔΘ͚Ͱ͸ͳ͍ɻ

View Slide

Terraform ͷ࣮ߦ؀ڥ
$ cd ~/axion/terraform
$ ls
dev modules prd stg
4 ؅ཧαʔόʹ Terraform ϦϙδτϦΛνΣο
ΫΞ΢τ
4 ߋ৽࣌ʹ pull ͨ͠Γ͢Δͷ͸౎౓खಈ… !

View Slide

Terraform ͷ࣮ߦ
$ cd stg
direnv: loading .envrc
direnv: export +OS_PROJECT_NAME +OS_REGION_NAME +OS_TENANT_ID
$ terraform plan
4 ૢ࡞ର৅ͷ؀ڥͷσΟϨΫτϦʹೖΔͱ
direnv ʹΑͬͯ OpenStack ؀ڥม਺͕ઃఆ
͞ΕΔ
4 terraform ίϚϯυΛ࣮ߦ΋खಈ… !

View Slide

OpenStack ؀ڥม਺
$ cat .envrc
export OS_AUTH_URL="http://openstack.local/v2.0/"
export OS_TENANT_ID=****
export OS_TENANT_NAME=****
export OS_PROJECT_NAME=****
export OS_USERNAME=****
export OS_PASSWORD=****
export OS_REGION_NAME=****
4 OpenStack ͷ API Λୟ͘πʔϧΛ࢖͏ͱ͖ʹඞཁͳ؀ڥม
਺
4 OpenStack ͷ Dashboard ͔ΒςϯϓϨʔτΛμ΢ϯϩʔυ
Մೳ

View Slide

.tfstate ʹ͍ͭͯ
$ cd dev; ls -1
main.tf
terraform.tfstate <-- ͜Ε
terraform.tfstate.backup
4 ࠷ޙʹ apply ͨ͠ͱ͖ͷΠϯϑϥͷঢ়ଶΛه࿥
4 ద༻͢΂͖มߋͷࠩ෼Λܭࢉ͢ΔͨΊʹ ඞਢ
4 refresh Ͱ࣮Πϯϑϥ͔Β͋Δఔ౓͸࠶ߏஙͰ͖Δ
͕ɺखಈͰϦιʔεঢ়ଶͷ੔߹ੑΛऔΔඞཁ͕͋Δ
4 Terraform Λӡ༻͢Δ্Ͱͷ೰ΈͲ͜Ζ

View Slide

.tfstate ͷ؅ཧ
4 ͻͱ·ͣ terraform ͷ࣮ߦΛ؅ཧαʔόʹݶ
ఆͯ͠ɺͦͷ··ϩʔΧϧσΟεΫʹஔ͍͍ͯ
Δ
4 Πϯϑϥͷมߋ͸ස౓͕௿͍ͷͰʮӡ༻Ͱ
Χόʔʯ
4 ֎෦ετϨʔδ(s3, artifactory ͳͲ)ʹ഑ஔ
͢Δ͜ͱ΋Մೳ

View Slide

Axion ͷॳظߏங
4 ݸਓ؀ڥΛαϯυϘοΫεʹઃఆΛॻ͘
4 ͻͨ͢Β apply and destroy !
4 ͋Δఔ౓ݻ·ͬͨΒຊ൪Ϧʔδϣϯ༻ʹม਺ͩ
͚มߋͯ͠ద༻
4 dev ؀ڥͰ࠷ऴௐ੔
4 stg / prd ͷߏங͸ apply ͢Δ͚ͩͰ׬ྃ

View Slide

Axion ͷ cloud-init
4 Πϯελϯε໊͔Β hostname ઃఆ
4 ಺෦ DNS ΁ͷϨίʔυొ࿥
4 LDAP ϩάΠϯͷηοτΞοϓ
4 chef-client ࣮ߦ
4 CIA ؅ཧͷجຊ Cookbook ͷ࣮ߦ
4 ϛυϧ΢ΣΞͷΠϯετʔϧ
4 ϩʔϧݻ༗ͷηοτΞοϓ
4 ϛυϧ΢ΣΞ΍ Docker ίϯςφͷىಈ
cloud-init ͚ͩͰηοτΞοϓ͕׬݁͢ΔΑ͏ʹ͍ͯ͠Δ

View Slide

Terraform ӡ༻ͷཧ૝
4 ΞϓϦͱಉ͘͡ GitHub Ͱมߋ͔ΒσϓϩΠ
·ͰͷαΠΫϧΛճ͍ͨ͠
4 PR ʹ plan ͷ݁ՌΛࣗಈతʹϙετͯ͠มߋ
఺ΛϨϏϡʔ
4 Ϛʔδ͢Δͱ CI ͕૸ͬͯมߋΛద༻

View Slide

Terraform ·ͱΊ
4 ࠓ·ͰΠϯελϯεͷ಺෦͸ Chef ͳͲͰߏ੒؅ཧͰ͖͍ͯͨ
͕ɺΠϯελϯεͦͷ΋ͷ͸γΣϧεΫϦϓτΛୟ͍ͯ࡞੒͠
͍ͯͨ
4 Adhoc ͳૢ࡞ + ࡞ۀϩά
4 ࣗಈԽͱ͍͏จ຺Ͱ͸͞΄Ͳखؒ͸มΘΒͳ͍͔΋͠Εͳ͍
4 ΠϯϑϥΛίʔυͱͯ͠දݱ͢ΔϝϦοτ
4 ࠶ݱੑͷ୲อ
4 ҉໧஌ͷഉআ
4 ཤྺͷ؅ཧ (git log)

View Slide

Terraform ·ͱΊ
4 Terraform ͦͷ΋ͷ͸ಛʹ໰୊ͳ͘ѻ͑ͨ !
4 Πϯελϯε಺෦Λ cloud-init Ͱߏங͢Δ
ͱ͜Ζ͸େมͩͬͨ… "
4 มߋద༻લʹ plan Λ֬ೝͰ͖Δͷ͸ྑ͍
4 ੲͷ CloudFormation ͸ dry-run ͕Ͱ͖
ͳͯ͘ා͔ͬͨ

View Slide

Terraform ·ͱΊ
4 ॳظߏஙޙͷӡ༻͕ઙ͍ͷͰࠓޙͭΒ͍͜ͱ͕͋
Δ͔΋͠Εͳ͍ !
4 ϓϩάϥϛϯάͱಉ͘͡ɺಡΈ΍͍͢ίʔυΛॻ
͘৺ֻ͚͕ඞཁ
4 Axion ͸ cloud-init ·ΘΓ͕͔ͳΓԚͳ͍…
4 ׬શͳ Container ϕʔεͷੈքͰ͋Ε͹Πϯελ
ϯε؅ཧͱͯ͠ͷ Terraform ͸ෆཁʹͳΓͦ͏

View Slide

Axion Πϯϑϥͷ
શମ૾

View Slide


View Slide

ΦϨ͸Α͏΍͘
ͷ΅Γ͸͡Ίͨ
͹͔Γ͔ͩΒͳ
͜ͷ͸ͯ͠ͳ͘ԕ͍
Docker ࡔΛΑ…

View Slide

ADC2016: Axion meets HashiCorp

ADC2016: Axion meets HashiCorp

AGAWA Koji

More Decks by AGAWA Koji

Other Decks in Technology

Featured

Transcript