ADC2016: Axion meets HashiCorp

by AGAWA Koji

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

DNS Ͱ A Ϩίʔυ໰͍߹Θͤ $ dig @127.0.0.1 -p 8600 kafka.service.consul ;; QUESTION SECTION: ;kafka.service.consul. IN A ;; ANSWER SECTION: kafka.service.consul. 0 IN A 192.168.0.21 kafka.service.consul. 0 IN A 192.168.0.20 kafka.service.consul. 0 IN A 192.168.0.19 © 2016 CyberAgent, Inc. 14

Slide 15

Slide 15 text

Slide 16

Slide 16 text

DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ $ dig @127.0.0.1 -p 8600 axion-api-admin.service.consul SRV ;; QUESTION SECTION: ;axion-api-admin.service.consul. IN SRV ;; ANSWER SECTION: axion-api-admin.service.consul. 0 IN SRV 1 1 42319 acd-stg-axion-dock-y002.node.openstack-stg.consul. axion-api-admin.service.consul. 0 IN SRV 1 1 38475 acd-stg-axion-dock-y001.node.openstack-stg.consul. ;; ADDITIONAL SECTION: acd-stg-axion-dock-y002.node.openstack-stg.consul. 0 IN A 192.168.0.26 acd-stg-axion-dock-y001.node.openstack-stg.consul. 0 IN A 192.168.0.27 © 2016 CyberAgent, Inc. 16

Slide 17

Slide 17 text

Slide 18

Slide 18 text

HTTP Ͱͷ໰͍߹Θͤ $ curl -s http://localhost:8500/v1/catalog/service/kafka | jq . [ { "Node": "acd-stg-axion-kafka-y001.local", "Address": "192.168.0.19", "ServiceID": "kafka", "ServiceName": "kafka", "ServiceTags": [], "ServiceAddress": "", "ServicePort": 9092, "ServiceEnableTagOverride": false, "CreateIndex": 55508, "ModifyIndex": 151479 }, ... ] © 2016 CyberAgent, Inc. 18

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

ྫ͑͹ Kafka Λొ࿥͢Δ { "service": { "name": "kafka-broker", "port": 9092, "checks": [ { "id": "kakfa-broker", "tcp": "localhost:9092", "interval": "5s", "timeout": "1s" } ] } } 4 Kafka ͕ಈ͍͍ͯΔϗετͰ Consul ͷઃఆϑΝΠϧʹهड़ 4 tcp/9200 ΛϔϧενΣοΫ © 2016 CyberAgent, Inc. 22

Slide 23

Slide 23 text

Docker ͱͷ࿈ܞ (registrator 3) docker run --name=registrator -d --net=host --restart=unless-stopped --log-driver=journald \ -v /var/run/docker.sock:/tmp/docker.sock \ gliderlabs/registrator:v6 \ -tags registrator -resync 3600 \ -ip $(curl -s http://169.254.169.254/2009-04-04/meta-data/local-ipv4) \ consul://localhost:8500 4 Docker ίϯςφΛࣗಈతʹ Consul ΁ Service ͱͯ͠ొ࿥ͯ͘͠ΕΔπʔϧ 3 https://github.com/gliderlabs/registrator © 2016 CyberAgent, Inc. 23

Slide 24

Slide 24 text

Docker registrator docker run --name=cadvisor -d \ --restart=unless-stopped --log-driver=journald \ --dns=192.168.168.1 --privileged=true \ -p 9101:8080 \ -e "SERVICE_NAME=cadvisor" \ -e "SERVICE_CHECK_HTTP=/" \ -e "SERVICE_CHECK_INTERVAL=30s" \ google/cadvisor:v0.22.0 4 ؀ڥม਺ SERVICE_* ʹ Service ͷఆٛΛॻ͍ͯίϯςφΛىಈ 4 registrator ͕ίϯςφىಈΛݕग़ͯ͠ Consul ΁ొ࿥ 4 ίϯςφ͕ࢭ·ͬͨͱ͖͸ Consul ͔Β࡟আ © 2016 CyberAgent, Inc. 24

Slide 25

Slide 25 text

Key-Value Storage $ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1 true $ curl http://localhost:8500/v1/kv/web/key1 [ { "CreateIndex":97, "ModifyIndex":97, "Key":"web/key1", "Flags":0, "Value":"dGVzdA==" } ] © 2016 CyberAgent, Inc. 25

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

consul-template ͱ HAProxy ͷྫ global daemon maxconn {{key "service/haproxy/maxconn"}} listen mysql-slave bind *:3306{{range service "mysql-slave"}} server {{.Node}} {{.Address}}:{{.Port}}{{end}} 4 haproxy.conf ͷςϯϓϨʔτΛ४උ $ consul-template -consul localhost:8500 -template \ haproxy.ctmpl:/etc/haproxy/haproxy.conf:systemctl reload haproxy 4 consul-template Λىಈ͓ͯ͘͠ © 2016 CyberAgent, Inc. 28

Slide 29

Slide 29 text

consul-template ͱ HAProxy ͷྫ global daemon maxconn 128 listen mysql-slave bind *:3306 server dbslave01 10.0.0.1:3306 server dbslave02 10.0.0.2:3306 4 Service ͷมߋΛݕ஌ͯ͠ HAProxy ͷઃఆϑΝΠ ϧΛߋ৽ͯ͠Ϧϩʔυͯ͘͠ΕΔ © 2016 CyberAgent, Inc. 29

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Consul ͱ dnsmasq # /etc/resolv.conf nameserver 127.0.0.1 # /etc/dnsmasq.conf server=/consul/127.0.0.1#8600 4 શͯͷϗετͰ Consul ͱ dnsmasq Λ࣮ߦ 4 Consul ͷଘࡏΛҙࣝ͠ͳͯ͘΋ *.consul Λ ໊લղܾͰ͖Δ © 2016 CyberAgent, Inc. 32

Slide 33

Slide 33 text

Slide 34

Slide 34 text

ϛυϧ΢ΣΞ΁ͷ઀ଓ 4 ΞϓϦ͔Βͷϛυϧ΢ΣΞ΁ͷ઀ଓͰ Consul ͷ DNS Λࢀর // Aerospike new com.aerospike.client.async.AsyncClient( policy, "aerospike.service.consul", 3000) // Kafka Consumer kafka.consumer.bootstrap.servers = "kafka.service.consul:9092" // ࣮ࡍ͸ Configurable Ͱ͢ © 2016 CyberAgent, Inc. 34

Slide 35

Slide 35 text

ΫϥελϦϯάͱ Consul 4 Kafka, Aerospike, Cassandra, Redis Cluster ͷ Α͏ͳΫϥελΛ૊Ήϛυϧ΢ΣΞͰ઀ଓઌͱͯ͠ ࢦఆ͢Δͷ͸ Ϋϥελʔ΁ͷίϯλΫτϙΠϯτ 4 ΫϥΠΞϯτ͸ੜ͖͍ͯΔϊʔυʹ઀ଓͰ͖Ε͹ɺ ͔ͦ͜ΒΫϥελʔϝϯόʔΛऔಘ͢Δ 4 Consul ͷ DNS ͸ੜ͖͍ͯΔϝϯόʔ͚ͩฦ͢ͷ ͰɺՄ༻ੑͷͨΊʹෳ਺ϗετΛྻڍ͢Δඞཁ͕ແ ͘ͳΔ © 2016 CyberAgent, Inc. 35

Slide 36

Slide 36 text

axion-client 4 Axion ͷ API αʔόʹ઀ଓ͢ΔΫϥΠΞϯτϥΠϒϥϦ 4 τϥϯεϙʔτʹ Scala ͷ RPC ϥΠϒϥϦͰ͋Δ finagle Λ ར༻ 4 finagle ͕ΫϥΠΞϯταΠυͷϩʔυόϥϯαʔΛ͍࣋ͬͯ ΔͨΊɺͦͷϝϯόʔΛ Consul ͷαʔϏεͱಉظ͢Δίʔυ Λ࣮૷ͯ͠ར༻ 4 Finagle Resolver ͷॻ͖ํ͸ finagle-consul 4 Λࢀߟʹ Blocking Query Λར༻࣮ͯ͠૷ 4 https://github.com/kachayev/finagle-consul © 2016 CyberAgent, Inc. 36

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Prometheus 4 Pull ܕͷ؂ࢹγεςϜ Prometheus Λ࢖͍ͬͯΔ 4 ؂ࢹର৅Λऔಘ͢Δͷʹ Consul Service Λར༻ (૬ੑൈ܈ !) scrape_configs: - job_name: 'consul' consul_sd_configs: - server: '192.168.168.1:8500' services: - prometheus-pushgateway - prometheus-node-exporter - prometheus-consul-exporter - prometheus-burrow - cadvisor - axion-api-admin © 2016 CyberAgent, Inc. 38

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Technology Radar 4 Technology Radar 2016-04 6 Ͱ ASSESS(ௐࠪͤΑ) 4 ASSESS = ࠓ͙͢࠾༻͢΂͖ͱ ͸ݴΘͳ͍͕ɺ஫໨͓ͯ͘͠΂ ͖ 4 ͪͳΈʹ Kubernetes ͸ TRIAL(ࢼߦͤΑ) 4 TRIAL = ϦεΫͷ௿͍ϓϩδΣ ΫτͰ࠾༻ͯ͠ΈΔ͜ͱΛਪ঑ ͢Δ 6 https://assets.thoughtworks.com/assets/ technology-radar-apr-2016-en.pdf © 2016 CyberAgent, Inc. 44

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Job ఆٛ job "axion-api" { type = "service" # or "batch" or "system" group "api" { count = 2 task "api" { driver = "docker" config { image = "internal-repository/axion-api:$TAG" port_map { thrift = 8000 } } } } } © 2016 CyberAgent, Inc. 46

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Job ͷ࣮ߦ $ nomad run axion-api.nomad ==> Monitoring evaluation "50f80881" Evaluation triggered by job "axion-api" Allocation "73526b78" created: node "0cc2a12f", group "api" Evaluation status changed: "pending" -> "complete" ==> Evaluation "50f80881" finished with status "complete" © 2016 CyberAgent, Inc. 49

Slide 50

Slide 50 text

Job ͷঢ়ଶ $ nomad status axion-api ID = axion-api Name = axion-api Type = service Priority = 50 Datacenters = openstack-stg Status = running Periodic = false ==> Evaluations ID Priority Triggered By Status 3bc903ef 50 rolling-update complete 3b772796 50 job-register complete ==> Allocations ID Eval ID Node ID Task Group Desired Status 45f2e143 3bc903ef 7a6b1272 api run running 546a9c55 3b772796 8959a37c api run running © 2016 CyberAgent, Inc. 50

Slide 51

Slide 51 text

഑ஔ(Alloc)ͷঢ়ଶ $ nomad alloc-status 45f2e143 ID = 45f2e143 Eval ID = 3bc903ef Name = axion-api.api[1] Node ID = 7a6b1272 Job ID = axion-api Client Status = running ==> Task Resources Task: "api" CPU Memory MB Disk MB IOPS Addresses 8000 4096 20000 0 admin: 10.4.85.27:40660 thrift: 10.4.85.27:9999 ==> Task "api" is "running" Recent Events: Time Type Description 07/07/16 16:38:49 JST Started Task started by client 07/07/16 16:38:45 JST Received Task received by client © 2016 CyberAgent, Inc. 51

Slide 52

Slide 52 text

ϩάͷ֬ೝ $ nomad fs ls 45f2e143 alloc/logs Mode Size Modfied Time Name -rw-r--r-- 3.5 kB 07/07/16 16:38:52 JST api.stderr.0 -rw-r--r-- 4.6 kB 08/07/16 09:14:31 JST api.stdout.0 # nomad fs cat 45f2e143 alloc/logs/api.stdout.0 © 2016 CyberAgent, Inc. 52

Slide 53

Slide 53 text

શ Job ͷ֬ೝ $ nomad status ID Type Priority Status axion-api service 50 running axion-another-service service 50 running axion-foo-batch batch 50 running axion-foo-batch/periodic-1467924600 batch 50 dead axion-foo-batch/periodic-1467928200 batch 50 dead axion-foo-batch/periodic-1467931800 batch 50 dead axion-foo-batch/periodic-1467935400 batch 50 dead © 2016 CyberAgent, Inc. 53

Slide 54

Slide 54 text

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Slide 63

Slide 63 text

3). PR ϚʔδͰ CircleCI ͕Ϗϧυ deployment: release: branch: deployment/release commands: - tar zcf deploy.tar.gz deploy/* - slack-post.sh @beckyy: deploy acd-axion #$CIRCLE_BUILD_NUM using deploy.tar.gz with deploy/deploy.sh stg general: artifacts: - deploy.tar.gz 4 ϦϙδτϦ಺ͷ deploy εΫϦϓτͳͲΛ artifact ͱͯ͠อଘ 4 Bot ΁ͷσϓϩΠίϚϯυΛ Slack ΁ϙετ © 2016 CyberAgent, Inc. 63

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

deploy.sh # Copy job spec to deploy server scp -pq *.nomad centos@deploy:/home/centos/axion/$ENV/nomad # Run new job spec on deploy server cat <

Slide 67

Slide 67 text

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Slide 73

Slide 73 text

Resource Providers Atlas / AWS / Azure (Service Management) / Azure (Resource Manager) Chef / CenturyLinkCloud / CloudFlare / CloudStack / Cobbler / Consul Datadog / DigitalOcean / DNSMadeEasy / DNSimple / Docker / Dyn GitHub / Fastly / Google Cloud / Heroku / InfluxDB / Librato Mailgun / MySQL / OpenStack / Packet / PostgreSQL / PowerDNS Rundeck / StatusCake / SoftLayer / Template / Terraform / TLS Triton / UltraDNS / VMware vCloud Director / VMware vSphere © 2016 CyberAgent, Inc. 73

Slide 74

Slide 74 text

ఆٛϑΝΠϧ resource "openstack_compute_instance_v2" "dock" { region = "${var.region}" name = "${format("acd-${var.env}-axion-dock-y%03d", count.index + 1)}" count = "10" image_name = "centos-7.2.1511" flavor_name = "s2.medium" security_groups = ["default"] network { name = "${var.network_name}" access_network = true } user_data = "${template_cloudinit_config.dock_cloudinit.rendered}" lifecycle { ignore_changes = ["user_data"] } } © 2016 CyberAgent, Inc. 74

Slide 75

Slide 75 text

terraform plan $ terraform plan + module.axion.openstack_compute_instance_v2.dock.1 access_ip_v4: "" => "" flavor_name: "" => "s2.medium" image_name: "" => "centos-7.2.1511" name: "" => "acd-stg-axion-dock-y001" security_groups.#: "" => "1" security_groups.3814588639: "" => "default" user_data: "" => "4a068b7a5a43f31d9bd280a20a04823bbf4082d8" © 2016 CyberAgent, Inc. 75

Slide 76

Slide 76 text

terraform apply $ terraform apply -parallelism=1 module.axion.openstack_compute_instance_v2.dock.1: Creating... access_ip_v4: "" => "" flavor_name: "" => "s2.medium" image_name: "" => "centos-7.2.1511" name: "" => "acd-stg-axion-dock-y001" security_groups.#: "" => "1" security_groups.3814588639: "" => "default" user_data: "" => "b0f6c60d289b151ef07f89fe543ee6edb46f0f1a" module.axion.openstack_compute_instance_v2.dock.1: Still creating... (10s elapsed) module.axion.openstack_compute_instance_v2.dock.1: Still creating... (20s elapsed) module.axion.openstack_compute_instance_v2.dock.1: Still creating... (30s elapsed) module.axion.openstack_compute_instance_v2.dock.1: Creation complete Apply complete! Resources: 1 added, 0 changed, 0 destroyed. © 2016 CyberAgent, Inc. 76

Slide 77

Slide 77 text

Slide 78

Slide 78 text

؅ཧର৅ʹ͍ͯ͠ΔϦιʔε 4 openstack_compute_instance_v2 4 openstack_compute_servergroup_v2 ηΩϡϦςΟάϧʔϓ΍ωοτϫʔΫ͸ Terraform ͷ؅ཧର৅֎ͱ͠ɺCIA ͕༻ҙͨ͠ ΋ͷΛ ID ΍໊લͰࢀর͢ΔΑ͏ʹ͍ͯ͠ΔɻΑͬ ͯɺεΫϥον͔Β Terraform ͚ͩͰߏஙͰ͖ ΔΘ͚Ͱ͸ͳ͍ɻ © 2016 CyberAgent, Inc. 78

Slide 79

Slide 79 text

Slide 80

Slide 80 text

Slide 81

Slide 81 text

Terraform ͷ࣮ߦ $ cd stg direnv: loading .envrc direnv: export +OS_PROJECT_NAME +OS_REGION_NAME +OS_TENANT_ID $ terraform plan 4 ૢ࡞ର৅ͷ؀ڥͷσΟϨΫτϦʹೖΔͱ direnv ʹΑͬͯ OpenStack ؀ڥม਺͕ઃఆ ͞ΕΔ 4 terraform ίϚϯυΛ࣮ߦ΋खಈ… ! © 2016 CyberAgent, Inc. 81

Slide 82

Slide 82 text

OpenStack ؀ڥม਺ $ cat .envrc export OS_AUTH_URL="http://openstack.local/v2.0/" export OS_TENANT_ID=**** export OS_TENANT_NAME=**** export OS_PROJECT_NAME=**** export OS_USERNAME=**** export OS_PASSWORD=**** export OS_REGION_NAME=**** 4 OpenStack ͷ API Λୟ͘πʔϧΛ࢖͏ͱ͖ʹඞཁͳ؀ڥม ਺ 4 OpenStack ͷ Dashboard ͔ΒςϯϓϨʔτΛμ΢ϯϩʔυ Մೳ © 2016 CyberAgent, Inc. 82

Slide 83

Slide 83 text

.tfstate ʹ͍ͭͯ $ cd dev; ls -1 main.tf terraform.tfstate <-- ͜Ε terraform.tfstate.backup 4 ࠷ޙʹ apply ͨ͠ͱ͖ͷΠϯϑϥͷঢ়ଶΛه࿥ 4 ద༻͢΂͖มߋͷࠩ෼Λܭࢉ͢ΔͨΊʹ ඞਢ 4 refresh Ͱ࣮Πϯϑϥ͔Β͋Δఔ౓͸࠶ߏஙͰ͖Δ ͕ɺखಈͰϦιʔεঢ়ଶͷ੔߹ੑΛऔΔඞཁ͕͋Δ 4 Terraform Λӡ༻͢Δ্Ͱͷ೰ΈͲ͜Ζ © 2016 CyberAgent, Inc. 83

Slide 84

Slide 84 text

Slide 85

Slide 85 text

Slide 86

Slide 86 text

Axion ͷ cloud-init 4 Πϯελϯε໊͔Β hostname ઃఆ 4 ಺෦ DNS ΁ͷϨίʔυొ࿥ 4 LDAP ϩάΠϯͷηοτΞοϓ 4 chef-client ࣮ߦ 4 CIA ؅ཧͷجຊ Cookbook ͷ࣮ߦ 4 ϛυϧ΢ΣΞͷΠϯετʔϧ 4 ϩʔϧݻ༗ͷηοτΞοϓ 4 ϛυϧ΢ΣΞ΍ Docker ίϯςφͷىಈ cloud-init ͚ͩͰηοτΞοϓ͕׬݁͢ΔΑ͏ʹ͍ͯ͠Δ © 2016 CyberAgent, Inc. 86

Slide 87

Slide 87 text

Slide 88

Slide 88 text

Terraform ·ͱΊ 4 ࠓ·ͰΠϯελϯεͷ಺෦͸ Chef ͳͲͰߏ੒؅ཧͰ͖͍ͯͨ ͕ɺΠϯελϯεͦͷ΋ͷ͸γΣϧεΫϦϓτΛୟ͍ͯ࡞੒͠ ͍ͯͨ 4 Adhoc ͳૢ࡞ + ࡞ۀϩά 4 ࣗಈԽͱ͍͏จ຺Ͱ͸͞΄Ͳखؒ͸มΘΒͳ͍͔΋͠Εͳ͍ 4 ΠϯϑϥΛίʔυͱͯ͠දݱ͢ΔϝϦοτ 4 ࠶ݱੑͷ୲อ 4 ҉໧஌ͷഉআ 4 ཤྺͷ؅ཧ (git log) © 2016 CyberAgent, Inc. 88