$30 off During Our Annual Pro Sale. View Details »

ADC2016: Axion meets HashiCorp

AGAWA Koji
August 09, 2016

ADC2016: Axion meets HashiCorp

社内イベント向けの資料

AGAWA Koji

August 09, 2016
Tweet

More Decks by AGAWA Koji

Other Decks in Technology

Transcript

  1. Axion meets HashiCorp
    @atty303 / AdTech Core Development Group
    © 2016 CyberAgent, Inc. 1

    View Slide

  2. ࣗݾ঺հ
    Ѩ઒ ߞ࢘
    4 2016-02 ʙ ج൫։ൃάϧʔϓ
    © 2016 CyberAgent, Inc. 2

    View Slide

  3. Agenda
    4 Consul : Service Discovery
    4 Nomad : Container Scheduling
    4 Terraform : Infrastructure as a Code
    © 2016 CyberAgent, Inc. 3

    View Slide

  4. Axion Overview
    © 2016 CyberAgent, Inc. 4

    View Slide

  5. What's Axion ?
    4 ࣾ಺޲͚(ϚΠΫϩ)αʔϏε
    4 σʔλϕʔεͷΑ͏ͳ΋ͷ
    4 ൺֱతߴ͍ࢀরස౓ (10k ops/s ~)
    4 εϥΠυ಺ͷ IP ΍ݻ༗໊ࢺͳͲ͸μϛʔͰ͢
    © 2016 CyberAgent, Inc. 5

    View Slide

  6. Axion ։ൃʹ͓͚Δ
    ٕज़త௅ઓ
    © 2016 CyberAgent, Inc. 6

    View Slide

  7. Docker ͱ
    ޲͖߹͏!
    (AWS/GCP ʹཔΒͣ)
    © 2016 CyberAgent, Inc. 7

    View Slide

  8. Docker - Technology Radar
    4 Technology Radar
    2016-04 1 Ͱ ADOPT(࠾
    ༻ͤΑ)
    4 ADOPT = ͜ͷτϨϯυ
    ʹࠓ͙͢৐Δ΂͖
    1 https://www.thoughtworks.com/radar/tools/
    docker
    © 2016 CyberAgent, Inc. 8

    View Slide

  9. Docker ͱ޲͖߹͏ͨΊʹ…
    Nomad ΫϥελʹΞϓϦΛσϓϩΠͯ͠ɺ
    σϓϩΠͨ͠ΞϓϦͷΤϯυϙΠϯτ؅ཧʹ
    Consul ͕ඞཁͰɺ͜ΕΒͷ؀ڥߏஙΛ
    Terraform ʹ΍ΒͤΑ͏ɻ
    © 2016 CyberAgent, Inc. 9

    View Slide

  10. Consul
    Service Discovery
    © 2016 CyberAgent, Inc. 10

    View Slide

  11. What's Consul ?
    4 Service ͷ؅ཧͱݕࡧ
    4 Key-Value Storage
    4 ߴ͍Մ༻ੑΛඋ͑ͨ෼ࢄγεςϜ
    © 2016 CyberAgent, Inc. 11

    View Slide

  12. Technology Radar
    4 Technology Radar
    2016-04 2 Ͱ ADOPT(࠾
    ༻ͤΑ)
    4 ADOPT = ͜ͷτϨϯυ
    ʹࠓ͙͢৐Δ΂͖
    2 https://www.thoughtworks.com/radar/tools/
    consul
    © 2016 CyberAgent, Inc. 12

    View Slide

  13. Service Discovery
    ωοτϫʔΫΛ௨ͯ͠ػೳΛఏڙ͢Δ Service Λ
    ൃݟ͢Δػߏ
    4 Service ͷྫ: api, mysql, kafka
    4 Service ͷϝϯόʔ͸ IP:PORT ͷϦετ
    4 ϔϧενΣοΫͰϝϯόʔͷࢮ׆؂ࢹ
    4 HTTP ͱ DNS Ͱ໰͍߹Θ͕ͤͰ͖Δ
    © 2016 CyberAgent, Inc. 13

    View Slide

  14. DNS Ͱ A Ϩίʔυ໰͍߹Θͤ
    $ dig @127.0.0.1 -p 8600 kafka.service.consul
    ;; QUESTION SECTION:
    ;kafka.service.consul. IN A
    ;; ANSWER SECTION:
    kafka.service.consul. 0 IN A 192.168.0.21
    kafka.service.consul. 0 IN A 192.168.0.20
    kafka.service.consul. 0 IN A 192.168.0.19
    © 2016 CyberAgent, Inc. 14

    View Slide

  15. DNS Ͱ A Ϩίʔυ໰͍߹Θͤ
    4 Consul agent Λ DNS αʔόʹࢦఆͯ͠໰͍߹
    Θͤ
    4 *.service.consul Λղܾ͢ΔͱαʔϏεͷϝ
    ϯόʔ͕ฦͬͯ͘Δ
    4 Ϧετͷฒͼ͸ϥϯμϜ (DNS ϥ΢ϯυϩϏϯ)
    4 ϔϧενΣοΫΛ௨ͬͨੜ͖͍ͯΔϝϯόʔͩ
    ؚ͚͕·ΕΔ
    © 2016 CyberAgent, Inc. 15

    View Slide

  16. DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ
    $ dig @127.0.0.1 -p 8600 axion-api-admin.service.consul SRV
    ;; QUESTION SECTION:
    ;axion-api-admin.service.consul. IN SRV
    ;; ANSWER SECTION:
    axion-api-admin.service.consul. 0 IN SRV 1 1 42319
    acd-stg-axion-dock-y002.node.openstack-stg.consul.
    axion-api-admin.service.consul. 0 IN SRV 1 1 38475
    acd-stg-axion-dock-y001.node.openstack-stg.consul.
    ;; ADDITIONAL SECTION:
    acd-stg-axion-dock-y002.node.openstack-stg.consul. 0 IN A 192.168.0.26
    acd-stg-axion-dock-y001.node.openstack-stg.consul. 0 IN A 192.168.0.27
    © 2016 CyberAgent, Inc. 16

    View Slide

  17. DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ
    4 SRV ϨίʔυͰ͋Ε͹ϙʔτ·Ͱ෼͔Δ
    4 ϙʔτΛಈతׂΓ౰ͯ͢Δͱ͖ʹ༗༻
    4 ͨͩ͠ར༻ଆͰ SRV ͷϨκϧό͕ඞཁ
    4 طʹ SRV ΛಡΉ࣮૷͕ͳ͚Ε͹ HTTP API
    ΛಡΉ࣮૷Λॻ͍ͨ΄͏ָ͕͔ͱ
    © 2016 CyberAgent, Inc. 17

    View Slide

  18. HTTP Ͱͷ໰͍߹Θͤ
    $ curl -s http://localhost:8500/v1/catalog/service/kafka | jq .
    [
    {
    "Node": "acd-stg-axion-kafka-y001.local",
    "Address": "192.168.0.19",
    "ServiceID": "kafka",
    "ServiceName": "kafka",
    "ServiceTags": [],
    "ServiceAddress": "",
    "ServicePort": 9092,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 55508,
    "ModifyIndex": 151479
    },
    ...
    ]
    © 2016 CyberAgent, Inc. 18

    View Slide

  19. HTTP Ͱͷ໰͍߹Θͤ
    4 ී௨ʹ JSON Ͱฦͬͯ͘Δ
    4 DNS ͱҧͬͯࢮΜͰΔϝϯόʔΛ஌Δ͜ͱ͕
    Ͱ͖Δ
    © 2016 CyberAgent, Inc. 19

    View Slide

  20. HTTP Blocking Query
    4 HTTP API Ͱఏڙ͞Ε͍ͯΔσʔλͷมߋΛ؂
    ࢹͰ͖Δ
    4 DNS ΍ HTTP ΛϙʔϦϯάͤͣʹଈ࠲ʹϝϯ
    όʔมߋΛݕ஌Ͱ͖Δ
    4 ͍ΘΏΔ Server Push (HTTP Long Polling)
    4 Non-Blocking ͳ HTTP Client Ͱ࢖͏͜ͱ
    © 2016 CyberAgent, Inc. 20

    View Slide

  21. αʔϏεͷొ࿥
    4 ઃఆϑΝΠϧͰ੩తʹొ࿥
    4 ϗετʹΠϯετʔϧ͞Ε͍ͯΔϛυϧ΢
    ΣΞͳͲ
    4 HTTP API Ͱಈతʹొ࿥
    4 ಈతʹσϓϩΠ͢ΔΞϓϦέʔγϣϯͳͲ
    4 Docker registrator ΍ Nomad ͱ౷߹
    © 2016 CyberAgent, Inc. 21

    View Slide

  22. ྫ͑͹ Kafka Λొ࿥͢Δ
    { "service": {
    "name": "kafka-broker",
    "port": 9092,
    "checks": [ {
    "id": "kakfa-broker",
    "tcp": "localhost:9092",
    "interval": "5s",
    "timeout": "1s"
    } ]
    } }
    4 Kafka ͕ಈ͍͍ͯΔϗετͰ Consul ͷઃఆϑΝΠϧʹهड़
    4 tcp/9200 ΛϔϧενΣοΫ
    © 2016 CyberAgent, Inc. 22

    View Slide

  23. Docker ͱͷ࿈ܞ (registrator 3)
    docker run --name=registrator -d --net=host
    --restart=unless-stopped --log-driver=journald \
    -v /var/run/docker.sock:/tmp/docker.sock \
    gliderlabs/registrator:v6 \
    -tags registrator -resync 3600 \
    -ip $(curl -s http://169.254.169.254/2009-04-04/meta-data/local-ipv4) \
    consul://localhost:8500
    4 Docker ίϯςφΛࣗಈతʹ Consul ΁
    Service ͱͯ͠ొ࿥ͯ͘͠ΕΔπʔϧ
    3 https://github.com/gliderlabs/registrator
    © 2016 CyberAgent, Inc. 23

    View Slide

  24. Docker registrator
    docker run --name=cadvisor -d \
    --restart=unless-stopped --log-driver=journald \
    --dns=192.168.168.1 --privileged=true \
    -p 9101:8080 \
    -e "SERVICE_NAME=cadvisor" \
    -e "SERVICE_CHECK_HTTP=/" \
    -e "SERVICE_CHECK_INTERVAL=30s" \
    google/cadvisor:v0.22.0
    4 ؀ڥม਺ SERVICE_* ʹ Service ͷఆٛΛॻ͍ͯίϯςφΛىಈ
    4 registrator ͕ίϯςφىಈΛݕग़ͯ͠ Consul ΁ొ࿥
    4 ίϯςφ͕ࢭ·ͬͨͱ͖͸ Consul ͔Β࡟আ
    © 2016 CyberAgent, Inc. 24

    View Slide

  25. Key-Value Storage
    $ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1
    true
    $ curl http://localhost:8500/v1/kv/web/key1
    [ {
    "CreateIndex":97,
    "ModifyIndex":97,
    "Key":"web/key1",
    "Flags":0,
    "Value":"dGVzdA=="
    } ]
    © 2016 CyberAgent, Inc. 25

    View Slide

  26. Key-Value Storage
    4 ֊૚Խ͞ΕͨΩʔͷ GET/PUT/DELETE
    4 Blocking Query Ͱ watch Ͱ͖Δ͠ɺ CAS ΋
    ͋Δ
    4 εέʔϥϏϦςΟ͸ແ͍ͷͰ஫ҙ
    4 ϗετ΍ΞϓϦͷઃఆͳͲࢀরස౓ͷ௿͍
    σʔλͷΈ֨ೲ
    © 2016 CyberAgent, Inc. 26

    View Slide

  27. consul-template
    4 Service ΍ KVS ͷมߋΛݕ஌ͯ͠ςϯϓϨʔ
    τϑΝΠϧΛߋ৽ɺίϚϯυΛ࣮ߦͯ͘͠ΕΔ
    πʔϧ
    4 Consul ʹରԠ͍ͯ͠ͳ͍πʔϧ/ϛυϧ΢ΣΞ
    ͱ Consul ͷΪϟοϓΛຒΊΔ
    © 2016 CyberAgent, Inc. 27

    View Slide

  28. consul-template ͱ HAProxy ͷྫ
    global
    daemon
    maxconn {{key "service/haproxy/maxconn"}}
    listen mysql-slave
    bind *:3306{{range service "mysql-slave"}}
    server {{.Node}} {{.Address}}:{{.Port}}{{end}}
    4 haproxy.conf ͷςϯϓϨʔτΛ४උ
    $ consul-template -consul localhost:8500 -template \
    haproxy.ctmpl:/etc/haproxy/haproxy.conf:systemctl reload haproxy
    4 consul-template Λىಈ͓ͯ͘͠
    © 2016 CyberAgent, Inc. 28

    View Slide

  29. consul-template ͱ HAProxy ͷྫ
    global
    daemon
    maxconn 128
    listen mysql-slave
    bind *:3306
    server dbslave01 10.0.0.1:3306
    server dbslave02 10.0.0.2:3306
    4 Service ͷมߋΛݕ஌ͯ͠ HAProxy ͷઃఆϑΝΠ
    ϧΛߋ৽ͯ͠Ϧϩʔυͯ͘͠ΕΔ
    © 2016 CyberAgent, Inc. 29

    View Slide

  30. Consul ͷΞʔΩςΫνϟ
    4 Server(Master) ઐ༻ϗετΛ 3 or 5 ୆
    4 ଞͷશͯͷϗετͰ Agent(Client) Λ࣮ߦ
    © 2016 CyberAgent, Inc. 30

    View Slide

  31. Axion ͱ Consul
    © 2016 CyberAgent, Inc. 31

    View Slide

  32. Consul ͱ dnsmasq
    # /etc/resolv.conf
    nameserver 127.0.0.1
    # /etc/dnsmasq.conf
    server=/consul/127.0.0.1#8600
    4 શͯͷϗετͰ Consul ͱ dnsmasq Λ࣮ߦ
    4 Consul ͷଘࡏΛҙࣝ͠ͳͯ͘΋ *.consul Λ
    ໊લղܾͰ͖Δ
    © 2016 CyberAgent, Inc. 32

    View Slide

  33. Service
    4 શͯͷωοτϫʔΫαʔϏεΛ Consul ʹొ࿥
    © 2016 CyberAgent, Inc. 33

    View Slide

  34. ϛυϧ΢ΣΞ΁ͷ઀ଓ
    4 ΞϓϦ͔Βͷϛυϧ΢ΣΞ΁ͷ઀ଓͰ Consul
    ͷ DNS Λࢀর
    // Aerospike
    new com.aerospike.client.async.AsyncClient(
    policy, "aerospike.service.consul", 3000)
    // Kafka Consumer
    kafka.consumer.bootstrap.servers = "kafka.service.consul:9092"
    // ࣮ࡍ͸ Configurable Ͱ͢
    © 2016 CyberAgent, Inc. 34

    View Slide

  35. ΫϥελϦϯάͱ Consul
    4 Kafka, Aerospike, Cassandra, Redis Cluster ͷ
    Α͏ͳΫϥελΛ૊Ήϛυϧ΢ΣΞͰ઀ଓઌͱͯ͠
    ࢦఆ͢Δͷ͸ Ϋϥελʔ΁ͷίϯλΫτϙΠϯτ
    4 ΫϥΠΞϯτ͸ੜ͖͍ͯΔϊʔυʹ઀ଓͰ͖Ε͹ɺ
    ͔ͦ͜ΒΫϥελʔϝϯόʔΛऔಘ͢Δ
    4 Consul ͷ DNS ͸ੜ͖͍ͯΔϝϯόʔ͚ͩฦ͢ͷ
    ͰɺՄ༻ੑͷͨΊʹෳ਺ϗετΛྻڍ͢Δඞཁ͕ແ
    ͘ͳΔ
    © 2016 CyberAgent, Inc. 35

    View Slide

  36. axion-client
    4 Axion ͷ API αʔόʹ઀ଓ͢ΔΫϥΠΞϯτϥΠϒϥϦ
    4 τϥϯεϙʔτʹ Scala ͷ RPC ϥΠϒϥϦͰ͋Δ finagle Λ
    ར༻
    4 finagle ͕ΫϥΠΞϯταΠυͷϩʔυόϥϯαʔΛ͍࣋ͬͯ
    ΔͨΊɺͦͷϝϯόʔΛ Consul ͷαʔϏεͱಉظ͢Δίʔυ
    Λ࣮૷ͯ͠ར༻
    4 Finagle Resolver ͷॻ͖ํ͸ finagle-consul 4 Λࢀߟʹ
    Blocking Query Λར༻࣮ͯ͠૷
    4 https://github.com/kachayev/finagle-consul
    © 2016 CyberAgent, Inc. 36

    View Slide

  37. axcsync
    4 Consul ͷ Service ͱ BIG-IP ͷ Pool
    Member Λಉظ
    4 consul-template + axc 5 + python εΫϦϓ
    τͰ࣮૷
    5 https://adtech.cyberagent.io/techblog/archives/59
    © 2016 CyberAgent, Inc. 37

    View Slide

  38. Prometheus
    4 Pull ܕͷ؂ࢹγεςϜ Prometheus Λ࢖͍ͬͯΔ
    4 ؂ࢹର৅Λऔಘ͢Δͷʹ Consul Service Λར༻ (૬ੑൈ܈ !)
    scrape_configs:
    - job_name: 'consul'
    consul_sd_configs:
    - server: '192.168.168.1:8500'
    services:
    - prometheus-pushgateway
    - prometheus-node-exporter
    - prometheus-consul-exporter
    - prometheus-burrow
    - cadvisor
    - axion-api-admin
    © 2016 CyberAgent, Inc. 38

    View Slide

  39. Consul Λ࠾༻ͨ͠ཧ༝
    Docker ίϯςφͷಈత഑ஔ͢Δ৔߹ɺαʔϏε
    ͷ഑ஔΛܾΊଧͪͰ͖ͳ͍ͷͰ Service
    Discovery ػߏ͕ඞਢʹͳΔ
    4 ZooKeeper ʹ͸ͳ͍ DNS ͱ HTTP ʹΑΔ؆
    ୯ͳΞΫηε
    4 etcd ʹ͸ͳ͍ Service ͷϔϧενΣοΫ
    4 HashiCorp ͱ͍͏ϒϥϯυ
    © 2016 CyberAgent, Inc. 39

    View Slide

  40. Consul ·ͱΊ
    4 Service Discovery ͕͋Δ͜ͱΛલఏʹγες
    ϜΛ૊Ή͜ͱͰίϯϙʔωϯτͷґଘؔ܎Λ៉
    ྷʹදݱͰ͖Δ
    4 αʔϏεͷಈతͳมԽ΁ͷରԠ͕ඇৗʹߦ͍΍
    ͍͢
    4 Docker ʹؔ܎ͳ͘༗༻
    © 2016 CyberAgent, Inc. 40

    View Slide

  41. Consul ·ͱΊ
    4 Consul ͸ϋϚΓͲ͜Ζ͕ͳ͘ૉ௚ʹ࢖͑ͨ !
    4 ඞཁ࠷খݶͷγϯϓϧͳػೳηοτͰ͋Γͳ͕
    ΒԠ༻ੑ͕ߴ͍
    4 ಋೖͷϦεΫ͕ແ͍Α͏ʹࢥ͑Δ
    4 Kubernetes ͳͲ SD Λ͍࣋ͬͯΔ΋ͷΛ࢖͏
    ৔߹͸ඞཁͳ͍͔΋
    © 2016 CyberAgent, Inc. 41

    View Slide

  42. Nomad
    Container Scheduling
    © 2016 CyberAgent, Inc. 42

    View Slide

  43. What's Nomad ?
    4 Docker Ϋϥελͷ؅ཧπʔϧ
    4 AWS ECS, Kubernetes, Docker Swarm,
    Mesos ͱಉ͡໾ׂ
    4 Ϋϥελʹίϯςφ(ΞϓϦ)Λεέδϡʔϧ
    (σϓϩΠ)͢Δ
    4 Consul ͱͷΠϯςάϨʔγϣϯ
    © 2016 CyberAgent, Inc. 43

    View Slide

  44. Technology Radar
    4 Technology Radar 2016-04 6 Ͱ
    ASSESS(ௐࠪͤΑ)
    4 ASSESS = ࠓ͙͢࠾༻͢΂͖ͱ
    ͸ݴΘͳ͍͕ɺ஫໨͓ͯ͘͠΂
    ͖
    4 ͪͳΈʹ Kubernetes ͸
    TRIAL(ࢼߦͤΑ)
    4 TRIAL = ϦεΫͷ௿͍ϓϩδΣ
    ΫτͰ࠾༻ͯ͠ΈΔ͜ͱΛਪ঑
    ͢Δ
    6 https://assets.thoughtworks.com/assets/
    technology-radar-apr-2016-en.pdf
    © 2016 CyberAgent, Inc. 44

    View Slide

  45. Nomad ͷΞʔΩςΫνϟ
    4 Server ઐ༻ϗετΛ 3 or 5 ୆
    4 Docker Ϋϥελʹ͢ΔϗετͰ Client Λ࣮ߦ
    © 2016 CyberAgent, Inc. 45

    View Slide

  46. Job ఆٛ
    job "axion-api" {
    type = "service" # or "batch" or "system"
    group "api" {
    count = 2
    task "api" {
    driver = "docker"
    config {
    image = "internal-repository/axion-api:$TAG"
    port_map {
    thrift = 8000
    }
    }
    }
    }
    }
    © 2016 CyberAgent, Inc. 46

    View Slide

  47. ϦιʔεׂΓ౰ͯ
    task "api" {
    resources {
    cpu = 8000 # MHz
    memory = 4096 # MB
    disk = 20000 # MB
    network {
    mbits = 100 # MBits
    port "thrift" {
    static = 8888
    }
    }
    }
    }
    © 2016 CyberAgent, Inc. 47

    View Slide

  48. Consul Service ొ࿥
    task "api" {
    service {
    name = "axion-api-thrift"
    port = "thrift"
    check {
    type = "tcp"
    interval = "2s"
    timeout = "1s"
    }
    }
    }
    © 2016 CyberAgent, Inc. 48

    View Slide

  49. Job ͷ࣮ߦ
    $ nomad run axion-api.nomad
    ==> Monitoring evaluation "50f80881"
    Evaluation triggered by job "axion-api"
    Allocation "73526b78" created: node "0cc2a12f", group "api"
    Evaluation status changed: "pending" -> "complete"
    ==> Evaluation "50f80881" finished with status "complete"
    © 2016 CyberAgent, Inc. 49

    View Slide

  50. Job ͷঢ়ଶ
    $ nomad status axion-api
    ID = axion-api
    Name = axion-api
    Type = service
    Priority = 50
    Datacenters = openstack-stg
    Status = running
    Periodic = false
    ==> Evaluations
    ID Priority Triggered By Status
    3bc903ef 50 rolling-update complete
    3b772796 50 job-register complete
    ==> Allocations
    ID Eval ID Node ID Task Group Desired Status
    45f2e143 3bc903ef 7a6b1272 api run running
    546a9c55 3b772796 8959a37c api run running
    © 2016 CyberAgent, Inc. 50

    View Slide

  51. ഑ஔ(Alloc)ͷঢ়ଶ
    $ nomad alloc-status 45f2e143
    ID = 45f2e143
    Eval ID = 3bc903ef
    Name = axion-api.api[1]
    Node ID = 7a6b1272
    Job ID = axion-api
    Client Status = running
    ==> Task Resources
    Task: "api"
    CPU Memory MB Disk MB IOPS Addresses
    8000 4096 20000 0 admin: 10.4.85.27:40660
    thrift: 10.4.85.27:9999
    ==> Task "api" is "running"
    Recent Events:
    Time Type Description
    07/07/16 16:38:49 JST Started Task started by client
    07/07/16 16:38:45 JST Received Task received by client
    © 2016 CyberAgent, Inc. 51

    View Slide

  52. ϩάͷ֬ೝ
    $ nomad fs ls 45f2e143 alloc/logs
    Mode Size Modfied Time Name
    -rw-r--r-- 3.5 kB 07/07/16 16:38:52 JST api.stderr.0
    -rw-r--r-- 4.6 kB 08/07/16 09:14:31 JST api.stdout.0
    # nomad fs cat 45f2e143 alloc/logs/api.stdout.0
    © 2016 CyberAgent, Inc. 52

    View Slide

  53. શ Job ͷ֬ೝ
    $ nomad status
    ID Type Priority Status
    axion-api service 50 running
    axion-another-service service 50 running
    axion-foo-batch batch 50 running
    axion-foo-batch/periodic-1467924600 batch 50 dead
    axion-foo-batch/periodic-1467928200 batch 50 dead
    axion-foo-batch/periodic-1467931800 batch 50 dead
    axion-foo-batch/periodic-1467935400 batch 50 dead
    © 2016 CyberAgent, Inc. 53

    View Slide

  54. Axion ͱ Nomad
    © 2016 CyberAgent, Inc. 54

    View Slide

  55. λεΫͷ഑ஔ
    4 Nomad Ϋϥελ͸ 1 ͚ͭͩ
    4 ͲͷϊʔυʹλεΫ͕഑ஔ͞ΕΔ͔͸෼͔Βͳ
    ͍
    4 ύϑΥʔϚϯεʹහײͳ api ͸ϊʔυΛ઎༗͠
    ͍ͨ
    4 ֤छ worker ΍όον͸ద౰ʹۭ͍͍ͯΔϊ
    ʔυͰಈ͍ͯ͘ΕΕ͹͍͍
    © 2016 CyberAgent, Inc. 55

    View Slide

  56. Job constraint
    constraint {
    attribute = "${meta.role}"
    value = "api"
    }
    4 Nomad ͷϊʔυʹઃఆͰ͖Δ meta ม਺Λ΋
    ͱʹλεΫ഑ஔΛ੍ݶ
    © 2016 CyberAgent, Inc. 56

    View Slide

  57. Nomad ͷ meta ઃఆ
    4 Consul ͷ KVS ͰϊʔυʹରԠ͢ΔΩʔʹ஋Λઃఆ͢Δͱ consul-template ͕൓Ԡ
    ͯ͠ Nomad ͷઃఆΛม͑Δ
    © 2016 CyberAgent, Inc. 57

    View Slide

  58. Nomad Λ࢖ͬͨ
    Axion ͷσϓϩΠϑϩʔ
    © 2016 CyberAgent, Inc. 58

    View Slide

  59. 1). Bot ʹϦϦʔε४උΛґཔ
    © 2016 CyberAgent, Inc. 59

    View Slide

  60. 2). ϦϦʔε PR ͕࡞੒͞ΕΔ
    4 master -> deployment/release ΁ͷϚʔδ
    © 2016 CyberAgent, Inc. 60

    View Slide

  61. master ʹϚʔδ͞Εͨ PR Λऩूͯ͠Ϧετ
    Խ
    © 2016 CyberAgent, Inc. 61

    View Slide

  62. ֤ PR ͷ ## Release Notes ηΫγϣϯΛ
    ू໿
    © 2016 CyberAgent, Inc. 62

    View Slide

  63. 3). PR ϚʔδͰ CircleCI ͕Ϗϧυ
    deployment:
    release:
    branch: deployment/release
    commands:
    - tar zcf deploy.tar.gz deploy/*
    - slack-post.sh @beckyy: deploy acd-axion #$CIRCLE_BUILD_NUM
    using deploy.tar.gz with deploy/deploy.sh stg
    general:
    artifacts:
    - deploy.tar.gz
    4 ϦϙδτϦ಺ͷ deploy εΫϦϓτͳͲΛ artifact
    ͱͯ͠อଘ
    4 Bot ΁ͷσϓϩΠίϚϯυΛ Slack ΁ϙετ
    © 2016 CyberAgent, Inc. 63

    View Slide

  64. 4). Bot ͕ stg ΁σϓϩΠΛ࣮ߦ
    4 CircleCI ͔Β deploy artifact Λऔಘͯ͠ίϚϯυΛ࣮ߦ
    © 2016 CyberAgent, Inc. 64

    View Slide

  65. 5). ୲౰͕ prd σϓϩΠΛࢦࣔ
    4 stg Ͱͷ֬ೝͱϦϦʔεͷ߹ҙΛͱ͔ͬͯΒ࣮
    ߦ
    © 2016 CyberAgent, Inc. 65

    View Slide

  66. deploy.sh
    # Copy job spec to deploy server
    scp -pq *.nomad centos@deploy:/home/centos/axion/$ENV/nomad
    # Run new job spec on deploy server
    cat <cd /home/centos/axion/$ENV
    nomad run nomad/axion-api.nomad
    EOF
    © 2016 CyberAgent, Inc. 66

    View Slide

  67. Nomad Λ࠾༻ͨ͠ཧ༝
    4 Kubernetes ͱ໎ͬͨ
    4 Nomad / Kubernetes ͱ΋ʹ΄΅஌͕ࣝແ͍
    ঢ়ଶ
    4 طʹ Consul Λ࢖͏͜ͱ͸ܾΊ͍ͯͨ
    4 ߏ੒͕γϯϓϧͰػೳ΋গͳ͍͜ͱ͔Βֶशί
    ετͷ௿ͦ͏ͳ Nomad Λબ୒
    © 2016 CyberAgent, Inc. 67

    View Slide

  68. Nomad Ͱࠔͬͨͱ͜Ζ
    4 Web UI (μογϡϘʔυ)͕ͳ͍
    4 Docker ͷ Volume Λαϙʔτ͍ͯ͠ͳ͍
    4 Rolling Upadte ͱϔϧενΣοΫ͕౷߹͞Ε
    ͍ͯͳ͍
    2015-09 ͷϦϦʔε͔Β·ͩ 1 ೥ܦ͍ͬͯͳ͍ͷ
    Ͱ࢓ํͳ͍໘΋…
    © 2016 CyberAgent, Inc. 68

    View Slide

  69. Nomad ·ͱΊ
    4 খ͞ͳ࢓ࣄΛ࣮֬ʹ͜ͳͯ͘͠ΕΔϠπͰ͸͋
    Δ
    4 ΞϓϦͷ഑ஔ͚ͩɺͳͲͱখ͘͞ར༻͢Δʹ͸
    ྑ͍
    4 ͓΋ͯͳ͠ײ͸ͳ͍
    4 ے͸͍͍ͷͰࠓޙͷ੒ख़ʹظ଴
    © 2016 CyberAgent, Inc. 69

    View Slide

  70. Terraform
    Infrastructure as a Code
    © 2016 CyberAgent, Inc. 70

    View Slide

  71. What's Terraform ?
    4 Πϯϑϥͷ࡞੒ɾมߋɾόʔδϣϯ؅ཧͷͨΊ
    ͷπʔϧ
    4 Πϯϑϥͷߏ੒ཁૉΛશͯίʔυͰهड़
    4 ༷ʑͳϦιʔεͷ؅ཧʹରԠ
    4 AWS CloudFormation ΍ OpenStack Heat
    ʹ૬౰͢Δπʔϧ
    © 2016 CyberAgent, Inc. 71

    View Slide

  72. Technology Radar
    4 Technology Rader 2015-01 7 Ͱ ASSESS
    (ௐࠪͤΑ)
    7 https://www.thoughtworks.com/radar/tools/terraform
    © 2016 CyberAgent, Inc. 72

    View Slide

  73. Resource Providers
    Atlas / AWS / Azure (Service Management) / Azure (Resource
    Manager)
    Chef / CenturyLinkCloud / CloudFlare / CloudStack / Cobbler /
    Consul
    Datadog / DigitalOcean / DNSMadeEasy / DNSimple / Docker /
    Dyn
    GitHub / Fastly / Google Cloud / Heroku / InfluxDB / Librato
    Mailgun / MySQL / OpenStack / Packet / PostgreSQL /
    PowerDNS
    Rundeck / StatusCake / SoftLayer / Template / Terraform / TLS
    Triton / UltraDNS / VMware vCloud Director / VMware vSphere
    © 2016 CyberAgent, Inc. 73

    View Slide

  74. ఆٛϑΝΠϧ
    resource "openstack_compute_instance_v2" "dock" {
    region = "${var.region}"
    name = "${format("acd-${var.env}-axion-dock-y%03d", count.index + 1)}"
    count = "10"
    image_name = "centos-7.2.1511"
    flavor_name = "s2.medium"
    security_groups = ["default"]
    network {
    name = "${var.network_name}"
    access_network = true
    }
    user_data = "${template_cloudinit_config.dock_cloudinit.rendered}"
    lifecycle {
    ignore_changes = ["user_data"]
    }
    }
    © 2016 CyberAgent, Inc. 74

    View Slide

  75. terraform plan
    $ terraform plan
    + module.axion.openstack_compute_instance_v2.dock.1
    access_ip_v4: "" => ""
    flavor_name: "" => "s2.medium"
    image_name: "" => "centos-7.2.1511"
    name: "" => "acd-stg-axion-dock-y001"
    security_groups.#: "" => "1"
    security_groups.3814588639: "" => "default"
    user_data: "" => "4a068b7a5a43f31d9bd280a20a04823bbf4082d8"
    © 2016 CyberAgent, Inc. 75

    View Slide

  76. terraform apply
    $ terraform apply -parallelism=1
    module.axion.openstack_compute_instance_v2.dock.1: Creating...
    access_ip_v4: "" => ""
    flavor_name: "" => "s2.medium"
    image_name: "" => "centos-7.2.1511"
    name: "" => "acd-stg-axion-dock-y001"
    security_groups.#: "" => "1"
    security_groups.3814588639: "" => "default"
    user_data: "" => "b0f6c60d289b151ef07f89fe543ee6edb46f0f1a"
    module.axion.openstack_compute_instance_v2.dock.1: Still creating... (10s elapsed)
    module.axion.openstack_compute_instance_v2.dock.1: Still creating... (20s elapsed)
    module.axion.openstack_compute_instance_v2.dock.1: Still creating... (30s elapsed)
    module.axion.openstack_compute_instance_v2.dock.1: Creation complete
    Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
    © 2016 CyberAgent, Inc. 76

    View Slide

  77. Axion ͱ Terraform
    © 2016 CyberAgent, Inc. 77

    View Slide

  78. ؅ཧର৅ʹ͍ͯ͠ΔϦιʔε
    4 openstack_compute_instance_v2
    4 openstack_compute_servergroup_v2
    ηΩϡϦςΟάϧʔϓ΍ωοτϫʔΫ͸
    Terraform ͷ؅ཧର৅֎ͱ͠ɺCIA ͕༻ҙͨ͠
    ΋ͷΛ ID ΍໊લͰࢀর͢ΔΑ͏ʹ͍ͯ͠ΔɻΑͬ
    ͯɺεΫϥον͔Β Terraform ͚ͩͰߏஙͰ͖
    ΔΘ͚Ͱ͸ͳ͍ɻ
    © 2016 CyberAgent, Inc. 78

    View Slide

  79. Terraform ઃఆͷϨΠΞ΢τ
    |-- modules
    | `-- axion
    | |-- main.tf | ڞ௨Ϧιʔεఆٛ
    | |-- dock.tf | dock ϩʔϧఆٛ
    | |-- council.tf | council ϩʔϧఆٛ
    | `-- variables.tf | Ϟδϡʔϧม਺ఆٛ
    |-- dev
    | |-- .envrc | OpenStack ؀ڥม਺ (not in repository)
    | `-- main.tf | dev ؀ڥઃఆ
    |-- stg
    | |-- .envrc
    | `-- main.tf | stg ؀ڥઃఆ
    `-- prd
    |-- .envrc
    `-- main.tf | prd ؀ڥઃఆ
    ࢀߟ: https://atlas.hashicorp.com/help/intro/use-cases/multiple-
    environments
    © 2016 CyberAgent, Inc. 79

    View Slide

  80. Terraform ͷ࣮ߦ؀ڥ
    $ cd ~/axion/terraform
    $ ls
    dev modules prd stg
    4 ؅ཧαʔόʹ Terraform ϦϙδτϦΛνΣο
    ΫΞ΢τ
    4 ߋ৽࣌ʹ pull ͨ͠Γ͢Δͷ͸౎౓खಈ… !
    © 2016 CyberAgent, Inc. 80

    View Slide

  81. Terraform ͷ࣮ߦ
    $ cd stg
    direnv: loading .envrc
    direnv: export +OS_PROJECT_NAME +OS_REGION_NAME +OS_TENANT_ID
    $ terraform plan
    4 ૢ࡞ର৅ͷ؀ڥͷσΟϨΫτϦʹೖΔͱ
    direnv ʹΑͬͯ OpenStack ؀ڥม਺͕ઃఆ
    ͞ΕΔ
    4 terraform ίϚϯυΛ࣮ߦ΋खಈ… !
    © 2016 CyberAgent, Inc. 81

    View Slide

  82. OpenStack ؀ڥม਺
    $ cat .envrc
    export OS_AUTH_URL="http://openstack.local/v2.0/"
    export OS_TENANT_ID=****
    export OS_TENANT_NAME=****
    export OS_PROJECT_NAME=****
    export OS_USERNAME=****
    export OS_PASSWORD=****
    export OS_REGION_NAME=****
    4 OpenStack ͷ API Λୟ͘πʔϧΛ࢖͏ͱ͖ʹඞཁͳ؀ڥม

    4 OpenStack ͷ Dashboard ͔ΒςϯϓϨʔτΛμ΢ϯϩʔυ
    Մೳ
    © 2016 CyberAgent, Inc. 82

    View Slide

  83. .tfstate ʹ͍ͭͯ
    $ cd dev; ls -1
    main.tf
    terraform.tfstate <-- ͜Ε
    terraform.tfstate.backup
    4 ࠷ޙʹ apply ͨ͠ͱ͖ͷΠϯϑϥͷঢ়ଶΛه࿥
    4 ద༻͢΂͖มߋͷࠩ෼Λܭࢉ͢ΔͨΊʹ ඞਢ
    4 refresh Ͱ࣮Πϯϑϥ͔Β͋Δఔ౓͸࠶ߏஙͰ͖Δ
    ͕ɺखಈͰϦιʔεঢ়ଶͷ੔߹ੑΛऔΔඞཁ͕͋Δ
    4 Terraform Λӡ༻͢Δ্Ͱͷ೰ΈͲ͜Ζ
    © 2016 CyberAgent, Inc. 83

    View Slide

  84. .tfstate ͷ؅ཧ
    4 ͻͱ·ͣ terraform ͷ࣮ߦΛ؅ཧαʔόʹݶ
    ఆͯ͠ɺͦͷ··ϩʔΧϧσΟεΫʹஔ͍͍ͯ
    Δ
    4 Πϯϑϥͷมߋ͸ස౓͕௿͍ͷͰʮӡ༻Ͱ
    Χόʔʯ
    4 ֎෦ετϨʔδ(s3, artifactory ͳͲ)ʹ഑ஔ
    ͢Δ͜ͱ΋Մೳ
    © 2016 CyberAgent, Inc. 84

    View Slide

  85. Axion ͷॳظߏங
    4 ݸਓ؀ڥΛαϯυϘοΫεʹઃఆΛॻ͘
    4 ͻͨ͢Β apply and destroy !
    4 ͋Δఔ౓ݻ·ͬͨΒຊ൪Ϧʔδϣϯ༻ʹม਺ͩ
    ͚มߋͯ͠ద༻
    4 dev ؀ڥͰ࠷ऴௐ੔
    4 stg / prd ͷߏங͸ apply ͢Δ͚ͩͰ׬ྃ
    © 2016 CyberAgent, Inc. 85

    View Slide

  86. Axion ͷ cloud-init
    4 Πϯελϯε໊͔Β hostname ઃఆ
    4 ಺෦ DNS ΁ͷϨίʔυొ࿥
    4 LDAP ϩάΠϯͷηοτΞοϓ
    4 chef-client ࣮ߦ
    4 CIA ؅ཧͷجຊ Cookbook ͷ࣮ߦ
    4 ϛυϧ΢ΣΞͷΠϯετʔϧ
    4 ϩʔϧݻ༗ͷηοτΞοϓ
    4 ϛυϧ΢ΣΞ΍ Docker ίϯςφͷىಈ
    cloud-init ͚ͩͰηοτΞοϓ͕׬݁͢ΔΑ͏ʹ͍ͯ͠Δ
    © 2016 CyberAgent, Inc. 86

    View Slide

  87. Terraform ӡ༻ͷཧ૝
    4 ΞϓϦͱಉ͘͡ GitHub Ͱมߋ͔ΒσϓϩΠ
    ·ͰͷαΠΫϧΛճ͍ͨ͠
    4 PR ʹ plan ͷ݁ՌΛࣗಈతʹϙετͯ͠มߋ
    ఺ΛϨϏϡʔ
    4 Ϛʔδ͢Δͱ CI ͕૸ͬͯมߋΛద༻
    © 2016 CyberAgent, Inc. 87

    View Slide

  88. Terraform ·ͱΊ
    4 ࠓ·ͰΠϯελϯεͷ಺෦͸ Chef ͳͲͰߏ੒؅ཧͰ͖͍ͯͨ
    ͕ɺΠϯελϯεͦͷ΋ͷ͸γΣϧεΫϦϓτΛୟ͍ͯ࡞੒͠
    ͍ͯͨ
    4 Adhoc ͳૢ࡞ + ࡞ۀϩά
    4 ࣗಈԽͱ͍͏จ຺Ͱ͸͞΄Ͳखؒ͸มΘΒͳ͍͔΋͠Εͳ͍
    4 ΠϯϑϥΛίʔυͱͯ͠දݱ͢ΔϝϦοτ
    4 ࠶ݱੑͷ୲อ
    4 ҉໧஌ͷഉআ
    4 ཤྺͷ؅ཧ (git log)
    © 2016 CyberAgent, Inc. 88

    View Slide

  89. Terraform ·ͱΊ
    4 Terraform ͦͷ΋ͷ͸ಛʹ໰୊ͳ͘ѻ͑ͨ !
    4 Πϯελϯε಺෦Λ cloud-init Ͱߏங͢Δ
    ͱ͜Ζ͸େมͩͬͨ… "
    4 มߋద༻લʹ plan Λ֬ೝͰ͖Δͷ͸ྑ͍
    4 ੲͷ CloudFormation ͸ dry-run ͕Ͱ͖
    ͳͯ͘ා͔ͬͨ
    © 2016 CyberAgent, Inc. 89

    View Slide

  90. Terraform ·ͱΊ
    4 ॳظߏஙޙͷӡ༻͕ઙ͍ͷͰࠓޙͭΒ͍͜ͱ͕͋
    Δ͔΋͠Εͳ͍ !
    4 ϓϩάϥϛϯάͱಉ͘͡ɺಡΈ΍͍͢ίʔυΛॻ
    ͘৺ֻ͚͕ඞཁ
    4 Axion ͸ cloud-init ·ΘΓ͕͔ͳΓԚͳ͍…
    4 ׬શͳ Container ϕʔεͷੈքͰ͋Ε͹Πϯελ
    ϯε؅ཧͱͯ͠ͷ Terraform ͸ෆཁʹͳΓͦ͏
    © 2016 CyberAgent, Inc. 90

    View Slide

  91. Axion Πϯϑϥͷ
    શମ૾
    © 2016 CyberAgent, Inc. 91

    View Slide

  92. © 2016 CyberAgent, Inc. 92

    View Slide

  93. ΦϨ͸Α͏΍͘
    ͷ΅Γ͸͡Ίͨ
    ͹͔Γ͔ͩΒͳ
    ͜ͷ͸ͯ͠ͳ͘ԕ͍
    Docker ࡔΛΑ…
    © 2016 CyberAgent, Inc. 93

    View Slide