Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Amazon ECS とマネージドサービスを活用したフルコンテナ構成によるゲームサーバの構築と運用 / Amazon-ECS-game-server

Amazon ECS とマネージドサービスを活用したフルコンテナ構成によるゲームサーバの構築と運用 / Amazon-ECS-game-server

FUJIWARA Shunichiro

August 22, 2018
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. Amazon ECS ͱ
    ϚωʔδυαʔϏεΛ׆༻ͨ͠
    ϑϧίϯςφߏ੒ʹΑΔ
    ήʔϜαʔόͷߏஙͱӡ༻
    2018.08.22 CEDEC 2018
    ໘ന๏ਓΧϠοΫ ౻ݪढ़Ұ࿠

    View Slide

  2. ࣗݾ঺հ
    @fujiwara
    github.com/fujiwara
    sfujiwara.hatenablog.com

    View Slide

  3. Game & Community

    View Slide

  4. Agenda
    • Amazon ECS Ͱͷαʔόߏஙͱӡ༻
    • σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ
    • ϩάͷू໿ͱετϦʔϛϯάॲཧ
    • GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ

    View Slide

  5. ࠷ॳʹ͓࿳ͼ
    ·ͨɺຊ൪ϦϦʔεલʹඞਢͰ͋Δෛՙࢼݧ΋ίϯςφΛ׆༻͢Δ͜
    ͱͰॊೈʹߦ͑ΔͨΊɺͦͷख๏ʹ͍ͭͯ΋આ໌͠·͢ɻ0
    ෛՙࢼݧʹ͍ͭͯ͸ऩ·Γ͖Βͳ͔ͬͨͨΊ
    ผͷࢿྉΛ͝ࢀর͍ͩ͘͞
    !
    GoͱίϯςφͰ࡞ΔWebΞϓϦέʔγϣϯϕϯνϚʔΧʔ
    speakerdeck.com/fujiwara3/benchmarker-built-with-go-and-container
    0 https://2018.cedec.cesa.or.jp/session/detail/s5abc66488c745

    View Slide

  6. Amazon ECS Ͱͷαʔόߏஙͱӡ༻

    View Slide

  7. Amazon ECS
    AWSͷίϯςφ؅ཧϚωʔδυαʔϏε
    ͍ΘΏΔʮΦʔέετϨʔγϣϯπʔϧʯ
    ෳ਺ͷίϯςφͰىಈͨ͠ϓϩηε܈Λ؅ཧ͢Δ΋ͷ
    ଞʹ༗໊ͳͷ͸ Kubernetes(k8s)

    View Slide

  8. ECS ͷ֓೦

    View Slide

  9. λεΫ - Task
    ؔ࿈͢ΔίϯςφϓϩηεΛҰͭʹ·ͱ
    Ίͨ΋ͷ
    λεΫ಺ͷίϯςφ͸૬ޓʹ௨৴Ͱ͖Δ
    (localhost or hostname)
    k8s ͷ Pod ʹ૬౰

    View Slide

  10. λεΫఆٛ - Task
    Definition
    λεΫͷߏ੒Λఆٛ͢Δ΋ͷ
    ؚ·ΕΔ(ෳ਺ͷ)ίϯςφɺλεΫࣗମ
    ͷωοτϫʔΫઃఆͳͲΛఆٛ
    • Πϝʔδ
    • CPU
    • ϝϞϦ
    • ϙʔτׂΓ౰ͯ ͳͲ

    View Slide

  11. αʔϏε - Service
    λεΫΛෳ਺ݸଋͶͨ΋ͷ
    • ϩʔυόϥϯαʔ (ALB, NLB) ʹऩ༰
    ͢Δ୯ҐʹͳΔ
    • ࢦఆͨ͠λεΫ਺Λҡ࣋͢Δ
    • λεΫ਺ΛΦʔτεέʔϧͰ͖Δ
    k8s ͷ Deployment ʹ૬౰

    View Slide

  12. Ϋϥελʔ - Cluster
    λεΫ͕഑ஔ͞ΕΔ EC2 Πϯελϯεͷू߹
    EC2 Πϯελϯεࣗମ͕Ϛωʔδυʹͳͬͨ Fargate ΋

    View Slide

  13. View Slide

  14. ࠓ·Ͱͷ EC2 Ͱͷαʔόӡ༻ͱҧ͏఺
    EC2 ্ʹΞϓϦέʔγϣϯ/ϛυϧ΢ΣΞΛىಈ
    → λεΫʹϓϩηεΛ·ͱΊΔ
    ϩʔυόϥϯαʔʹEC2ΠϯελϯεΛొ࿥
    → αʔϏεΛొ࿥
    Φʔτεέʔϧ͸EC2ΠϯελϯεΛ૿ݮ͢Δ
    → λεΫΛ૿ݮ͢Δ
    …ͨͩ͠ EC2 ίϯςφΠϯελϯε΋૿ݮ͠ͳ͍ͱ࣮ࡍʹ࢖͑ΔϦιʔ
    ε͸มΘΒͳ͍ / Fargate Ͱ͸ղܾ

    View Slide

  15. ECS Ͱͷαʔόߏ੒ͷํ਑
    ঢ়ଶΛ࣋ͨͳ͍ΞϓϦέʔγϣϯͷΈΛಈ͔͢
    ௕࣌ؒͷঢ়ଶΛ࣋ͭϛυϧ΢ΣΞ͸ಈ͔͞ͳ͍
    (RDBMSͳͲͷετϨʔδ)
    ঢ়ଶ͸͢΂ͯϚωʔδυαʔϏε΁อଘ͢Δ
    (RDS, S3, ElastiCache...)

    View Slide

  16. ECS Ͱͷαʔόߏ੒ͷํ਑
    ঢ়ଶΛ࣋ͨͳ͍ΞϓϦέʔγϣϯͷΈΛಈ͔͢
    σϓϩΠͷͨͼʹ৽͍͠λεΫ͕ىಈ͠ɺݹ͍λεΫ͸མͪΔ
    λεΫ͕ऴྃ/མͪΔͱϑΝΠϧ΁ͷॻ͖ࠐΈ͸ࣦΘΕΔ
    EC2ϗετͷϑΝΠϧγεςϜΛϚ΢ϯτ͸Ͱ͖Δ͕…
    Ϋϥελ಺ͷͲͷEC2ͰλεΫΛىಈ͢Δ͔͸ECS͕ܾΊΔ
    → Ӭଓ͢Δঢ়ଶ͸͢΂ͯϚωʔδυαʔϏε΁ʂ
    ϩά΋ϩʔΧϧϑΝΠϧʹ࣋ͨͣϚωʔδυαʔϏε΁సૹ

    View Slide

  17. ECS ʹͯ͠Α͔ͬͨ͜ͱ
    EC2্Ͱͷϛυϧ΢ΣΞͷߏ੒؅ཧ͕ෆཁ
    ͜Ε·Ͱ͸ Chef Ͱ؅ཧ
    ੈؒͰ͍͏΄Ͳਏ͘͸ͳ͍͕…
    ႈ౳ʹ͢Δͷ͕໘౗ɺΞϓϦέʔγϣϯΤϯδχΞʹ͸ෛ୲
    → Dockerfile ͰΫϦʔϯͳঢ়ଶ͔ΒΠϯετʔϧ͢Δ͚ͩ

    View Slide

  18. ECS ʹͯ͠Α͔ͬͨ͜ͱ
    αʔόͷ௥Ճ࡟আָ͕
    ΫϥελͷEC2Πϯελϯε͸͢΂ͯಉ͡΋ͷ
    ͜Ε·Ͱ͸αʔόͷछྨ͝ͱʹΠϯελϯεΛ࡞͍ͬͯͨ
    App, WebSocket, Batch, ϩάू໿ etc...
    OS ʹ࠶ىಈඞਢͳύονΛ౰͍ͯͨ৔߹
    1.৽͍͠ AMI ͔ΒΠϯελϯεىಈ
    2.λεΫΛ৽͍͠ΠϯελϯεʹҠಈ
    3.ݹ͍ΠϯελϯεΛࣺͯΔ

    View Slide

  19. ECS ʹͯ͠େมͩͬͨ͜ͱ
    ͜Ε·Ͱͱͷҧ͍Λҙࣝͯ͠৐Γӽ͑Δ
    ֓೦ΛΞϓϦέʔγϣϯΤϯδχΞʹཧղͯ͠΋Β͏ͷ͕େม
    ϩʔΧϧϑΝΠϧʹґଘ͠ͳ͍࢓૊ΈΛ࡞Δඞཁ͕͋Δ
    → ঢ়ଶΛ࣋ͨͳ͍͜ͱΛపఈ͢Δ
    ʲྫʳϩάΛ tail -f ͍͚ͨ͠ͲϑΝΠϧʹ͸ॻ͚ͳ͍…
    → ϚωʔδυαʔϏεʴࣗ࡞πʔϧͰղܾ (ޙड़)

    View Slide

  20. ECS ʹͯ͠େมͩͬͨ͜ͱ
    ௚઀ϓϩηεͷঢ়ଶΛΈΔௐ͕ࠪ໘౗
    ʮstrace ͍ͨ͠ʂʯ
    ɾૂͬͨλεΫ͕ಈ࡞͍ͯ͠ΔEC2ͷϗετΛݟ͚ͭΔ
    ɾEC2 ʹ ssh ͯ͠ docker exec sh & strace
    ίϯςφʹೖͬͯ΋ඞཁͳπʔϧ͕ͳ͔ͬͨΓ͢Δ

    View Slide

  21. ECS / ίϯςφԽͷ෭࡞༻
    ঢ়ଶΛ࣋ͨͳ͍ɺϗετ͸͍ͭফ͑ͯ΋໰୊ͳ͍Α͏ʹͳͬͨ
    → Spot Πϯελϯε͕׆༻Ͱ͖Δ
    ௨ৗͷΦϯσϚϯυΠϯελϯεͷ༨৒ϦιʔεΛ҆͘
    (30%͙Β͍Ͱ) ఏڙ͢Δ΋ͷ
    ͨͩ͠༨৒Ϧιʔε͕ͳ͘ͳΔͱམͱ͞ΕΔ

    View Slide

  22. Spot Fleet
    ෳ਺ͷΠϯελϯελΠϓ
    AZ Λ૊Έ߹ΘͤͯϦιʔεΛ֬อ
    མͪͨ෼͸ࣗಈతʹ୅ସ͕ىಈ

    View Slide

  23. ECS Ͱ Spot ΠϯελϯεΛ҆શʹ੾Γ཭͢
    ΦϯσϚϯυͷधཁ͕૿͑ͯ Spot ΁ׂΓ౰ͯΔϦιʔε͕
    ͳ͘ͳΔͱ120ඵલʹ termination ௨஌͕དྷΔ
    http://169.254.169.254/latest/meta-data/spot/instance-action
    Λ polling
    → ঢ়ଶมԽΛݕ஌ͨ͠Βࣗ෼ࣗ਎ΛΫϥελ͔Β֎͢
    ͜ΕΛ͠ͳ͍ͱλεΫ͕ಥવࢮ͢Δ

    View Slide

  24. ֤ΠϯελϯεͰୀ໾ॲཧλεΫΛಈ͔͢
    #!/bin/bash
    while sleep 5; do
    CONTENT=$(curl -sf http://169.254.169.254/latest/meta-data/spot/instance-action)
    if [ -z "$CONTENT" ]; then
    continue
    fi
    CLUSTER=$(curl -s http://localhost:51678/v1/metadata | jq -r .Cluster)
    CONTAINER_INSTANCE=$(curl -s http://localhost:51678/v1/metadata | jq -r .ContainerInstanceArn)
    aws ecs update-container-instances-state \
    --cluster "$CLUSTER" \
    --container-instances "$CONTAINER_INSTANCE" \
    --status DRAINING \
    && exit 0
    done
    localhost:51678 = ECS agent
    !
    ਖ਼௚͜Ε͙Β͍͸ ECS agent ͕΍ͬͯ΄͍͠

    View Slide

  25. Amazon ECS Ͱͷαʔόߏஙͱӡ༻ ·ͱΊ
    • ΞϓϦέʔγϣϯʹঢ়ଶΛ࣋ͨͳ͍͜ͱΛపఈ͢Δ
    • ঢ়ଶΛ࣋ͨͳ͍͜ͱͰಘΒΕΔϝϦοτΛڗड͠Α͏

    View Slide

  26. σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ

    View Slide

  27. ͜Ε·Ͱͷ EC2 ্ͷσϓϩΠ
    • ֤ EC2 ʹΞʔΧΠϒϑΝΠϧΛ഑෍
    • S3 ʹ tar.gz ΛΞοϓϩʔυ
    • ֤ϗετ͕ S3 ͔Βμ΢ϯϩʔυ
    ͯ͠ల։
    • ϓϩηεΛ graceful ʹ࠶ىಈ
    Consul + Stretcher (ࣗ࡞OSS) Ͱ࣮ݱ

    View Slide

  28. ECSʹ͓͚ΔσϓϩΠ
    Ϣʔβ͕΍Δ͜ͱ
    ৽͍͠ΞϓϦέʔγϣϯؚ͕·ΕͨΠ
    ϝʔδΛ ECR ʹొ࿥
    ৽͍͠ΠϝʔδΛ࢖͏λεΫఆٛΛొ࿥
    αʔϏεઃఆͰ৽͍͠λεΫఆٛΛ࢖͏
    Α͏ʹมߋ

    View Slide

  29. ECSʹ͓͚ΔσϓϩΠ
    ECS ͕΍Δ͜ͱ
    ৽͍͠λεΫఆٛͰλεΫΛىಈ
    (LB ͕͋Δ৔߹͸ LB ʹ૊ΈࠐΉ)
    ਖ਼ৗʹىಈɺ૊ΈࠐΊͨΒݹ͍λεΫΛ
    Λམͱ͢
    ਖ਼ৗʹىಈ͠ͳ͔ͬͨΒݹ͍΋ͷ͸མͪ
    ͳ͍
    αʔϏε಺ͷ͢΂ͯͷλεΫ͕৽͍͠΋
    ͷʹͳͬͨΒ׬ྃ

    View Slide

  30. ϩʔϧόοΫ
    ݹ͍λεΫఆٛΛ࢖͏Α͏ʹαʔϏεΛมߋ͢Δ
    ಛʹ ECS ʹػೳ͕͋ΔΘ͚Ͱ͸ͳ͍
    Ϣʔβ͕ࣗ෼ͰʮҰͭલʹσϓϩΠ͍ͯͨ͠λεΫఆٛʯ
    ΛσϓϩΠ͠௚͢
    !
    ىಈޙʹಈతͳ͜ͱΛͳΔ΂͘͠ͳ͍ͷ͕ॏཁ
    !
    ίϯςφىಈޙʹ࠷৽ίʔυΛϨϙδτϦ͔Βऔಘ
    "
    ίʔυ͸Πϝʔδʹম͖ࠐΉ

    View Slide

  31. σϓϩΠπʔϧ
    ͍Ζ͍Ζ͋Δ
    ecs-cli github.com/aws/amazon-ecs-cli
    ɹGo / docker-compose Ͱఆٛͨ͠ઃఆΛ ECS ʹσϓϩΠ
    hako github.com/eagletmt/hako
    ɹRuby / ఆٛ͸ YAML
    ecs-deploy github.com/silinternational/ecs-deploy
    ɹbash + aws-cli

    View Slide

  32. σϓϩΠπʔϧࣗ࡞
    ecspresso github.com/kayac/ecspresso
    1. ݩʑίϯιʔϧૢ࡞ͰσϓϩΠ͍ͯͨ͠΋ͷ͕͋ͬͨ
    2. aws-cli ͰऔಘͰ͖ΔλεΫఆٛJSONΛͦͷ··࢖͏ͨΊʹ
    shell script Λॻ͍ͨ
    3. ίʔυ͕ෳࡶʹͳͬͨͷͰ Go + aws-sdk-go Ͱॻ͖௚ͦ͏
    ͍ͭͷؒʹ͔ҭͬͯ͠·ͬͨ

    View Slide

  33. ecspresso ͷಛ৭
    λεΫఆٛ͸ aws-cli Ͱѻ͏JSONΛͦͷ··࢖͑Δ
    →ݱঢ়ಈ͍͍ͯΔ΋ͷΛͦͷ··؅ཧର৅ʹͰ͖Δ
    JSON ͷதͷ஋Λ࣮ߦ࣌ͷ؀ڥม਺Ͱஔ׵ͯ͠ొ࿥Ͱ͖Δ
    {
    "taskDefinition": {
    "cpu": "4 vCPU",
    "containerDefinitions": [
    {
    "name": "app",
    "image": "prod/app:f6fb4dcc20",

    View Slide

  34. {
    "taskDefinition": {
    "cpu": "{{ env `VCPU_NUM` `4` }} vCPU",
    "containerDefinitions": [
    {
    "name": "app",
    "image": "{{ must_env `SERVICE` }}/app:{{ must_env `TAG` }}",
    VCPU_NUM : prod | staging ͰҟͳΔCPUΛՄมʹ
    SERVICE : prod | staging σϓϩΠઌͷ؀ڥΛ੾Γସ͑Δ
    TAG : σϓϩΠ͝ͱʹมΘΔΠϝʔδͷλά
    ͻͱͭͷ JSON Λෳ਺؀ڥʹ࢖͍ճͤΔͷͰҡ࣋؅ཧָ͕

    View Slide

  35. ίϯςφ΁ͷൿಗ৘ใͷઃఆ
    Πϝʔδʹൿಗ৘ใΛম͖ࠐΉͷ͸ආ͚͍ͨ
    ম͖ࠐΉͱ docker pull ͨ͠Βͦ͜ʹ࢒Δ
    ഁغ͍ͨ͠৔߹ɹΠϝʔδΛ͢΂ͯݸผʹഁغ͢Δ͔͠ͳ͍
    → ؀ڥม਺Ͱ౉͢
    λεΫఆٛͰ environment ઃఆ͸Ͱ͖Δ͕
    → λεΫఆٛࣗମ͸ฏจͰอଘ͞ΕΔ
    ɹݹ͍৘ใͷഁغ΋Πϝʔδಉ༷ʹ໰୊ʹͳΔ

    View Slide

  36. ͲͷΑ͏ʹ؀ڥม਺Λઃఆ͢Δ͔
    λεΫىಈ࣌ʹ҆શͳͱ͜Ζ͔Βऔಘ
    ؀ڥม਺ʹઃఆ͔ͯ͠ΒϓϩηεΛىಈ͢Δ
    ҆શͳઃఆอଘ৔ॴ
    AWS Systems Manager (SSM) ύϥϝʔλετΞ

    View Slide

  37. SSMύϥϝʔλετΞ

    View Slide

  38. SSMύϥϝʔλετΞ
    • ֊૚Խͨ͠ path ໊Ͱ஋Λอଘ
    • ઃఆ஋ͷόʔδϣχϯάɺมߋऀͷཤ
    ྺ؅ཧ
    • KMSͰ҉߸Խͨ͠஋ΛอଘͰ͖Δ
    • KMS΁ͷݖݶ͕ͳ͚Ε͹஋͸ಡΊ
    ͳ͍
    • ίϯςφىಈ࣌ʹ͔͜͜Β஋ΛऔΓ
    ग़ͯ͠؀ڥม਺ʹઃఆ͢Ε͹Α͍

    View Slide

  39. ύϥϝʔλετΞ͔Β஋ΛऔΓग़͢
    aws-ssm-env github.com/jamietsao/aws-ssm-env
    $ aws-ssm-env --paths=/prod/
    API_KEY=xxxxxx
    DB_PASS=productionpass
    entry point ͷ sh Ͱ࣮ߦɺexport ͔ͯ͠Β exec ͢Δ
    #!/bin/sh
    export AWS_REGION=ap-northeast-1
    export $(aws-ssm-env --paths=/prod/)
    exec /path/to/myapp

    View Slide

  40. ύϥϝʔλετΞͷ஋Λ export & exec
    ssmwrap github.com/handlename/ssmwrap
    ฐࣾಉ྅࡞
    $ ssmwrap -paths=/prod/ -- /path/to/myapp
    ssm ͔Βऔಘͨ͠஋Λ env ʹઃఆͯ͠ exec ·Ͱͯ͘͠ΕΔ
    ɾexec ·Ͱ͢ΔͷͰ Docker ͷ entrypoint ʹ௚઀ࢦఆͰ͖Δ
    ɾshell scirpt Λܦ༝͠ͳ͍ͷͰվߦΛؚΉ஋ͷ໰୊͕ͳ͍
    ɾ-retries ΦϓγϣϯͰϦτϥΠճ਺ΛࢦఆͰ͖Δ

    View Slide

  41. SSMύϥϝʔλετΞͷ໰୊
    API Rate Limit ͕ݫ͠Ί
    େྔͷλεΫΛҰؾʹىಈ͢Δͱ Rate Limit Error
    (੍ݶ஋͸αϙʔτʹ໰͍߹Θ͕ͤͨඇެ։)
    ɾaws-ssm-env ͸ΤϥʔʹͳΔͱ panic Ͱࢮ͵
    ɹσϓϩΠ࣌ʹࢮΜͩλεΫ͸࠶౓ىಈ͞ΕΔ͕…
    ɾssmwrap ͸ -retries ઃఆͰϦτϥΠͰ͖Δ(͓નΊ)
    ɾىಈ࣌ʹ random sleep ͰλΠϛϯάΛͣΒͯ͠؇࿨
    !
    ਖ਼௚ECSଆͰ؀ڥม਺ʹઃఆͯ͠ىಈͯ͠΄͍͠

    View Slide

  42. σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ ·ͱΊ
    • Blue-Green ͷར఺Λੜ͔ͤΔΑ͏ʹ࢖͏
    • SSMύϥϝʔλετΞ͸ศར
    • Ͱ΋΋ͬͱڧ͘ͳͬͯ΄͍͠
    • ֤ࣗ޻෉ͯ͠΍͍͖ͬͯ·͠ΐ͏

    View Slide

  43. ϩάͷू໿ͱετϦʔϛϯάॲཧ

    View Slide

  44. ϩάू໿ͷੲͱࠓ
    ੲ (ʙ2011 Before Fluentd)
    ɾϑΝΠϧΛఆظతʹճऩ
    ɾλΠϜϥάେ

    View Slide

  45. ϩάू໿ͷੲͱࠓ
    ۙ୅ (2012ʙ After Fluentd)
    ϑΝΠϧΛ fluentd ͕ஞ࣍ಡΈऔΓ ૹ৴
    ΞϓϦέʔγϣϯ͕ fluentd ΁௚઀ૹ৴
    λΠϜϥάখ

    View Slide

  46. ECS ͔Βͷϩάू໿
    ൃੜ͢Δϩά͸େ·͔ʹ3छྨ
    1.ΞΫηεϩά
    2.ΞϓϦέʔγϣϯ/ϛυϧ΢ΣΞ͕
    ɹඪ४ग़ྗɺඪ४Τϥʔग़ྗʹు͖ग़͢ϩά
    3.ΞϓϦέʔγϣϯతʹҙຯ͕͋Δߦಈϩά
    ΞΫηεϩά͸ nginx ͔Β /dev/stdout ʹग़ͤ͹2छྨ
    ͲͷΑ͏ʹूΊΔ͔

    View Slide

  47. ֤ίϯςφ͕ STDOUT, STDERR ʹు͖ग़ͨ͠΋ͷ
    Docker logging driver ʹΑͬͯѻΘΕΔ
    λεΫఆٛͰίϯςφ͝ͱʹࢦఆͰ͖Δ
    awslogs : CloudWatch Logs ΁ૹ৴
    fluend : Fluentd ΁ૹ৴
    json-file : ϑΝΠϧ΁อଘ
    syslog : syslogd ΁ૹ৴
    (ଞʹ΋͋Δ)

    View Slide

  48. awslogs or fluentd?
    CloudWatch Logs
    ɹpros: ϚωʔδυͳͷͰԿ΋ߟ͑ͳͯ͘Α͍
    ɹcons: ͔ͦ͜Βߋʹผʹྲྀ͢ͷ͕ଟগ໘౗ (Lambda ͕ඞཁ)
    Fluentd:
    ɹpros: ߦಈϩάଞͱಉ༷ͷѻ͍͕Ͱ͖Δ
    ɹcons: fluentdΛࣗ෼ͰཱͯΔඞཁ͕͋Δ

    View Slide

  49. Fluentd ʹ౷Ұ͢Δ
    ߦಈϩά͸ͲͪΒʹ͠Ζ Fluentd Ͱѻ͍͍ͨͷͰඞཁ
    ߦಈϩά
    ɾλεΫʹΞϓϦέʔγϣϯͱҰॹʹ fluentd Λ഑ஔ (sidecar)
    ɾΞϓϦ͔Β localhost:24224 ʹૹ৴͢Δ(ߏ଄Խϩά)
    STDOUT, STDERR
    ɾECSαʔϏεͱͯ͠ NLB Λհͯ͠ fluentd Λ഑ஔ
    ɾ֤λεΫ͸ Docker logging driver Ͱ NLB:24224 ΁ૹ৴͢Δ

    View Slide

  50. View Slide

  51. ECS Ͱ fluentd ΛͲ͏࢖͏͔
    AWS ͷ৔߹ fluentd + plugin S3 Ͱ S3 ΁อ؅͢Δͷ͕ఆ൪
    S3 ΁ͷॻ͖ग़͠͸ 1ʙ5 ෼ఔ౓ʹ͢Δ
    ɹ͋·Γ୹͍ͱ S3 ͷΦϒδΣΫτ͕ࡉ੾ΕʹͳΔ)
    ো֐࣌ͷ͜ͱΛߟ͑Δͱ਺෼ఔ౓Ͱ΋ίϯςφ಺ͷόοϑΝ
    (ϝϞϦ or ϩʔΧϧϑΝΠϧ)ʹอଘ͢Δͷ͸ফࣦϦεΫ
    → ৴པͷஔ͚ΔόοϑΝ͕΄͍͠

    View Slide

  52. Kinesis Data Streams
    ౤͛ࠐΜͩϝοηʔδ(ϩάʹݶΒͳ͍)Λ24࣌ؒɺॱংΛอͬͯอ؅ͯ͘͠ΕΔετϦʔϛϯάॲཧج൫ͷϚωʔδυαʔϏε
    OSS Ͱ͸ Apache Kafka ͕ྨࣅ
    ϩάΛ S3 ʹॻ͖ग़͢લͷʮ৴པͰ͖ΔʯόοϑΝͱͯ͠࠷ద

    View Slide

  53. Kinesis Data Firehose
    ετϦʔϛϯάσʔλΛఆظతʹ
    ɾS3
    ɾRedshift
    ɾElasticsearch
    ʹॻ͖ग़͢ϚωʔδυαʔϏε
    ιʔεͱͯ͠ Kinesis Streams Λ࢖͑Δ

    View Slide

  54. ࠷ऴతʹ͸͜͏

    View Slide

  55. ʮtail -f ͍ͨ͠ΜͰ͕͢ʯ
    ϩά͕ϑΝΠϧʹॻ͔Εͳ͘ͳͬͨ
    → tail -f ͸Ͳ͏͢Δ?
    kinesis-tailf
    github.com/fujiwara/kinesis-tailf
    Go + aws-sdk-go Ͱࣗ࡞
    $ kinesis-tailf -stream docker-logs
    Kinesis Streams Λ௥ඌ͠ඪ४ग़ྗʹు
    ͖ग़͢ CLI
    -start -end ΦϓγϣϯͰಛఆ࣌ؒଳΛ
    நग़Մೳ

    View Slide

  56. Τϥʔϩά͚ͩநग़͍ͨ͠
    શϩάΛ kinesis-tailf | grep ?
    ྲྀྔ͕ଟ͍ͱͭΒ͍
    Streams ͷσʔλΛ Lambda Ͱॲཧ
    ύλʔϯʹϚονͨ͠ϩά͚ͩผͷ
    Streams ʹྲྀ͠௚͢
    ɾαʔόΤϥʔ status:5
    ɾॲཧʹ͕࣌ؒ3ඵҎ্ֻ͔ͬͨ΋ͷ
    ɹapptime:([3-9]|[1-9][0-9]+)\.
    ɾ(ERROR|WARN)

    View Slide

  57. ϩά͔ΒͷΤϥʔ௨஌
    S3 Event notification
    S3ʹΦϒδΣΫτ͕࡞੒/࡟আ͞ΕͨΒ
    LambdaΛݺͿ
    Τϥʔநग़ετϦʔϜ͔Β S3 ͷΦϒ
    δΣΫτ͕ੜ੒͞ΕͨΒ Slack ʹ௨஌
    ΠϕϯτυϦϒϯͳόονॲཧʹ
    Firehose + S3 + Lambda ͕࢖͑Δ

    View Slide

  58. ϩά͔ΒͷΤϥʔ௨஌
    நग़͞Εͨλάɺߦ਺ͱͱ΋ʹsnippet౤ߘ
    Τϥʔൃੜ͔Β1෼Ͱ௨஌
    େྔʹΤϥʔ͕ൃੜͯ͠΋1෼͝ͱʹ͔͠௨஌͸དྷͳ͍(ॏཁ)

    View Slide

  59. ϩάͷू໿ͱετϦʔϛϯάॲཧ ·ͱΊ
    • ϩά͸ STDOUT/STDERR ʹग़ྗ
    Docker Logging Driver Ͱૹ৴
    • Fluentd + ৴པͰ͖ΔόοϑΝͱͯ͠ͷ Kinesis Streams
    • Firehose + S3 + Lambda ͰΠϕϯτυϦϒϯͳσʔλॲཧ΋

    View Slide

  60. GoݴޠʹΑΔ
    ӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ

    View Slide

  61. AWS / ECS ͸伱͕ؒଟ͍
    ࠓճࣗલͰ։ൃͨ͠΋ͷ
    ecspresso github.com/kayac/ecspresso
    σϓϩΠπʔϧ
    ssmwrap github.com/handlename/ssmwrap
    SSM ύϥϝʔλετΞͷ஋Λ؀ڥม਺ʹઃఆͯ͠ exec
    kinesis-tailf github.com/fujiwara/kinesis-tailf
    Kinesis StreamsΛ௥ඌ͢Δ

    View Slide

  62. 伱ؒՈ۩Λࣗ࡞͢Δ
    伱ؒΛຒΊΔπʔϧɺϛυϧ΢ΣΞΛ։ൃͯ͠ӡ༻վળ
    খ͘͞ɺద౓ʹ൚༻తͳ΋ͷΛ࡞Δ
    ຊՈ͕伱ؒΛຒΊͨΒࣺͯΒΕΔΑ͏ʹ

    View Slide

  63. 伱ؒՈ۩ͷྫ
    Rin github.com/fujiwara/Rin
    Redshift data Importer by SQS
    messaging.
    S3 Πϕϯτ௨஌Ͱ SQS ʹૹ৴
    SQS ͷϝοηʔδ͔Β Redshiftʹ copy
    ͯ͠औΓࠐΈΛߦ͏πʔϧ
    2015.05 ʹ։ൃ
    2015.10 Firehose ൃද
    2017.07 ౦ژϦʔδϣϯʹ Firehose

    View Slide

  64. Before / After
    Fluentd ͔ΒϩάΛૹ৴
    S3 / Redshift ʹॱ࣍औΓ͜·ΕΔ
    ͱ͍͏ߏ଄͸ͦͷ··Ϛωʔδυʹ

    View Slide

  65. πʔϧ։ൃݴޠͱͯ͠ͷ Go
    γϯάϧόΠφϦʹͳΔ
    ϥϯλΠϜෆཁ
    ίϯςφͷϕʔεΠϝʔδΛબ͹ͳ͍
    ॻ͖΍͢͞ΑΓಡΈ΍͢͞Λॏࢹ
    → ϝϯςφϯεੑ͕ߴ͍
    aws-sdk-go
    ػೳɺΞοϓσʔτ͸ਃ͠෼ͳ͍
    ࢖͍উख͸ਖ਼௚ඍົͳͱ͜Ζ͕…

    View Slide

  66. ίϯςφ؀ڥ޲͚ͷ։ൃ Tips
    ίϚϯυϥΠϯΦϓγϣϯͷ஋Λ؀ڥม਺͔Β΋ಡΉ1
    func main() {
    var port int
    var host string
    flag.IntVar(&port, "port", 8080, "port number")
    flag.StringVar(&host, "host", "localhost", "hostname")
    flag.VisitAll(func(f *flag.Flag) {
    if s := os.Getenv(strings.ToUpper(f.Name)); s != "" {
    f.Value.Set(s)
    }
    })
    flag.Parse()
    fmt.Printf("%s:%d\n", host, port)
    }
    1 https://mattn.kaoriya.net/software/lang/go/20170609110526.htm

    View Slide

  67. ίϚϯυϥΠϯΦϓγϣϯͷ஋Λ؀ڥม਺͔Β΋ಡΉ
    $ myapp -host example.com -port 9999
    example.com:9999
    $ HOST=example.com PORT=9999 myapp
    example.com:9999
    ίϯςφͰ͸ίϚϯυϥΠϯΑΓ
    ؀ڥม਺ͷ΄͏͕ઃఆ͠΍͍͢

    View Slide

  68. ઃఆϑΝΠϧʹ؀ڥม਺͔Β஋ΛຒΊࠐΉ
    github.com/kayac/go-config
    Go ͷઃఆϑΝΠϧಡΈࠐΈ package
    # yaml
    account_id: "{{ must_env `MY_ACCOUNT_ID` }}"
    profile: "{{ env `PROFILE` `default` }}"
    must_env: ઃఆ͞Εͳ͍ঢ়ଶͰ࣮ߦ͞ΕΔͱ panic
    env: ઃఆ͞Ε͍ͯͳ͍৔߹͸σϑΥϧτ஋Λࢦఆ
    YAML / JSON / TOML ʹରԠ

    View Slide

  69. ઃఆϑΝΠϧΛϦϞʔτ͔Βऔಘ͢Δ
    ઃఆ͕ϑΝΠϧʹͳ͍ͬͯΔͱ
    มߋͷࡍʹίϯςφͷ࠶Ϗϧυ & Ξοϓϩʔυ͕ඞཁ
    (ࢼߦࡨޡ͕໘౗)
    File / S3 / HTTP ͰऔಘͰ͖ΔΑ͏ʹ࡞͓ͬͯ͘ͱศར2
    $ CONFIG=s3://my-config-bucket/config.yaml mydaemon
    2 GoͰHTTPͱS3Λಁաతʹѻ͏ https://shogo82148.github.io/blog/2018/06/09/go-s3-protocol/

    View Slide

  70. OSS ͱͯ͠࡞Δ
    OSS ʹͯ͠͠·͏ (Ծʹࣗ෼Β͔͠࢖Θͳͯ͘΋)
    README ͙Β͍͸ؤுͬͯॻ͘(ϞνϕʔγϣϯʹͳΔ)
    ա౓ͳࣾ಺ࣄ৘ͷࠞೖΛ๷͙
    ʮͦͷϓϩδΣΫτҎ֎ʹҙຯ͕͋Δػೳͳͷ͔?ʯ
    ίϐϖͰຐվ଄൛͕૿৩͢ΔͷΛ๷͙
    ʮGitHubͰόΠφϦఏڙͯ͠ΔͷͰͦΕΛ࢖͍ͬͯͩ͘͞ʯ

    View Slide

  71. GoݴޠʹΑΔ
    ӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ ·ͱΊ
    • AWS / ECS ͷ伱ؒΛখ͘͞ॻ͍ͨπʔϧͰຒΊΑ͏
    • ͋͑ͯOSSʹ͢Δ͜ͱͰ͖Ε͍ʹอͭ
    • Go ͸ίϯςφ؀ڥͰ࢖͍উख͕Α͍

    View Slide

  72. Questions?
    • Amazon ECS Ͱͷαʔόߏஙͱӡ༻
    • σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ
    • ϩάͷू໿ͱετϦʔϛϯάॲཧ
    • GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ

    View Slide