Upgrade to Pro — share decks privately, control downloads, hide ads and more …

タップル誕生: オンプレからAWS Fargateまで @ JAWS DAYS 2020

タップル誕生: オンプレからAWS Fargateまで @ JAWS DAYS 2020

PDF版: https://share.getcloudapp.com/E0uqmebA (リンク踏みたい方向け)

本セッションでは、タップル誕生というサービスのオンプレ環境からAWS Fargateになるまでのストーリーと技術的なチャレンジについて紹介している。歴史的背景、移設前の構成を説明し、EC2、ECS/EC2、ECS/Fargateに継続的に、段階的にアーキテクチャを変えて行ったことについて解説する。

トークの後半では、Fargateのメリットデメリット、ECSの落とし所、スケーリング設計に役立つアドバイス、クラウド移設で気を付けるべきこと、ECSの最新機能の検証結果についても触れる。

Speaker Profile:

株式会社サイバーエージェント
Torgayev Tamirlan 氏

株式会社サイバーエージェント 技術本部 サービスリライアビリティグループ所属 クラウド技術アドバイザ / 2017年~内定者アルバイトを経て、2018年に新卒入社 / Kinesis、ECS、Lambda、Elasticsearch、Terraformを専門領域とし、複数の新規サービス立ち上げや、AWA・タップル誕生などのシステム改善に従事している

Tamirlan 893 Torgayev

March 22, 2020
Tweet

More Decks by Tamirlan 893 Torgayev

Other Decks in Technology

Transcript

  1. λοϓϧ஀ੜ:
    ΦϯϓϨ͔ΒAWS Fargate·Ͱ
    גࣜձࣾαΠόʔΤʔδΣϯτ
    ٕज़ຊ෦ αʔϏεϦϥΠΞϏϦςΟάϧʔϓ
    ϓϩάϥϛϯάϠΫβ (@prog893)

    View Slide

  2. ࣗݾ঺հ
    • Torgayev Tamirlan / ϓϩάϥϛϯάϠΫβ / @prog893
    τ ϧ Ψ Τ ϑ λ ϝ ϧ ϥ ϯ
    • αΠόʔΤʔδΣϯτ ٕज़ຊ෦ 

    αʔϏεϦϥΠΞϏϦςΟάϧʔϓ (௨শ SRG)
    • झຯ: Χϝϥ
    • ޷͖ͳ΋ͷ: TerraformɺPythonɺElasticsearchɺServerless ❤
    • ޷͖ͳAWSͷαʔϏε: ECS
    • ΞϧόΠτ 14ϲ݄ 2018೥ ৽ଔೖࣾ (ࣗশ4೥໨)
    AWA վળʢͦͷ1ɺͦͷ2ʣɺτϧςɺCARTEɺREQU ্ཱͪ͛αϙʔτ
    ݱ AWAɺλοϓϧ஀ੜɺCROSS MEɺͳͲ୲౰
    2

    View Slide

  3. αʔϏεϦϥΠΞϏϦςΟάϧʔϓ is Կ
    • ฐࣾʹ͓͚ΔϝσΟΞ؅׋ͱݺ͹ΕΔ࿮૊Έʹଐ͢ΔαʔϏεͷ

    ʮΠϯϑϥʯͳͲΛԣஅతʹݟ͍ͯΔ૊৫ (AbemaTVͱͦͷؔ࿈Λআ)
    యܕతͳΠϯϑϥ԰Ͱ͸ͳ͍
    SREͱDevOpsͱCREͬΆ͍໾ׂ΋Ռ͍ͨͯ͠Δ (ৄࡉ͸ͪ͜Β)
    3

    View Slide

  4. αʔϏεϦϥΠΞϏϦςΟάϧʔϓ is Կॲ
    4

    View Slide

  5. ࠓ೔࿩͢͜ͱ
    • λοϓϧ஀ੜͷҠઃ෺ޠ
    2೥ؒҎ্ʹΘͨΔετʔϦʔ
    • લ൒: ྺ࢙తܦҢɺҠઃ͋Δ͋Δ
    • ޙ൒: ECS EC2/FargateͷҠઃͱ͔ΑΓσΟʔϓͳ࿩
    ऴ൫ʹ࠷ۙग़ͨ৽ػೳΛࢼͯ͠Έͨ࿩΋͋ΔΑ
    5

    View Slide

  6. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    6

    View Slide

  7. λοϓϧ஀ੜ
    αʔϏε֓ཁ

    View Slide

  8. View Slide

  9. λοϓϧ஀ੜ is Կ
    • झຯͰͭͳ͕Δ࿀׆αʔϏε
    • ձһ਺400ສਓҎ্ɺϚονϯά਺͸Ԇ΂1ԯ2,000ສ૊
    • ࠃ಺࠷େن໛
    9

    View Slide

  10. https://tapple.me
    10

    View Slide

  11. ྺ࢙తܦҢ

    View Slide

  12. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    12

    View Slide

  13. ΦϯϓϨ࣌୅ͷߏ੒֓ཁ
    • Φϯϓr…ϓϥΠϕʔτΫϥ΢υ
    • GitHubͰίʔυ؅ཧɺJenkinsͰϏϧυˍσϓϩΠ
    • αʔόͷϓϩϏδϣχϯά͸Ansible
    • Node.js + MongoDB + Elasticsearch
    • ϩάपΓ͸ɺFluentd tail -> aggregator -> in-house analyticsج൫
    13

    View Slide

  14. 14

    View Slide

  15. ΦϯϓϨߏ੒ʹ͓͚Δ՝୊఺
    • ύϒϦοΫΫϥ΢υͰ͍͏ͱ͜ΖͷManagedͳαʔϏε͕গͳ͍
    ΄ͱΜͲͷίϯϙʔωϯτ͸ࣗ෼ͨͪͰ؅ཧ͠ͳ͚Ε͹ͳΒͳ͍ͨΊɺ؅ཧίετߴ
    • ΦʔτεέʔϦϯά͸ɺ૬౰࡞Γࠐ·ͳ͍ͱ࣮ݱ͕೉͍͠
    ίετ࠷దԽ͕Ͱ͖ͣɺৗʹ࠷େΩϟύγςΟ͕ىಈ͍ͯͯ͠ແବ
    ͦͯ͠࠷େΩϟύγςΟ͕มΘΕ͹…खಈ૿ઃ…
    • ࣗಈԽ͕೉͘͠ɺϗετো֐ͳͲ͕ى͖ΔͨͼʹखಈΦϖൃੜ
    • ϓϥΠϕʔτΫϥ΢υͷӡ༻ϧʔϧ্ɺϦʔδϣϯʹ͸EOL͕͋ΓҠઃൃੜ
    ౰࣌࢖͍ͬͯͨϦʔδϣϯͷEOL͕͍͖͍ۙͮͯͯͨ

    Disclaimer: ผʹϓϥΠϕʔτΫϥ΢υΛdisͬͯΔΘ͚͡Όͳ͍͔Βצҧ͍͠ͳ͍ͰΑͶͬ
    15

    View Slide

  16. AWSʹҠઃ͢Δ͜ͱʹͳͬͨཧ༝
    • ͳͥύϒϦοΫΫϥ΢υ
    ΦʔτεέʔϦϯά͍ͨ͠
    ϚωʔδυαʔϏεͰ؅ཧɾӡ༻ίετԼ͛ͯ։ൃʹूத͍ͨ͠
    • ͳͥAWS
    ϚονϯάαʔϏεͷதͰ͸͢ͰʹAWSͷར༻࣮੷͕͋ͬͨ
    ϚονϯάαʔϏεΛAWSʹἧ͍͍ͨͱ͍͏૝͍΋
    ϝσΟΞ؅׋ͷதͰAWSͷ஌ݟ͕ଟ͔ͬͨ
    ౰࣌͸།Ұ౦ژϦʔδϣϯΛఏڙ͍ͯͨ͠
    16

    View Slide

  17. ҠઃܭըΛ
    ཱͯΑ͏

    View Slide

  18. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    18

    View Slide

  19. Ҡઃޙͷߏ੒ v.1: EC2
    • ΞϓϦ૚͸ͦͷ··
    Ծ૝αʔό EC2 Πϯελϯεͱ͍͚ͯ࣋ͬͯͩ͘͠
    • MongoDBͱElasticsearch΋ͦͷ··͍࣋ͬͯ͘
    MongoDB΁ͷΞΫηε͸ɺ֤ΞϓϦαʔόʹ৐͍ͬͯΔmongosΛϓϩΩγʹ࢖͏
    Elasticsearch͸ɺALBͰΞΫηεΛ෼ࢄ
    • RedisɺMySQL͸ElastiCacheɺAuroraʹ
    • nginxΛALBʹϦϓϨΠε
    • ੩తίϯςϯπͷ഑৴͸ɺS3+CloudFront
    19

    View Slide

  20. 20

    View Slide

  21. ҠઃϩʔυϚοϓ
    1. σʔλ૚͸ϝϯςΛೖΕͨঢ়ଶͰҠઃ͠ɺͦΕҎ֎͸ΦϯϥΠϯͰҠઃ
    σʔλ૚ͷҠઃͷͨΊͷϝϯς͸௒ਖ਼͍͠൑அɺঝ୚ͨؔ͠܎ऀ֤Ґʹѹ౗తײँ
    ϓϥΠϕʔτΫϥ΢υ͔Βσʔλ૚΁ͷΞΫηε͸ɺઐ༻ճઢܦ༝
    2. ΞϓϦ૚ɺݕূ͠ͳ͕ΒঃʑʹҠઃ͍ͯ͘͠
    21

    View Slide

  22. ࣮ߦ
    ͢Δͧ

    View Slide

  23. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    23

    View Slide

  24. εςοϓ2: ΞϓϦ૚ͷҠઃ…ʁ
    • EC2 AutoscalingͷͨΊͷɺAMIϏϧυ༻Packer࡞ΓࠐΈɺݕূΛ…

    ͢Δ…ͭ΋Γ…͕ͩͬͨ…
    Ω < ͔ͤͬ͘ͷҠઃͳΒɺΞʔΩςΫνϟมߋ΋͍ͨ͠Ͷͬ
    Ω < ϚΠΫϩαʔϏεԽͰ͖ͨΒϦϦʔεͷ୯ҐΛখ͘͞Ͱ͖Δ͠੾Γ໭͠ͱָ͔ͩ͠
    Ω < Dockerfile࡞ͬͨͧ
    Ω < KubernetesͰͱΓ͋͑ͣಈ͍ͨͧ
    Ω < KubernetesͰߦ͜͏ͥ
    Ω < ͔ͤͬͩ͘͠ଞ΋վળ͍ͯ͜͠͏ Μɺɺ͋Δ͐ʁɹҰ୴EC2Ͱߦ͘Μ͡Όͳ͔͚ͬͨͬ
    24

    View Slide

  25. Ҡઃޙͷߏ੒ v.2: Kubernetes
    • σʔλ૚͸มΘΒͣ
    • NLBΛIngestͱͯ͠࢖༻
    • kops + helm
    • ap-northeast-1 3 AZ (a/c/d)
    • ASGͰworkerϊʔυΛ؅ཧɺΦʔτεέʔϦϯά
    25

    View Slide

  26. 26

    View Slide

  27. Ҡઃޙͷߏ੒ v.2: ϩάपΓ
    • APIαʔϏεͳͲͰ૬৐Γfluentdίϯςφఆٛ
    • ڞ༗volume͔Βtail Kinesis Lambda in-houseͳanalytics
    KinesisͰόοϑΝϦϯάɺin-houseͳج൫͕ࢮΜͰ͍ͯ΋ɺ

    ʮKinesis Lambdaʯͷ෦෼Ͱ͍͍ײ͡ʹϦτϥΠ͞ΕΔ
    ৄࡉ: KinesisͱLambdaͰͭ͘ΔServerlessͳϩάج൫ @ AWS DevDay Tokyo 2019
    27

    View Slide

  28. 28

    View Slide


  29. Ω < ΫϥελͱVPCͱαϒωοτपΓ࡞ͬͨͧ
    Ω < manifestΧΩΧΩͯ͠Δͧ
    Ω < Α͘ݟͨΒͪΌΜͱಈ͍ͯͳ͍ͳ
    Ω < Kubernetesͷόά͔ͳ
    Ω < Dockerͷόά͔ͳ
    Ω < ͋ΕΕ
    29

    View Slide

  30. Ҡઃ͕׬ྃ͠ͳ͍
    ··1೥͕ܦͭ

    View Slide

  31. ͦͯ࣌͠ݶര஄͕
    େരൃ

    View Slide

  32. ࣌ݶര஄ #ͱ͸
    • ࣌ݶര஄ = ઐ༻ճઢ
    • ϙΠϯτ:
    1. σʔλ૚΁ͷΞΫηε͕Ϋϥ΢υΛލ͍Ͱ͍ͨ
    2. ઐ༻ճઢ͕อূͳ͠Ͱఏڙ͞Ε͍ͯͨ 

    (౰࣌͸ SLA ͕ఏڙ͞Ε͍ͯͳ͔ͬͨ)
    • Ҡઃ͕׬ྃ͠ͳ͍ঢ়ଶͰ1೥͕ܦͬͨ
    • ͦͯ͠ࣄ͕݅ى͖ͨ
    32

    View Slide

  33. ࣌ݶര஄େരൃࣄ݅
    • ઐ༻ճઢͷো֐ͰશAPI஗Ԇര૿ɺDBෛՙߴಅɺ౤ߘ؂ࢹࣦഊɺ

    ΞϓϦͷΤϥʔදࣔʹΑΔେྔ࠶ϩάΠϯ
    ϦʔνҰൃ໳લਗ਼ࣗ໚࿨ฏ࿨υϥ2ɺ௓ຬɺ௓Μͩ…
    • શମϝϯςφϯεΛೖΕͯɺઐ༻ճઢఏڙݩʹ໰͍߹Θͤͭͭ͠

    ༷ࢠΛݟΔ͜ͱʹ
    • ࡾ࣌ؒޙɺճઢ෮چɺϝϯς໌͚
    33

    View Slide

  34. ࣌ݶര஄େരൃࣄ݅ɺೋճ໨
    • ೋ೔࿈ଓͰ·ͨಉ͡ো֐͕ى͖Δ
    • ϝϯςΠϯ͔Β3࣌ؒޙɺ෮چͷݟࠐΈ͕ͳ͘ͳΓɺ

    ΋͏Ҡઃͯ͠͠·͑͹͍͍͡Όͳ͍͔આΛࢲ͕ޠΓʹ͍͘
    • ආ೉ઌͷ४උ։࢝
    • ϝϯςΠϯ͔Β5࣌ؒޙɺճઢ͕෮چɺϝϯς໌͚
    • ࠓ౓͸ɺ5࣌ؒҎ্ͷϝϯςφϯε
    • ·͍ͨͭى͖Δ͔Θ͔Βͳ͍͠ා͍͔Βɺආ೉ઌͷ४උΛଓߦ
    • ͦͷ৔Ͱ࠷௿ݶಈ͘؀ڥͷߏஙͱσϓϩΠϑϩʔͷཱ֬·Ͱ׬ྃͯ͠͠·͏
    ͜ͷ͋ͱ໓஡ۤ஡͓ञΛҿΈ·ͨ͠
    34

    View Slide

  35. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    35

    View Slide

  36. View Slide

  37. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    37

    View Slide

  38. ҠઃϩʔυϚοϓ
    • 3ϑΣʔζͰ࣮ࢪ
    1. ۓٸEC2Ҡઃ: ͱΓ͋͑ͣ ΦϯϓϨ EC2 (࣌ؒՔ͗)
    • ϢʔβʹͰ͖Δ͚ͩૣ͘ΑΓ҆ఆͨ͠؀ڥͷఏڙ͢ΔͨΊ
    2. EC2 ECS (EC2 Type)
    3. ECS (EC2 Type) ECS (Fargate Type)
    • ͙͢ʹ͸Fargateʹ͠ͳ͔ͬͨཧ༝:
    • Fargateͷ஌ݟ͕গͳ͔ͬͨ
    • ౰࣌଍Γͳ͍ػೳ͕͋ͬͨΓɺߴ͔ͬͨΓʢ࠷ۙ൒ֹʹͳͬͨΓػೳ͕૿͑ͨΓʣ
    • λοϓϧϝϯόʔͷ஌ࣝΛਂΊΔͨΊʹ΋ɺֶ͋͑ͯशͷͨΊEC2ʹ
    • Fargateͱ͸ҟͳΓɺEC2Ͱ͸docker exec -it ͳͲͰίϯςφʹϩάΠϯͯ͠ௐࠪɺσόοά͕Ͱ͖Δ
    38

    View Slide

  39. KubernetesΛఘΊͯECSΛ࠾༻ͨ͠ཧ༝
    • KubernetesͰؤுͬͯݕূ͍͕ͯͨ͠ɺ

    1೥͔͔ͬͯ݁ہຊ൪ʹग़ͤΔϨϕϧʹ͸ࢸΒͳ͔ͬͨʢׂѪʣ
    • ECSͷํ͕AWSͷଞͷαʔϏεͱͷ਌࿨ੑ͕ߴ͍
    • Kubernetes͡Όͳ͍ͱ͍͚ͳ͍ϢʔεέʔεͰ͸ͳ͔ͬͨ
    Service MeshͷΑ͏ͳෳࡶͳߏ੒Ͱ͸ͳ͘ɺಠཱͨ͠API͍͔ͭ͘ͱόον͍͔͚ͭͩ͘
    • SRGͱͯ͠΋ฐࣾϝσΟΞࢠձࣾͱͯ͠΋ɺECSͷϊ΢ϋ΢͕ଟ͍
    ࣗݾ঺հʹ͋ΔϦϯΫࢀর
    ߏஙͷΈͰ4೔ؒ + ΦʔτεέʔϦϯάͷઃܭ 1िؒʢׂ߹มߋͱͱ΋ʹ࣮ࢪʣ
    39

    View Slide

  40. ۓٸEC2Ҡઃ: ϩʔυϚοϓ
    • ߏ੒Λେ͖͘ม͑ͣɺαʔόΛͦͷ··EC2Ͱ͍࣋ͬͯ͘
    QCɺෛՙࢼݧͳͲΛ௨ͯ͡Ϣʔβʹग़ͯ͠΋͍͍͜ͱ͕ূ໌Ͱ͖Δ·Ͱ͸੾Γସ͑ͳ͍
    ͨͩ͠ɺ΋͠·ͨେো֐͕ى͖ͨ৔߹ɺ৽؀ڥΛආ೉ઌͱͯ͠ར༻
    • ίϯςφԽߏ੒Λߏஙɺςετ
    • EC2΁ͷ੾Γସ͑ޙɺίϯςφԽߏ੒ʹ੾Γସ͍͑ͯ͘
    KubernetesΛఘΊͯɺECSΛ࠾༻ʢޙʹղઆʣ
    KubernetesݕূΛͨ͠ͱ͖ͷDockerfile΍VPCपΓͳͲྲྀ༻
    40

    View Slide

  41. ۓٸEC2Ҡઃ: ߏங
    • Packerͱ͔Φʔτεέʔϧͱ͔Ұ୴ؾʹͤͣߏங
    • ര଎ͰTerraformͰEC2ܥɺηΩϡϦςΟάϧʔϓɺALBपΓΛ੔උʢԶʣ
    • AnsibleΛमਖ਼͠ͳ͕Βྲྀ͢ʢλοϓϧϝϯόʔʣ
    • ΞϓϦσϓϩΠɺಈ࡞֬ೝɺमਖ਼ͷ܁Γฦ͠
    • ߹ܭ12࣌ؒͰͱΓ͋͑ͣಈ͘΋ͷ͕׬੒
    • ·ͨো֐͕ى͖ͨΒආ೉ઌͱͯ͠࢖͏
    41

    View Slide

  42. ۓٸEC2Ҡઃ: ࠷ऴ֬ೝͱ੾Γସ͑
    • ߏஙͷཌ೔: QCɺϦάϨογϣϯςετɺෛՙࢼݧʹ߹֨
    • ҰिؒΛ͔͚ͯঃʑʹϦΫΤετΛ৽͍͠؀ڥʹدͤΔ
    • tapple.me AϨίʔυΛALIASͰRoute 53 Traffic Flowʹ޲͚Δ
    • ϓϥΠϕʔτΫϥ΢υLBͱAWS ALBͷׂ߹ΛweightͰ੍ޚ
    • Θ͔Γ΍͘͢͢ΔͨΊʹɺweightͷ߹ܭ͕100ʹͳΔΑ͏ʹઃఆ
    • ࣮ࡍͷΞΫηεϩάͰׂ߹Λ֬ೝ
    42

    View Slide

  43. 43

    View Slide

  44. 44

    View Slide

  45. Route 53 Traffic Flowॴײ
    • weightͷมߋΛద༻͔ͯ͠Βɺ࣮ࡍͷϦΫΤετͷׂ߹͕มΘΔ·Ͱ ~ 5෼
    • DNSΩϟογϡ͕มʹӨڹ͢Δ͜ͱͳ͘ɺ

    ࢦఆͨ͠weightͱ࣮ࡍͷΞΫηεͷׂ߹͸2%΄Ͳͷޡࠩ
    • Ϋϥ΢υҠઃϕετϓϥΫςΟε͗͢Δ…
    • KibanaͰϦΞϧλΠϜͰׂ߹͕มΘΔͷΛݟΔͷָ͍͠…
    45

    View Slide

  46. 2िؒͰEC2Ҡઃ׬ྃ
    ര଎

    View Slide

  47. ͋Δ΂͖࢟΁

    View Slide

  48. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    48

    View Slide

  49. Ҡઃޙͷߏ੒ v.3: ECS EC2
    ԦಓͳECS EC2:
    • ALB
    • ECS Service (EC2 Type)
    ͨͩ͠ɺBatchܥ͸શ෦Fargate (ޙʹղઆ)
    • EC2 Auto-Scaling Group (ΫϥελͷόοΫΤϯυ)
    49

    View Slide

  50. 50

    View Slide

  51. ECS EC2Ҡઃ: ECSΫϥελ
    • ECSόοΫΤϯυ͸ɺEC2 + Autoscaling Group
    • VanillaͳAmazon Linux (ECS-optimized) AMIͰىಈɺ

    user_dataͰͪΐͬͱͨ͠ॳظԽ͚࣮ͩߦ
    • εέʔϦϯάϝτϦοΫε͸ɺECS ClusterͷCPUReservationͷMaximum
    51

    View Slide

  52. ECS EC2Ҡઃ: APIܥ
    • API༻αʔϏε͸EC2 Type
    • εέʔϦϯάϝτϦοΫε͸ɺECS ServiceͷCPUUtilizationͷMaximum
    • Dockerfile౳ɺΞϓϦΛίϯςφͰಈ͔ͨ͢Ίͷ΋ͷҰࣜ͸ɺ

    Kubernetesݕূͷͱ͖ͷ΋ͷΛྲྀ༻
    • ϚΠΫϩαʔϏεʹ෼͚Δͱ͜Ζ͸Ұ୴΍Βͳ͍
    ཧ༝ɿALBΛར༻͍ͯ͠ΔͨΊɺޙ͔Βಛఆͷύε͚ͩผͷαʔϏεʹϧʔςΟϯάͤ͞Δ

    ͜ͱ͕Մೳɺ͙͢΍Δඞཁ͕ͳ͍
    52

    View Slide

  53. ECS EC2Ҡઃ: όονܥ
    • όον༻αʔϏε͸Fargate Type
    • ݁ߏසൟʹ࣮ߦ͞ΕΔόον͕͋ͬͯɺ

    EC2Ͱಈ͔͢ͱASGͷΦʔτεέʔϦϯάͷઃܭ͕໘౗ͳ͜ͱʹͳΔ…
    όον༻ͷΫϥελΛ࡞ΔͱϦιʔε͕ແବʹͳΔ͠ɺ

    ࡞Βͳ͍ͱεέʔϦϯάΞΫγϣϯͷढ़හੑ…
    • CloudWatch Eventsͷcron͕ࣜͪΐͬͱಛघͳͨΊɺ

    طଘͷcronࣜΛͪΐͬͱ͍͡Δඞཁ͕͋ͬͨ
    • Baikonur OSS fargate-scheduled-task ϞδϡʔϧΛར༻
    53

    View Slide

  54. ECS EC2Ҡઃ: DB
    • MongoDBɺElasticsearch͸ʢ΄΅ʣͦͷ··
    • EC2ɺϓϥΠϕʔτΫϥ΢υͰΠϯελϯε͝ͱʹىಈ͍ͤͯͨ͞

    mongos͸ɺઐ༻EC2 x 3୆ʹͨ͠
    • mongos͕ݱΕͨΓফ͑ͨΓ͢Δͱɺconfig DBʹ΋͏͍ͳ͍mongosͷ৘ใ͕ཷ·Δ
    • ͢Ͱʹ͍ͳ͍mongosʹmongoc͕ping͠Α͏ͱ͍ͯ͠Δʁ
    • ޙʹ૿ઃ͞Ε…ͦͯ͠·ͨ૿ઃ͞Εͨ…
    54

    View Slide

  55. ECS EC2Ҡઃ: ϩάपΓ
    • Kubernetes༻ͷ΋ͷͱҰॹɺ૬৐Γfluentdίϯςφར༻
    • ڞ༗volume͔Βtail aggregator Kinesis Lambda 

    in-houseͳanalytics
    KinesisసૹϓϥάΠϯͱϩάʹର͢Δཁ݅ͷ౎߹Ͱ࠷ऴతʹfluentd aggregatorḉര஀
    55

    View Slide

  56. ECS EC2Ҡઃ: Cluster ASGͷεέʔϦϯάઃܭ
    • ΦʔτεέʔϦϯάͱ͍͏֓೦͕ECSҠઃ·Ͱ͸ଘࡏ͠ͳ͔ͬͨ
    • Ұ୴ɺෛՙͷ͔͔Γํ౳͕ࣅ͍ͯΔαʔϏεͷᮢ஋ΛύΫͬͨ
    • 1िׂ͔͚ؒͯ߹Λมߋ͢ΔΞϨͱಉ࣌ʹɺ

    εέʔϦϯάΛݟकΓͭͭࡉ͔͍ௐ੔Λߦͳͬͨ
    • εέʔϦϯάͷௐ੔Λߦͳͬͨࡍͷߟ͑ํΛ͝঺հ
    56

    View Slide

  57. ECS EC2Ҡઃ: Cluster ASGͷεέʔϦϯάઃܭ
    • CPUReservationͷAverage͕75%-85%ͷؒΛӭ͙Α͏ʹɺ

    Step ScalingΛखͰௐ੔
    εύΠΫɺϐʔΫΛܴ͑Δͱ͖ʹɺΫϥελͷΩϟύ͕

    Taskͷ഑ஔͷ଍ΛҾͬுΒͳ͍Α͏ʹஸೡʹௐ੔
    57

    View Slide

  58. Step ScalingɺͩΔ͍Ͱ͠ΐʁ
    • ઃܭ͕ͩΔ͍͠೉͍͠
    • ͦͯ͠ରࡦΛ͠ͳ͍ͱASG scale-in࣌ʹੜ͖͍ͯΔίϯςφ͕ࡴ͞ΕΔ
    • ରࡦ: EC2 Auto-ScalingͰECS૊ΜͰ·͔͢ʁΤϥʔϨʔτߴ͍Ͱ͔͢ʁٹͬͯ͋͛ͯ΋͍͍Ͱ͔͢ʁ
    • ECS Cluster Auto Scalingͱ͍͏ਆػೳ͕Ͱͨʂ
    • ͍͍ײ͡ʹεέʔϦϯάͯ͘͠ΕΔɺStep Scaling͢Δඞཁͳ͠
    • CPUɺϝϞϦɺENIͳͲɺ͍ͣΕ͔ҰͭͷΩϟύγςΟͷछྨͰ͸ͳ͘ɺ

    ্ཱ͍ͪ͛ͨTask͕࣮ࡍ্ཱͪ͛ΒΕΔͷ͔ͱ͍͏γϛϡϨʔγϣϯͰΩϟύෆ଍Λಛఆ
    • ىಈ͍ͯ͠Δίϯςφͷ͋ΔEC2Λࡴ͞ͳ͍ઃఆ΋Ͱ͖Δɺൿ఻ͷλϨͳ͠Ͱಈ͘ʂ
    • ౰࣌ͳ͔ͬͨͷͰͱͯ΋ͭΒ͔ͬͨɺECSͷதͷਓ·ͫ͋Γ͕ͱ❤
    58

    View Slide

  59. ECS EC2Ҡઃ: ECS ServiceͷεέʔϦϯάઃܭ
    • CPUUtilizationͷMaximum͕40%-75%ͷؒΛӭ͙Α͏ʹɺ

    Step ScalingΛखͰௐ੔ɺ֤ίϯςφͷෛՙΛ֬ೝ
    ͜Ε͙Β͍ͷઃఆͰɺ֤ίϯςφ͕݈શతͳෛՙ (~50%) Λड͚Δ
    ίεύͱϨΠςϯγͷόϥϯγϯά
    59

    View Slide

  60. Step ScalingɺͩΔ͍Ͱ͠ΐʁ
    • ઃܭ͕ͩΔ͍͠೉͍͠
    • Target Tracking Scaling (for ECS)ͱ͍͏΋ͷ͕ग़ͨ
    • ʮCPU࢖༻཰͕ฏۉͰ60%͙Β͍ʹͳΔΑ͏ʹͯ͠ɺϤϩʯͱ͍͏ઃఆ͚ͩͰ

    ͋ͱ͸ΑΖ͘͠΍ͬͯ͘ΕΔ
    • ౰੍࣌໿͕͋ͬͯಋೖ͕೉͔ͬͨ͠
    • ࢖͑ΔϝτϦΫε͕ݶΒΕ͍ͯͨ…هԱ
    • ޙʹෛՙࢼݧ؀ڥͰࢼ͢΋ɺզʑͷϢʔεέʔεʹ߹Θͣ
    • Step Scaling͔Β͍·ͩʹಀ͛ΒΕ͍ͯͳ͍
    60

    View Slide

  61. ECS EC2Ҡઃ: ECS ServiceͷεέʔϦϯάઃܭ
    61
    ίϯςφͷCPU࢖༻཰͕40-60%

    ͙Β͍ʹͳ݈ͬͯͯ߁తͳෛՙ

    View Slide

  62. ECS EC2Ҡઃ: ࠷ऴ֬ೝͱ੾Γସ͑
    • ۓٸEC2ҠઃͱҰॹͩΑ
    • QCɺϦάϨογϣϯςετɺෛՙࢼݧʹ߹֨͢Δ·Ͱमਖ਼࣮ࢪ
    • ҰिؒΛ͔͚ͯঃʑʹϦΫΤετΛدͤΔ
    • tapple.me AϨίʔυΛALIASͰRoute 53 Traffic Flowʹ޲͚Δ
    • EC2؀ڥͷALBͱECS؀ڥͷALBͷׂ߹ΛweightͰ੍ޚ
    • Θ͔Γ΍͘͢͢ΔͨΊʹɺweightͷ߹ܭ͕100ʹͳΔΑ͏ʹઃఆ
    • ࣮ࡍͷΞΫηεϩάͰׂ߹Λ֬ೝ
    62

    View Slide

  63. ECS EC2Ҡઃ: Ҡઃޙͷ͋Ε͜Ε
    • ઐ༻ճઢ͕ͳ͘ͳͬͨ͜ͱͰɺAWSྉ͕͔ۚͳΓ҆͘ͳͬͨ
    • ϦΫΤετΛฦͨ͢Ίͷ௨৴͸AWS಺௨৴ͷΈʹͳͬͯɺ

    API͕ര଎ʹͳͬͨ
    ࣮૷͸΋ͱ΋ͱͱͯ΋༏लͩͬͨ
    ϨΠςϯγ͕ฏۉͰ120ms଎͘ͳΓɺmsೋܻ୆΁
    63

    View Slide

  64. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ۓٸҠઃ͔Β͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    64

    View Slide

  65. View Slide

  66. Fargate!
    • ECS EC2Ҡઃ͔Β΄΅1೥͕ܦͬͨ
    • EC2ʹϩάΠϯͯ͠docker execͱ͔͢Δ͜ͱ͕΄΅ͳ͘ͳͬͯ
    ΞϓϦέʔγϣϯ΋ίϯςφʹదͨ͠ܗʹ͍͖ۙͮͯͨ
    • ϝϯόʔ΋ECSɺίϯςφΛ࢖ͬͨΞϓϦέʔγϣϯͷӡ༻ʹ׳Ε͖ͯͨ
    • ͦͯ͠ECS EC2ʹର͢Δෆຬ͕ཷ·͖ͬͯͨ
    • EC2 ASGͷεέʔϧΛ଴ͨͳ͚Ε͹ͳΒͳ͍ɺͦͷͨΊεέʔϦϯάɺσϓϩΠ͕஗͍
    • Ωϟϯϖʔϯ౳Λ࣮ࢪ͢Δࡍʹscheduled scaling͕ඞཁɺͦͷϙϦγʔΛઃܭɾܭࢉ͢Δͷ͸େม
    • AMIߋ৽͢ΔͨͼʹASGͷEC2ΠϯελϯεΛస͕͢ͷ͕େม
    • Α͠ɺFargateԽ͠Α͏ʂ
    66
    ࢲ͸࠷ॳ͔Βɺ࠷ऴతʹFargateʹ͢Δͭ΋Γͩͬͨ

    View Slide

  67. 67

    View Slide

  68. Fargateͷίετʹؔͯ͠
    • Ϧιʔείετ͸On-demand EC2 (RIͳͲͷׂҾͳ͠) ͱൺֱͯ͠

    Ή͠Ζ҆͘ͳΔ
    • EC2ͷόοΫΤϯυΠϯελϯεΛASGͱ͔Ͱ༻ҙ͢ΔࡍʹɺCPU/ϝϞϦͷۭ͖͕ৗʹ

    15-20%ʹͳΔΑ͏ʹ͍ͯ͠Δ͔Βʢεέʔϧͷढ़හੑʣ
    • ͦͯ͠൒ֹʹ஋Լ͛͞ΕͨΓͨ͠
    • ͞ΒʹɺEC2ϨΠϠͷΦϖϨʔγϣϯɾ؅ཧίετ͕࡟ݮ
    • ηΩϡϦςΟύονɺDockerͷύον࡞ۀ͞Α͏ͳΒ
    • ͍҆͠ɺ͍͍͡ΌΜ
    68

    View Slide

  69. FargateͷΠΠͱ͜Ζ
    • Firecrackerͷొ৔Ͱىಈ͕଎͘ͳͬͨ
    • ENIͷallocation͸ಛʹੲͱൺ΂ͯ଎͘ͳ͍ͬͯΔ
    • EC2ͷΩϟύ଴ͭඞཁͳ͍͠ɺىಈ΋଎͍ͷͰσϓϩΠɺ੾Γ໭͠ɺεέʔϦϯάͷߴ଎Խ
    • EC2Ͱͷ՝୊: bridge network͕஗͍͕ɺhostΛ࢖͏ͱϙʔτ͕େมͳ͜ͱʹ
    ͳΔ͔ΒawsvpcΛݕ౼͕ͨ͠ɺΠϯελϯελΠϓͱαΠζ͋ͨΓͷENI

    ෇༩্ݶ਺͕ͱͯ΋গͳ͍
    • ࠷ۙENI TrunkingͰ؇࿨͞Ε͍ͯΔ͕ͦΕͰ΋଍Γͳ͔ͬͨ
    • FargateΛ࢖͏ͱ͖awsvpc͔͠࢖͑ͳ͍͕ɺͦΕͰ͍͍
    • Docker Links࢖ͬͯΔϚφʔͷΑΖ͘͠ͳ͍Task͸ɺFargateԽ (+awsvpcԽ) Ͱͳͤ͘Δ
    69

    View Slide

  70. Fargateͷͪΐͬͱ໘౗ͳͱ͜Ζ
    • TaskͷCPUͱϝϞϦΛܾΊΒΕͨ૊Έ߹Θͤͷத͔Βબ͹ͳ͍ͱμϝ
    • ૊Έ߹Θͤ਺΋ଟ͍͠෯޿͘ઃఆͰ͖Δ͕ࣗ෼ͷχʔζʹtailoredͳͪΐ͏Ͳ͍͍Ωϟύʹ͸
    ͳΒͳ͍
    • AWS͞Μ͕ΫϥελΛ؅ཧͯ͘͠ΕͯΔ͠ɺڐͤΔ
    • Docker VolumeͰͷϑΝΠϧڞ༗͸ɺ1 volumeͰhard limit 4GB
    70

    View Slide

  71. ͍ͭͰʹ΍Δ͜ͱʹͳͬͨ͜ͱ
    • ͔ͤͬ͘ServiceɺALBΛ࡞Γ௚͔͢Β΍Γ͍ͨͳͱࢥͬͨ͜ͱ
    • ALB Least Outstanding RequestsΞϧΰϦζϜͷݕূ
    • Internal LB༻ͷtarget groupΛServiceʹඥ෇͚Δ
    • Internal/Global༻ͷTGΛҰͭͷServiceʹඥ෇͚
    • લ͸ෳ਺TGͷඥ෇͚͕Ͱ͖ͳ͔ͬͨͨΊInternalઐ༻Service…
    71

    View Slide

  72. ALB Least Outstanding Requests
    Ͳ͕ͬͪRound-RobinͰͲ͕ͬͪLORͩͱࢥ͍·͔͢ʁ
    72

    View Slide

  73. ALB Least Outstanding Requests
    ਖ਼ղɿ
    Round-Robin LOR
    ͙͢໭͢͜ͱʹͳͬͨ
    73

    View Slide

  74. εέʔϦϯάपΓͷௐ੔
    • Datadog APMͷϨΠςϯγɺFargateͷCPU࢖༻཰͕ಉఔ౓ʹͳΔΑ͏ௐ੔
    • APMͰͷEC2/Fargateͷ൑ผ͸DD_ENV
    • DatadogϝτϦΫε͸໊લۭؒ(docker.* vs ecs.*)ͱTask Family໊
    • ϩά͸ɺplatform = {“ec2”, “fargate”} ͱ͍͏ϑΟʔϧυ௥Ճ
    ո͍͠ ո͍͠ چ؀ڥಉ༷
    Ϥγʂ



    چ؀ڥϨΠςϯγ ৽؀ڥϨΠςϯγ
    74

    View Slide

  75. ׂΓ߹͍มߋͷ֬ೝ on Kibana
    • platform = {“ec2”, “fargate”} ͳϑΟʔϧυͰ

    ΞΫηεϩά͔ΒϦΞϧλΠϜͰՄࢹԽ
    75

    View Slide

  76. ׂ߹Λ֬ೝɺϤγʂ Τϥʔ֬ೝɺϤγʂ
    76

    View Slide

  77. LOR੾Γ໭͠ Ϥγʂ
    ϨΠςϯγ Ϥγʂ
    ΤϥʔϨʔτ Ϥγʂ
    εέʔϦϯά Ϥγʂ
    77

    View Slide

  78. Ҡઃ׬ྃ

    View Slide

  79. λοϓϧͷΠϯϑϥपΓͷ͜Ε͔Β
    • Fargate SpotΛ࢖ͬͨover-provisioningͰٸͳεύΠΫʹରԠ
    • FireLens + KinesisͳϩάपΓ
    • FireLens΄Μͱ͍͢͝ΑɺϩάपΓͷ՝୊͸(container nativeͳܗͰ)େ఍ͳΜͱ͔ͳΔ
    • ͨͩ͠ɺloggingDriverࢦఆʹͳΔͷͰCloudWatch Logsʹ΋FireLensʹ΋ྲྀ͢ͷ͸೉͍͠
    • ࠓޙ͜Ε͕վળ͞ΕΔ͜ͱΛفΓͳ͕ΒFeature Request౤͛ ʢͬ'-')‸ = ͟
    ͟͞͞ŲƄőƃɹ
    • ΧελϜεέʔϦϯά
    • CPUͰ͸ͳ͘ΞΫηε਺ɺͳͲͰΠΠײ͡ͰŧŔŕŰƅźͳεέʔϥΛ࡞͍ͬͯΔ
    • ϝτϦΫεΛ଴ͨͣʹ1෼ະຬͷϦΫΤετมԽʹରԠ͠ϨΠςϯγΛ͞Βʹ҆ఆԽ
    • @pahudnet ͞ΜͷŧŔŕŰƅźεέʔϥΛࢀߟʹͭͭ͠
    • ΧελϜσϓϩΠ
    Deployment Controller: EXTERNALͰTask Set APIΛ࢖ͬͨߴ౓ͳαϜγϯάΛ࡞͍ͬͯΔ
    79

    View Slide

  80. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ۓٸҠઃ͔Β͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    80

    View Slide

  81. ॴײ
    • ॊೈͰ͋Ε
    • ౰ॳͷ໨తΛ๨ΕΔͳ
    • ϩʔυϚοϓΛҾ͘͜ͱ͕େࣄ
    • ເΛݟΔ͜ͱ͸ѱ͍͜ͱ͡Όͳ͍͕ɺ΍Δ΂͖͜ͱ͸͔ͬ͠Γɺ

    ॱ൪ʹ΍͍ͬͯ͘΂͠
    • Ϣʔβͷ͜ͱΛߟ͑ͳ͕ΒɺϢʔβମݧΛ࠷༏ઌʹͯ͠࢓ࣄʹऔΓ૊Ή΂͠
    • ECS͸͍͍ͧʢAWS Managedͱͷ਌࿨ੑͱֶशۂઢʣ
    • Fargate͸͍͍ͧʢόοΫΤϯυͷ͜ͱΛ๨ΕͯαʔόϨεʹಀ͍͛ͨਓੜʣ
    81

    View Slide

  82. એ఻
    82
    ← https://baikonur.dev 

    ʹԿ͔Λग़͠·ͨ͠

    View Slide

  83. बۀܕΠϯλʔϯγοϓɺ௨೥ืूதʂ
    • ΤϯτϦʔక੾: Πϯλʔϯγοϓ։݄࢝ͷ3ϲ݄લͷ݄຤
    • ৄࡉ: https://www.cyberagent.co.jp/careers/students/event/detail/id=24422
    83

    View Slide

  84. ࠷ޙʹ΋͏Ұճʂ

    View Slide

  85. 85
    ͜ΕΛ

    View Slide

  86. 86
    ͜͏ͯ͠

    View Slide

  87. 87
    ͜͏ͯ͠

    View Slide

  88. 88
    ͜͏͡Όʂ

    View Slide

  89. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide