タップル誕生: オンプレからAWS Fargateまで @ JAWS DAYS 2020

PDF版: https://share.getcloudapp.com/E0uqmebA (リンク踏みたい方向け)

本セッションでは、タップル誕生というサービスのオンプレ環境からAWS Fargateになるまでのストーリーと技術的なチャレンジについて紹介している。歴史的背景、移設前の構成を説明し、EC2、ECS/EC2、ECS/Fargateに継続的に、段階的にアーキテクチャを変えて行ったことについて解説する。

トークの後半では、Fargateのメリットデメリット、ECSの落とし所、スケーリング設計に役立つアドバイス、クラウド移設で気を付けるべきこと、ECSの最新機能の検証結果についても触れる。

Speaker Profile:

株式会社サイバーエージェント
Torgayev Tamirlan 氏

株式会社サイバーエージェント 技術本部 サービスリライアビリティグループ所属 クラウド技術アドバイザ / 2017年~内定者アルバイトを経て、2018年に新卒入社 / Kinesis、ECS、Lambda、Elasticsearch、Terraformを専門領域とし、複数の新規サービス立ち上げや、AWA・タップル誕生などのシステム改善に従事している

Transcript

1. λοϓϧ஀ੜ:
   ΦϯϓϨ͔ΒAWS Fargate·Ͱ
   גࣜձࣾαΠόʔΤʔδΣϯτ
   ٕज़ຊ෦ αʔϏεϦϥΠΞϏϦςΟάϧʔϓ
   ϓϩάϥϛϯάϠΫβ (@prog893)
   

   View Slide

2. ࣗݾ঺հ
   • Torgayev Tamirlan / ϓϩάϥϛϯάϠΫβ / @prog893
   τ ϧ Ψ Τ ϑ λ ϝ ϧ ϥ ϯ
   • αΠόʔΤʔδΣϯτ ٕज़ຊ෦ 

   αʔϏεϦϥΠΞϏϦςΟάϧʔϓ (௨শ SRG)
   • झຯ: Χϝϥ
   • ޷͖ͳ΋ͷ: TerraformɺPythonɺElasticsearchɺServerless ❤
   • ޷͖ͳAWSͷαʔϏε: ECS
   • ΞϧόΠτ 14ϲ݄ 2018೥ ৽ଔೖࣾ (ࣗশ4೥໨)
   AWA վળʢͦͷ1ɺͦͷ2ʣɺτϧςɺCARTEɺREQU ্ཱͪ͛αϙʔτ
   ݱ AWAɺλοϓϧ஀ੜɺCROSS MEɺͳͲ୲౰
   2
   

   View Slide

3. αʔϏεϦϥΠΞϏϦςΟάϧʔϓ is Կ
   • ฐࣾʹ͓͚ΔϝσΟΞ؅׋ͱݺ͹ΕΔ࿮૊Έʹଐ͢ΔαʔϏεͷ

   ʮΠϯϑϥʯͳͲΛԣஅతʹݟ͍ͯΔ૊৫ (AbemaTVͱͦͷؔ࿈Λআ)
   యܕతͳΠϯϑϥ԰Ͱ͸ͳ͍
   SREͱDevOpsͱCREͬΆ͍໾ׂ΋Ռ͍ͨͯ͠Δ (ৄࡉ͸ͪ͜Β)
   3
   

   View Slide

4. αʔϏεϦϥΠΞϏϦςΟάϧʔϓ is Կॲ
   4
   

   View Slide

5. ࠓ೔࿩͢͜ͱ
   • λοϓϧ஀ੜͷҠઃ෺ޠ
   2೥ؒҎ্ʹΘͨΔετʔϦʔ
   • લ൒: ྺ࢙తܦҢɺҠઃ͋Δ͋Δ
   • ޙ൒: ECS EC2/FargateͷҠઃͱ͔ΑΓσΟʔϓͳ࿩
   ऴ൫ʹ࠷ۙग़ͨ৽ػೳΛࢼͯ͠Έͨ࿩΋͋ΔΑ
   5
   

   View Slide

6. ໨࣍
   1. αʔϏε֓ཁ
   2. ྺ࢙తܦҢ
   ΦϯϓϨߏ੒
   Ҡઃܭը
   Ҡઃ࣮ߦ
   3. ʁʁʁ
   4. ʁʁʁ
   ʁʁʁ
   ʁʁʁ
   5. ॴײ
   6
   

   View Slide

7. λοϓϧ஀ੜ
   αʔϏε֓ཁ
   

   View Slide

8. View Slide

9. λοϓϧ஀ੜ is Կ
   • झຯͰͭͳ͕Δ࿀׆αʔϏε
   • ձһ਺400ສਓҎ্ɺϚονϯά਺͸Ԇ΂1ԯ2,000ສ૊
   • ࠃ಺࠷େن໛
   9
   

   View Slide

10. https://tapple.me
    10
    

    View Slide

11. ྺ࢙తܦҢ
    

    View Slide

12. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    12
    

    View Slide

13. ΦϯϓϨ࣌୅ͷߏ੒֓ཁ
    • Φϯϓr…ϓϥΠϕʔτΫϥ΢υ
    • GitHubͰίʔυ؅ཧɺJenkinsͰϏϧυˍσϓϩΠ
    • αʔόͷϓϩϏδϣχϯά͸Ansible
    • Node.js + MongoDB + Elasticsearch
    • ϩάपΓ͸ɺFluentd tail -> aggregator -> in-house analyticsج൫
    13
    

    View Slide

14. View Slide

15. ΦϯϓϨߏ੒ʹ͓͚Δ՝୊఺
    • ύϒϦοΫΫϥ΢υͰ͍͏ͱ͜ΖͷManagedͳαʔϏε͕গͳ͍
    ΄ͱΜͲͷίϯϙʔωϯτ͸ࣗ෼ͨͪͰ؅ཧ͠ͳ͚Ε͹ͳΒͳ͍ͨΊɺ؅ཧίετߴ
    • ΦʔτεέʔϦϯά͸ɺ૬౰࡞Γࠐ·ͳ͍ͱ࣮ݱ͕೉͍͠
    ίετ࠷దԽ͕Ͱ͖ͣɺৗʹ࠷େΩϟύγςΟ͕ىಈ͍ͯͯ͠ແବ
    ͦͯ͠࠷େΩϟύγςΟ͕มΘΕ͹…खಈ૿ઃ…
    • ࣗಈԽ͕೉͘͠ɺϗετো֐ͳͲ͕ى͖ΔͨͼʹखಈΦϖൃੜ
    • ϓϥΠϕʔτΫϥ΢υͷӡ༻ϧʔϧ্ɺϦʔδϣϯʹ͸EOL͕͋ΓҠઃൃੜ
    ౰࣌࢖͍ͬͯͨϦʔδϣϯͷEOL͕͍͖͍ۙͮͯͯͨ

    Disclaimer: ผʹϓϥΠϕʔτΫϥ΢υΛdisͬͯΔΘ͚͡Όͳ͍͔Βצҧ͍͠ͳ͍ͰΑͶͬ
    15
    

    View Slide

16. AWSʹҠઃ͢Δ͜ͱʹͳͬͨཧ༝
    • ͳͥύϒϦοΫΫϥ΢υ
    ΦʔτεέʔϦϯά͍ͨ͠
    ϚωʔδυαʔϏεͰ؅ཧɾӡ༻ίετԼ͛ͯ։ൃʹूத͍ͨ͠
    • ͳͥAWS
    ϚονϯάαʔϏεͷதͰ͸͢ͰʹAWSͷར༻࣮੷͕͋ͬͨ
    ϚονϯάαʔϏεΛAWSʹἧ͍͍ͨͱ͍͏૝͍΋
    ϝσΟΞ؅׋ͷதͰAWSͷ஌ݟ͕ଟ͔ͬͨ
    ౰࣌͸།Ұ౦ژϦʔδϣϯΛఏڙ͍ͯͨ͠
    16
    

    View Slide

17. ҠઃܭըΛ
    ཱͯΑ͏
    

    View Slide

18. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    18
    

    View Slide

19. Ҡઃޙͷߏ੒ v.1: EC2
    • ΞϓϦ૚͸ͦͷ··
    Ծ૝αʔό EC2 Πϯελϯεͱ͍͚ͯ࣋ͬͯͩ͘͠
    • MongoDBͱElasticsearch΋ͦͷ··͍࣋ͬͯ͘
    MongoDB΁ͷΞΫηε͸ɺ֤ΞϓϦαʔόʹ৐͍ͬͯΔmongosΛϓϩΩγʹ࢖͏
    Elasticsearch͸ɺALBͰΞΫηεΛ෼ࢄ
    • RedisɺMySQL͸ElastiCacheɺAuroraʹ
    • nginxΛALBʹϦϓϨΠε
    • ੩తίϯςϯπͷ഑৴͸ɺS3+CloudFront
    19
    

    View Slide

20. View Slide

21. ҠઃϩʔυϚοϓ
    1. σʔλ૚͸ϝϯςΛೖΕͨঢ়ଶͰҠઃ͠ɺͦΕҎ֎͸ΦϯϥΠϯͰҠઃ
    σʔλ૚ͷҠઃͷͨΊͷϝϯς͸௒ਖ਼͍͠൑அɺঝ୚ͨؔ͠܎ऀ֤Ґʹѹ౗తײँ
    ϓϥΠϕʔτΫϥ΢υ͔Βσʔλ૚΁ͷΞΫηε͸ɺઐ༻ճઢܦ༝
    2. ΞϓϦ૚ɺݕূ͠ͳ͕ΒঃʑʹҠઃ͍ͯ͘͠
    21
    

    View Slide

22. ࣮ߦ
    ͢Δͧ
    

    View Slide

23. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    23
    

    View Slide

24. εςοϓ2: ΞϓϦ૚ͷҠઃ…ʁ
    • EC2 AutoscalingͷͨΊͷɺAMIϏϧυ༻Packer࡞ΓࠐΈɺݕূΛ…

    ͢Δ…ͭ΋Γ…͕ͩͬͨ…
    Ω < ͔ͤͬ͘ͷҠઃͳΒɺΞʔΩςΫνϟมߋ΋͍ͨ͠Ͷͬ
    Ω < ϚΠΫϩαʔϏεԽͰ͖ͨΒϦϦʔεͷ୯ҐΛখ͘͞Ͱ͖Δ͠੾Γ໭͠ͱָ͔ͩ͠
    Ω < Dockerfile࡞ͬͨͧ
    Ω < KubernetesͰͱΓ͋͑ͣಈ͍ͨͧ
    Ω < KubernetesͰߦ͜͏ͥ
    Ω < ͔ͤͬͩ͘͠ଞ΋վળ͍ͯ͜͠͏ Μɺɺ͋Δ͐ʁɹҰ୴EC2Ͱߦ͘Μ͡Όͳ͔͚ͬͨͬ
    24
    

    View Slide

25. Ҡઃޙͷߏ੒ v.2: Kubernetes
    • σʔλ૚͸มΘΒͣ
    • NLBΛIngestͱͯ͠࢖༻
    • kops + helm
    • ap-northeast-1 3 AZ (a/c/d)
    • ASGͰworkerϊʔυΛ؅ཧɺΦʔτεέʔϦϯά
    25
    

    View Slide

26. View Slide

27. Ҡઃޙͷߏ੒ v.2: ϩάपΓ
    • APIαʔϏεͳͲͰ૬৐Γfluentdίϯςφఆٛ
    • ڞ༗volume͔Βtail Kinesis Lambda in-houseͳanalytics
    KinesisͰόοϑΝϦϯάɺin-houseͳج൫͕ࢮΜͰ͍ͯ΋ɺ

    ʮKinesis Lambdaʯͷ෦෼Ͱ͍͍ײ͡ʹϦτϥΠ͞ΕΔ
    ৄࡉ: KinesisͱLambdaͰͭ͘ΔServerlessͳϩάج൫ @ AWS DevDay Tokyo 2019
    27
    

    View Slide

28. View Slide

29. …
    Ω < ΫϥελͱVPCͱαϒωοτपΓ࡞ͬͨͧ
    Ω < manifestΧΩΧΩͯ͠Δͧ
    Ω < Α͘ݟͨΒͪΌΜͱಈ͍ͯͳ͍ͳ
    Ω < Kubernetesͷόά͔ͳ
    Ω < Dockerͷόά͔ͳ
    Ω < ͋ΕΕ
    29
    

    View Slide

30. Ҡઃ͕׬ྃ͠ͳ͍
    ··1೥͕ܦͭ
    

    View Slide

31. ͦͯ࣌͠ݶര஄͕
    େരൃ
    

    View Slide

32. ࣌ݶര஄ #ͱ͸
    • ࣌ݶര஄ = ઐ༻ճઢ
    • ϙΠϯτ:
    1. σʔλ૚΁ͷΞΫηε͕Ϋϥ΢υΛލ͍Ͱ͍ͨ
    2. ઐ༻ճઢ͕อূͳ͠Ͱఏڙ͞Ε͍ͯͨ 

    (౰࣌͸ SLA ͕ఏڙ͞Ε͍ͯͳ͔ͬͨ)
    • Ҡઃ͕׬ྃ͠ͳ͍ঢ়ଶͰ1೥͕ܦͬͨ
    • ͦͯ͠ࣄ͕݅ى͖ͨ
    32
    

    View Slide

33. ࣌ݶര஄େരൃࣄ݅
    • ઐ༻ճઢͷো֐ͰશAPI஗Ԇര૿ɺDBෛՙߴಅɺ౤ߘ؂ࢹࣦഊɺ

    ΞϓϦͷΤϥʔදࣔʹΑΔେྔ࠶ϩάΠϯ
    ϦʔνҰൃ໳લਗ਼ࣗ໚࿨ฏ࿨υϥ2ɺ௓ຬɺ௓Μͩ…
    • શମϝϯςφϯεΛೖΕͯɺઐ༻ճઢఏڙݩʹ໰͍߹Θͤͭͭ͠

    ༷ࢠΛݟΔ͜ͱʹ
    • ࡾ࣌ؒޙɺճઢ෮چɺϝϯς໌͚
    33
    

    View Slide

34. ࣌ݶര஄େരൃࣄ݅ɺೋճ໨
    • ೋ೔࿈ଓͰ·ͨಉ͡ো֐͕ى͖Δ
    • ϝϯςΠϯ͔Β3࣌ؒޙɺ෮چͷݟࠐΈ͕ͳ͘ͳΓɺ

    ΋͏Ҡઃͯ͠͠·͑͹͍͍͡Όͳ͍͔આΛࢲ͕ޠΓʹ͍͘
    • ආ೉ઌͷ४උ։࢝
    • ϝϯςΠϯ͔Β5࣌ؒޙɺճઢ͕෮چɺϝϯς໌͚
    • ࠓ౓͸ɺ5࣌ؒҎ্ͷϝϯςφϯε
    • ·͍ͨͭى͖Δ͔Θ͔Βͳ͍͠ා͍͔Βɺආ೉ઌͷ४උΛଓߦ
    • ͦͷ৔Ͱ࠷௿ݶಈ͘؀ڥͷߏஙͱσϓϩΠϑϩʔͷཱ֬·Ͱ׬ྃͯ͠͠·͏
    ͜ͷ͋ͱ໓஡ۤ஡͓ञΛҿΈ·ͨ͠
    34
    

    View Slide

35. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. ʁʁʁ
    4. ʁʁʁ
    ʁʁʁ
    ʁʁʁ
    5. ॴײ
    35
    

    View Slide

36. View Slide

37. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    37
    

    View Slide

38. ҠઃϩʔυϚοϓ
    • 3ϑΣʔζͰ࣮ࢪ
    1. ۓٸEC2Ҡઃ: ͱΓ͋͑ͣ ΦϯϓϨ EC2 (࣌ؒՔ͗)
    • ϢʔβʹͰ͖Δ͚ͩૣ͘ΑΓ҆ఆͨ͠؀ڥͷఏڙ͢ΔͨΊ
    2. EC2 ECS (EC2 Type)
    3. ECS (EC2 Type) ECS (Fargate Type)
    • ͙͢ʹ͸Fargateʹ͠ͳ͔ͬͨཧ༝:
    • Fargateͷ஌ݟ͕গͳ͔ͬͨ
    • ౰࣌଍Γͳ͍ػೳ͕͋ͬͨΓɺߴ͔ͬͨΓʢ࠷ۙ൒ֹʹͳͬͨΓػೳ͕૿͑ͨΓʣ
    • λοϓϧϝϯόʔͷ஌ࣝΛਂΊΔͨΊʹ΋ɺֶ͋͑ͯशͷͨΊEC2ʹ
    • Fargateͱ͸ҟͳΓɺEC2Ͱ͸docker exec -it ͳͲͰίϯςφʹϩάΠϯͯ͠ௐࠪɺσόοά͕Ͱ͖Δ
    38
    

    View Slide

39. KubernetesΛఘΊͯECSΛ࠾༻ͨ͠ཧ༝
    • KubernetesͰؤுͬͯݕূ͍͕ͯͨ͠ɺ

    1೥͔͔ͬͯ݁ہຊ൪ʹग़ͤΔϨϕϧʹ͸ࢸΒͳ͔ͬͨʢׂѪʣ
    • ECSͷํ͕AWSͷଞͷαʔϏεͱͷ਌࿨ੑ͕ߴ͍
    • Kubernetes͡Όͳ͍ͱ͍͚ͳ͍ϢʔεέʔεͰ͸ͳ͔ͬͨ
    Service MeshͷΑ͏ͳෳࡶͳߏ੒Ͱ͸ͳ͘ɺಠཱͨ͠API͍͔ͭ͘ͱόον͍͔͚ͭͩ͘
    • SRGͱͯ͠΋ฐࣾϝσΟΞࢠձࣾͱͯ͠΋ɺECSͷϊ΢ϋ΢͕ଟ͍
    ࣗݾ঺հʹ͋ΔϦϯΫࢀর
    ߏஙͷΈͰ4೔ؒ + ΦʔτεέʔϦϯάͷઃܭ 1िؒʢׂ߹มߋͱͱ΋ʹ࣮ࢪʣ
    39
    

    View Slide

40. ۓٸEC2Ҡઃ: ϩʔυϚοϓ
    • ߏ੒Λେ͖͘ม͑ͣɺαʔόΛͦͷ··EC2Ͱ͍࣋ͬͯ͘
    QCɺෛՙࢼݧͳͲΛ௨ͯ͡Ϣʔβʹग़ͯ͠΋͍͍͜ͱ͕ূ໌Ͱ͖Δ·Ͱ͸੾Γସ͑ͳ͍
    ͨͩ͠ɺ΋͠·ͨେো֐͕ى͖ͨ৔߹ɺ৽؀ڥΛආ೉ઌͱͯ͠ར༻
    • ίϯςφԽߏ੒Λߏஙɺςετ
    • EC2΁ͷ੾Γସ͑ޙɺίϯςφԽߏ੒ʹ੾Γସ͍͑ͯ͘
    KubernetesΛఘΊͯɺECSΛ࠾༻ʢޙʹղઆʣ
    KubernetesݕূΛͨ͠ͱ͖ͷDockerfile΍VPCपΓͳͲྲྀ༻
    40
    

    View Slide

41. ۓٸEC2Ҡઃ: ߏங
    • Packerͱ͔Φʔτεέʔϧͱ͔Ұ୴ؾʹͤͣߏங
    • ര଎ͰTerraformͰEC2ܥɺηΩϡϦςΟάϧʔϓɺALBपΓΛ੔උʢԶʣ
    • AnsibleΛमਖ਼͠ͳ͕Βྲྀ͢ʢλοϓϧϝϯόʔʣ
    • ΞϓϦσϓϩΠɺಈ࡞֬ೝɺमਖ਼ͷ܁Γฦ͠
    • ߹ܭ12࣌ؒͰͱΓ͋͑ͣಈ͘΋ͷ͕׬੒
    • ·ͨো֐͕ى͖ͨΒආ೉ઌͱͯ͠࢖͏
    41
    

    View Slide

42. ۓٸEC2Ҡઃ: ࠷ऴ֬ೝͱ੾Γସ͑
    • ߏஙͷཌ೔: QCɺϦάϨογϣϯςετɺෛՙࢼݧʹ߹֨
    • ҰिؒΛ͔͚ͯঃʑʹϦΫΤετΛ৽͍͠؀ڥʹدͤΔ
    • tapple.me AϨίʔυΛALIASͰRoute 53 Traffic Flowʹ޲͚Δ
    • ϓϥΠϕʔτΫϥ΢υLBͱAWS ALBͷׂ߹ΛweightͰ੍ޚ
    • Θ͔Γ΍͘͢͢ΔͨΊʹɺweightͷ߹ܭ͕100ʹͳΔΑ͏ʹઃఆ
    • ࣮ࡍͷΞΫηεϩάͰׂ߹Λ֬ೝ
    42
    

    View Slide

43. View Slide

44. View Slide

45. Route 53 Traffic Flowॴײ
    • weightͷมߋΛద༻͔ͯ͠Βɺ࣮ࡍͷϦΫΤετͷׂ߹͕มΘΔ·Ͱ ~ 5෼
    • DNSΩϟογϡ͕มʹӨڹ͢Δ͜ͱͳ͘ɺ

    ࢦఆͨ͠weightͱ࣮ࡍͷΞΫηεͷׂ߹͸2%΄Ͳͷޡࠩ
    • Ϋϥ΢υҠઃϕετϓϥΫςΟε͗͢Δ…
    • KibanaͰϦΞϧλΠϜͰׂ߹͕มΘΔͷΛݟΔͷָ͍͠…
    45
    

    View Slide

46. 2िؒͰEC2Ҡઃ׬ྃ
    ര଎
    

    View Slide

47. ͋Δ΂͖࢟΁
    

    View Slide

48. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    48
    

    View Slide

49. Ҡઃޙͷߏ੒ v.3: ECS EC2
    ԦಓͳECS EC2:
    • ALB
    • ECS Service (EC2 Type)
    ͨͩ͠ɺBatchܥ͸શ෦Fargate (ޙʹղઆ)
    • EC2 Auto-Scaling Group (ΫϥελͷόοΫΤϯυ)
    49
    

    View Slide

50. View Slide

51. ECS EC2Ҡઃ: ECSΫϥελ
    • ECSόοΫΤϯυ͸ɺEC2 + Autoscaling Group
    • VanillaͳAmazon Linux (ECS-optimized) AMIͰىಈɺ

    user_dataͰͪΐͬͱͨ͠ॳظԽ͚࣮ͩߦ
    • εέʔϦϯάϝτϦοΫε͸ɺECS ClusterͷCPUReservationͷMaximum
    51
    

    View Slide

52. ECS EC2Ҡઃ: APIܥ
    • API༻αʔϏε͸EC2 Type
    • εέʔϦϯάϝτϦοΫε͸ɺECS ServiceͷCPUUtilizationͷMaximum
    • Dockerfile౳ɺΞϓϦΛίϯςφͰಈ͔ͨ͢Ίͷ΋ͷҰࣜ͸ɺ

    Kubernetesݕূͷͱ͖ͷ΋ͷΛྲྀ༻
    • ϚΠΫϩαʔϏεʹ෼͚Δͱ͜Ζ͸Ұ୴΍Βͳ͍
    ཧ༝ɿALBΛར༻͍ͯ͠ΔͨΊɺޙ͔Βಛఆͷύε͚ͩผͷαʔϏεʹϧʔςΟϯάͤ͞Δ

    ͜ͱ͕Մೳɺ͙͢΍Δඞཁ͕ͳ͍
    52
    

    View Slide

53. ECS EC2Ҡઃ: όονܥ
    • όον༻αʔϏε͸Fargate Type
    • ݁ߏසൟʹ࣮ߦ͞ΕΔόον͕͋ͬͯɺ

    EC2Ͱಈ͔͢ͱASGͷΦʔτεέʔϦϯάͷઃܭ͕໘౗ͳ͜ͱʹͳΔ…
    όον༻ͷΫϥελΛ࡞ΔͱϦιʔε͕ແବʹͳΔ͠ɺ

    ࡞Βͳ͍ͱεέʔϦϯάΞΫγϣϯͷढ़හੑ…
    • CloudWatch Eventsͷcron͕ࣜͪΐͬͱಛघͳͨΊɺ

    طଘͷcronࣜΛͪΐͬͱ͍͡Δඞཁ͕͋ͬͨ
    • Baikonur OSS fargate-scheduled-task ϞδϡʔϧΛར༻
    53
    

    View Slide

54. ECS EC2Ҡઃ: DB
    • MongoDBɺElasticsearch͸ʢ΄΅ʣͦͷ··
    • EC2ɺϓϥΠϕʔτΫϥ΢υͰΠϯελϯε͝ͱʹىಈ͍ͤͯͨ͞

    mongos͸ɺઐ༻EC2 x 3୆ʹͨ͠
    • mongos͕ݱΕͨΓফ͑ͨΓ͢Δͱɺconfig DBʹ΋͏͍ͳ͍mongosͷ৘ใ͕ཷ·Δ
    • ͢Ͱʹ͍ͳ͍mongosʹmongoc͕ping͠Α͏ͱ͍ͯ͠Δʁ
    • ޙʹ૿ઃ͞Ε…ͦͯ͠·ͨ૿ઃ͞Εͨ…
    54
    

    View Slide

55. ECS EC2Ҡઃ: ϩάपΓ
    • Kubernetes༻ͷ΋ͷͱҰॹɺ૬৐Γfluentdίϯςφར༻
    • ڞ༗volume͔Βtail aggregator Kinesis Lambda 

    in-houseͳanalytics
    KinesisసૹϓϥάΠϯͱϩάʹର͢Δཁ݅ͷ౎߹Ͱ࠷ऴతʹfluentd aggregatorḉര஀
    55
    

    View Slide

56. ECS EC2Ҡઃ: Cluster ASGͷεέʔϦϯάઃܭ
    • ΦʔτεέʔϦϯάͱ͍͏֓೦͕ECSҠઃ·Ͱ͸ଘࡏ͠ͳ͔ͬͨ
    • Ұ୴ɺෛՙͷ͔͔Γํ౳͕ࣅ͍ͯΔαʔϏεͷᮢ஋ΛύΫͬͨ
    • 1िׂ͔͚ؒͯ߹Λมߋ͢ΔΞϨͱಉ࣌ʹɺ

    εέʔϦϯάΛݟकΓͭͭࡉ͔͍ௐ੔Λߦͳͬͨ
    • εέʔϦϯάͷௐ੔Λߦͳͬͨࡍͷߟ͑ํΛ͝঺հ
    56
    

    View Slide

57. ECS EC2Ҡઃ: Cluster ASGͷεέʔϦϯάઃܭ
    • CPUReservationͷAverage͕75%-85%ͷؒΛӭ͙Α͏ʹɺ

    Step ScalingΛखͰௐ੔
    εύΠΫɺϐʔΫΛܴ͑Δͱ͖ʹɺΫϥελͷΩϟύ͕

    Taskͷ഑ஔͷ଍ΛҾͬுΒͳ͍Α͏ʹஸೡʹௐ੔
    57
    

    View Slide

58. Step ScalingɺͩΔ͍Ͱ͠ΐʁ
    • ઃܭ͕ͩΔ͍͠೉͍͠
    • ͦͯ͠ରࡦΛ͠ͳ͍ͱASG scale-in࣌ʹੜ͖͍ͯΔίϯςφ͕ࡴ͞ΕΔ
    • ରࡦ: EC2 Auto-ScalingͰECS૊ΜͰ·͔͢ʁΤϥʔϨʔτߴ͍Ͱ͔͢ʁٹͬͯ͋͛ͯ΋͍͍Ͱ͔͢ʁ
    • ECS Cluster Auto Scalingͱ͍͏ਆػೳ͕Ͱͨʂ
    • ͍͍ײ͡ʹεέʔϦϯάͯ͘͠ΕΔɺStep Scaling͢Δඞཁͳ͠
    • CPUɺϝϞϦɺENIͳͲɺ͍ͣΕ͔ҰͭͷΩϟύγςΟͷछྨͰ͸ͳ͘ɺ

    ্ཱ͍ͪ͛ͨTask͕࣮ࡍ্ཱͪ͛ΒΕΔͷ͔ͱ͍͏γϛϡϨʔγϣϯͰΩϟύෆ଍Λಛఆ
    • ىಈ͍ͯ͠Δίϯςφͷ͋ΔEC2Λࡴ͞ͳ͍ઃఆ΋Ͱ͖Δɺൿ఻ͷλϨͳ͠Ͱಈ͘ʂ
    • ౰࣌ͳ͔ͬͨͷͰͱͯ΋ͭΒ͔ͬͨɺECSͷதͷਓ·ͫ͋Γ͕ͱ❤
    58
    

    View Slide

59. ECS EC2Ҡઃ: ECS ServiceͷεέʔϦϯάઃܭ
    • CPUUtilizationͷMaximum͕40%-75%ͷؒΛӭ͙Α͏ʹɺ

    Step ScalingΛखͰௐ੔ɺ֤ίϯςφͷෛՙΛ֬ೝ
    ͜Ε͙Β͍ͷઃఆͰɺ֤ίϯςφ͕݈શతͳෛՙ (~50%) Λड͚Δ
    ίεύͱϨΠςϯγͷόϥϯγϯά
    59
    

    View Slide

60. Step ScalingɺͩΔ͍Ͱ͠ΐʁ
    • ઃܭ͕ͩΔ͍͠೉͍͠
    • Target Tracking Scaling (for ECS)ͱ͍͏΋ͷ͕ग़ͨ
    • ʮCPU࢖༻཰͕ฏۉͰ60%͙Β͍ʹͳΔΑ͏ʹͯ͠ɺϤϩʯͱ͍͏ઃఆ͚ͩͰ

    ͋ͱ͸ΑΖ͘͠΍ͬͯ͘ΕΔ
    • ౰੍࣌໿͕͋ͬͯಋೖ͕೉͔ͬͨ͠
    • ࢖͑ΔϝτϦΫε͕ݶΒΕ͍ͯͨ…هԱ
    • ޙʹෛՙࢼݧ؀ڥͰࢼ͢΋ɺզʑͷϢʔεέʔεʹ߹Θͣ
    • Step Scaling͔Β͍·ͩʹಀ͛ΒΕ͍ͯͳ͍
    60
    

    View Slide

61. ECS EC2Ҡઃ: ECS ServiceͷεέʔϦϯάઃܭ
    61
    ίϯςφͷCPU࢖༻཰͕40-60%

    ͙Β͍ʹͳ݈ͬͯͯ߁తͳෛՙ
    

    View Slide

62. ECS EC2Ҡઃ: ࠷ऴ֬ೝͱ੾Γସ͑
    • ۓٸEC2ҠઃͱҰॹͩΑ
    • QCɺϦάϨογϣϯςετɺෛՙࢼݧʹ߹֨͢Δ·Ͱमਖ਼࣮ࢪ
    • ҰिؒΛ͔͚ͯঃʑʹϦΫΤετΛدͤΔ
    • tapple.me AϨίʔυΛALIASͰRoute 53 Traffic Flowʹ޲͚Δ
    • EC2؀ڥͷALBͱECS؀ڥͷALBͷׂ߹ΛweightͰ੍ޚ
    • Θ͔Γ΍͘͢͢ΔͨΊʹɺweightͷ߹ܭ͕100ʹͳΔΑ͏ʹઃఆ
    • ࣮ࡍͷΞΫηεϩάͰׂ߹Λ֬ೝ
    62
    

    View Slide

63. ECS EC2Ҡઃ: Ҡઃޙͷ͋Ε͜Ε
    • ઐ༻ճઢ͕ͳ͘ͳͬͨ͜ͱͰɺAWSྉ͕͔ۚͳΓ҆͘ͳͬͨ
    • ϦΫΤετΛฦͨ͢Ίͷ௨৴͸AWS಺௨৴ͷΈʹͳͬͯɺ

    API͕ര଎ʹͳͬͨ
    ࣮૷͸΋ͱ΋ͱͱͯ΋༏लͩͬͨ
    ϨΠςϯγ͕ฏۉͰ120ms଎͘ͳΓɺmsೋܻ୆΁
    63
    

    View Slide

64. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ۓٸҠઃ͔Β͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    64
    

    View Slide

65. View Slide

66. Fargate!
    • ECS EC2Ҡઃ͔Β΄΅1೥͕ܦͬͨ
    • EC2ʹϩάΠϯͯ͠docker execͱ͔͢Δ͜ͱ͕΄΅ͳ͘ͳͬͯ
    ΞϓϦέʔγϣϯ΋ίϯςφʹదͨ͠ܗʹ͍͖ۙͮͯͨ
    • ϝϯόʔ΋ECSɺίϯςφΛ࢖ͬͨΞϓϦέʔγϣϯͷӡ༻ʹ׳Ε͖ͯͨ
    • ͦͯ͠ECS EC2ʹର͢Δෆຬ͕ཷ·͖ͬͯͨ
    • EC2 ASGͷεέʔϧΛ଴ͨͳ͚Ε͹ͳΒͳ͍ɺͦͷͨΊεέʔϦϯάɺσϓϩΠ͕஗͍
    • Ωϟϯϖʔϯ౳Λ࣮ࢪ͢Δࡍʹscheduled scaling͕ඞཁɺͦͷϙϦγʔΛઃܭɾܭࢉ͢Δͷ͸େม
    • AMIߋ৽͢ΔͨͼʹASGͷEC2ΠϯελϯεΛస͕͢ͷ͕େม
    • Α͠ɺFargateԽ͠Α͏ʂ
    66
    ࢲ͸࠷ॳ͔Βɺ࠷ऴతʹFargateʹ͢Δͭ΋Γͩͬͨ
    

    View Slide

67. View Slide

68. Fargateͷίετʹؔͯ͠
    • Ϧιʔείετ͸On-demand EC2 (RIͳͲͷׂҾͳ͠) ͱൺֱͯ͠

    Ή͠Ζ҆͘ͳΔ
    • EC2ͷόοΫΤϯυΠϯελϯεΛASGͱ͔Ͱ༻ҙ͢ΔࡍʹɺCPU/ϝϞϦͷۭ͖͕ৗʹ

    15-20%ʹͳΔΑ͏ʹ͍ͯ͠Δ͔Βʢεέʔϧͷढ़හੑʣ
    • ͦͯ͠൒ֹʹ஋Լ͛͞ΕͨΓͨ͠
    • ͞ΒʹɺEC2ϨΠϠͷΦϖϨʔγϣϯɾ؅ཧίετ͕࡟ݮ
    • ηΩϡϦςΟύονɺDockerͷύον࡞ۀ͞Α͏ͳΒ
    • ͍҆͠ɺ͍͍͡ΌΜ
    68
    

    View Slide

69. FargateͷΠΠͱ͜Ζ
    • Firecrackerͷొ৔Ͱىಈ͕଎͘ͳͬͨ
    • ENIͷallocation͸ಛʹੲͱൺ΂ͯ଎͘ͳ͍ͬͯΔ
    • EC2ͷΩϟύ଴ͭඞཁͳ͍͠ɺىಈ΋଎͍ͷͰσϓϩΠɺ੾Γ໭͠ɺεέʔϦϯάͷߴ଎Խ
    • EC2Ͱͷ՝୊: bridge network͕஗͍͕ɺhostΛ࢖͏ͱϙʔτ͕େมͳ͜ͱʹ
    ͳΔ͔ΒawsvpcΛݕ౼͕ͨ͠ɺΠϯελϯελΠϓͱαΠζ͋ͨΓͷENI

    ෇༩্ݶ਺͕ͱͯ΋গͳ͍
    • ࠷ۙENI TrunkingͰ؇࿨͞Ε͍ͯΔ͕ͦΕͰ΋଍Γͳ͔ͬͨ
    • FargateΛ࢖͏ͱ͖awsvpc͔͠࢖͑ͳ͍͕ɺͦΕͰ͍͍
    • Docker Links࢖ͬͯΔϚφʔͷΑΖ͘͠ͳ͍Task͸ɺFargateԽ (+awsvpcԽ) Ͱͳͤ͘Δ
    69
    

    View Slide

70. Fargateͷͪΐͬͱ໘౗ͳͱ͜Ζ
    • TaskͷCPUͱϝϞϦΛܾΊΒΕͨ૊Έ߹Θͤͷத͔Βબ͹ͳ͍ͱμϝ
    • ૊Έ߹Θͤ਺΋ଟ͍͠෯޿͘ઃఆͰ͖Δ͕ࣗ෼ͷχʔζʹtailoredͳͪΐ͏Ͳ͍͍Ωϟύʹ͸
    ͳΒͳ͍
    • AWS͞Μ͕ΫϥελΛ؅ཧͯ͘͠ΕͯΔ͠ɺڐͤΔ
    • Docker VolumeͰͷϑΝΠϧڞ༗͸ɺ1 volumeͰhard limit 4GB
    70
    

    View Slide

71. ͍ͭͰʹ΍Δ͜ͱʹͳͬͨ͜ͱ
    • ͔ͤͬ͘ServiceɺALBΛ࡞Γ௚͔͢Β΍Γ͍ͨͳͱࢥͬͨ͜ͱ
    • ALB Least Outstanding RequestsΞϧΰϦζϜͷݕূ
    • Internal LB༻ͷtarget groupΛServiceʹඥ෇͚Δ
    • Internal/Global༻ͷTGΛҰͭͷServiceʹඥ෇͚
    • લ͸ෳ਺TGͷඥ෇͚͕Ͱ͖ͳ͔ͬͨͨΊInternalઐ༻Service…
    71
    

    View Slide

72. ALB Least Outstanding Requests
    Ͳ͕ͬͪRound-RobinͰͲ͕ͬͪLORͩͱࢥ͍·͔͢ʁ
    72
    

    View Slide

73. ALB Least Outstanding Requests
    ਖ਼ղɿ
    Round-Robin LOR
    ͙͢໭͢͜ͱʹͳͬͨ
    73
    

    View Slide

74. εέʔϦϯάपΓͷௐ੔
    • Datadog APMͷϨΠςϯγɺFargateͷCPU࢖༻཰͕ಉఔ౓ʹͳΔΑ͏ௐ੔
    • APMͰͷEC2/Fargateͷ൑ผ͸DD_ENV
    • DatadogϝτϦΫε͸໊લۭؒ(docker.* vs ecs.*)ͱTask Family໊
    • ϩά͸ɺplatform = {“ec2”, “fargate”} ͱ͍͏ϑΟʔϧυ௥Ճ
    ո͍͠ ո͍͠ چ؀ڥಉ༷
    Ϥγʂ
    
    
    
    چ؀ڥϨΠςϯγ ৽؀ڥϨΠςϯγ
    74
    

    View Slide

75. ׂΓ߹͍มߋͷ֬ೝ on Kibana
    • platform = {“ec2”, “fargate”} ͳϑΟʔϧυͰ

    ΞΫηεϩά͔ΒϦΞϧλΠϜͰՄࢹԽ
    75
    

    View Slide

76. ׂ߹Λ֬ೝɺϤγʂ Τϥʔ֬ೝɺϤγʂ
    76
    

    View Slide

77. LOR੾Γ໭͠ Ϥγʂ
    ϨΠςϯγ Ϥγʂ
    ΤϥʔϨʔτ Ϥγʂ
    εέʔϦϯά Ϥγʂ
    77
    

    View Slide

78. Ҡઃ׬ྃ
    

    View Slide

79. λοϓϧͷΠϯϑϥपΓͷ͜Ε͔Β
    • Fargate SpotΛ࢖ͬͨover-provisioningͰٸͳεύΠΫʹରԠ
    • FireLens + KinesisͳϩάपΓ
    • FireLens΄Μͱ͍͢͝ΑɺϩάपΓͷ՝୊͸(container nativeͳܗͰ)େ఍ͳΜͱ͔ͳΔ
    • ͨͩ͠ɺloggingDriverࢦఆʹͳΔͷͰCloudWatch Logsʹ΋FireLensʹ΋ྲྀ͢ͷ͸೉͍͠
    • ࠓޙ͜Ε͕վળ͞ΕΔ͜ͱΛفΓͳ͕ΒFeature Request౤͛ ʢͬ'-')‸ = ͟
    ͟͞͞ŲƄőƃɹ
    • ΧελϜεέʔϦϯά
    • CPUͰ͸ͳ͘ΞΫηε਺ɺͳͲͰΠΠײ͡ͰŧŔŕŰƅźͳεέʔϥΛ࡞͍ͬͯΔ
    • ϝτϦΫεΛ଴ͨͣʹ1෼ະຬͷϦΫΤετมԽʹରԠ͠ϨΠςϯγΛ͞Βʹ҆ఆԽ
    • @pahudnet ͞ΜͷŧŔŕŰƅźεέʔϥΛࢀߟʹͭͭ͠
    • ΧελϜσϓϩΠ
    Deployment Controller: EXTERNALͰTask Set APIΛ࢖ͬͨߴ౓ͳαϜγϯάΛ࡞͍ͬͯΔ
    79
    

    View Slide

80. ໨࣍
    1. αʔϏε֓ཁ
    2. ྺ࢙తܦҢ
    ΦϯϓϨߏ੒
    Ҡઃܭը
    Ҡઃ࣮ߦ
    3. Re: ো֐͔Β࢝ΊΔۓٸҠઃੜ׆
    4. ۓٸҠઃ͔Β͋Δ΂͖࢟΁
    EC2 -> ECS (EC2 Type)
    ECS (EC2 Type) -> ECS (Fargate Type)
    5. ॴײ
    80
    

    View Slide

81. ॴײ
    • ॊೈͰ͋Ε
    • ౰ॳͷ໨తΛ๨ΕΔͳ
    • ϩʔυϚοϓΛҾ͘͜ͱ͕େࣄ
    • ເΛݟΔ͜ͱ͸ѱ͍͜ͱ͡Όͳ͍͕ɺ΍Δ΂͖͜ͱ͸͔ͬ͠Γɺ

    ॱ൪ʹ΍͍ͬͯ͘΂͠
    • Ϣʔβͷ͜ͱΛߟ͑ͳ͕ΒɺϢʔβମݧΛ࠷༏ઌʹͯ͠࢓ࣄʹऔΓ૊Ή΂͠
    • ECS͸͍͍ͧʢAWS Managedͱͷ਌࿨ੑͱֶशۂઢʣ
    • Fargate͸͍͍ͧʢόοΫΤϯυͷ͜ͱΛ๨ΕͯαʔόϨεʹಀ͍͛ͨਓੜʣ
    81
    

    View Slide

82. એ఻
    82
    ← https://baikonur.dev 

    ʹԿ͔Λग़͠·ͨ͠
    

    View Slide

83. बۀܕΠϯλʔϯγοϓɺ௨೥ืूதʂ
    • ΤϯτϦʔక੾: Πϯλʔϯγοϓ։݄࢝ͷ3ϲ݄લͷ݄຤
    • ৄࡉ: https://www.cyberagent.co.jp/careers/students/event/detail/id=24422
    83
    

    View Slide

84. ࠷ޙʹ΋͏Ұճʂ
    

    View Slide

85. 85
    ͜ΕΛ
    

    View Slide

86. 86
    ͜͏ͯ͠
    

    View Slide

87. 87
    ͜͏ͯ͠
    

    View Slide

88. 88
    ͜͏͡Όʂ
    

    View Slide

89. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide