Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WEARのEKSコストを救いたい

mi.kobaaaa
February 26, 2023

 WEARのEKSコストを救いたい

【Autify x ZOZO x dip共同開催】AWSコスト削減事例祭り
https://zozotech-inc.connpass.com/event/273113/

mi.kobaaaa

February 26, 2023
Tweet

More Decks by mi.kobaaaa

Other Decks in Programming

Transcript

  1. WEARͷEKSίετΛٹ͍͍ͨ
    גࣜձࣾZOZO
    ϒϥϯυιϦϡʔγϣϯ։ൃຊ෦
    όοΫΤϯυ෦ SREϒϩοΫ
    খྛ ະདྷ
    Copyright © ZOZO, Inc.
    AWSίετ࡟ݮࣄྫࡇΓ

    View full-size slide

  2. © ZOZO, Inc.
    ࣗݾ঺հ

    View full-size slide

  3. © ZOZO, Inc.
    גࣜձࣾZOZO
    ϒϥϯυιϦϡʔγϣϯ։ൃຊ෦ όοΫΤϯυ෦
    SREϒϩοΫ
    খྛ ະདྷ
    ● @mirai_kobaaaaaa
    ● ΏΔ;ΘܥSRE
    ● ޷͖ͳαʔϏεɿAWS CDK

    View full-size slide

  4. © ZOZO, Inc.

    View full-size slide

  5. © ZOZO, Inc.
    https://zozo.jp/
    6
    ● ϑΝογϣϯEC
    ● 1,500Ҏ্ͷγϣοϓɺ8,500Ҏ্ͷϒϥϯυͷऔΓѻ͍
    ● ৗ࣌90ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱຖ೔ฏۉ2,600఺Ҏ্ͷ৽ண
    ঎඼Λܝࡌʢ2022೥12݄຤࣌఺ʣ
    ● ϒϥϯυݹணͷϑΝογϣϯκʔϯʮZOZOUSEDʯ΍ίεϝઐ໳
    ϞʔϧʮZOZOCOSMEʯɺۺͷઐ໳Ϟʔϧ
    ʮZOZOSHOESʯɺϥάδϡΞϦʔˍσβΠφʔζκʔϯ
    ʮZOZOVILLAʯΛల։
    ● ଈ೔഑ૹαʔϏε
    ● ΪϑτϥοϐϯάαʔϏε
    ● πέ෷͍ ͳͲ

    View full-size slide

  6. © ZOZO, Inc.
    https://wear.jp/
    7
    ● ϑΝογϣϯίʔσΟωʔτΞϓϦ
    ● 1,600ສμ΢ϯϩʔυಥഁɺίʔσΟωʔτ౤ߘ૯਺͸1,300ສ݅
    Ҏ্ʢ2022೥12݄຤࣌఺ʣ
    ● ϐοΫΞοϓλά͔Β࠷৽ͷτϨϯυΛνΣοΫ
    ● ίʔσΟωʔτண༻ΞΠςϜΛެࣜαΠτͰߪೖՄೳ
    ● WEARެೝͷਓؾϢʔβʔΛWEARISTAͱೝఆɻϞσϧɾλϨϯ
    τɾσβΠφʔɾΠϯϑϧΤϯαʔͱ͍֤ͬͨքஶ໊ਓ΋ࢀՃ

    View full-size slide

  7. © ZOZO, Inc.
    ຊ೔͓࿩͢͠Δ͜ͱ

    View full-size slide

  8. © ZOZO, Inc.
    ຊ೔͓࿩͢͠Δ͜ͱ
    ● WEAR ʹ͓͚Δ EKS ίετ࠷దԽʹ޲͚ͨଧͪख
    ○ ݱঢ়ͷऔΓ૊Έͱࠓޙͷଧͪख
    ○ ௐࠪͨ݁͠ՌɺԿ͕՝୊ͩͬͨͷ͔
    ○ ͳͥ࠷దԽ͕ඞཁͳͷ͔
    ຊ೔͓࿩͠͠ͳ͍͜ͱ
    ● EKS (Amazon Elastic Kubernetes Service) ͷࡉ͔ͳ࢓༷
    ● ಺෦ΞϓϦέʔγϣϯͷৄࡉ
    ● Kubernetes ͷ֓ཁɺৄࡉ

    View full-size slide

  9. © ZOZO, Inc.
    ͋Ε͸2022೥๭೔

    View full-size slide

  10. © ZOZO, Inc.
    AWS ίετ૿Λݕ஌
    ● EKS ίετ͕૿Ճ
    ○ ౰࣌ ECS => EKS ΁ͷϦϓϨΠεϓϩδΣΫτ͕ਐߦத
    ○ ͦͷͨΊଟগͷίετ૿Ճ͸૝ఆൣғͱ͍ͯͨ͠

    View full-size slide

  11. © ZOZO, Inc.
    AWS ίετ૿Λݕ஌
    ● ཌ݄ɺߋʹ EKS ίετ͕૿Ճ
    ○ 2ϲ݄લͱൺ΂ͯ໿10ഒ΁
    ○ ҟৗ஋ͱ൑அ͠ௐࠪ։࢝
    10ഒ

    View full-size slide

  12. © ZOZO, Inc.
    ௐࠪ݁Ռ

    View full-size slide

  13. © ZOZO, Inc.
    ௐࠪ݁Ռ
    ● όονॲཧϫʔΫϑϩʔΛ࣮ߦ͢Δ Cluster ͷίετ͕ٸ૿͍ͯͨ͠
    ○ 2,400pod/day ͕࣮ߦ͞Ε͍ͯΔ
    ○ Fargate ͷΈͰߏ੒
    Mecha Ippai Fargate
    Batch Cluster

    View full-size slide

  14. © ZOZO, Inc.
    ௐࠪ݁Ռ
    ● όονॲཧϫʔΫϑϩʔΛ࣮ߦ͢Δ Cluster ͷίετ͕ٸ૿͍ͯͨ͠
    ○ 2,400pod/day ͕࣮ߦ͞Ε͍ͯΔ؀ڥ
    ○ Fargate ͷ࢖༻ྔ͕૿Ճ͠ίετ͕ਹΜͰ͍ͨ
    Mecha Ippai Fargate
    Batch Cluster
    ͕͜͜໰୊ʂ

    View full-size slide

  15. © ZOZO, Inc.
    උߟɿEKSίετͷߟ͑ํ
    ● EKSίετ͸Լهͷ߹ࢉͰܭࢉ͞ΕΔ
    ○ Kubernetes Worker Node ࣮ߦ؀ڥͷίετ
    ○ EKS Cluster ͷίετ
    Amazon EKS
    +
    AWS Fargate Amazon EC2
    EKS Cluster Kubernetes Worker Nodes

    View full-size slide

  16. © ZOZO, Inc.
    උߟɿWorker Node ͷίετ
    ● Fargate ͷ৔߹
    ○ Image Pull ։͔࢝Β Pod ͕ऴྃ͢Δ·Ͱʹ࢖༻͞ΕͨϦιʔεͰܭࢉ
    ○ ࣮ߦ͞ΕΔ Fargate ͷεϖοΫʹΑΓࢉग़
    AWS Fargate
    1࣌ؒ౰ͨΓͷ vCPU ୯Ґ
    1࣌ؒ౰ͨΓͷ GB ୯Ґ
    Pod਺
    ×
    ×

    View full-size slide

  17. © ZOZO, Inc.
    උߟɿWorker Node ͷίετ
    Ex. 2୆ͷ EKS Pod Λ30೔ϑϧՔಇͤͨ͞৔߹
    Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ
    ߹ܭ vCPU ྉۚ = (Pod ਺) × (vCPU ਺) × (CPU ࣌ؒ͋ͨΓͷྉۚ) × (࣌ؒ୯Ґͷ 1 ೔͋ͨΓͷ CPU ࢖༻࣌ؒ) × (೔਺)
    ߹ܭϝϞϦྉۚ = (Pod ਺) × (ϝϞϦ (GB)) × (GB ͋ͨΓͷྉۚ) × (࣌ؒ୯Ґͷ 1 ೔͋ͨΓͷ ϝϞϦ ࢖༻࣌ؒ) × (೔਺)
    Fargate ͷίϯϐϡʔςΟϯάͷֹ݄ྉۚ = (CPU ֹ݄ྉۚ) + (ϝϞϦͷֹ݄ྉۚ)
    ྫ୊ Fargate ͷίϯϐϡʔςΟϯάͷֹ݄ྉۚ = 72.8064USD + 2.4348USD = 88.74USD
    ྫ୊ vCPU ྉۚ = 2 × 1 × 0.05056USD × 24 × 30 = 72.81USD
    ྫ୊ϝϞϦྉۚ = 2 × 2 × 0.00553USD × 24 × 30 = 15.93USD
    ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ

    View full-size slide

  18. © ZOZO, Inc.
    උߟɿWorker Node ͷίετ
    ● EC2 ͷ৔߹
    ○ Managed Node Group ͷ৔߹΋ࢉग़ํ๏͸ಉ͡
    ○ Worker Node ࣮ߦͷͨΊʹ࢖༻ͨ͠ AWS ϦιʔεͰࢉग़
    EC2 Πϯελϯείετ
    EBS ϘϦϡʔϜ
    Πϯελϯε਺
    ×
    ×
    Amazon EC2

    View full-size slide

  19. © ZOZO, Inc.
    උߟɿWorker Node ͷίετ
    Ex. 2୆ͷ EKS Pod Λ30೔ϑϧՔಇͤͨ͞৔߹
    ߹ܭΠϯελϯεྉۚ = (ΠϯελϯελΠϓͷ࣌ؒ౰ͨΓྉۚ) × (Πϯελϯε਺) × (Πϯελϯε࣌ؒ)
    ߹ܭ EBS ྉۚ = (1 ϲ݄ʹϓϩϏδϣχϯά͞Εͨ GB) × (1 ϲ݄͋ͨΓͷΠϯελϯε࣌ؒ) × (1 GB ͋ͨΓͷֹ݄ྉۚ)
    ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ
    Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ
    ྫ୊߹ܭ EBS ྉۚ = 20 × (720 ࣌ؒ / 730࣌ؒ) × 0.12USD = 2.38USD
    ྫ୊߹ܭΠϯελϯεྉۚ = 0.107USD × 1 × 720 = 77.04 USD
    ྫ୊Ͱ͋Ε͹ 2vCPU, 4GBϝϞϦ͕ඞཁͰ͋ΔͨΊɺc5.large ͔ͭ ൚༻SSD (gp2: 20GB)Ͱܭࢉ͢Δ
    ྫ୊߹ܭ EC2 ྉۚ = 77.04 USD + 2.38USD = 79.42USD

    View full-size slide

  20. © ZOZO, Inc.
    උߟɿWorker Node ͷίετ
    Ex. 2୆ͷ EKS Pod Λ30೔ϑϧՔಇͤͨ͞৔߹
    ߹ܭΠϯελϯεྉۚ = (ΠϯελϯελΠϓͷ࣌ؒ౰ͨΓྉۚ) × (Πϯελϯε਺) × (Πϯελϯε࣌ؒ)
    ߹ܭ EBS ྉۚ = (1 ϲ݄ʹϓϩϏδϣχϯά͞Εͨ GB) × (1 ϲ݄͋ͨΓͷΠϯελϯε࣌ؒ) × (1 GB ͋ͨΓͷֹ݄ྉۚ)
    ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ
    Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ
    ྫ୊߹ܭ EBS ྉۚ = 20 × (720 ࣌ؒ / 730࣌ؒ) × 0.12USD = 2.38USD
    ྫ୊߹ܭΠϯελϯεྉۚ = 0.107USD × 1 × 720 = 77.04 USD
    ྫ୊Ͱ͋Ε͹ 2vCPU, 4GBϝϞϦ͕ඞཁͰ͋ΔͨΊɺc5.large ͔ͭ ൚༻SSD (gp2: 20GB)Ͱܭࢉ͢Δ
    ྫ୊߹ܭ EC2 ྉۚ = 77.04 USD + 2.38USD = 79.42USD
    Fargateͷࠩ͸10%
    ※Fargate or EC2 ͸ӡ༻ίετ΋ؑΈΔඞཁ͕͋Γ·͢ɻ͝஫ҙԼ͍͞ɻ

    View full-size slide

  21. © ZOZO, Inc.
    ଧͪख

    View full-size slide

  22. © ZOZO, Inc.
    ଧͪख
    ● ଧͪखͱͯ͠ԼهΛݕ౼
    ■ ࣮ࡍͷϝτϦΫε͔Β Fargate Pod ͷ࣮ߦ spec Λௐ੔
    ○ Fargate spec ͷ optimize
    ○ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ
    ○ Fargate ࣮ߦ࣌ؒͷ࡟ݮ
    ■ ෆཁͳ Fargate Pod ͷࣗಈ࡟আ΍ Image Pull ଎౓ͷվળ
    ■ EKS on EC2 ΁Ҡߦ͢Δ͜ͱʹΑΔίετѹॖ

    View full-size slide

  23. © ZOZO, Inc.
    Fargate spec ͷ optimize
    ● Datadog μογϡϘʔυ͔Β pod ͷར༻ϝτϦΫεΛνΣοΫ

    View full-size slide

  24. © ZOZO, Inc.
    Fargate spec ͷ optimize
    ● ࣮ࡍͷϝτϦΫε͔Βඞཁ spec Λ࠶ࢉग़͠ɺManifest Λमਖ਼
    ● ෳ਺ͷ Deployment શͯʹରͯ͠ௐ੔

    View full-size slide

  25. © ZOZO, Inc.
    Fargate ࣮ߦ࣌ؒͷ࡟ݮ
    ● ϫʔΫϑϩʔ࣮ߦ׬ྃޙʹ Fargate Pod ͕࡟আ͞ΕΔ·Ͱͷ࣌ؒΛઃఆ
    ● كʹϫʔΫϑϩʔ࣮ߦ׬ྃޙʹ࡟আ͞Εͳ͍ Fargate Pod ͕ଘࡏ͍ͯͨ͠
    ○ job ʹରͯ͠ .spec.ttlSecondsAfterFinished Λ௥Ճ

    View full-size slide

  26. © ZOZO, Inc.
    Fargate ࣮ߦ࣌ؒͷ࡟ݮ
    ● Image ѹॖΞϧΰϦζϜมߋʹΑΔ Image Pull ࣌ؒͷ࡟ݮ
    ● gzip ѹॖํ͔ࣜΒ zstd ѹॖํࣜʹมߋ͢Δ͜ͱͰ Image αΠζΛѹॖ

    View full-size slide

  27. © ZOZO, Inc.
    Fargate ࣮ߦ࣌ؒͷ࡟ݮ
    ● Image ѹॖΞϧΰϦζϜมߋʹΑΔ Image Pull ࣌ؒͷ࡟ݮ
    ● gzip ѹॖํ͔ࣜΒ zstd ѹॖํࣜʹมߋ͢Δ͜ͱͰ Image αΠζΛѹॖ
    ● WEARͰ͸ Image Pull ࣌ؒ΁ͷӨڹ͸গͳ͔ͬͨͨΊҰ୴ޙճ͠΁
    5%ఔ౓ѹॖ

    View full-size slide

  28. © ZOZO, Inc.
    Fargate ࣮ߦ࣌ؒͷ࡟ݮ
    ● AWS ެࣜΑΓϒϩά͕ެ։͞Ε͍ͯ·͢ͷͰੋඇ͝ҰಡԼ͍͞ʂ
    ref. https://aws.amazon.com/jp/blogs/news/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/

    View full-size slide

  29. © ZOZO, Inc.
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    Batch Cluster
    Fargate Fargate
    Batch Cluster
    EC2 EC2

    View full-size slide

  30. © ZOZO, Inc.
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    Batch Cluster
    Fargate Fargate
    Batch Cluster
    EC2 EC2
    ○ Fargate Λഇࢭ͠ɺશͯ EC2 ʹࡌͤସ͑Δ
    ○ EC2 Managed Node Group Λར༻͠ɺӡ༻ίετ΋཈͑Δ
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத

    View full-size slide

  31. © ZOZO, Inc.
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    ● ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
    Batch Cluster
    EC2 Critical Group
    EC2 Common
    Group
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத

    View full-size slide

  32. © ZOZO, Inc.
    Batch Cluster
    EC2 Critical Group
    EC2 Common
    Group
    ࣮ߦ؀ڥʹߴ͍


    Πϯελϯεੑೳ͕ෆཁͳPod


    Ex. Operator౳
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    ● ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔

    View full-size slide

  33. © ZOZO, Inc.
    Batch Cluster
    EC2 Critical Group
    EC2 Common
    Group
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    ● ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
    ࣮ߦ؀ڥʹߴ͍


    Πϯελϯεੑೳ͕ඞཁͳPod


    Ex. σʔλॲཧ౳

    View full-size slide

  34. © ZOZO, Inc.
    ○ Manifest ʹ Node Affinity Λ௥Ճ͠એݴతʹϊʔυΛબ୒
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    ● ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔

    View full-size slide

  35. © ZOZO, Inc.
    Batch Cluster
    EC2 Critical Group
    EC2 Common
    Group
    Deployment Critical
    Deployment Common
    EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
    ○ Manifest ʹ Node Affinity Λ௥Ճ͠એݴతʹϊʔυΛબ୒
    ● Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ
    ● ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔

    View full-size slide

  36. © ZOZO, Inc.
    ·ͱΊ

    View full-size slide

  37. © ZOZO, Inc.
    ·ͱΊ
    WEARʹ͓͚Δ EKS ίετ࡟ݮʹ·ͭΘΔ͋Ε͜ΕΛ঺հ͠·ͨ͠
    ● ·ͣ͸ίετΛ Watch ͢Δ͜ͱ
    ● ݱঢ়ʹ߹Θͤͨ࠷దԽΛଓ͚͍ͯ͘͜ͱ
    ● αʔϏεͷϦιʔεΛ؍ଌ͠ଓ͚Δ͜ͱ
    ৭ʑ͓࿩͠͠·͕ͨ͠ɺେ੾ͳ͜ͱ͸Լه3఺ͩͱߟ͍͑ͯ·͢
    ਐߦதͰ͸͋Γ·͕͢ɺϐʔΫ࣌ͷ50%ఔ౓·Ͱίετ࡟ݮ͕Ͱ͖͍ͯ·͢

    View full-size slide

  38. © ZOZO, Inc.
    Զͨͪͷઓ͍͸͜Ε͔Βͩʂ

    View full-size slide

  39. © ZOZO, Inc.
    We are hiring!

    View full-size slide

  40. © ZOZO, Inc.
    ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ

    View full-size slide