Upgrade to Pro — share decks privately, control downloads, hide ads and more …

WEARのEKSコストを救いたい

mi.kobaaaa
February 26, 2023

 WEARのEKSコストを救いたい

【Autify x ZOZO x dip共同開催】AWSコスト削減事例祭り
https://zozotech-inc.connpass.com/event/273113/

mi.kobaaaa

February 26, 2023
Tweet

More Decks by mi.kobaaaa

Other Decks in Programming

Transcript

  1. © ZOZO, Inc. https://zozo.jp/ 6 • ϑΝογϣϯEC • 1,500Ҏ্ͷγϣοϓɺ8,500Ҏ্ͷϒϥϯυͷऔΓѻ͍ •

    ৗ࣌90ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱຖ೔ฏۉ2,600఺Ҏ্ͷ৽ண ঎඼Λܝࡌʢ2022೥12݄຤࣌఺ʣ • ϒϥϯυݹணͷϑΝογϣϯκʔϯʮZOZOUSEDʯ΍ίεϝઐ໳ ϞʔϧʮZOZOCOSMEʯɺۺͷઐ໳Ϟʔϧ ʮZOZOSHOESʯɺϥάδϡΞϦʔˍσβΠφʔζκʔϯ ʮZOZOVILLAʯΛల։ • ଈ೔഑ૹαʔϏε • ΪϑτϥοϐϯάαʔϏε • πέ෷͍ ͳͲ
  2. © ZOZO, Inc. https://wear.jp/ 7 • ϑΝογϣϯίʔσΟωʔτΞϓϦ • 1,600ສμ΢ϯϩʔυಥഁɺίʔσΟωʔτ౤ߘ૯਺͸1,300ສ݅ Ҏ্ʢ2022೥12݄຤࣌఺ʣ

    • ϐοΫΞοϓλά͔Β࠷৽ͷτϨϯυΛνΣοΫ • ίʔσΟωʔτண༻ΞΠςϜΛެࣜαΠτͰߪೖՄೳ • WEARެೝͷਓؾϢʔβʔΛWEARISTAͱೝఆɻϞσϧɾλϨϯ τɾσβΠφʔɾΠϯϑϧΤϯαʔͱ͍֤ͬͨքஶ໊ਓ΋ࢀՃ
  3. © ZOZO, Inc. ຊ೔͓࿩͢͠Δ͜ͱ • WEAR ʹ͓͚Δ EKS ίετ࠷దԽʹ޲͚ͨଧͪख ◦

    ݱঢ়ͷऔΓ૊Έͱࠓޙͷଧͪख ◦ ௐࠪͨ݁͠ՌɺԿ͕՝୊ͩͬͨͷ͔ ◦ ͳͥ࠷దԽ͕ඞཁͳͷ͔ ຊ೔͓࿩͠͠ͳ͍͜ͱ • EKS (Amazon Elastic Kubernetes Service) ͷࡉ͔ͳ࢓༷ • ಺෦ΞϓϦέʔγϣϯͷৄࡉ • Kubernetes ͷ֓ཁɺৄࡉ
  4. © ZOZO, Inc. AWS ίετ૿Λݕ஌ • EKS ίετ͕૿Ճ ◦ ౰࣌

    ECS => EKS ΁ͷϦϓϨΠεϓϩδΣΫτ͕ਐߦத ◦ ͦͷͨΊଟগͷίετ૿Ճ͸૝ఆൣғͱ͍ͯͨ͠
  5. © ZOZO, Inc. AWS ίετ૿Λݕ஌ • ཌ݄ɺߋʹ EKS ίετ͕૿Ճ ◦

    2ϲ݄લͱൺ΂ͯ໿10ഒ΁ ◦ ҟৗ஋ͱ൑அ͠ௐࠪ։࢝ 10ഒ
  6. © ZOZO, Inc. ௐࠪ݁Ռ • όονॲཧϫʔΫϑϩʔΛ࣮ߦ͢Δ Cluster ͷίετ͕ٸ૿͍ͯͨ͠ ◦ 2,400pod/day

    ͕࣮ߦ͞Ε͍ͯΔ ◦ Fargate ͷΈͰߏ੒ Mecha Ippai Fargate Batch Cluster
  7. © ZOZO, Inc. ௐࠪ݁Ռ • όονॲཧϫʔΫϑϩʔΛ࣮ߦ͢Δ Cluster ͷίετ͕ٸ૿͍ͯͨ͠ ◦ 2,400pod/day

    ͕࣮ߦ͞Ε͍ͯΔ؀ڥ ◦ Fargate ͷ࢖༻ྔ͕૿Ճ͠ίετ͕ਹΜͰ͍ͨ Mecha Ippai Fargate Batch Cluster ͕͜͜໰୊ʂ
  8. © ZOZO, Inc. උߟɿEKSίετͷߟ͑ํ • EKSίετ͸Լهͷ߹ࢉͰܭࢉ͞ΕΔ ◦ Kubernetes Worker Node

    ࣮ߦ؀ڥͷίετ ◦ EKS Cluster ͷίετ Amazon EKS + AWS Fargate Amazon EC2 EKS Cluster Kubernetes Worker Nodes
  9. © ZOZO, Inc. උߟɿWorker Node ͷίετ • Fargate ͷ৔߹ ◦

    Image Pull ։͔࢝Β Pod ͕ऴྃ͢Δ·Ͱʹ࢖༻͞ΕͨϦιʔεͰܭࢉ ◦ ࣮ߦ͞ΕΔ Fargate ͷεϖοΫʹΑΓࢉग़ AWS Fargate 1࣌ؒ౰ͨΓͷ vCPU ୯Ґ 1࣌ؒ౰ͨΓͷ GB ୯Ґ Pod਺ × ×
  10. © ZOZO, Inc. උߟɿWorker Node ͷίετ Ex. 2୆ͷ EKS Pod

    Λ30೔ϑϧՔಇͤͨ͞৔߹ Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ ߹ܭ vCPU ྉۚ = (Pod ਺) × (vCPU ਺) × (CPU ࣌ؒ͋ͨΓͷྉۚ) × (࣌ؒ୯Ґͷ 1 ೔͋ͨΓͷ CPU ࢖༻࣌ؒ) × (೔਺) ߹ܭϝϞϦྉۚ = (Pod ਺) × (ϝϞϦ (GB)) × (GB ͋ͨΓͷྉۚ) × (࣌ؒ୯Ґͷ 1 ೔͋ͨΓͷ ϝϞϦ ࢖༻࣌ؒ) × (೔਺) Fargate ͷίϯϐϡʔςΟϯάͷֹ݄ྉۚ = (CPU ֹ݄ྉۚ) + (ϝϞϦͷֹ݄ྉۚ) ྫ୊ Fargate ͷίϯϐϡʔςΟϯάͷֹ݄ྉۚ = 72.8064USD + 2.4348USD = 88.74USD ྫ୊ vCPU ྉۚ = 2 × 1 × 0.05056USD × 24 × 30 = 72.81USD ྫ୊ϝϞϦྉۚ = 2 × 2 × 0.00553USD × 24 × 30 = 15.93USD ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ
  11. © ZOZO, Inc. උߟɿWorker Node ͷίετ • EC2 ͷ৔߹ ◦

    Managed Node Group ͷ৔߹΋ࢉग़ํ๏͸ಉ͡ ◦ Worker Node ࣮ߦͷͨΊʹ࢖༻ͨ͠ AWS ϦιʔεͰࢉग़ EC2 Πϯελϯείετ EBS ϘϦϡʔϜ Πϯελϯε਺ × × Amazon EC2
  12. © ZOZO, Inc. උߟɿWorker Node ͷίετ Ex. 2୆ͷ EKS Pod

    Λ30೔ϑϧՔಇͤͨ͞৔߹ ߹ܭΠϯελϯεྉۚ = (ΠϯελϯελΠϓͷ࣌ؒ౰ͨΓྉۚ) × (Πϯελϯε਺) × (Πϯελϯε࣌ؒ) ߹ܭ EBS ྉۚ = (1 ϲ݄ʹϓϩϏδϣχϯά͞Εͨ GB) × (1 ϲ݄͋ͨΓͷΠϯελϯε࣌ؒ) × (1 GB ͋ͨΓͷֹ݄ྉۚ) ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ ྫ୊߹ܭ EBS ྉۚ = 20 × (720 ࣌ؒ / 730࣌ؒ) × 0.12USD = 2.38USD ྫ୊߹ܭΠϯελϯεྉۚ = 0.107USD × 1 × 720 = 77.04 USD ྫ୊Ͱ͋Ε͹ 2vCPU, 4GBϝϞϦ͕ඞཁͰ͋ΔͨΊɺc5.large ͔ͭ ൚༻SSD (gp2: 20GB)Ͱܭࢉ͢Δ ྫ୊߹ܭ EC2 ྉۚ = 77.04 USD + 2.38USD = 79.42USD
  13. © ZOZO, Inc. උߟɿWorker Node ͷίετ Ex. 2୆ͷ EKS Pod

    Λ30೔ϑϧՔಇͤͨ͞৔߹ ߹ܭΠϯελϯεྉۚ = (ΠϯελϯελΠϓͷ࣌ؒ౰ͨΓྉۚ) × (Πϯελϯε਺) × (Πϯελϯε࣌ؒ) ߹ܭ EBS ྉۚ = (1 ϲ݄ʹϓϩϏδϣχϯά͞Εͨ GB) × (1 ϲ݄͋ͨΓͷΠϯελϯε࣌ؒ) × (1 GB ͋ͨΓͷֹ݄ྉۚ) ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ ྫ୊߹ܭ EBS ྉۚ = 20 × (720 ࣌ؒ / 730࣌ؒ) × 0.12USD = 2.38USD ྫ୊߹ܭΠϯελϯεྉۚ = 0.107USD × 1 × 720 = 77.04 USD ྫ୊Ͱ͋Ε͹ 2vCPU, 4GBϝϞϦ͕ඞཁͰ͋ΔͨΊɺc5.large ͔ͭ ൚༻SSD (gp2: 20GB)Ͱܭࢉ͢Δ ྫ୊߹ܭ EC2 ྉۚ = 77.04 USD + 2.38USD = 79.42USD Fargateͷࠩ͸10% ※Fargate or EC2 ͸ӡ༻ίετ΋ؑΈΔඞཁ͕͋Γ·͢ɻ͝஫ҙԼ͍͞ɻ
  14. © ZOZO, Inc. ଧͪख • ଧͪखͱͯ͠ԼهΛݕ౼ ▪ ࣮ࡍͷϝτϦΫε͔Β Fargate Pod

    ͷ࣮ߦ spec Λௐ੔ ◦ Fargate spec ͷ optimize ◦ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ◦ Fargate ࣮ߦ࣌ؒͷ࡟ݮ ▪ ෆཁͳ Fargate Pod ͷࣗಈ࡟আ΍ Image Pull ଎౓ͷվળ ▪ EKS on EC2 ΁Ҡߦ͢Δ͜ͱʹΑΔίετѹॖ
  15. © ZOZO, Inc. Fargate spec ͷ optimize • ࣮ࡍͷϝτϦΫε͔Βඞཁ spec

    Λ࠶ࢉग़͠ɺManifest Λमਖ਼ • ෳ਺ͷ Deployment શͯʹରͯ͠ௐ੔
  16. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • ϫʔΫϑϩʔ࣮ߦ׬ྃޙʹ Fargate Pod ͕࡟আ͞ΕΔ·Ͱͷ࣌ؒΛઃఆ

    • كʹϫʔΫϑϩʔ࣮ߦ׬ྃޙʹ࡟আ͞Εͳ͍ Fargate Pod ͕ଘࡏ͍ͯͨ͠ ◦ job ʹରͯ͠ .spec.ttlSecondsAfterFinished Λ௥Ճ
  17. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • Image ѹॖΞϧΰϦζϜมߋʹΑΔ Image Pull

    ࣌ؒͷ࡟ݮ • gzip ѹॖํ͔ࣜΒ zstd ѹॖํࣜʹมߋ͢Δ͜ͱͰ Image αΠζΛѹॖ
  18. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • Image ѹॖΞϧΰϦζϜมߋʹΑΔ Image Pull

    ࣌ؒͷ࡟ݮ • gzip ѹॖํ͔ࣜΒ zstd ѹॖํࣜʹมߋ͢Δ͜ͱͰ Image αΠζΛѹॖ • WEARͰ͸ Image Pull ࣌ؒ΁ͷӨڹ͸গͳ͔ͬͨͨΊҰ୴ޙճ͠΁ 5%ఔ౓ѹॖ
  19. © ZOZO, Inc. EKS on Fargate ͔Β EKS on EC2

    ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ Batch Cluster Fargate Fargate Batch Cluster EC2 EC2
  20. © ZOZO, Inc. • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ Batch Cluster

    Fargate Fargate Batch Cluster EC2 EC2 ◦ Fargate Λഇࢭ͠ɺશͯ EC2 ʹࡌͤସ͑Δ ◦ EC2 Managed Node Group Λར༻͠ɺӡ༻ίετ΋཈͑Δ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
  21. © ZOZO, Inc. • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ

    EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔ Batch Cluster EC2 Critical Group EC2 Common Group EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
  22. © ZOZO, Inc. Batch Cluster EC2 Critical Group EC2 Common

    Group ࣮ߦ؀ڥʹߴ͍ Πϯελϯεੑೳ͕ෆཁͳPod Ex. Operator౳ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
  23. © ZOZO, Inc. Batch Cluster EC2 Critical Group EC2 Common

    Group EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔ ࣮ߦ؀ڥʹߴ͍ Πϯελϯεੑೳ͕ඞཁͳPod Ex. σʔλॲཧ౳
  24. © ZOZO, Inc. ◦ Manifest ʹ Node Affinity Λ௥Ճ͠એݴతʹϊʔυΛબ୒ EKS

    on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
  25. © ZOZO, Inc. Batch Cluster EC2 Critical Group EC2 Common

    Group Deployment Critical Deployment Common EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத ◦ Manifest ʹ Node Affinity Λ௥Ճ͠એݴతʹϊʔυΛબ୒ • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
  26. © ZOZO, Inc. ·ͱΊ WEARʹ͓͚Δ EKS ίετ࡟ݮʹ·ͭΘΔ͋Ε͜ΕΛ঺հ͠·ͨ͠ • ·ͣ͸ίετΛ Watch

    ͢Δ͜ͱ • ݱঢ়ʹ߹Θͤͨ࠷దԽΛଓ͚͍ͯ͘͜ͱ • αʔϏεͷϦιʔεΛ؍ଌ͠ଓ͚Δ͜ͱ ৭ʑ͓࿩͠͠·͕ͨ͠ɺେ੾ͳ͜ͱ͸Լه3఺ͩͱߟ͍͑ͯ·͢ ਐߦதͰ͸͋Γ·͕͢ɺϐʔΫ࣌ͷ50%ఔ౓·Ͱίετ࡟ݮ͕Ͱ͖͍ͯ·͢