WEARのEKSコストを救いたい

【Autify x ZOZO x dip共同開催】AWSコスト削減事例祭り
https://zozotech-inc.connpass.com/event/273113/

Transcript

1. WEARͷEKSίετΛٹ͍͍ͨ גࣜձࣾZOZO ϒϥϯυιϦϡʔγϣϯ։ൃຊ෦ όοΫΤϯυ෦ SREϒϩοΫ খྛ ະདྷ Copyright © ZOZO, Inc. AWSίετ࡟ݮࣄྫࡇΓ
2. © ZOZO, Inc. ࣗݾ঺հ
3. © ZOZO, Inc. גࣜձࣾZOZO ϒϥϯυιϦϡʔγϣϯ։ൃຊ෦ όοΫΤϯυ෦ SREϒϩοΫ খྛ ະདྷ • @mirai_kobaaaaaa • ΏΔ;ΘܥSRE • ޷͖ͳαʔϏεɿAWS CDK
4. © ZOZO, Inc.
6. © ZOZO, Inc. https://zozo.jp/ 6 • ϑΝογϣϯEC • 1,500Ҏ্ͷγϣοϓɺ8,500Ҏ্ͷϒϥϯυͷऔΓѻ͍ • ৗ࣌90ສ఺Ҏ্ͷ঎඼ΞΠςϜ਺ͱຖ೔ฏۉ2,600఺Ҏ্ͷ৽ண ঎඼Λܝࡌʢ2022೥12݄຤࣌఺ʣ • ϒϥϯυݹணͷϑΝογϣϯκʔϯʮZOZOUSEDʯ΍ίεϝઐ໳ ϞʔϧʮZOZOCOSMEʯɺۺͷઐ໳Ϟʔϧ ʮZOZOSHOESʯɺϥάδϡΞϦʔˍσβΠφʔζκʔϯ ʮZOZOVILLAʯΛల։ • ଈ೔഑ૹαʔϏε • ΪϑτϥοϐϯάαʔϏε • πέ෷͍ ͳͲ
7. © ZOZO, Inc. https://wear.jp/ 7 • ϑΝογϣϯίʔσΟωʔτΞϓϦ • 1,600ສμ΢ϯϩʔυಥഁɺίʔσΟωʔτ౤ߘ૯਺͸1,300ສ݅ Ҏ্ʢ2022೥12݄຤࣌఺ʣ • ϐοΫΞοϓλά͔Β࠷৽ͷτϨϯυΛνΣοΫ • ίʔσΟωʔτண༻ΞΠςϜΛެࣜαΠτͰߪೖՄೳ • WEARެೝͷਓؾϢʔβʔΛWEARISTAͱೝఆɻϞσϧɾλϨϯ τɾσβΠφʔɾΠϯϑϧΤϯαʔͱ͍֤ͬͨքஶ໊ਓ΋ࢀՃ
8. © ZOZO, Inc. ຊ೔͓࿩͢͠Δ͜ͱ
9. © ZOZO, Inc. ຊ೔͓࿩͢͠Δ͜ͱ • WEAR ʹ͓͚Δ EKS ίετ࠷దԽʹ޲͚ͨଧͪख ◦ ݱঢ়ͷऔΓ૊Έͱࠓޙͷଧͪख ◦ ௐࠪͨ݁͠ՌɺԿ͕՝୊ͩͬͨͷ͔ ◦ ͳͥ࠷దԽ͕ඞཁͳͷ͔ ຊ೔͓࿩͠͠ͳ͍͜ͱ • EKS (Amazon Elastic Kubernetes Service) ͷࡉ͔ͳ࢓༷ • ಺෦ΞϓϦέʔγϣϯͷৄࡉ • Kubernetes ͷ֓ཁɺৄࡉ
10. © ZOZO, Inc. ͋Ε͸2022೥๭೔
11. © ZOZO, Inc. AWS ίετ૿Λݕ஌ • EKS ίετ͕૿Ճ ◦ ౰࣌ ECS => EKS ΁ͷϦϓϨΠεϓϩδΣΫτ͕ਐߦத ◦ ͦͷͨΊଟগͷίετ૿Ճ͸૝ఆൣғͱ͍ͯͨ͠
12. © ZOZO, Inc. AWS ίετ૿Λݕ஌ • ཌ݄ɺߋʹ EKS ίετ͕૿Ճ ◦ 2ϲ݄લͱൺ΂ͯ໿10ഒ΁ ◦ ҟৗ஋ͱ൑அ͠ௐࠪ։࢝ 10ഒ
13. © ZOZO, Inc. ௐࠪ݁Ռ
14. © ZOZO, Inc. ௐࠪ݁Ռ • όονॲཧϫʔΫϑϩʔΛ࣮ߦ͢Δ Cluster ͷίετ͕ٸ૿͍ͯͨ͠ ◦ 2,400pod/day ͕࣮ߦ͞Ε͍ͯΔ ◦ Fargate ͷΈͰߏ੒ Mecha Ippai Fargate Batch Cluster
15. © ZOZO, Inc. ௐࠪ݁Ռ • όονॲཧϫʔΫϑϩʔΛ࣮ߦ͢Δ Cluster ͷίετ͕ٸ૿͍ͯͨ͠ ◦ 2,400pod/day ͕࣮ߦ͞Ε͍ͯΔ؀ڥ ◦ Fargate ͷ࢖༻ྔ͕૿Ճ͠ίετ͕ਹΜͰ͍ͨ Mecha Ippai Fargate Batch Cluster ͕͜͜໰୊ʂ
16. © ZOZO, Inc. උߟɿEKSίετͷߟ͑ํ • EKSίετ͸Լهͷ߹ࢉͰܭࢉ͞ΕΔ ◦ Kubernetes Worker Node ࣮ߦ؀ڥͷίετ ◦ EKS Cluster ͷίετ Amazon EKS + AWS Fargate Amazon EC2 EKS Cluster Kubernetes Worker Nodes
17. © ZOZO, Inc. උߟɿWorker Node ͷίετ • Fargate ͷ৔߹ ◦ Image Pull ։͔࢝Β Pod ͕ऴྃ͢Δ·Ͱʹ࢖༻͞ΕͨϦιʔεͰܭࢉ ◦ ࣮ߦ͞ΕΔ Fargate ͷεϖοΫʹΑΓࢉग़ AWS Fargate 1࣌ؒ౰ͨΓͷ vCPU ୯Ґ 1࣌ؒ౰ͨΓͷ GB ୯Ґ Pod਺ × ×
18. © ZOZO, Inc. උߟɿWorker Node ͷίετ Ex. 2୆ͷ EKS Pod Λ30೔ϑϧՔಇͤͨ͞৔߹ Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ ߹ܭ vCPU ྉۚ = (Pod ਺) × (vCPU ਺) × (CPU ࣌ؒ͋ͨΓͷྉۚ) × (࣌ؒ୯Ґͷ 1 ೔͋ͨΓͷ CPU ࢖༻࣌ؒ) × (೔਺) ߹ܭϝϞϦྉۚ = (Pod ਺) × (ϝϞϦ (GB)) × (GB ͋ͨΓͷྉۚ) × (࣌ؒ୯Ґͷ 1 ೔͋ͨΓͷ ϝϞϦ ࢖༻࣌ؒ) × (೔਺) Fargate ͷίϯϐϡʔςΟϯάͷֹ݄ྉۚ = (CPU ֹ݄ྉۚ) + (ϝϞϦͷֹ݄ྉۚ) ྫ୊ Fargate ͷίϯϐϡʔςΟϯάͷֹ݄ྉۚ = 72.8064USD + 2.4348USD = 88.74USD ྫ୊ vCPU ྉۚ = 2 × 1 × 0.05056USD × 24 × 30 = 72.81USD ྫ୊ϝϞϦྉۚ = 2 × 2 × 0.00553USD × 24 × 30 = 15.93USD ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ
19. © ZOZO, Inc. උߟɿWorker Node ͷίετ • EC2 ͷ৔߹ ◦ Managed Node Group ͷ৔߹΋ࢉग़ํ๏͸ಉ͡ ◦ Worker Node ࣮ߦͷͨΊʹ࢖༻ͨ͠ AWS ϦιʔεͰࢉग़ EC2 Πϯελϯείετ EBS ϘϦϡʔϜ Πϯελϯε਺ × × Amazon EC2
20. © ZOZO, Inc. උߟɿWorker Node ͷίετ Ex. 2୆ͷ EKS Pod Λ30೔ϑϧՔಇͤͨ͞৔߹ ߹ܭΠϯελϯεྉۚ = (ΠϯελϯελΠϓͷ࣌ؒ౰ͨΓྉۚ) × (Πϯελϯε਺) × (Πϯελϯε࣌ؒ) ߹ܭ EBS ྉۚ = (1 ϲ݄ʹϓϩϏδϣχϯά͞Εͨ GB) × (1 ϲ݄͋ͨΓͷΠϯελϯε࣌ؒ) × (1 GB ͋ͨΓͷֹ݄ྉۚ) ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ ྫ୊߹ܭ EBS ྉۚ = 20 × (720 ࣌ؒ / 730࣌ؒ) × 0.12USD = 2.38USD ྫ୊߹ܭΠϯελϯεྉۚ = 0.107USD × 1 × 720 = 77.04 USD ྫ୊Ͱ͋Ε͹ 2vCPU, 4GBϝϞϦ͕ඞཁͰ͋ΔͨΊɺc5.large ͔ͭ ൚༻SSD (gp2: 20GB)Ͱܭࢉ͢Δ ྫ୊߹ܭ EC2 ྉۚ = 77.04 USD + 2.38USD = 79.42USD
21. © ZOZO, Inc. උߟɿWorker Node ͷίετ Ex. 2୆ͷ EKS Pod Λ30೔ϑϧՔಇͤͨ͞৔߹ ߹ܭΠϯελϯεྉۚ = (ΠϯελϯελΠϓͷ࣌ؒ౰ͨΓྉۚ) × (Πϯελϯε਺) × (Πϯελϯε࣌ؒ) ߹ܭ EBS ྉۚ = (1 ϲ݄ʹϓϩϏδϣχϯά͞Εͨ GB) × (1 ϲ݄͋ͨΓͷΠϯελϯε࣌ؒ) × (1 GB ͋ͨΓͷֹ݄ྉۚ) ※2023/2/10࣌఺Ͱͷ౦ژϦʔδϣϯྉ͔ۚΒࢼࢉ Pod ͷεϖοΫ͸ 1vCPU, 2GBϝϞϦͱ͢Δ ྫ୊߹ܭ EBS ྉۚ = 20 × (720 ࣌ؒ / 730࣌ؒ) × 0.12USD = 2.38USD ྫ୊߹ܭΠϯελϯεྉۚ = 0.107USD × 1 × 720 = 77.04 USD ྫ୊Ͱ͋Ε͹ 2vCPU, 4GBϝϞϦ͕ඞཁͰ͋ΔͨΊɺc5.large ͔ͭ ൚༻SSD (gp2: 20GB)Ͱܭࢉ͢Δ ྫ୊߹ܭ EC2 ྉۚ = 77.04 USD + 2.38USD = 79.42USD Fargateͷࠩ͸10% ※Fargate or EC2 ͸ӡ༻ίετ΋ؑΈΔඞཁ͕͋Γ·͢ɻ͝஫ҙԼ͍͞ɻ
22. © ZOZO, Inc. ଧͪख
23. © ZOZO, Inc. ଧͪख • ଧͪखͱͯ͠ԼهΛݕ౼ ▪ ࣮ࡍͷϝτϦΫε͔Β Fargate Pod ͷ࣮ߦ spec Λௐ੔ ◦ Fargate spec ͷ optimize ◦ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ◦ Fargate ࣮ߦ࣌ؒͷ࡟ݮ ▪ ෆཁͳ Fargate Pod ͷࣗಈ࡟আ΍ Image Pull ଎౓ͷվળ ▪ EKS on EC2 ΁Ҡߦ͢Δ͜ͱʹΑΔίετѹॖ
24. © ZOZO, Inc. Fargate spec ͷ optimize • Datadog μογϡϘʔυ͔Β pod ͷར༻ϝτϦΫεΛνΣοΫ
25. © ZOZO, Inc. Fargate spec ͷ optimize • ࣮ࡍͷϝτϦΫε͔Βඞཁ spec Λ࠶ࢉग़͠ɺManifest Λमਖ਼ • ෳ਺ͷ Deployment શͯʹରͯ͠ௐ੔
26. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • ϫʔΫϑϩʔ࣮ߦ׬ྃޙʹ Fargate Pod ͕࡟আ͞ΕΔ·Ͱͷ࣌ؒΛઃఆ • كʹϫʔΫϑϩʔ࣮ߦ׬ྃޙʹ࡟আ͞Εͳ͍ Fargate Pod ͕ଘࡏ͍ͯͨ͠ ◦ job ʹରͯ͠ .spec.ttlSecondsAfterFinished Λ௥Ճ
27. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • Image ѹॖΞϧΰϦζϜมߋʹΑΔ Image Pull ࣌ؒͷ࡟ݮ • gzip ѹॖํ͔ࣜΒ zstd ѹॖํࣜʹมߋ͢Δ͜ͱͰ Image αΠζΛѹॖ
28. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • Image ѹॖΞϧΰϦζϜมߋʹΑΔ Image Pull ࣌ؒͷ࡟ݮ • gzip ѹॖํ͔ࣜΒ zstd ѹॖํࣜʹมߋ͢Δ͜ͱͰ Image αΠζΛѹॖ • WEARͰ͸ Image Pull ࣌ؒ΁ͷӨڹ͸গͳ͔ͬͨͨΊҰ୴ޙճ͠΁ 5%ఔ౓ѹॖ
29. © ZOZO, Inc. Fargate ࣮ߦ࣌ؒͷ࡟ݮ • AWS ެࣜΑΓϒϩά͕ެ։͞Ε͍ͯ·͢ͷͰੋඇ͝ҰಡԼ͍͞ʂ ref. https://aws.amazon.com/jp/blogs/news/reducing-aws-fargate-startup-times-with-zstd-compressed-container-images/
30. © ZOZO, Inc. EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ Batch Cluster Fargate Fargate Batch Cluster EC2 EC2
31. © ZOZO, Inc. • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ Batch Cluster Fargate Fargate Batch Cluster EC2 EC2 ◦ Fargate Λഇࢭ͠ɺશͯ EC2 ʹࡌͤସ͑Δ ◦ EC2 Managed Node Group Λར༻͠ɺӡ༻ίετ΋཈͑Δ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
32. © ZOZO, Inc. • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔ Batch Cluster EC2 Critical Group EC2 Common Group EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத
33. © ZOZO, Inc. Batch Cluster EC2 Critical Group EC2 Common Group ࣮ߦ؀ڥʹߴ͍ Πϯελϯεੑೳ͕ෆཁͳPod Ex. Operator౳ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
34. © ZOZO, Inc. Batch Cluster EC2 Critical Group EC2 Common Group EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔ ࣮ߦ؀ڥʹߴ͍ Πϯελϯεੑೳ͕ඞཁͳPod Ex. σʔλॲཧ౳
35. © ZOZO, Inc. ◦ Manifest ʹ Node Affinity Λ௥Ճ͠એݴతʹϊʔυΛબ୒ EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
36. © ZOZO, Inc. Batch Cluster EC2 Critical Group EC2 Common Group Deployment Critical Deployment Common EKS on Fargate ͔Β EKS on EC2 ΁ͷҠߦ ※ݕূத ◦ Manifest ʹ Node Affinity Λ௥Ճ͠એݴతʹϊʔυΛબ୒ • Pod ࣮ߦج൫Λ EC2 ʹมߋ͢Δ͜ͱͰɺߋͳΔίετѹॖΛਤΔ • ୯७ͳ EKS on EC2 ΁ͷҠߦͰ͸ͳ͘ɺPod ͷੑ࣭ຖʹ഑ஔ͢ΔϊʔυΛௐ੔
37. © ZOZO, Inc. ·ͱΊ
38. © ZOZO, Inc. ·ͱΊ WEARʹ͓͚Δ EKS ίετ࡟ݮʹ·ͭΘΔ͋Ε͜ΕΛ঺հ͠·ͨ͠ • ·ͣ͸ίετΛ Watch ͢Δ͜ͱ • ݱঢ়ʹ߹Θͤͨ࠷దԽΛଓ͚͍ͯ͘͜ͱ • αʔϏεͷϦιʔεΛ؍ଌ͠ଓ͚Δ͜ͱ ৭ʑ͓࿩͠͠·͕ͨ͠ɺେ੾ͳ͜ͱ͸Լه3఺ͩͱߟ͍͑ͯ·͢ ਐߦதͰ͸͋Γ·͕͢ɺϐʔΫ࣌ͷ50%ఔ౓·Ͱίετ࡟ݮ͕Ͱ͖͍ͯ·͢
39. © ZOZO, Inc. Զͨͪͷઓ͍͸͜Ε͔Βͩʂ
40. © ZOZO, Inc. We are hiring!
41. © ZOZO, Inc. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ