ISUCON運営を支えるAmazon ECSとAurora Serverless v2 / AWS Dev Day 2022 Japan

https://aws.amazon.com/jp/events/devday/japan/

Transcript

1. ISUCONӡӦΛࢧ͑Δ Amazon ECSͱ Aurora Serverless v2 ໘ന๏ਓΧϠοΫ ౻ݪढ़Ұ࿠ (@fujiwara) AWS Dev Day 2022 Japan 2022.11.10
2. @fujiwara SREνʔϜ ISUCON 1,2,5,11 ! ༏উ4ճ ISUCON 3,8,12 ӡӦ(ग़୊)3ճ github.com/kayac/ecspresso Amazon ECS σϓϩΠπʔϧ github.com/fujiwara/lambroll AWS Lambda σϓϩΠπʔϧ
3. ISUCONͱ͸ LINEגࣜձࣾओ࠵ ʮͳΜͰ΋͋ΓʯͷWebαʔϏεύϑΥʔϚϯε νϡʔχϯάίϯςετ 2011೥։࢝ʙ2022೥Ͱ12ճ໨
4. ISUCONͱ͸ WebΞϓϦέʔγϣϯ͕ಈ࡞͍ͯ͠ΔαʔόʔΠϯελϯε͕ ༻ҙ͞ΕΔ(1୆ʙ࠷େ5୆) Πϯελϯε֎෦͔ΒϕϯνϚʔΫ(HTTPϦΫΤετ)࣮ߦ ॲཧͰ͖ͨϦΫΤετ਺ͳͲ͔ΒείΞΛࢉग़ ڝٕ࣌ؒ8࣌ؒ ϕϯνϚʔΫείΞ͕Ұ൪ߴ͍νʔϜ͕༏উʂ
5. ISUCON͸ʮͳΜͰ΋͋Γʯ ʮͳΜͰ΋ʯʹʮ༩͑ΒΕͨڝٕ༻Πϯελϯεͷ্Ͱ ಈ͘΋ͷ͸ͳΜͰ΋1มߋͯ͠Α͍ʯ • WebΞϓϦέʔγϣϯͷ࣮૷(ݴޠؚΊͯ) • ϛυϧ΢ΣΞ΍OSͷઃఆ • ೚ҙͷιϑτ΢ΣΞ΍ϛυϧ΢ΣΞͷಋೖ ֎ܗతͳڍಈΛҡ͍࣋ͯ͠Ε͹ɺߴ଎Խ͢Δख๏ʹ੍ݶ͸ͳ͍ 1 Ұ෦ϨΪϡϨʔγϣϯʹ໌ه͞Εͨࣄ߲Λআ͘
6. ISUCON12֓ཁ ΦϯϥΠϯ༧બ 2022-07-23(౔) 10:00-18:00 ࢀՃνʔϜ਺ 698૊ʢֶੜ 98૊ʣ ࢀՃऀ߹ܭ 1657໊ʢֶੜ 246໊ʣ ΦϯϥΠϯຊબ 2022-08-27೔(౔) 10:00-18:00 ࢀՃνʔϜ਺ 30૊ʢֶੜ 8૊ʣ ࢀՃऀ߹ܭ 84໊ʢֶੜ 22໊ʣ
7. ISUCONϙʔλϧαΠτ
8. ISUCONϙʔλϧͷ࢓ࣄ ਃ͠ࠐΈ͔Β༧બ·Ͱ νʔϜɾϝϯόʔࢀՃड͚෇͚ ɹGitHub, DiscordΞΧ΢ϯτ࿈ܞ AWSಈ࡞ݕূ༻CloudFormation(CFn) ςϯϓϨʔτ഑෍ ɹ࣮ࡍʹEC2ΛىಈɺsshՄೳ͔ݕূ Ϋʔϙϯ഑෍ ɹAWS༷ɺ͘͞ΒΠϯλʔωοτ༷
9. ISUCONϙʔλϧͷ࢓ࣄ ༧બ / ຊબ ڝٕ༻ΠϯελϯεͷIPΞυϨεΛ೺Ѳ ɹϗετ্ͷagent͕ϙʔλϧͱ௨৴ ϕϯνϚʔΫ࣮ߦ ɹࢦఆͨ͠Πϯελϯεʹର࣮ͯ͠ߦ ɹϕϯνϚʔΫ݁Ռొ࿥API ڝٕதͷ݁ՌӾཡ ɹϕϯνϚʔΫ݁ՌҰཡͱৄࡉ ɹશνʔϜͷείΞਪҠ
10. ISUCON12ϙʔλϧͷ࣮૷ https://github.com/isucon/portal ࣮͸OSS(MIT Lisense)ͳͷͰͲͳͨͰ ΋࢖͑·͢2 @sorah ͕ISUCON10Ͱ࣮૷ վमɺҾ͖ܧ͍Ͱ3୅໨ 2 υΩϡϝϯτ͸΄΅ͳ͍ͨΊ؆୯ʹಈ͔ͤΔΘ͚Ͱ͸ͳ͍…
11. ISUCON12ϙʔλϧͷߏ੒ Rails + Shoryuken (SQS job worker) ɹSPAʹ͍ۙߏ੒ ɹReact + REST API (Protobuf) ɹDiscord, GitHub࿈ܞͳͲ͸ඇಉظjobͰ࣮ߦ ɹDiscord bot ϕϯνϚʔΧʔ ɹϙʔλϧAPI͔ΒjobΛऔಘ (Rust੡daemon) ɹ→ ର৅ΠϯελϯεʹϕϯνϚʔΫ࣮ߦ (Go੡CLI) ɹϕϯνϚʔΫ݁ՌΛϙʔλϧAPIʹૹ৴
12. ISUCON12ϙʔλϧ͸AWS্ʹσϓϩΠ Amazon CloudFront Application Load Balancer (ALB) Amazon Simple Queue Service (SQS) Amazon ElastiCache Redis Amazon ECS Fargate Aurora MySQL Serverless v2 [New!] ɹ2022-04 GA ɹͪΐ͏Ͳ࢖ͬͯΈ͔ͨͬͨʂ
13. ISUCON11→12Ͱͷ͓΋ͳมߋ఺ • ϕϯνϚʔΧʔ͕ EC2 → ECS Fargate ʹ • ISUCON11 νʔϜ͝ͱʹ1୆ͷ EC2 Λ੩తׂΓ౰ͯ • ISUCON12 700νʔϜʹରͯ͠ Fargate 200 λεΫΛڞ༗ • RDS ͕ Aurora Serverless v2 ʹʂ
14. ISUCONϙʔλϧͷར༻ঢ়گ ࢀՃਃ͠ࠐΈ 220ʙ230νʔϜΛ3ճʹ෼͚ͯࢀՃొ࿥ 1. ॳճ 2෼40ඵͰ׬ച 2. 2ճ໨ 1෼10ඵͰ׬ച 3. 3ճ໨ 20෼ఔ౓Ͱ׬ച ਃ͠ࠐΈ࿮͕ຒ·Δ਺෼͚ͩ ۃ୺ʹΞΫηε͕ଟ͍
15. ISUCONϙʔλϧͷར༻ঢ়گ ༧બ౰೔ ڝٕ։࢝࣌ࠁʹڝٕ༻ϖʔδ͕open ࢀՃऀ1700໊͕͍͍ͬͤʹΞΫηε CFnςϯϓϨʔτμ΢ϯϩʔυ ϕϯνϚʔΫ࣮ߦ ɹ༧બ࣌ 40,000ճ/8࣌ؒ = ฏۉ83ճ/෼ ɹϐʔΫ࣌120ճ/෼
16. ISUCONϙʔλϧʹ͸Aurora Serverless v2͕࠷ద? ΞΫηεͷ૿ݮ͕ۃΊͯݦஶ ΞΫηε͕ଟ͍ظؒ͸ಛʹ୹͍(਺෼ʙ਺࣌ؒ) 3ϲ݄ͷӡӦظؒதɺ΄ͱΜͲ͸Ջ ࠷খ Aurora Capacity Unit(ACU) 0.5Ͱे෼ ΞΫηε͕૿ݮ͢ΔλΠϛϯά͸ಡΊΔɺ͕… աڈͷܦݧ͕ͳ͍ͨΊ૿Ճ཰͕ಡΈͮΒ͍ ݟੵ΋ΓΛޡΔͱ଍Γͳ͍ = ো֐ͷՄೳੑ͕͋Δ
17. ISUCONϙʔλϧʹ͸Aurora Serverless v2͕࠷ద! ෛՙ͕૿ݮ͢ΔλΠϛϯά = ਃ͠ࠐΈ΍ڝٕ։࢝ɺऴྃ࣌ → ӡӦ͕ଟ๩ ࣄલεέʔϧΞοϓɺࣄޙεέʔϧμ΢ϯͷखΛֻ͚ͨ͘ͳ͍ ίετ΋ൺֱత͍҆ ΠϯελϯελΠϓ USD/h USD/month db.t3.small 0.063 45.99 Serverless v2 (0.5 ACU) 0.10 73.00 db.t4g.medium 0.113 82.49
18. ࣮ࡍͷར༻ঢ়گΛϝτϦΫεͰಡΈղ͘ υΩϡϝϯτ Aurora Serverless v2ͷॏཁͳAmazon CloudWatchϝτϦΫε3 ͔Β ServerlessDatabaseCapacity ʹ஫໨ ʼ ΠϯελϯεϨϕϧͷϝτϦΫεͱͯ͠ɺݱࡏͷDBΠϯελϯεͷ༰ྔͰ ද͞ΕΔ ACU ஋Λใࠂ͠·͢ 3 https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.setting-capacity.html#aurora- serverless-v2.parameters-based-on-scaling
19. ࢀՃొ࿥ୈ2ظ 70ඵͰ230νʔϜ͕׬ച 42k/min = 700req/secఔ౓ ɹSPAͳͷͰ΄΅͢΂ͯAPI Fargate 16 task (1 vCPU 4GiB) ɹCPU ΄΅100%ʹ ɹFargate͸ࣄલʹtask਺खಈઃఆ άϥϑ͸CloudFrontͷϦΫΤετ਺
20. ࢀՃొ࿥ୈ2ظ Queries & ServerlessDatabaseCapacity Queries 30 qps → 600 qps ServerlessDatabaseCapacity 0.5 → 2 Կ΋͠ͳ͍Ͱ΋ෛՙʹ௥ैͯ͠໰୊ͳ͠ 1෼ͰͷϐʔΫ͸΋ͬͱӶ͍4 4 ϝτϦΫεऔಘ࣌ʹ5෼ཻ౓ͷ஋͔͠࢒͍ͬͯͳ͔ͬͨͨΊ
21. ༧બ։࢝࣌ Queries & ServerlessDatabaseCapacity 9:40 ͡Θ͡ΘͱΞΫηε͕૿͑Δ 10:00 1700໊͕Ұ੪ʹΞΫηε 10:00ͷΫΤϦ਺ ɹ9:55ͱൺֱͯ͠10ഒ ɹ9:59ͱൺֱͯ͠5ഒ ServerlessDatabaseCapacity 0.5 → 3ʹॠ࣌ʹ௥ै ઌճΓ্͕͍ͯͬͯ͠ΔΑ͏ʹ΋ʁ
22. ༧બ։࢝࣌ SelectLatency & ServerlessDatabaseCapacity ։࢝લ͸͍Ζ͍ΖͳΫΤϦ͕গ਺ → ϨΠςϯγ͸҆ఆ͠ͳ͍ ։࢝௚ޙ͸Ұ੪ʹಉछͷΞΫηε → ϨΠςϯγ͕҆ఆ → Queries, ACU͕ٸ૿ͯ͠΋҆ఆ
23. ༧બ1೔Λ௨ͯ͠ 10:00 ɹQueries 1.2k/sec ɹServerlessDatabaseCapacity 2.5 18:00 ɹQueries 2.8k/sec ɹServerlessDatabaseCapacity 8.5 20࣌Ҏ߱ʹ௓Ͷ͍ͯΔͷ͸ӡӦʹΑΔ ௥ࢼͷӨڹ(ޙड़) Fargate͸๨Ε͍ͯͨ (ΦʔτεέʔϧͰ͍͍ײ͡ʹ૿͑ͨ)
24. ༧બ1೔Λ௨ͯ͠ SelectLatency & ServerlessDatabaseCapacity 10:00ʙ17:00·ͰLatency͸҆ఆ 17:00ʙ18:00͸ϙʔλϧͷڍಈ (ൃߦ͢ΔΫΤϦ)͕มΘͬͨӨڹͷมԽ
25. ༧બऴྃޙͷӡӦ௥ࢼ ڝٕ༻ΠϯελϯεΛશ୆࠶ىಈޙ ڝٕதͷείΞ͕࠶ݱͰ͖Δ͔௥ࢼ 700νʔϜͷϕϯνϚʔΫҰؾʹ౤ೖ × 3ճ CLIΛ࣮ߦ͍ͯͨ͠ମײͰ͸ 1ճ໨࠷ॳ਺ඵҾ͔͔ͬΔ → ͦͷޙεϜʔζʹྲྀΕͨ
26. ༧બऴྃޙͷӡӦ௥ࢼ ࠷ॳͷϨΠςϯγ͕େ͖Ί Capacity͸एׯ஗Εͯ૿Ճ 2, 3ճ໨͸Capacity͕མ͖͍ͪͬͯͳ͍ →εϜʔζʹ௥ै͍ͯ͠Δ "Aurora Serverless v2Ͱͷ ύϑΥʔϚϯεͱεέʔϦϯά"5 ʼ ݱࡏͷ༰ྔ͕େ͖͍΄ͲɺεέʔϧΞοϓ͕ ଎͘ͳΓ·͢ 5 https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/AuroraUserGuide/ aurora-serverless-v2.setting-capacity.html
27. εέʔϧΠϯ࣌ͷڍಈ Queries(੨) & ServerlessDatabaseCapacity(ᒵ) 18:00ʙ ঃʑʹԼ͕Δ͕2.5ఔ౓Λҡ࣋ 21:00ʙ 0.5·Ͱݮগ ૿Ճ࣌͸ෛՙʹԠͯ͡Ұؾʹ্͕Δ ݮগ࣌͸ෛՙ͕ݮͬͯ΋͙͢ʹ࠷௿·Ͱ ͸Լ͕Βͣ ͋Δఔ౓ҡ࣋ͨ͋͠ͱঃʑʹ௿Լ͢Δ
28. ·ͱΊ Aurora Serverless v2ΛISUCON12ϙʔλϧʹ࣮ઓ౤ೖ͠·ͨ͠ ಉҰੑೳͰprovisionedͳΠϯελϯεͱൺֱ͢ΔͱߴՁ͕ͩ ࣗಈͰෛՙ௥ैͯ͘͠ΕΔͨΊӡ༻ίετ͕ۃΊͯ௿͍ ීஈ͸Ջɺ౰೔ʹಥൃෛՙ͕ൃੜ͢ΔΠϕϯτ༻ͷαΠτӡӦ ʹ͸ۃΊ࣮ͯ༻తͳಛੑ