SQS を使った非同期処理の導入

AWS cafeteria #1 〜サイバーエージェント×ゆめみ×クラスメソッド 3社共催LT会〜 で発表した資料です。
https://cyberagent.connpass.com/event/303690/

#AWScafeteria

Transcript

1. SQS Λ࢖ͬͨඇಉظॲཧͷಋೖ AWS cafeteria #1 גࣜձࣾαΠόʔΤʔδΣϯτ AI ࣄۀຊ෦ Ἒ໦ ܒޞ
2. Ἒ໦ܒޞ ɹ @oyasumi_pants 2022೥৽ଔೖࣾ CyberAgentʛAIࣄۀຊ෦ Go, GoogleCloud, AWS, Terraform ٸ͕͹ճΕ 🏢 💻 🗣 2
3. ຊ೔ͷ಺༰ 1. ීஈͲΜͳ͜ͱΛ͍ͯ͠Δ͔ʁ 2. ඇಉظॲཧͷಋೖͱ͸ʁ 3. SQS ͱ͸ʁ 4. ࣮૷લޙͷΞʔΩςΫνϟ 5. SQS બఆ࣌ʹߟྀͨ͠఺ 6. SQS બఆޙʹߟྀͨ͠఺ 7. ෛՙࢼݧͷ࿩ 8. ࠓޙͷల๬ 3
4. 1. ීஈͲΜͳ͜ͱΛ͍ͯ͠Δͷ͔ 4
5. ීஈͲΜͳ͜ͱΛ͍ͯ͠Δͷ͔ 5 • ޿ࠂ഑৴ϓϩμΫτΛ࡞͍ͬͯ·͢ʂ
6. ޿ࠂ഑৴ϓϩμΫτͷ֓ཁ
7. ޿ࠂ഑৴ϓϩμΫτͷ֓ཁ ޿ࠂओʹͱͬͯ࠷େͷ ίετύϑΥʔϚϯεΛ ࣮ݱ͢ΔΑ͏੍ޚ
8. ޿ࠂ഑৴ϓϩμΫτͷ֓ཁ ϝσΟΞଆͷऩӹ͕ ࠷େͱͳΔΑ͏ʹ੍ޚ
9. ࣗ෼ୡ͕ߏங͍ͯ͠Δ΋ͷ ޿ࠂ഑৴ϓϩμΫτͷ֓ཁ
10. 2. ඇಉظॲཧͷಋೖͱ͸ʁ 10
11. • ޿ࠂओ͔Β͍͍ͨͩͨ༧ࢉΛద੾ʹ࢖͏ͨΊʹɺࠓ͍͘Β༧ࢉΛফԽ͔ͨ͠ ͱ͍͏৘ ใΛ DynamoDB ʹॻ͖ࠐΜͰ͍Δ • ༧ࢉ͸͓ۚʹؔΘΔ͜ͱͳͷͰ ࣮֬ʹɾਖ਼֬ʹ ॻ͖ࠐ·ΕΔඞཁ͕͋Δ • ॲཧ͕ࣦഊ͢Δͱɺ༧ࢉരൃͷݒ೦͕͋ΔͷͰɺ҆શଆʹ౗ͤΔΑ͏ʹ͍ͨ͠ ༧ࢉফԽػೳͷઆ໌ 11
12. • ༧ࢉܭ্Λߦ͍ͬͯΔαʔόʔʹෛՙ͕ूதͨ͠ࡍʹ༧ࢉͷॻ͖ࠐΈ͕ࣦഊͯ͠͠ ·ͬͨ ͋Δ೔εϩοτϦϯά͕ى͖ͨ 12
13. • ༧ࢉॻ͖ࠐΈʹؔͯ͠͸ଈ࣌ੑΑΓ΋࣮֬ੑΛॏࢹ͔ͨͬͨͨ͠ΊɺSQS Λ࢖ͬͯ͜ ͷػೳͷΈ෼཭͔ͨͬͨ͠ • ·ͨɺDLQ ͷ࢓૊ΈΛ࢖ͬͯػೳͷ࠶ࢼߦɾఀࢭΛࢼΈ͔ͨͬͨ • SQS Λ࢖ͬͯҰ෦ͷॲཧΛ෼཭͢Δ͜ͱΛʮඇಉظॲཧͷಋೖʯͱݺΜͰ͍Δ εϩοτϦϯάରࡦΛ͍ͨ͠ 13
14. 3. SQS ͱ͸ʁ 14
15. • Amazon Simple Queue ServiceʢSQSʣͷ͜ͱ • ιϑτ΢ΣΞίϯϙʔωϯτؒͰ೚ҙͷྔͷϝοηʔδΛૹ৴ɺอଘɺड৴͢Δ͜ͱ ͕Ͱ͖ΔαʔϏε • ϝοηʔδΛࣦͬͨΓɺଞͷαʔϏεΛར༻͢Δඞཁ΋ͳ͍ SQS ͱ͸ʁ 15 ग़య : https://aws.amazon.com/jp/sqs/
16. SQS ͱ͸ʁ 16 ࢀߟ : https://aws.amazon.com/jp/sqs/ https://d1.awsstatic.com/webinars/jp/pdf/services/20190717_AWS-BlackBelt_AmazonSQS.pdf ʮΩϡʔʯͷʮϝοηʔδʯΛ औಘ͢ΔΞϓϦέʔγϣϯ ίϯγϡʔϚ ϓϩσϡʔα͕ੜ੒͢Δσʔλ ϝοηʔδ ʮΩϡʔʯʹʮϝοηʔδʯΛ ૹ৴͢ΔΞϓϦέʔγϣϯ ϓϩσϡʔα ϝοηʔδΛΩϡʔΠϯά͢Δ ෼ࢄΩϡʔ Ωϡʔ
17. 4. ࣮૷લޙͷΞʔΩςΫνϟ 17
18. • ECS ͕௚઀ DynamoDB ͱ Aurora ʹ ʮಡΈऔΓʯʮॻ͖ࠐΈʯΛߦ͍ͬͯͨ લޙͷΞʔΩςΫνϟ • ؒʹ SQS ͱ Lambda ΛڬΉ͜ͱͰɺ ʮ༧ࢉফԽʯ෦෼ͷΈΛ੾Γ཭ͨ͠ ࣮૷લ ࣮૷ޙ 18
19. 5. SQS બఆ࣌ʹߟྀͨ͜͠ͱ 19
20. SQS બఆ࣌ʹߟྀͨ͜͠ͱ 20 • εϧʔϓοτ͸ߴ͍ͨ͘͠...ʂ • ॻ͖ࠐΈ͸ 1౓͚ͩʹ͍ͨ͠...ʂ
21. ඪ४Ωϡʔ vs FIFO Ωϡʔͷ࿩ ͋ • API ΞΫγϣϯ͝ͱʹ 1ඵ͋ͨΓ 300τϥϯβΫγϣϯ ͷΫΥʔλΛα ϙʔτ • ϝοηʔδ͸ ඞͣ1ճ ͚ͩ഑৴ ඪ४Ωϡʔ FIFO Ωϡʔ • API ΞΫγϣϯ͝ͱʹ 1ඵ͋ͨΓ ΄΅ແ੍ݶ ͷ API ίʔϧΛαϙʔτ • ϝοηʔδ͸ গͳ͘ͱ΋1ճ ഑৴ 21 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html
22. • FIFO ΩϡʔͷߴεϧʔϓοτϞʔυΛద༻͢Δͱɺ1ඵʹ͖ͭ 3000݅ ͷϦΫΤετʹ ରԠͰ͖Δ ◦ 2023.10.04 ࣌఺Ͱ౦ژϦʔδϣϯͰ͸ඵؒ 9000τϥϯβΫγϣϯ • ϝοηʔδάϧʔϓΛ෼ࢄͤ͞Δ͜ͱͰɺͦΕͧΕͷ ϝοηʔδάϧʔϓ͕1ඵ͋ͨΓ 300݅ͷϦΫΤετΛαϙʔτ ͢Δ High Throughput Queue ʹ͍ͭͯ 22 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/high-throughput-fifo.html https://aws.amazon.com/jp/about-aws/whats-new/2023/10/amazon-sqs-increased-throughput-quota-fifo-high-throughput-mode/
23. SQS બఆ࣌ʹߟྀͨ͜͠ͱ 23 • ॲཧ͕ࣦഊͨ࣌͠ʹݕ஌Ͱ͖ΔΑ͏ʹ͍ͨ͠...ʂ
24. Dead Letter Queue ʹ͍ͭͯ 24 ϓϩσϡʔα͕ϝοηʔδΛ ૹ৴͢ΔͨΊͷओཁͳΩϡʔ ιʔεΩϡʔ ॲཧ͕ࣦഊͨ͠ࡍͷ ૹ৴ઌͱͳΔΩϡʔ σουϨλʔΩϡʔ ίϯγϡʔϚ͕ϝοηʔδΛ ड৴͠Α͏ͱ͢Δճ਺ maxReceiveCount ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
25. Dead Letter Queue ʹ͍ͭͯ 25 • DynamoDB ΁ͷॻ͖ࠐΈʹࣦഊͨ࣌͠ʹ DeadLetterQueue ʹϝοηʔδΛૹ৴͠ɺ Lambda Ͱ ഑৴ΛҰ࣌ఀࢭ͢Δ ϑϥάΛཱͨͤΔΑ͏ʹͨ͠
26. 6. SQS બఆޙʹߟྀͨ͜͠ͱ 26
27. SQS બఆޙʹߟྀͨ͜͠ͱ 27 • SQS ͷϝοηʔδΛ Lambda ʹॲཧͯ͠ཉ͍͠...ʂ
28. ΠϕϯτιʔεϚοϐϯά 28 • ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͔ΒΠϕϯτΛಡΈऔΓɺLambda ؔ਺Λݺͼग़ ͢ Lambda Ϧιʔεͷ͜ͱ • Πϕϯτιʔεͷ SQS ʹϝοηʔδ͕ૹ৴͞Εͨ͜ͱΛݕ஌ͯ͠ɺLambda ؔ਺͕࣮ ߦ͞ΕΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/invocation-eventsourcemapping.html
29. ΠϕϯτιʔεϚοϐϯάͷ batch sizeʢಉ࣌ݺͼग़্͠ݶʣ • ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͔Β Lambda ͕ಉ࣌ʹಡΈऔΕΔΠϕϯτ਺ • FIFO Ωϡʔͷ৔߹͸࠷େ஋͸ 10 • Lambda ͷಉ࣮࣌ߦ਺੍ݶ͕͋ΔͷͰɺ1ճͷ Lambda ىಈͰΑΓଟ͘ͷϝοηʔδ ΛॲཧͰ͖ΔΑ͏ʹͨ͠ 29 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html
30. ΠϕϯτιʔεϚοϐϯάͷ Lambda ࠷େಉ࣮࣌ߦ਺ 30 • ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͕ಉ࣌ʹݺͼग़ͤΔ Lambda ؔ਺ͷ࠷େ਺ • FIFO Ωϡʔͷ৔߹ɺಉ࣌ݺͼग़͠ͷ্ݶ͸ɺʮϝοηʔδάϧʔϓ ID ͷ਺ʯ·ͨ͸ ʮ࠷େಉ࣮࣌ߦ਺ͷઃఆʯͷͲͪΒ͔ খ͍͞ํ ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html
31. ΠϕϯτιʔεϚοϐϯάͷ࠷େಉ࣮࣌ߦ਺ 31 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html
32. ΠϕϯτιʔεϚοϐϯάͷ࠷େಉ࣮࣌ߦ਺ 32 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html
33. SQS બఆޙʹߟྀͨ͜͠ͱ 33 • ϝοηʔδ͕ద੾ʹॲཧ͞Εͯཉ͍͠...ʂ
34. SQS ͷՄࢹੑλΠϜΞ΢τʹ͍ͭͯ 34 • ϝοηʔδ͕ड৴͞Εͨ௚ޙʹɺଞͷίϯγϡʔϚ͕ಉ͡ϝοηʔδͷड৴΍ॲཧΛ Ͱ͖ͳ͘ͳΔظؒͷ͜ͱ • ՄࢹੑλΠϜΞ΢τ͸ Lambda ͷ࣮ߦ࣌ؒͷগͳ͘ͱ΋ 6ഒ ʹઃఆ͞ΕΔ͜ͱ͕ਪ঑ ͞Ε͍ͯΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
35. SQS ͷ maxReceiveCount ʹ͍ͭͯ 35 • ίϯγϡʔϚʔ͕ϝοηʔδΛड৴͠Α͏ͱͨ͠ࡍʹɺmaxReceiveCount ΛॳΊͯ ௒͑ͨ࣌ʹ DLQ ʹߦ͘࢓༷ʹͳ͍ͬͯΔ • ϝοηʔδ͕ॲཧ͞ΕΔ֬཰ΛߴΊΔʹ͸ maxReceiveCount Λ 5 Ҏ্ʹઃఆ͢Δ͜ ͱ͕ਪ঑͞Ε͍ͯΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
36. 7. ෛՙࢼݧͷ࿩ 36
37. • k6 Λ࢖ͬͨෛՙࢼݧΛߦͬͨ • ౰࣌ͷฏۉ qps ͷ 100 ഒ·Ͱ଱͑ΒΕΔ͔ͷෛՙࢼݧΛߦ͑ͨ ෛՙࢼݧͷ࿩ 37 ࢀߟ : https://k6.io/
38. • ࠷ॳ͸ϝοηʔδάϧʔϓ ID Λ୯Ұʹ͍ͯͨ͠ͷͰύϑΥʔϚϯε͕Ͱͳ͔ͬͨ • SQS FIFO ΩϡʔͷߴεϧʔϓοτϞʔυͰ͸ɺϝοηʔδάϧʔϓ ʹ͖ͭ 1ඵؒ 300 ݅ͷϦΫΤετΛαϙʔτ͢ΔͷͰɺͦ͜Λద੾ʹ෼ࢄͤͨ͞ Queue ͷ API Limit ͷ্ݶʹର͢Δݟམͱ͠ 38 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
39. 8. ࠓޙͷల๬ 39
40. • LT ͷࢿྉ࡞੒Λ͢Δ·ͰΠϕϯτιʔεϚοϐϯάͷ Lambda ಉ࣮࣌ߦ਺੍ޚΛ஌Β ͳ͔ͬͨ... ࠓޙͷల๬ 40
41. • SQS ͷ API Limit Λ௒͑ͯ͠·ͬͨ৔߹ʹෳ਺ͷ Queue ʹ෼ࢄͤ͞Δඞཁ͕͋Δ ࠓޙͷల๬ 41
42. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ 42