SQS を使った非同期処理の導入

by Keigo Ibaraki

Slide 1

Slide 1 text

SQS Λ࢖ͬͨඇಉظॲཧͷಋೖ AWS cafeteria #1 גࣜձࣾαΠόʔΤʔδΣϯτ AI ࣄۀຊ෦ Ἒ໦ ܒޞ

Slide 2

Slide 2 text

Ἒ໦ܒޞ ɹ @oyasumi_pants 2022೥৽ଔೖࣾ CyberAgentʛAIࣄۀຊ෦ Go, GoogleCloud, AWS, Terraform ٸ͕͹ճΕ 🏢 💻 🗣 2

Slide 3

Slide 3 text

ຊ೔ͷ಺༰ 1. ීஈͲΜͳ͜ͱΛ͍ͯ͠Δ͔ʁ 2. ඇಉظॲཧͷಋೖͱ͸ʁ 3. SQS ͱ͸ʁ 4. ࣮૷લޙͷΞʔΩςΫνϟ 5. SQS બఆ࣌ʹߟྀͨ͠఺ 6. SQS બఆޙʹߟྀͨ͠఺ 7. ෛՙࢼݧͷ࿩ 8. ࠓޙͷల๬ 3

Slide 4

Slide 4 text

1. ීஈͲΜͳ͜ͱΛ͍ͯ͠Δͷ͔ 4

Slide 5

Slide 5 text

ීஈͲΜͳ͜ͱΛ͍ͯ͠Δͷ͔ 5 ● ޿ࠂ഑৴ϓϩμΫτΛ࡞͍ͬͯ·͢ʂ

Slide 6

Slide 6 text

޿ࠂ഑৴ϓϩμΫτͷ֓ཁ

Slide 7

Slide 7 text

޿ࠂ഑৴ϓϩμΫτͷ֓ཁ ޿ࠂओʹͱͬͯ࠷େͷ ίετύϑΥʔϚϯεΛ ࣮ݱ͢ΔΑ͏੍ޚ

Slide 8

Slide 8 text

޿ࠂ഑৴ϓϩμΫτͷ֓ཁ ϝσΟΞଆͷऩӹ͕ ࠷େͱͳΔΑ͏ʹ੍ޚ

Slide 9

Slide 9 text

ࣗ෼ୡ͕ߏங͍ͯ͠Δ΋ͷ ޿ࠂ഑৴ϓϩμΫτͷ֓ཁ

Slide 10

Slide 10 text

2. ඇಉظॲཧͷಋೖͱ͸ʁ 10

Slide 11

Slide 11 text

● ޿ࠂओ͔Β͍͍ͨͩͨ༧ࢉΛద੾ʹ࢖͏ͨΊʹɺࠓ͍͘Β༧ࢉΛফԽ͔ͨ͠ ͱ͍͏৘ ใΛ DynamoDB ʹॻ͖ࠐΜͰ͍Δ ● ༧ࢉ͸͓ۚʹؔΘΔ͜ͱͳͷͰ ࣮֬ʹɾਖ਼֬ʹ ॻ͖ࠐ·ΕΔඞཁ͕͋Δ ● ॲཧ͕ࣦഊ͢Δͱɺ༧ࢉരൃͷݒ೦͕͋ΔͷͰɺ҆શଆʹ౗ͤΔΑ͏ʹ͍ͨ͠ ༧ࢉফԽػೳͷઆ໌ 11

Slide 12

Slide 12 text

● ༧ࢉܭ্Λߦ͍ͬͯΔαʔόʔʹෛՙ͕ूதͨ͠ࡍʹ༧ࢉͷॻ͖ࠐΈ͕ࣦഊͯ͠͠ ·ͬͨ ͋Δ೔εϩοτϦϯά͕ى͖ͨ 12

Slide 13

Slide 13 text

● ༧ࢉॻ͖ࠐΈʹؔͯ͠͸ଈ࣌ੑΑΓ΋࣮֬ੑΛॏࢹ͔ͨͬͨͨ͠ΊɺSQS Λ࢖ͬͯ͜ ͷػೳͷΈ෼཭͔ͨͬͨ͠ ● ·ͨɺDLQ ͷ࢓૊ΈΛ࢖ͬͯػೳͷ࠶ࢼߦɾఀࢭΛࢼΈ͔ͨͬͨ ● SQS Λ࢖ͬͯҰ෦ͷॲཧΛ෼཭͢Δ͜ͱΛʮඇಉظॲཧͷಋೖʯͱݺΜͰ͍Δ εϩοτϦϯάରࡦΛ͍ͨ͠ 13

Slide 14

Slide 14 text

3. SQS ͱ͸ʁ 14

Slide 15

Slide 15 text

● Amazon Simple Queue ServiceʢSQSʣͷ͜ͱ ● ιϑτ΢ΣΞίϯϙʔωϯτؒͰ೚ҙͷྔͷϝοηʔδΛૹ৴ɺอଘɺड৴͢Δ͜ͱ ͕Ͱ͖ΔαʔϏε ● ϝοηʔδΛࣦͬͨΓɺଞͷαʔϏεΛར༻͢Δඞཁ΋ͳ͍ SQS ͱ͸ʁ 15 ग़య : https://aws.amazon.com/jp/sqs/

Slide 16

Slide 16 text

SQS ͱ͸ʁ 16 ࢀߟ : https://aws.amazon.com/jp/sqs/ https://d1.awsstatic.com/webinars/jp/pdf/services/20190717_AWS-BlackBelt_AmazonSQS.pdf ʮΩϡʔʯͷʮϝοηʔδʯΛ औಘ͢ΔΞϓϦέʔγϣϯ ίϯγϡʔϚ ϓϩσϡʔα͕ੜ੒͢Δσʔλ ϝοηʔδ ʮΩϡʔʯʹʮϝοηʔδʯΛ ૹ৴͢ΔΞϓϦέʔγϣϯ ϓϩσϡʔα ϝοηʔδΛΩϡʔΠϯά͢Δ ෼ࢄΩϡʔ Ωϡʔ

Slide 17

Slide 17 text

4. ࣮૷લޙͷΞʔΩςΫνϟ 17

Slide 18

Slide 18 text

● ECS ͕௚઀ DynamoDB ͱ Aurora ʹ ʮಡΈऔΓʯʮॻ͖ࠐΈʯΛߦ͍ͬͯͨ લޙͷΞʔΩςΫνϟ ● ؒʹ SQS ͱ Lambda ΛڬΉ͜ͱͰɺ ʮ༧ࢉফԽʯ෦෼ͷΈΛ੾Γ཭ͨ͠ ࣮૷લ ࣮૷ޙ 18

Slide 19

Slide 19 text

5. SQS બఆ࣌ʹߟྀͨ͜͠ͱ 19

Slide 20

Slide 20 text

SQS બఆ࣌ʹߟྀͨ͜͠ͱ 20 ● εϧʔϓοτ͸ߴ͍ͨ͘͠...ʂ ● ॻ͖ࠐΈ͸ 1౓͚ͩʹ͍ͨ͠...ʂ

Slide 21

Slide 21 text

ඪ४Ωϡʔ vs FIFO Ωϡʔͷ࿩ ͋ ● API ΞΫγϣϯ͝ͱʹ 1ඵ͋ͨΓ 300τϥϯβΫγϣϯ ͷΫΥʔλΛα ϙʔτ ● ϝοηʔδ͸ ඞͣ1ճ ͚ͩ഑৴ ඪ४Ωϡʔ FIFO Ωϡʔ ● API ΞΫγϣϯ͝ͱʹ 1ඵ͋ͨΓ ΄΅ແ੍ݶ ͷ API ίʔϧΛαϙʔτ ● ϝοηʔδ͸ গͳ͘ͱ΋1ճ ഑৴ 21 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html

Slide 22

Slide 22 text

● FIFO ΩϡʔͷߴεϧʔϓοτϞʔυΛద༻͢Δͱɺ1ඵʹ͖ͭ 3000݅ ͷϦΫΤετʹ ରԠͰ͖Δ ○ 2023.10.04 ࣌఺Ͱ౦ژϦʔδϣϯͰ͸ඵؒ 9000τϥϯβΫγϣϯ ● ϝοηʔδάϧʔϓΛ෼ࢄͤ͞Δ͜ͱͰɺͦΕͧΕͷ ϝοηʔδάϧʔϓ͕1ඵ͋ͨΓ 300݅ͷϦΫΤετΛαϙʔτ ͢Δ High Throughput Queue ʹ͍ͭͯ 22 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/high-throughput-fifo.html https://aws.amazon.com/jp/about-aws/whats-new/2023/10/amazon-sqs-increased-throughput-quota-fifo-high-throughput-mode/

Slide 23

Slide 23 text

SQS બఆ࣌ʹߟྀͨ͜͠ͱ 23 ● ॲཧ͕ࣦഊͨ࣌͠ʹݕ஌Ͱ͖ΔΑ͏ʹ͍ͨ͠...ʂ

Slide 24

Slide 24 text

Dead Letter Queue ʹ͍ͭͯ 24 ϓϩσϡʔα͕ϝοηʔδΛ ૹ৴͢ΔͨΊͷओཁͳΩϡʔ ιʔεΩϡʔ ॲཧ͕ࣦഊͨ͠ࡍͷ ૹ৴ઌͱͳΔΩϡʔ σουϨλʔΩϡʔ ίϯγϡʔϚ͕ϝοηʔδΛ ड৴͠Α͏ͱ͢Δճ਺ maxReceiveCount ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html

Slide 25

Slide 25 text

Dead Letter Queue ʹ͍ͭͯ 25 ● DynamoDB ΁ͷॻ͖ࠐΈʹࣦഊͨ࣌͠ʹ DeadLetterQueue ʹϝοηʔδΛૹ৴͠ɺ Lambda Ͱ ഑৴ΛҰ࣌ఀࢭ͢Δ ϑϥάΛཱͨͤΔΑ͏ʹͨ͠

Slide 26

Slide 26 text

6. SQS બఆޙʹߟྀͨ͜͠ͱ 26

Slide 27

Slide 27 text

SQS બఆޙʹߟྀͨ͜͠ͱ 27 ● SQS ͷϝοηʔδΛ Lambda ʹॲཧͯ͠ཉ͍͠...ʂ

Slide 28

Slide 28 text

ΠϕϯτιʔεϚοϐϯά 28 ● ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͔ΒΠϕϯτΛಡΈऔΓɺLambda ؔ਺Λݺͼग़ ͢ Lambda Ϧιʔεͷ͜ͱ ● Πϕϯτιʔεͷ SQS ʹϝοηʔδ͕ૹ৴͞Εͨ͜ͱΛݕ஌ͯ͠ɺLambda ؔ਺͕࣮ ߦ͞ΕΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/invocation-eventsourcemapping.html

Slide 29

Slide 29 text

ΠϕϯτιʔεϚοϐϯάͷ batch sizeʢಉ࣌ݺͼग़্͠ݶʣ ● ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͔Β Lambda ͕ಉ࣌ʹಡΈऔΕΔΠϕϯτ਺ ● FIFO Ωϡʔͷ৔߹͸࠷େ஋͸ 10 ● Lambda ͷಉ࣮࣌ߦ਺੍ݶ͕͋ΔͷͰɺ1ճͷ Lambda ىಈͰΑΓଟ͘ͷϝοηʔδ ΛॲཧͰ͖ΔΑ͏ʹͨ͠ 29 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html

Slide 30

Slide 30 text

ΠϕϯτιʔεϚοϐϯάͷ Lambda ࠷େಉ࣮࣌ߦ਺ 30 ● ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͕ಉ࣌ʹݺͼग़ͤΔ Lambda ؔ਺ͷ࠷େ਺ ● FIFO Ωϡʔͷ৔߹ɺಉ࣌ݺͼग़͠ͷ্ݶ͸ɺʮϝοηʔδάϧʔϓ ID ͷ਺ʯ·ͨ͸ ʮ࠷େಉ࣮࣌ߦ਺ͷઃఆʯͷͲͪΒ͔ খ͍͞ํ ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html

Slide 31

Slide 31 text

ΠϕϯτιʔεϚοϐϯάͷ࠷େಉ࣮࣌ߦ਺ 31 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html

Slide 32

Slide 32 text

ΠϕϯτιʔεϚοϐϯάͷ࠷େಉ࣮࣌ߦ਺ 32 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html

Slide 33

Slide 33 text

SQS બఆޙʹߟྀͨ͜͠ͱ 33 ● ϝοηʔδ͕ద੾ʹॲཧ͞Εͯཉ͍͠...ʂ

Slide 34

Slide 34 text

SQS ͷՄࢹੑλΠϜΞ΢τʹ͍ͭͯ 34 ● ϝοηʔδ͕ड৴͞Εͨ௚ޙʹɺଞͷίϯγϡʔϚ͕ಉ͡ϝοηʔδͷड৴΍ॲཧΛ Ͱ͖ͳ͘ͳΔظؒͷ͜ͱ ● ՄࢹੑλΠϜΞ΢τ͸ Lambda ͷ࣮ߦ࣌ؒͷগͳ͘ͱ΋ 6ഒ ʹઃఆ͞ΕΔ͜ͱ͕ਪ঑ ͞Ε͍ͯΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

Slide 35

Slide 35 text

SQS ͷ maxReceiveCount ʹ͍ͭͯ 35 ● ίϯγϡʔϚʔ͕ϝοηʔδΛड৴͠Α͏ͱͨ͠ࡍʹɺmaxReceiveCount ΛॳΊͯ ௒͑ͨ࣌ʹ DLQ ʹߦ͘࢓༷ʹͳ͍ͬͯΔ ● ϝοηʔδ͕ॲཧ͞ΕΔ֬཰ΛߴΊΔʹ͸ maxReceiveCount Λ 5 Ҏ্ʹઃఆ͢Δ͜ ͱ͕ਪ঑͞Ε͍ͯΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html

Slide 36

Slide 36 text

7. ෛՙࢼݧͷ࿩ 36

Slide 37

Slide 37 text

● k6 Λ࢖ͬͨෛՙࢼݧΛߦͬͨ ● ౰࣌ͷฏۉ qps ͷ 100 ഒ·Ͱ଱͑ΒΕΔ͔ͷෛՙࢼݧΛߦ͑ͨ ෛՙࢼݧͷ࿩ 37 ࢀߟ : https://k6.io/

Slide 38

Slide 38 text

● ࠷ॳ͸ϝοηʔδάϧʔϓ ID Λ୯Ұʹ͍ͯͨ͠ͷͰύϑΥʔϚϯε͕Ͱͳ͔ͬͨ ● SQS FIFO ΩϡʔͷߴεϧʔϓοτϞʔυͰ͸ɺϝοηʔδάϧʔϓ ʹ͖ͭ 1ඵؒ 300 ݅ͷϦΫΤετΛαϙʔτ͢ΔͷͰɺͦ͜Λద੾ʹ෼ࢄͤͨ͞ Queue ͷ API Limit ͷ্ݶʹର͢Δݟམͱ͠ 38 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html

Slide 39

Slide 39 text

8. ࠓޙͷల๬ 39

Slide 40

Slide 40 text

● LT ͷࢿྉ࡞੒Λ͢Δ·ͰΠϕϯτιʔεϚοϐϯάͷ Lambda ಉ࣮࣌ߦ਺੍ޚΛ஌Β ͳ͔ͬͨ... ࠓޙͷల๬ 40

Slide 41

Slide 41 text

● SQS ͷ API Limit Λ௒͑ͯ͠·ͬͨ৔߹ʹෳ਺ͷ Queue ʹ෼ࢄͤ͞Δඞཁ͕͋Δ ࠓޙͷల๬ 41

Slide 42

Slide 42 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ɻ 42