サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt

by kasacchiful

Slide 1

Slide 1 text

αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ ּݪ޺ /*//05FDI'FTU

Slide 2

Slide 2 text

ຊ೔͓࿩͢͠Δ͜ͱ wαʔόʔϨεͰσʔλ࿈ܞج൫Λߏஙͨ͠ࡍɺϋϚͬͨͱ͜Ζ͕৭ʑ ͋ͬͨͷͰɺ؆୯ʹ͓࿩͠͠·͢ɻ w"84Ͱߏங͍ͯ͠·͢ɻ"84͋·ΓΘ͔Βͳ͍ํ͸ɺ಺༰ࠐΈೖͬͯ ΔͷͰ೉͍͔͠΋͠Ε·ͤΜɻ w-5ͳͷʹϖʔδ΋࡞ͬͨͷͰɺۦ͚଍ͰਐΈ·͢ɻ

Slide 3

Slide 3 text

ࣗݾ঺հ w ּݪ޺ w ΫϥεϝιουגࣜձࣾσʔλΞφϦςΟΫεࣄۀຊ෦ w ৽ׁݝ৽ׁࢢࡏॅ w ޷͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ +B445৽ׁʗ48"/**ʗFUD @kasacchiful @kasacchiful

Slide 4

Slide 4 text

೔ ։࢝ w IUUQTQZNMOJJHBUBDPOOQBTTDPNFWFOU 1ZUIPOػցֶशษڧձJO৽ׁ

Slide 5

Slide 5 text

w IUUQTKBXTVHOJJHBUBDPOOQBTTDPN +"846(/JJHBUB

Slide 6

Slide 6 text

+"841"/,3"5*0/ ౔ ʙ ೔ ͷ࣌ؒΠϕϯτ w IUUQTKBXTQBOLSBUJPOKBXTVHKQ

Slide 7

Slide 7 text

ຊ୊ αʔόʔϨεͷσʔλج൫

Slide 8

Slide 8 text

ྫ͑͹͜Μͳ΍ͭ σʔλ෼ੳʹ͓͚Δ֤छ"84αʔϏε

Slide 9

Slide 9 text

ྫ͑͹͜Μͳ΍ͭ 4ʹσʔλೖΕͯɺ-BNCEBͰՃ޻ɺ݁ՌΛ4ʹ͓͘ɻ

Slide 10

Slide 10 text

ྫ͑͹͜Μͳ΍ͭ 4ʹՃ޻ࡁσʔλ͓͍ͨΒɺ͋ͱ͸෼ੳʹͲ͏ͧɻ

Slide 11

Slide 11 text

ࠓճ͸͜Μͳײ͡ͷ΍ͭ 4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ޻ w ೔ճεέδϡʔϧ࣮ߦ

Slide 12

Slide 12 text

4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ޻ w ೔ճεέδϡʔϧ࣮ߦ ࠓճ͸͜Μͳײ͡ͷ΍ͭ σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷௿ݶͷσʔ λՃ޻Λͯ͠ɺ4ʹอଘ 2VJDL4JHIU #* ༻ʹ ෳ਺ϑΝΠϧͷσʔλΛ·ͱΊ ͯద੾ʹ੔ܗ͢Δ

Slide 13

Slide 13 text

͍Ζ͍ΖϋϚͬͨ ϋϚͬͨ఺Λ4ͭ঺հ

Slide 14

Slide 14 text

ϋϚͬͨ఺ͦͷ ಛఆͷσʔλϑΝΠϧଟ͗͢

Slide 15

Slide 15 text

೰Έ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ w ෼ؒͷσʔλ͕ϑΝΠϧʹ͋Δ w த਎͸ϛϦඵ୯ҐͷϨίʔυ w ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ ಛఆͷσʔλϑΝΠϧଟ͗͢

Slide 16

Slide 16 text

ಛఆͷσʔλϑΝΠϧଟ͗͢ ରॲ๏͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲཧʹมߋ w ݅਺ଟ͍σʔλ͸ɺ#*ʹग़ྗ͠ͳ͍߲໨ͩͬͨ w ೔࣍ॲཧ͔Β੾Γ཭ͯ͠ɺຖ࣌ॲཧʹมߋ w ೔࣍ॲཧͷϘτϧωοΫΛআ͍ͨ

Slide 17

Slide 17 text

ϋϚͬͨ఺ͦͷ Athena ͷΫΥʔλ

Slide 18

Slide 18 text

೰Έ"UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ w ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ "UIFOBͷΫΥʔλ

Slide 19

Slide 19 text

"UIFOBͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

Slide 20

Slide 20 text

ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ w .BQεςʔτ ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ ͷ࠷େಉ࣮࣌ߦ਺Λઃ ఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ͓͑͞Δ "UIFOBͷΫΥʔλ

Slide 21

Slide 21 text

.BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ "UIFOBͷΫΥʔλ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

Slide 22

Slide 22 text

ϋϚͬͨ఺ͦͷ Step Functions ͷΫΥʔλ

Slide 23

Slide 23 text

೰Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ w ͋Δಛఆͷ೔͚ͩɺຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍ w ࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ Πϕϯτ w ্ݶ؇࿨ෆՄͷ߲໨ 4UFQ'VODUJPOTͷΫΥʔλ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }

Slide 24

Slide 24 text

4UFQ'VODUJPOTͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

Slide 25

Slide 25 text

ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ w εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ʹͨ͠ w -BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ 㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥ 㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ 4UFQ'VODUJPOTͷΫΥʔλ

Slide 26

Slide 26 text

4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ

Slide 27

Slide 27 text

4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ

Slide 28

Slide 28 text

ϋϚͬͨ఺ͦͷ Lambdaͷεέʔϧ͕௥͍͔ͭͳ͍

Slide 29

Slide 29 text

೰Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰ w ಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ w ͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶΓɺಉ࣮࣌ ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ { "error": "Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }

Slide 30

Slide 30 text

-BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

Slide 31

Slide 31 text

ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠ w 4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ௚͠ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍

Slide 32

Slide 32 text

ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w 3FUSZͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ $ node -e '((i,m,b)=>{for(let w=i,c=0;c

Slide 33

Slide 33 text

ରॲ๏-BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠ w -BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

Slide 34

Slide 34 text

ิ଍ σʔλͷՃ޻ʹ͸ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ

Slide 35

Slide 35 text

(MVF࢖ΘͣʹɺΘ͟Θ͟4UFQ'VODUJPOT-BNCEBͰ૊Ήඞཁ͋Δͷ͔ʁ w 4UFQ'VODUJPOT-BNCEBͷ৔߹ɺΑ͘࢖ΘΕΔ։ൃϑϨʔϜϫʔΫ͕࢖͑ΔͷͰɺෳ਺ਓ Ͱͷ։ൃ͕͠΍͍͢ɻ 㾎 ࠓճ͸4FSWFSMFTT'SBNFXPSL࢖ͬͨɻ w σʔλϑΝΠϧ਺͕ଟͯ͘΋ɺσʔλ݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Ε͹ɺ޻෉࣍ୈ Ͱ-BNCEBͰॲཧ͕Ͱ͖Δɻ w -BNCEBͰ͸৐Γ੾Εͳ͍σʔλ༰ྔΛѻ͏৔߹͸ɺ(MVF࢖ͬͨํ͕͍͍ɻ 㾎 ࠷େϝϞϦׂ౰.#ɺ࠷େ࣮ߦ࣌ؒ෼ɺUNQσΟϨΫτϦαΠζ.# σʔλͷՃ޻ͳΒ(MVF͕͋Δ

Slide 36

Slide 36 text

·ͱΊ • αʔόʔϨεαʔϏεΛۦ࢖ͯ͠ɺσʔλ෼ੳج൫Λ ߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏ ·͘࢖͍͜ͳ͠·͠ΐ͏

Slide 37

Slide 37 text

No content