Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt

サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt

2021/10/22 (金) NINNO Tech Fest #2 の LT で発表した資料

82d6167c4d14393c2e20b37a74b363c5?s=128

kasacchiful

October 22, 2021
Tweet

More Decks by kasacchiful

Other Decks in Programming

Transcript

  1. αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ ϋϚͬͨͱ͜Ζ ּݪ޺  /*//05FDI'FTU

  2. ຊ೔͓࿩͢͠Δ͜ͱ  wαʔόʔϨεͰσʔλ࿈ܞج൫Λߏஙͨ͠ࡍɺϋϚͬͨͱ͜Ζ͕৭ʑ ͋ͬͨͷͰɺ؆୯ʹ͓࿩͠͠·͢ɻ w"84Ͱߏங͍ͯ͠·͢ɻ"84͋·ΓΘ͔Βͳ͍ํ͸ɺ಺༰ࠐΈೖͬͯ ΔͷͰ೉͍͔͠΋͠Ε·ͤΜɻ w-5ͳͷʹϖʔδ΋࡞ͬͨͷͰɺۦ͚଍ͰਐΈ·͢ɻ

  3. ࣗݾ঺հ  w ּݪ޺ w ΫϥεϝιουגࣜձࣾσʔλΞφϦςΟΫεࣄۀຊ෦ w ৽ׁݝ৽ׁࢢࡏॅ w ޷͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS

    w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ +B445৽ׁʗ48"/**ʗFUD @kasacchiful @kasacchiful
  4.  ೔ ։࢝ w IUUQTQZNMOJJHBUBDPOOQBTTDPNFWFOU 1ZUIPOػցֶशษڧձJO৽ׁ 

  5. w IUUQTKBXTVHOJJHBUBDPOOQBTTDPN +"846(/JJHBUB 

  6. +"841"/,3"5*0/   ౔ ʙ ೔ ͷ࣌ؒΠϕϯτ w IUUQTKBXTQBOLSBUJPOKBXTVHKQ

  7. ຊ୊  αʔόʔϨεͷσʔλج൫

  8. ྫ͑͹͜Μͳ΍ͭ  σʔλ෼ੳʹ͓͚Δ֤छ"84αʔϏε

  9. ྫ͑͹͜Μͳ΍ͭ  4ʹσʔλೖΕͯɺ-BNCEBͰՃ޻ɺ݁ՌΛ4ʹ͓͘ɻ

  10. ྫ͑͹͜Μͳ΍ͭ  4ʹՃ޻ࡁσʔλ͓͍ͨΒɺ͋ͱ͸෼ੳʹͲ͏ͧɻ

  11. ࠓճ͸͜Μͳײ͡ͷ΍ͭ  4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ޻ w ೔ճεέδϡʔϧ࣮ߦ

  12. 4UFQ'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ޻ w ೔ճεέδϡʔϧ࣮ߦ ࠓճ͸͜Μͳײ͡ͷ΍ͭ  σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷௿ݶͷσʔ λՃ޻Λͯ͠ɺ4ʹอଘ

    2VJDL4JHIU #* ༻ʹ ෳ਺ϑΝΠϧͷσʔλΛ·ͱΊ ͯద੾ʹ੔ܗ͢Δ
  13. ͍Ζ͍ΖϋϚͬͨ  ϋϚͬͨ఺Λ4ͭ঺հ

  14. ϋϚͬͨ఺ͦͷ  ಛఆͷσʔλϑΝΠϧଟ͗͢

  15. ೰Έ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍ w ෼ؒͷσʔλ͕ϑΝΠϧʹ͋Δ w த਎͸ϛϦඵ୯ҐͷϨίʔυ w ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ ಛఆͷσʔλϑΝΠϧଟ͗͢ 

  16. ಛఆͷσʔλϑΝΠϧଟ͗͢  ରॲ๏͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲཧʹมߋ w ݅਺ଟ͍σʔλ͸ɺ#*ʹग़ྗ͠ͳ͍߲໨ͩͬͨ w ೔࣍ॲཧ͔Β੾Γ཭ͯ͠ɺຖ࣌ॲཧʹมߋ w ೔࣍ॲཧͷϘτϧωοΫΛআ͍ͨ

  17. ϋϚͬͨ఺ͦͷ  Athena ͷΫΥʔλ

  18. ೰Έ"UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ w

    ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ "UIFOBͷΫΥʔλ 
  19. "UIFOBͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

  20. ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ w .BQεςʔτ ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ ͷ࠷େಉ࣮࣌ߦ਺Λઃ ఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ͓͑͞Δ "UIFOBͷΫΥʔλ 

  21. .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ "UIFOBͷΫΥʔλ  IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

  22. ϋϚͬͨ఺ͦͷ  Step Functions ͷΫΥʔλ

  23. ೰Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ w ͋Δಛఆͷ೔͚ͩɺຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍ w ࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ  Πϕϯτ  w ্ݶ؇࿨ෆՄͷ߲໨

    4UFQ'VODUJPOTͷΫΥʔλ  { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
  24. 4UFQ'VODUJPOTͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

  25. ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ w εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ʹͨ͠ w -BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ 㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥ 㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ 4UFQ'VODUJPOTͷΫΥʔλ

    
  26. 4UFQ'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ

  27. 4UFQ'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ

  28. ϋϚͬͨ఺ͦͷ  Lambdaͷεέʔϧ͕௥͍͔ͭͳ͍

  29. ೰Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰ w ಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ w ͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶΓɺಉ࣮࣌ ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  { "error":

    "Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
  30. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

  31. ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠ w 4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ௚͠ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ 

  32. ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w 3FUSZͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)'

    2 3.85 7.272500000000001 13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
  33. ରॲ๏-BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠ w -BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍ -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

  34. ิ଍  σʔλͷՃ޻ʹ͸ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ

  35. (MVF࢖ΘͣʹɺΘ͟Θ͟4UFQ'VODUJPOT -BNCEBͰ૊Ήඞཁ͋Δͷ͔ʁ w 4UFQ'VODUJPOT -BNCEBͷ৔߹ɺΑ͘࢖ΘΕΔ։ൃϑϨʔϜϫʔΫ͕࢖͑ΔͷͰɺෳ਺ਓ Ͱͷ։ൃ͕͠΍͍͢ɻ 㾎 ࠓճ͸4FSWFSMFTT'SBNFXPSL࢖ͬͨɻ w σʔλϑΝΠϧ਺͕ଟͯ͘΋ɺσʔλ݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Ε͹ɺ޻෉࣍ୈ

    Ͱ-BNCEBͰॲཧ͕Ͱ͖Δɻ w -BNCEBͰ͸৐Γ੾Εͳ͍σʔλ༰ྔΛѻ͏৔߹͸ɺ(MVF࢖ͬͨํ͕͍͍ɻ 㾎 ࠷େϝϞϦׂ౰.#ɺ࠷େ࣮ߦ࣌ؒ෼ɺUNQσΟϨΫτϦαΠζ.# σʔλͷՃ޻ͳΒ(MVF͕͋Δ 
  36. ·ͱΊ  • αʔόʔϨεαʔϏεΛۦ࢖ͯ͠ɺσʔλ෼ੳج൫Λ ߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏ ·͘࢖͍͜ͳ͠·͠ΐ͏

  37. None