サーバーレスでデータ連携する際にハマったところ / ninno-techfest-20211022-lt

Transcript

1. αʔόʔϨεͰσʔλ࿈ܞ͢Δࡍʹ
   ϋϚͬͨͱ͜Ζ ּݪ
   ޺ 
   /*//0
   5FDI
   'FTU
   

2. ຊ೔͓࿩͢͠Δ͜ͱ  wαʔόʔϨεͰσʔλ࿈ܞج൫Λߏஙͨ͠ࡍɺϋϚͬͨͱ͜Ζ͕৭ʑ ͋ͬͨͷͰɺ؆୯ʹ͓࿩͠͠·͢ɻ
   w"84Ͱߏங͍ͯ͠·͢ɻ"84͋·ΓΘ͔Βͳ͍ํ͸ɺ಺༰ࠐΈೖͬͯ ΔͷͰ೉͍͔͠΋͠Ε·ͤΜɻ
   w-5ͳͷʹϖʔδ΋࡞ͬͨͷͰɺۦ͚଍ͰਐΈ·͢ɻ

3. ࣗݾ঺հ  w ּݪ
   ޺
   w Ϋϥεϝιουגࣜձࣾ
   σʔλΞφϦςΟΫεࣄۀຊ෦
   w ৽ׁݝ৽ׁࢢࡏॅ
   w ޷͖ͳ"84αʔϏε
   4ɺ-BNCEBɺ4BHF.BLFS
   

   w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ +B445৽ׁʗ48"/**ʗFUD @kasacchiful @kasacchiful

4. 
   ೔
   
   ։࢝
   w IUUQTQZNMOJJHBUBDPOOQBTTDPNFWFOU 1ZUIPOػցֶशษڧձ
   JO
   ৽ׁ
    

5. w IUUQTKBXTVHOJJHBUBDPOOQBTTDPN +"846(
   /JJHBUB 

6. +"84
   1"/,3"5*0/
     
   ౔
   
   ʙ
   
   ೔
   
   ͷ
   ࣌ؒΠϕϯτ
   w IUUQTKBXTQBOLSBUJPOKBXTVHKQ

7. ຊ୊  αʔόʔϨεͷσʔλج൫

8. ྫ͑͹͜Μͳ΍ͭ  σʔλ෼ੳʹ͓͚Δ֤छ"84αʔϏε

9. ྫ͑͹͜Μͳ΍ͭ  4ʹσʔλೖΕͯɺ-BNCEBͰՃ޻ɺ݁ՌΛ4ʹ͓͘ɻ

10. ྫ͑͹͜Μͳ΍ͭ  4ʹՃ޻ࡁσʔλ͓͍ͨΒɺ͋ͱ͸෼ੳʹͲ͏ͧɻ

11. ࠓճ͸͜Μͳײ͡ͷ΍ͭ  4UFQ
    'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ޻
    w ೔ճεέδϡʔϧ࣮ߦ

12. 4UFQ
    'VODUJPOTͷεςʔτϚγϯͰ-BNCEBͷϫʔΫϑϩʔΛ੍ޚͯ͠ɺσʔλ ΛՃ޻
    w ೔ճεέδϡʔϧ࣮ߦ ࠓճ͸͜Μͳײ͡ͷ΍ͭ  σʔλൃੜݩ͔ΒɺσʔλΛऔ ಘͯ͠4ʹอଘ ֤ϑΝΠϧຖʹɺ࠷௿ݶͷσʔ λՃ޻Λͯ͠ɺ4ʹอଘ

    2VJDL4JHIU
    #*
    ༻ʹ
    ෳ਺ϑΝΠϧͷσʔλΛ·ͱΊ ͯద੾ʹ੔ܗ͢Δ

13. ͍Ζ͍ΖϋϚͬͨ  ϋϚͬͨ఺Λ4ͭ঺հ

14. ϋϚͬͨ఺
    ͦͷ  ಛఆͷσʔλϑΝΠϧଟ͗͢

15. ೰Έ
    ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩҟৗʹଟ͍
    w ෼ؒͷσʔλ͕ϑΝΠϧʹ͋Δ
    w த਎͸ϛϦඵ୯ҐͷϨίʔυ
    w ಛఆͷॲཧ͚͕͔͔ͩ࣌ؒΔ 
    ಛఆͷσʔλϑΝΠϧଟ͗͢ 

16. 
    ಛఆͷσʔλϑΝΠϧଟ͗͢  ରॲ๏
    ͋ΔಛఆͷσʔλϑΝΠϧ͚ͩɺຖ࣌ॲཧʹมߋ
    w ݅਺ଟ͍σʔλ͸ɺ#*ʹग़ྗ͠ͳ͍߲໨ͩͬͨ
    w ೔࣍ॲཧ͔Β੾Γ཭ͯ͠ɺຖ࣌ॲཧʹมߋ
    w ೔࣍ॲཧͷϘτϧωοΫΛআ͍ͨ

17. ϋϚͬͨ఺
    ͦͷ  Athena ͷΫΥʔλ

18. ೰Έ
    "UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ
    w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ
    w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ
    w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ
    w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ
    w

    ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ 
    "UIFOBͷΫΥʔλ 

19. 
    "UIFOBͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

20. ରॲ๏
    4UFQ
    'VODUJPOT
    ͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ
    w .BQεςʔτ
    ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ
    ͷ࠷େಉ࣮࣌ߦ਺Λઃ ఆ͠ɺ"UIFOBͷ
    TUBSURVFSZFYFDVUJPO
    "1*ίʔϧΛ࠷େ·Ͱʹ͓͑͞Δ 
    "UIFOBͷΫΥʔλ 

21. .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ 
    "UIFOBͷΫΥʔλ  IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

22. ϋϚͬͨ఺
    ͦͷ  Step Functions ͷΫΥʔλ

23. ೰Έ
    4UFQ
    'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ
    w ͋Δಛఆͷ೔͚ͩɺຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍
    w ࣌ؒܦͬͯҟৗऴྃɻ4UFQ
    'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ
     Πϕϯτ
    w ্ݶ؇࿨ෆՄͷ߲໨

    
    4UFQ
    'VODUJPOTͷΫΥʔλ  { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }

24. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

25. ରॲ๏
    4UFQ
    'VODUJPOT
    ͷεςʔτϚγϯΛೖΕࢠʹ
    w εςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ʹͨ͠
    w -BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ
    㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥
    㾎 4UFQ
    'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ 
    4UFQ
    'VODUJPOTͷΫΥʔλ

    

26. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ

27. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ

28. ϋϚͬͨ఺
    ͦͷ  Lambdaͷεέʔϧ͕௥͍͔ͭͳ͍

29. ೰Έ
    ճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰
    w ಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ
    w ͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶΓɺಉ࣮࣌ ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  { "error":

    "Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }

30. 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

31. ରॲ๏
    -BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w 4UFQ
    'VODUJPOT
    ͷ
    .BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠
    w 4UFQ
    'VODUJPOT
    Ͱఆٛ͢Δ
    -BNCEB
    ͷ
    3FUSZ
    ઃఆΛݟ௚͠ 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ 

32. ରॲ๏
    -BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w 3FUSZ
    ͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)'

    2 3.85 7.272500000000001 13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ

33. ରॲ๏
    -BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠
    w -BNCEB
    ͷ
    1SPWJTJPOFE
    $PODVSSFODZ
    ઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍ 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

34. ิ଍  σʔλͷՃ޻ʹ͸ AWS Glueͱ͍͏αʔϏε͋ΔΑʁ

35. (MVF࢖ΘͣʹɺΘ͟Θ͟4UFQ
    'VODUJPOT
    
    -BNCEBͰ૊Ήඞཁ͋Δͷ͔ʁ
    w 4UFQ
    'VODUJPOT
    
    -BNCEBͷ৔߹ɺΑ͘࢖ΘΕΔ։ൃϑϨʔϜϫʔΫ͕࢖͑ΔͷͰɺෳ਺ਓ Ͱͷ։ൃ͕͠΍͍͢ɻ
    㾎 ࠓճ͸
    4FSWFSMFTT
    'SBNFXPSL
    ࢖ͬͨɻ
    w σʔλϑΝΠϧ਺͕ଟͯ͘΋ɺσʔλ݅͋ͨΓͷ༰ྔ͕ͦ͜·Ͱେ͖͘ͳ͚Ε͹ɺ޻෉࣍ୈ

    Ͱ-BNCEBͰॲཧ͕Ͱ͖Δɻ
    w -BNCEBͰ͸৐Γ੾Εͳ͍σʔλ༰ྔΛѻ͏৔߹͸ɺ(MVF࢖ͬͨํ͕͍͍ɻ
    㾎 ࠷େϝϞϦׂ౰
    .#ɺ࠷େ࣮ߦ࣌ؒ
    ෼ɺ
    UNQ
    σΟϨΫτϦαΠζ
    .# σʔλͷՃ޻ͳΒ(MVF͕͋Δ 

36. ·ͱΊ  • αʔόʔϨεαʔϏεΛۦ࢖ͯ͠ɺσʔλ෼ੳج൫Λ ߏஙՄೳ • αʔόʔϨεͷΑ͋͘ΔΞʔΩςΫνϟύλʔϯΛ͏ ·͘࢖͍͜ͳ͠·͠ΐ͏

37. None