Upgrade to Pro — share decks privately, control downloads, hide ads and more …

サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

DevelopersIO 2022 ビデオセッション資料。
https://youtu.be/-YN1Z2oH-G0

DevelopersIO 2022 公式サイト:
https://classmethod.jp/m/developers-io/

82d6167c4d14393c2e20b37a74b363c5?s=128

kasacchiful

July 27, 2022
Tweet

More Decks by kasacchiful

Other Decks in Programming

Transcript

  1.  σʔλΞφϦςΟΫεࣄۀຊ෦ּݪ޺ αʔόʔϨεύλʔϯ͔ΒֶͿ σʔλ෼ੳج൫ߏங

  2. ͓࿩͢͠Δ͜ͱ  wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳʹ࢖͑Δύλʔ ϯΛத৺ʹɺ࣮ࡍͷߏஙࣄྫΛަ͑ͳ͕Β͓࿩͠͠·͢ɻ wಛʹ4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔߏஙͰϋϚͬͨ ͱ͜ΖΛத৺ʹ͓࿩͠͠·͢ɻ

  3. ࣗݾ঺հ  w ּݪ޺ ೥݄δϣΠϯ  w σʔλΞφϦςΟΫεࣄۀຊ෦ϏδωειϦϡʔγϣϯ෦Ϗοάσʔλ νʔϜ w

    ৽ׁݝ৽ׁࢢࡏॅ w "1/"845PQ&OHJOFFST "OBMZUJDT "1/"--"84 $FSUJ fi DBUJPOT&OHJOFFST w ޷͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ+B445৽ׁʗ 48"/**ʗFUD @kasacchiful @kasacchiful
  4. ࠓճͷςʔϚ  αʔόʔϨεύλʔϯ

  5. αʔόʔϨεύλʔϯ  IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

  6. Ϣʔεέʔεผʹύλʔϯ͕͋Δ  IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

  7. αʔόʔϨεύλʔϯΛ࢖ͬͨσʔλ෼ੳج൫  wൺֱతن໛ͷখ͍͞σʔλ෼ੳج൫͔ΒɺαʔόʔϨεύ λʔϯΛ࢖ͬͯΈΔͱɺͲ͏ͳΔ͔ݟͯΈΔ wΑ͋͘Δύλʔϯ͔ΒϐοΫΞοϓͯ͠঺հ

  8. ը૾ॲཧγϯϓϧͳσʔλՃ޻ 

  9. ·ͣ͸γϯϓϧͳσʔλՃ޻͔Β࢝ΊͯΈΔ  w֨ೲ͢Δύε͸ɺ੔ܗલͱޙͰม ͑Δ͜ͱ wແݶϧʔϓΛճආ w-BNCEB಺ͷUNQྖҬʹҰ࣌อ ଘ wॲཧޙʹ໌ࣔతʹ࡟আ͢Δ͜ͱ w੔ܗޙ͸ผͷόέοτʹ֨ೲͯ͠ ΋ྑ͍

  10. ෼ࢄฒྻॲཧ 

  11. ྫ͑͹೥౓ຖʹॲཧΛ෼͚Δ  wॲཧ͢Δσʔλຖʹ-BNCEBؔ ਺ىಈ wσʔλຖʹ෼ࢄॲཧͰ͖Δͷ Ͱɺτʔλϧͷ࣮ߦ͕࣌ؒ୹͘ ࡁΉ wσʔλຖʹॲཧ಺༰͕ҟͳΔ৔ ߹͸ɺ౎౓ؔ਺Λఆٛ͢Δඞཁ ͕͋Δέʔε΋

  12. Πϕϯτۦಈͷۀ຿ॲཧ࿈ܞ 

  13. 4/4ɾ424Λ૊߹ͤͯॲཧΛ࿈ܞ  w4/4ͰϑΝϯΞ΢τ wฒྻ࣮ߦ w୯Ұ࣮ߦͰ΋σόοά༻్ʹ w424ͷΩϡʔʹ֨ೲ͢Δ͜ͱͰ ৴པੑ֬อ w'*'04/4'*'0424Ͱॱংੑ ΛߟྀՄೳ

  14. 424ͷલʹ4/4τϐοΫڬΜͰ͓͜͏  IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTOTUPQJDTIPVMECFQMBDFECFIJOE TRTRVFVF

  15. ࣮ࡍʹߏஙͨ͠ج൫  wσʔλϑΝΠϧຖʹ4/4ɾ424 Λܦ༝ͯ͠-BNCEBͰॲཧ wॲཧͨ͠ϑΝΠϧΛ4ʹ֨ೲ͠ ͨΒɺޙଓͷॲཧ͕ىಈ w࠷ޙͷ-BNCEBॲཧͰϚʔδ

  16. σʔλϑΝΠϧ͕ଟ͍৔߹ॲཧ͕ෳࡶͳ৔߹  Step Functionsͷ εςʔτϚγϯΛ׆༻

  17. ΞϓϦέʔγϣϯॲཧ 

  18. "844UFQ'VODUJPOTͰϫʔΫϑϩʔ੍ޚ  w4UFQ'VODUJPOTͷεςʔτϚγ ϯΛߏஙͯ͠ϫʔΫϑϩʔΛ੍ ޚ w1BSBMMFMλεΫɺ.BQλεΫΛ׆ ༻ͯ͠-BNCEBؔ਺Λฒྻ࣮ߦ wσʔλϑΝΠϧαΠζ͕খ͘͞ ͯେྔʹ͋ΔέʔεͰ༗ޮ

  19. ࣮ࡍʹߏஙͨ͠ج൫  w೔࣍ॲཧͰར༻ w֤छ%#΍ΦϒδΣΫτετϨʔ δʹΞΫηεͯ͠σʔλΛऔಘ wσʔλͷछྨ͕ଟ͍ͷͰɺछྨ ຖʹ1BSBMMFMλεΫͰॲཧ wσʔλϑΝΠϧͷ਺΋ଟ͍ͷ ͰɺϑΝΠϧຖʹ.BQλεΫͰ ॲཧ

    w࠷ޙʹ෼ੳ༻ͷσʔλΛੜ੒
  20. ࣮ࡍʹߏஙͨ͠ج൫  wຖ࣌ॲཧͰར༻ wϑΝΠϧ਺͕ඇৗʹଟ͍ͨ ΊɺͭͷεςʔτϚγϯͰ ͸ࡹ͖͖Εͳ͍ w࣮ߦཤྺ ݅ͷ ΫΥʔλʹҾ͔͔ͬΔ wεςʔτϚγϯΛೖΕࢠʹ

    ͢Δͱɺ-BNCEBͷಉ࣮࣌ ߦ਺΋૿Ճ
  21. 4UFQ'VODUJPOT৭ʑϋϚͬͨ  ϋϚͬͨͱ͜ΖΛ 3ͭ঺հ

  22. 4UFQ'VODUJPOTͷΫΥʔλ  ೰Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ wຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍ w࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ  Π ϕϯτ  w্ݶ؇࿨ෆՄͷ߲໨

    { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
  23. 4UFQ'VODUJPOTͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

  24. 4UFQ'VODUJPOTͷΫΥʔλ  ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ wεςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ ʹͨ͠ w-BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ 㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥ 㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ

  25. 4UFQ'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ

  26. 4UFQ'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ

  27. ೰Έ"UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ w

    ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ "UIFOBͷΫΥʔλ 
  28. "UIFOBͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

  29. "UIFOBͷΫΥʔλ  ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ w.BQεςʔτ ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ ͷ࠷େಉ࣌ ࣮ߦ਺Λઃఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ ͓͑͞Δ

  30. "UIFOBͷΫΥʔλ  .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

  31. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ೰Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰ wಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ w͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶ Γɺಉ࣮࣌ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ { "error": "Lambda.TooManyRequestsException", "cause":

    "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
  32. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

  33. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠ w4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ௚͠

  34. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w3FUSZͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)' 2

    3.85 7.272500000000001 13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
  35. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ରॲ๏-BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠ w-BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

  36. ن໛ͷେ͖͍σʔλॲཧʹ͸  GlueΛ࢖͏ύλʔϯ

  37. σʔλϨΠΫपΓͷσʔλՃ޻ 

  38. -BNCEBͰॲཧ͖͠Εͳ͍σʔλՃ޻͸(MVFͰ  w-BNCEBؔ਺ͷ࠷େ࣮ߦ࣌ؒ  ෼ Λ௒͑ΔσʔλՃ޻ͷ৔߹͸ (MVFΛ࢖༻ w4UFQ'VODUJPOT (MVF %BUBCSFXͷ૊Έ߹Θͤ΋ΞϦ

  39. -BNCEBͱ(MVFɺͲͬͪ࢖͏ʁ  wॲཧ͢ΔϑΝΠϧ਺͕ଟ͘ɺ֤ϑΝΠϧͷ༰ྔ͕খ͍͞৔߹ ͸-BNCEBͷىಈͷ଎͞ɺಉ࣮࣌ߦ਺ͷଟ͞ͷར఺͕׆͖Δ wϑΝΠϧ༰ྔ͕େ͖͘ɺॲཧ͕࣌ؒଟ͔͔͘ΔσʔλΛॲཧ ͢Δ৔߹͸(MVFΛ࢖ͬͨॲཧ͕ྑ͍ w-BNCEBͰ΋UNQͷ༰ྔ͕(#·Ͱ֦ுՄೳͳͷͰɺ ͋Δఔ౓ͷ༰ྔͳΒॲཧͰ͖ͳ͘΋ͳ͍

  40. ·ͱΊ  wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳج൫ʹ࢖͑Δύ λʔϯΛத৺ʹ͝આ໌ w4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔ੍ޚͰ৭ʑϋϚͬͨ ͷͰɺͦͷ಺༰Λ͝঺հ w·ͣ͸γϯϓϧͳσʔλՃ޻ͷج൫Λߏஙͯ͠Έ·͠ΐ͏ w༻్ʹԠͯ͡αʔϏεΛ͏·͘࢖͍·͠ΐ͏

  41. None