サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

DevelopersIO 2022 ビデオセッション資料。
https://youtu.be/-YN1Z2oH-G0

DevelopersIO 2022 公式サイト:
https://classmethod.jp/m/developers-io/

Transcript

1.  σʔλΞφϦςΟΫεࣄۀຊ෦
   ּݪ
   ޺ αʔόʔϨεύλʔϯ͔ΒֶͿ σʔλ෼ੳج൫ߏங
2. ͓࿩͢͠Δ͜ͱ  wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳʹ࢖͑Δύλʔ ϯΛத৺ʹɺ࣮ࡍͷߏஙࣄྫΛަ͑ͳ͕Β͓࿩͠͠·͢ɻ
   wಛʹ4UFQ
   'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔߏஙͰϋϚͬͨ ͱ͜ΖΛத৺ʹ͓࿩͠͠·͢ɻ
3. ࣗݾ঺հ  w ּݪ
   ޺
   ೥݄δϣΠϯ
   w σʔλΞφϦςΟΫεࣄۀຊ෦
   ϏδωειϦϡʔγϣϯ෦
   Ϗοάσʔλ νʔϜ
   w ৽ׁݝ৽ׁࢢࡏॅ
   w 
   "1/
   "84
   5PQ
   &OHJOFFST
   "OBMZUJDT
   
   
   "1/
   "--
   "84
   $FSUJ fi DBUJPOT
   &OHJOFFST
   w ޷͖ͳ"84αʔϏε
   4ɺ-BNCEBɺ4BHF.BLFS
   w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ+B445৽ׁʗ 48"/**ʗFUD @kasacchiful @kasacchiful
4. ࠓճͷςʔϚ  αʔόʔϨεύλʔϯ
5. αʔόʔϨεύλʔϯ  IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
6. Ϣʔεέʔεผʹύλʔϯ͕͋Δ  IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
7. αʔόʔϨεύλʔϯΛ࢖ͬͨσʔλ෼ੳج൫  wൺֱతن໛ͷখ͍͞σʔλ෼ੳج൫͔ΒɺαʔόʔϨεύ λʔϯΛ࢖ͬͯΈΔͱɺͲ͏ͳΔ͔ݟͯΈΔ
   wΑ͋͘Δύλʔϯ͔ΒϐοΫΞοϓͯ͠঺հ
8. ը૾ॲཧ
   
   γϯϓϧͳσʔλՃ޻ 
9. ·ͣ͸γϯϓϧͳσʔλՃ޻͔Β࢝ΊͯΈΔ  w֨ೲ͢Δύε͸ɺ੔ܗલͱޙͰม ͑Δ͜ͱ
   wແݶϧʔϓΛճආ
   w-BNCEB಺ͷ
   UNQ
   ྖҬʹҰ࣌อ ଘ
   wॲཧޙʹ໌ࣔతʹ࡟আ͢Δ͜ͱ
   w੔ܗޙ͸ผͷόέοτʹ֨ೲͯ͠ ΋ྑ͍
10. ෼ࢄฒྻॲཧ 
11. ྫ͑͹೥౓ຖʹॲཧΛ෼͚Δ  wॲཧ͢Δσʔλຖʹ-BNCEBؔ ਺ىಈ
    wσʔλຖʹ෼ࢄॲཧͰ͖Δͷ Ͱɺτʔλϧͷ࣮ߦ͕࣌ؒ୹͘ ࡁΉ
    wσʔλຖʹॲཧ಺༰͕ҟͳΔ৔ ߹͸ɺ౎౓ؔ਺Λఆٛ͢Δඞཁ ͕͋Δέʔε΋
12. Πϕϯτۦಈͷۀ຿ॲཧ࿈ܞ 
13. 4/4ɾ424Λ૊߹ͤͯॲཧΛ࿈ܞ  w4/4ͰϑΝϯΞ΢τ
    wฒྻ࣮ߦ
    w୯Ұ࣮ߦͰ΋σόοά༻్ʹ
    w424ͷΩϡʔʹ֨ೲ͢Δ͜ͱͰ ৴པੑ֬อ
    w'*'0
    4/4'*'0
    424Ͱॱংੑ ΛߟྀՄೳ
14. 424ͷલʹ4/4τϐοΫڬΜͰ͓͜͏  IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTOTUPQJDTIPVMECFQMBDFECFIJOE TRTRVFVF
15. ࣮ࡍʹߏஙͨ͠ج൫  wσʔλϑΝΠϧຖʹ4/4ɾ424 Λܦ༝ͯ͠-BNCEBͰॲཧ
    wॲཧͨ͠ϑΝΠϧΛ4ʹ֨ೲ͠ ͨΒɺޙଓͷॲཧ͕ىಈ
    w࠷ޙͷ-BNCEBॲཧͰϚʔδ
16. σʔλϑΝΠϧ͕ଟ͍৔߹
    
    ॲཧ͕ෳࡶͳ৔߹  Step Functionsͷ εςʔτϚγϯΛ׆༻
17. ΞϓϦέʔγϣϯॲཧ 
18. "84
    4UFQ
    'VODUJPOTͰϫʔΫϑϩʔ੍ޚ  w4UFQ
    'VODUJPOTͷεςʔτϚγ ϯΛߏஙͯ͠ϫʔΫϑϩʔΛ੍ ޚ
    w1BSBMMFMλεΫɺ.BQλεΫΛ׆ ༻ͯ͠-BNCEBؔ਺Λฒྻ࣮ߦ
    wσʔλϑΝΠϧαΠζ͕খ͘͞ ͯେྔʹ͋ΔέʔεͰ༗ޮ
19. ࣮ࡍʹߏஙͨ͠ج൫  w೔࣍ॲཧͰར༻
    w֤छ%#΍ΦϒδΣΫτετϨʔ δʹΞΫηεͯ͠σʔλΛऔಘ
    wσʔλͷछྨ͕ଟ͍ͷͰɺछྨ ຖʹ1BSBMMFMλεΫͰॲཧ
    wσʔλϑΝΠϧͷ਺΋ଟ͍ͷ ͰɺϑΝΠϧຖʹ.BQλεΫͰ ॲཧ
    w࠷ޙʹ෼ੳ༻ͷσʔλΛੜ੒
20. ࣮ࡍʹߏஙͨ͠ج൫  wຖ࣌ॲཧͰར༻
    wϑΝΠϧ਺͕ඇৗʹଟ͍ͨ ΊɺͭͷεςʔτϚγϯͰ ͸ࡹ͖͖Εͳ͍
    w࣮ߦཤྺ ݅ͷ ΫΥʔλʹҾ͔͔ͬΔ
    wεςʔτϚγϯΛೖΕࢠʹ ͢Δͱɺ-BNCEBͷಉ࣮࣌ ߦ਺΋૿Ճ
21. 4UFQ
    'VODUJPOT৭ʑϋϚͬͨ  ϋϚͬͨͱ͜ΖΛ 3ͭ঺հ
22. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  ೰Έ
    4UFQ
    'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ
    wຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍
    w࣌ؒܦͬͯҟৗऴྃɻ4UFQ
    'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ
     Π ϕϯτ
    w্ݶ؇࿨ෆՄͷ߲໨ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }
23. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
24. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  ରॲ๏
    4UFQ
    'VODUJPOT
    ͷεςʔτϚγϯΛೖΕࢠʹ
    wεςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ ʹͨ͠
    w-BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ
    㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥
    㾎 4UFQ
    'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ
25. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ
26. 
    4UFQ
    'VODUJPOTͷΫΥʔλ  มߋલ มߋޙ
27. ೰Έ
    "UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ
    w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ
    w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ
    w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ
    w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ
    w ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ 
    "UIFOBͷΫΥʔλ 
28. 
    "UIFOBͷΫΥʔλ  IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
29. 
    "UIFOBͷΫΥʔλ  ରॲ๏
    4UFQ
    'VODUJPOT
    ͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ
    w.BQεςʔτ
    ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ
    ͷ࠷େಉ࣌ ࣮ߦ਺Λઃఆ͠ɺ"UIFOBͷ
    TUBSURVFSZFYFDVUJPO
    "1*ίʔϧΛ࠷େ·Ͱʹ ͓͑͞Δ
30. 
    "UIFOBͷΫΥʔλ  .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM
31. 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ೰Έ
    ճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰
    wಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ
    w͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶ Γɺಉ࣮࣌ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ { "error": "Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }
32. 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM
33. 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ରॲ๏
    -BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w4UFQ
    'VODUJPOT
    ͷ
    .BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠
    w4UFQ
    'VODUJPOT
    Ͱఆٛ͢Δ
    -BNCEB
    ͷ
    3FUSZ
    ઃఆΛݟ௚͠
34. 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ରॲ๏
    -BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w3FUSZ
    ͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ $ node -e '((i,m,b)=>{for(let w=i,c=0;c<m;c++){console.log(w+=(c==0?0:b**c))}})(2,7,1.85)' 2 3.85 7.272500000000001 13.604125000000002 25.317631250000005 46.987617812500005 87.07709295312502 IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
35. 
    -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍  ରॲ๏
    -BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠
    w-BNCEB
    ͷ
    1SPWJTJPOFE
    $PODVSSFODZ
    ઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU
36. ن໛ͷେ͖͍σʔλॲཧʹ͸  GlueΛ࢖͏ύλʔϯ
37. σʔλϨΠΫपΓͷσʔλՃ޻ 
38. -BNCEBͰॲཧ͖͠Εͳ͍σʔλՃ޻͸(MVFͰ  w-BNCEBؔ਺ͷ࠷େ࣮ߦ࣌ؒ  ෼ Λ௒͑ΔσʔλՃ޻ͷ৔߹͸ (MVFΛ࢖༻
    w4UFQ
    'VODUJPOT
    
    (MVF
    %BUBCSFXͷ૊Έ߹Θͤ΋ΞϦ
39. -BNCEBͱ(MVFɺͲͬͪ࢖͏ʁ  wॲཧ͢ΔϑΝΠϧ਺͕ଟ͘ɺ֤ϑΝΠϧͷ༰ྔ͕খ͍͞৔߹ ͸-BNCEBͷىಈͷ଎͞ɺಉ࣮࣌ߦ਺ͷଟ͞ͷར఺͕׆͖Δ
    wϑΝΠϧ༰ྔ͕େ͖͘ɺॲཧ͕࣌ؒଟ͔͔͘ΔσʔλΛॲཧ ͢Δ৔߹͸(MVFΛ࢖ͬͨॲཧ͕ྑ͍
    w-BNCEBͰ΋
    UNQ
    ͷ༰ྔ͕(#·Ͱ֦ுՄೳͳͷͰɺ ͋Δఔ౓ͷ༰ྔͳΒॲཧͰ͖ͳ͘΋ͳ͍
40. ·ͱΊ  wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳج൫ʹ࢖͑Δύ λʔϯΛத৺ʹ͝આ໌
    w4UFQ
    'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔ੍ޚͰ৭ʑϋϚͬͨ ͷͰɺͦͷ಺༰Λ͝঺հ
    w·ͣ͸γϯϓϧͳσʔλՃ޻ͷج൫Λߏஙͯ͠Έ·͠ΐ͏
    w༻్ʹԠͯ͡αʔϏεΛ͏·͘࢖͍·͠ΐ͏