サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

DevelopersIO 2022 ビデオセッション資料。
https://youtu.be/-YN1Z2oH-G0

DevelopersIO 2022 公式サイト:
https://classmethod.jp/m/developers-io/

Transcript

1. 
   σʔλΞφϦςΟΫεࣄۀຊ෦ּݪ޺
   αʔόʔϨεύλʔϯ͔ΒֶͿ
   σʔλ෼ੳج൫ߏங
   

   View Slide

2. ͓࿩͢͠Δ͜ͱ
   wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳʹ࢖͑Δύλʔ
   ϯΛத৺ʹɺ࣮ࡍͷߏஙࣄྫΛަ͑ͳ͕Β͓࿩͠͠·͢ɻ
   wಛʹ4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔߏஙͰϋϚͬͨ
   ͱ͜ΖΛத৺ʹ͓࿩͠͠·͢ɻ
   

   View Slide

3. ࣗݾ঺հ
   w ּݪ޺ ೥݄δϣΠϯ
   
   w σʔλΞφϦςΟΫεࣄۀຊ෦ϏδωειϦϡʔγϣϯ෦Ϗοάσʔλ
   νʔϜ
   w ৽ׁݝ৽ׁࢢࡏॅ
   w "1/"845PQ&OHJOFFST "OBMZUJDT
   "1/"--"84
   $FSUJ
   fi
   DBUJPOT&OHJOFFST
   w ޷͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS
   w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ+B445৽ׁʗ
   48"/**ʗFUD
   @kasacchiful
   @kasacchiful
   

   View Slide

4. ࠓճͷςʔϚ
   αʔόʔϨεύλʔϯ
   

   View Slide

5. αʔόʔϨεύλʔϯ
   IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
   

   View Slide

6. Ϣʔεέʔεผʹύλʔϯ͕͋Δ
   IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO
   

   View Slide

7. αʔόʔϨεύλʔϯΛ࢖ͬͨσʔλ෼ੳج൫
   wൺֱతن໛ͷখ͍͞σʔλ෼ੳج൫͔ΒɺαʔόʔϨεύ
   λʔϯΛ࢖ͬͯΈΔͱɺͲ͏ͳΔ͔ݟͯΈΔ
   wΑ͋͘Δύλʔϯ͔ΒϐοΫΞοϓͯ͠঺հ
   

   View Slide

8. ը૾ॲཧγϯϓϧͳσʔλՃ޻
   

   View Slide

9. ·ͣ͸γϯϓϧͳσʔλՃ޻͔Β࢝ΊͯΈΔ
   w֨ೲ͢Δύε͸ɺ੔ܗલͱޙͰม
   ͑Δ͜ͱ
   wແݶϧʔϓΛճආ
   w-BNCEB಺ͷUNQྖҬʹҰ࣌อ
   ଘ
   wॲཧޙʹ໌ࣔతʹ࡟আ͢Δ͜ͱ
   w੔ܗޙ͸ผͷόέοτʹ֨ೲͯ͠
   ΋ྑ͍
   

   View Slide

10. ෼ࢄฒྻॲཧ
    

    View Slide

11. ྫ͑͹೥౓ຖʹॲཧΛ෼͚Δ
    wॲཧ͢Δσʔλຖʹ-BNCEBؔ
    ਺ىಈ
    wσʔλຖʹ෼ࢄॲཧͰ͖Δͷ
    Ͱɺτʔλϧͷ࣮ߦ͕࣌ؒ୹͘
    ࡁΉ
    wσʔλຖʹॲཧ಺༰͕ҟͳΔ৔
    ߹͸ɺ౎౓ؔ਺Λఆٛ͢Δඞཁ
    ͕͋Δέʔε΋
    

    View Slide

12. Πϕϯτۦಈͷۀ຿ॲཧ࿈ܞ
    

    View Slide

13. 4/4ɾ424Λ૊߹ͤͯॲཧΛ࿈ܞ
    w4/4ͰϑΝϯΞ΢τ
    wฒྻ࣮ߦ
    w୯Ұ࣮ߦͰ΋σόοά༻్ʹ
    w424ͷΩϡʔʹ֨ೲ͢Δ͜ͱͰ
    ৴པੑ֬อ
    w'*'04/4'*'0424Ͱॱংੑ
    ΛߟྀՄೳ
    

    View Slide

14. 424ͷલʹ4/4τϐοΫڬΜͰ͓͜͏
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTOTUPQJDTIPVMECFQMBDFECFIJOE
    TRTRVFVF
    

    View Slide

15. ࣮ࡍʹߏஙͨ͠ج൫
    wσʔλϑΝΠϧຖʹ4/4ɾ424
    Λܦ༝ͯ͠-BNCEBͰॲཧ
    wॲཧͨ͠ϑΝΠϧΛ4ʹ֨ೲ͠
    ͨΒɺޙଓͷॲཧ͕ىಈ
    w࠷ޙͷ-BNCEBॲཧͰϚʔδ
    

    View Slide

16. σʔλϑΝΠϧ͕ଟ͍৔߹ॲཧ͕ෳࡶͳ৔߹
    Step Functionsͷ
    
    
    εςʔτϚγϯΛ׆༻
    

    View Slide

17. ΞϓϦέʔγϣϯॲཧ
    

    View Slide

18. "844UFQ'VODUJPOTͰϫʔΫϑϩʔ੍ޚ
    w4UFQ'VODUJPOTͷεςʔτϚγ
    ϯΛߏஙͯ͠ϫʔΫϑϩʔΛ੍
    ޚ
    w1BSBMMFMλεΫɺ.BQλεΫΛ׆
    ༻ͯ͠-BNCEBؔ਺Λฒྻ࣮ߦ
    wσʔλϑΝΠϧαΠζ͕খ͘͞
    ͯେྔʹ͋ΔέʔεͰ༗ޮ
    

    View Slide

19. ࣮ࡍʹߏஙͨ͠ج൫
    w೔࣍ॲཧͰར༻
    w֤छ%#΍ΦϒδΣΫτετϨʔ
    δʹΞΫηεͯ͠σʔλΛऔಘ
    wσʔλͷछྨ͕ଟ͍ͷͰɺछྨ
    ຖʹ1BSBMMFMλεΫͰॲཧ
    wσʔλϑΝΠϧͷ਺΋ଟ͍ͷ
    ͰɺϑΝΠϧຖʹ.BQλεΫͰ
    ॲཧ
    w࠷ޙʹ෼ੳ༻ͷσʔλΛੜ੒
    

    View Slide

20. ࣮ࡍʹߏஙͨ͠ج൫
    wຖ࣌ॲཧͰར༻
    wϑΝΠϧ਺͕ඇৗʹଟ͍ͨ
    ΊɺͭͷεςʔτϚγϯͰ
    ͸ࡹ͖͖Εͳ͍
    w࣮ߦཤྺ ݅ͷ
    ΫΥʔλʹҾ͔͔ͬΔ
    wεςʔτϚγϯΛೖΕࢠʹ
    ͢Δͱɺ-BNCEBͷಉ࣮࣌
    ߦ਺΋૿Ճ
    

    View Slide

21. 4UFQ'VODUJPOT৭ʑϋϚͬͨ
    ϋϚͬͨͱ͜ΖΛ
    
    
    3ͭ঺հ
    

    View Slide

22. 4UFQ'VODUJPOTͷΫΥʔλ
    ೰Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ
    wຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍
    w࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ Π
    ϕϯτ
    
    w্ݶ؇࿨ෆՄͷ߲໨
    {
    
    
    "error": "States.Runtime",
    
    
    "cause": "The execution reached the maximum number of history events (25000)."
    
    
    }
    

    View Slide

23. 4UFQ'VODUJPOTͷΫΥʔλ
    IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
    

    View Slide

24. 4UFQ'VODUJPOTͷΫΥʔλ
    ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ
    wεςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏
    ʹͨ͠
    w-BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ
    㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥
    㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ
    

    View Slide

25. 4UFQ'VODUJPOTͷΫΥʔλ
    มߋલ มߋޙ
    

    View Slide

26. 4UFQ'VODUJPOTͷΫΥʔλ
    มߋલ มߋޙ
    

    View Slide

27. ೰Έ"UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ
    w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ
    w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ
    w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ
    w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ
    w ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ
    "UIFOBͷΫΥʔλ
    

    View Slide

28. "UIFOBͷΫΥʔλ
    IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM
    

    View Slide

29. "UIFOBͷΫΥʔλ
    ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ
    w.BQεςʔτ ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ
    ͷ࠷େಉ࣌
    ࣮ߦ਺Λઃఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ
    ͓͑͞Δ
    

    View Slide

30. "UIFOBͷΫΥʔλ
    .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM
    

    View Slide

31. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ೰Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰
    wಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ
    w͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶ
    Γɺಉ࣮࣌ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍
    {
    
    
    "error": "Lambda.TooManyRequestsException",
    
    
    "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID:
    xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)"
    
    
    }
    

    View Slide

32. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM
    

    View Slide

33. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠
    w4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ௚͠
    

    View Slide

34. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w3FUSZͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠
    $ node -e '((i,m,b)=>{for(let w=i,c=0;c
    
    2
    
    
    3.85
    
    
    7.272500000000001
    
    
    13.604125000000002
    
    
    25.317631250000005
    
    
    46.987617812500005
    
    
    87.07709295312502
    
    
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ
    

    View Slide

35. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ରॲ๏-BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠
    w-BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU
    

    View Slide

36. ن໛ͷେ͖͍σʔλॲཧʹ͸
    GlueΛ࢖͏ύλʔϯ
    

    View Slide

37. σʔλϨΠΫपΓͷσʔλՃ޻
    

    View Slide

38. -BNCEBͰॲཧ͖͠Εͳ͍σʔλՃ޻͸(MVFͰ
    w-BNCEBؔ਺ͷ࠷େ࣮ߦ࣌ؒ
    ෼
    Λ௒͑ΔσʔλՃ޻ͷ৔߹͸
    (MVFΛ࢖༻
    w4UFQ'VODUJPOT(MVF
    %BUBCSFXͷ૊Έ߹Θͤ΋ΞϦ
    

    View Slide

39. -BNCEBͱ(MVFɺͲͬͪ࢖͏ʁ
    wॲཧ͢ΔϑΝΠϧ਺͕ଟ͘ɺ֤ϑΝΠϧͷ༰ྔ͕খ͍͞৔߹
    ͸-BNCEBͷىಈͷ଎͞ɺಉ࣮࣌ߦ਺ͷଟ͞ͷར఺͕׆͖Δ
    wϑΝΠϧ༰ྔ͕େ͖͘ɺॲཧ͕࣌ؒଟ͔͔͘ΔσʔλΛॲཧ
    ͢Δ৔߹͸(MVFΛ࢖ͬͨॲཧ͕ྑ͍
    w-BNCEBͰ΋UNQͷ༰ྔ͕(#·Ͱ֦ுՄೳͳͷͰɺ
    ͋Δఔ౓ͷ༰ྔͳΒॲཧͰ͖ͳ͘΋ͳ͍
    

    View Slide

40. ·ͱΊ
    wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳج൫ʹ࢖͑Δύ
    λʔϯΛத৺ʹ͝આ໌
    w4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔ੍ޚͰ৭ʑϋϚͬͨ
    ͷͰɺͦͷ಺༰Λ͝঺հ
    w·ͣ͸γϯϓϧͳσʔλՃ޻ͷج൫Λߏஙͯ͠Έ·͠ΐ͏
    w༻్ʹԠͯ͡αʔϏεΛ͏·͘࢖͍·͠ΐ͏
    

    View Slide

41. View Slide