$30 off During Our Annual Pro Sale. View Details »

サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

DevelopersIO 2022 ビデオセッション資料。
https://youtu.be/-YN1Z2oH-G0

DevelopersIO 2022 公式サイト:
https://classmethod.jp/m/developers-io/

kasacchiful

July 27, 2022
Tweet

More Decks by kasacchiful

Other Decks in Programming

Transcript


  1. σʔλΞφϦςΟΫεࣄۀຊ෦ּݪ޺
    αʔόʔϨεύλʔϯ͔ΒֶͿ
    σʔλ෼ੳج൫ߏங

    View Slide

  2. ͓࿩͢͠Δ͜ͱ
    wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳʹ࢖͑Δύλʔ
    ϯΛத৺ʹɺ࣮ࡍͷߏஙࣄྫΛަ͑ͳ͕Β͓࿩͠͠·͢ɻ
    wಛʹ4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔߏஙͰϋϚͬͨ
    ͱ͜ΖΛத৺ʹ͓࿩͠͠·͢ɻ

    View Slide

  3. ࣗݾ঺հ
    w ּݪ޺ ೥݄δϣΠϯ

    w σʔλΞφϦςΟΫεࣄۀຊ෦ϏδωειϦϡʔγϣϯ෦Ϗοάσʔλ
    νʔϜ
    w ৽ׁݝ৽ׁࢢࡏॅ
    w "1/"845PQ&OHJOFFST "OBMZUJDT
    "1/"--"84
    $FSUJ
    fi
    DBUJPOT&OHJOFFST
    w ޷͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS
    w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ+B445৽ׁʗ
    48"/**ʗFUD
    @kasacchiful
    @kasacchiful

    View Slide

  4. ࠓճͷςʔϚ
    αʔόʔϨεύλʔϯ

    View Slide

  5. αʔόʔϨεύλʔϯ
    IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

    View Slide

  6. Ϣʔεέʔεผʹύλʔϯ͕͋Δ
    IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

    View Slide

  7. αʔόʔϨεύλʔϯΛ࢖ͬͨσʔλ෼ੳج൫
    wൺֱతن໛ͷখ͍͞σʔλ෼ੳج൫͔ΒɺαʔόʔϨεύ
    λʔϯΛ࢖ͬͯΈΔͱɺͲ͏ͳΔ͔ݟͯΈΔ
    wΑ͋͘Δύλʔϯ͔ΒϐοΫΞοϓͯ͠঺հ

    View Slide

  8. ը૾ॲཧγϯϓϧͳσʔλՃ޻

    View Slide

  9. ·ͣ͸γϯϓϧͳσʔλՃ޻͔Β࢝ΊͯΈΔ
    w֨ೲ͢Δύε͸ɺ੔ܗલͱޙͰม
    ͑Δ͜ͱ
    wແݶϧʔϓΛճආ
    w-BNCEB಺ͷUNQྖҬʹҰ࣌อ

    wॲཧޙʹ໌ࣔతʹ࡟আ͢Δ͜ͱ
    w੔ܗޙ͸ผͷόέοτʹ֨ೲͯ͠
    ΋ྑ͍

    View Slide

  10. ෼ࢄฒྻॲཧ

    View Slide

  11. ྫ͑͹೥౓ຖʹॲཧΛ෼͚Δ
    wॲཧ͢Δσʔλຖʹ-BNCEBؔ
    ਺ىಈ
    wσʔλຖʹ෼ࢄॲཧͰ͖Δͷ
    Ͱɺτʔλϧͷ࣮ߦ͕࣌ؒ୹͘
    ࡁΉ
    wσʔλຖʹॲཧ಺༰͕ҟͳΔ৔
    ߹͸ɺ౎౓ؔ਺Λఆٛ͢Δඞཁ
    ͕͋Δέʔε΋

    View Slide

  12. Πϕϯτۦಈͷۀ຿ॲཧ࿈ܞ

    View Slide

  13. 4/4ɾ424Λ૊߹ͤͯॲཧΛ࿈ܞ
    w4/4ͰϑΝϯΞ΢τ
    wฒྻ࣮ߦ
    w୯Ұ࣮ߦͰ΋σόοά༻్ʹ
    w424ͷΩϡʔʹ֨ೲ͢Δ͜ͱͰ
    ৴པੑ֬อ
    w'*'04/4'*'0424Ͱॱংੑ
    ΛߟྀՄೳ

    View Slide

  14. 424ͷલʹ4/4τϐοΫڬΜͰ͓͜͏
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTOTUPQJDTIPVMECFQMBDFECFIJOE
    TRTRVFVF

    View Slide

  15. ࣮ࡍʹߏஙͨ͠ج൫
    wσʔλϑΝΠϧຖʹ4/4ɾ424
    Λܦ༝ͯ͠-BNCEBͰॲཧ
    wॲཧͨ͠ϑΝΠϧΛ4ʹ֨ೲ͠
    ͨΒɺޙଓͷॲཧ͕ىಈ
    w࠷ޙͷ-BNCEBॲཧͰϚʔδ

    View Slide

  16. σʔλϑΝΠϧ͕ଟ͍৔߹ॲཧ͕ෳࡶͳ৔߹
    Step Functionsͷ


    εςʔτϚγϯΛ׆༻

    View Slide

  17. ΞϓϦέʔγϣϯॲཧ

    View Slide

  18. "844UFQ'VODUJPOTͰϫʔΫϑϩʔ੍ޚ
    w4UFQ'VODUJPOTͷεςʔτϚγ
    ϯΛߏஙͯ͠ϫʔΫϑϩʔΛ੍
    ޚ
    w1BSBMMFMλεΫɺ.BQλεΫΛ׆
    ༻ͯ͠-BNCEBؔ਺Λฒྻ࣮ߦ
    wσʔλϑΝΠϧαΠζ͕খ͘͞
    ͯେྔʹ͋ΔέʔεͰ༗ޮ

    View Slide

  19. ࣮ࡍʹߏஙͨ͠ج൫
    w೔࣍ॲཧͰར༻
    w֤छ%#΍ΦϒδΣΫτετϨʔ
    δʹΞΫηεͯ͠σʔλΛऔಘ
    wσʔλͷछྨ͕ଟ͍ͷͰɺछྨ
    ຖʹ1BSBMMFMλεΫͰॲཧ
    wσʔλϑΝΠϧͷ਺΋ଟ͍ͷ
    ͰɺϑΝΠϧຖʹ.BQλεΫͰ
    ॲཧ
    w࠷ޙʹ෼ੳ༻ͷσʔλΛੜ੒

    View Slide

  20. ࣮ࡍʹߏஙͨ͠ج൫
    wຖ࣌ॲཧͰར༻
    wϑΝΠϧ਺͕ඇৗʹଟ͍ͨ
    ΊɺͭͷεςʔτϚγϯͰ
    ͸ࡹ͖͖Εͳ͍
    w࣮ߦཤྺ ݅ͷ
    ΫΥʔλʹҾ͔͔ͬΔ
    wεςʔτϚγϯΛೖΕࢠʹ
    ͢Δͱɺ-BNCEBͷಉ࣮࣌
    ߦ਺΋૿Ճ

    View Slide

  21. 4UFQ'VODUJPOT৭ʑϋϚͬͨ
    ϋϚͬͨͱ͜ΖΛ


    3ͭ঺հ

    View Slide

  22. 4UFQ'VODUJPOTͷΫΥʔλ
    ೰Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ
    wຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍
    w࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ Π
    ϕϯτ

    w্ݶ؇࿨ෆՄͷ߲໨
    {


    "error": "States.Runtime",


    "cause": "The execution reached the maximum number of history events (25000)."


    }

    View Slide

  23. 4UFQ'VODUJPOTͷΫΥʔλ
    IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

    View Slide

  24. 4UFQ'VODUJPOTͷΫΥʔλ
    ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ
    wεςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏
    ʹͨ͠
    w-BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ
    㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥
    㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ

    View Slide

  25. 4UFQ'VODUJPOTͷΫΥʔλ
    มߋલ มߋޙ

    View Slide

  26. 4UFQ'VODUJPOTͷΫΥʔλ
    มߋલ มߋޙ

    View Slide

  27. ೰Έ"UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ
    w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ
    w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ
    w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ
    w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ
    w ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ
    "UIFOBͷΫΥʔλ

    View Slide

  28. "UIFOBͷΫΥʔλ
    IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

    View Slide

  29. "UIFOBͷΫΥʔλ
    ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ
    w.BQεςʔτ ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ
    ͷ࠷େಉ࣌
    ࣮ߦ਺Λઃఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ
    ͓͑͞Δ

    View Slide

  30. "UIFOBͷΫΥʔλ
    .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

    View Slide

  31. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ೰Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰
    wಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ
    w͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶ
    Γɺಉ࣮࣌ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍
    {


    "error": "Lambda.TooManyRequestsException",


    "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID:
    xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)"


    }

    View Slide

  32. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

    View Slide

  33. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠
    w4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ௚͠

    View Slide

  34. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠
    w3FUSZͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠
    $ node -e '((i,m,b)=>{for(let w=i,c=0;c

    2


    3.85


    7.272500000000001


    13.604125000000002


    25.317631250000005


    46.987617812500005


    87.07709295312502


    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTXBJU@UJNF@BOE@QBSBNT@JO@TUFQ@GVODUJPO@SFUSZ

    View Slide

  35. -BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍
    ରॲ๏-BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠
    w-BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍
    IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

    View Slide

  36. ن໛ͷେ͖͍σʔλॲཧʹ͸
    GlueΛ࢖͏ύλʔϯ

    View Slide

  37. σʔλϨΠΫपΓͷσʔλՃ޻

    View Slide

  38. -BNCEBͰॲཧ͖͠Εͳ͍σʔλՃ޻͸(MVFͰ
    w-BNCEBؔ਺ͷ࠷େ࣮ߦ࣌ؒ

    Λ௒͑ΔσʔλՃ޻ͷ৔߹͸
    (MVFΛ࢖༻
    w4UFQ'VODUJPOT(MVF
    %BUBCSFXͷ૊Έ߹Θͤ΋ΞϦ

    View Slide

  39. -BNCEBͱ(MVFɺͲͬͪ࢖͏ʁ
    wॲཧ͢ΔϑΝΠϧ਺͕ଟ͘ɺ֤ϑΝΠϧͷ༰ྔ͕খ͍͞৔߹
    ͸-BNCEBͷىಈͷ଎͞ɺಉ࣮࣌ߦ਺ͷଟ͞ͷར఺͕׆͖Δ
    wϑΝΠϧ༰ྔ͕େ͖͘ɺॲཧ͕࣌ؒଟ͔͔͘ΔσʔλΛॲཧ
    ͢Δ৔߹͸(MVFΛ࢖ͬͨॲཧ͕ྑ͍
    w-BNCEBͰ΋UNQͷ༰ྔ͕(#·Ͱ֦ுՄೳͳͷͰɺ
    ͋Δఔ౓ͷ༰ྔͳΒॲཧͰ͖ͳ͘΋ͳ͍

    View Slide

  40. ·ͱΊ
    wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳج൫ʹ࢖͑Δύ
    λʔϯΛத৺ʹ͝આ໌
    w4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔ੍ޚͰ৭ʑϋϚͬͨ
    ͷͰɺͦͷ಺༰Λ͝঺հ
    w·ͣ͸γϯϓϧͳσʔλՃ޻ͷج൫Λߏஙͯ͠Έ·͠ΐ͏
    w༻్ʹԠͯ͡αʔϏεΛ͏·͘࢖͍·͠ΐ͏

    View Slide

  41. View Slide