サーバーレスパターンから学ぶデータ分析基盤構築 / devio2022

by kasacchiful

Slide 1

Slide 1 text

σʔλΞφϦςΟΫεࣄۀຊ෦ּݪ޺ αʔόʔϨεύλʔϯ͔ΒֶͿ σʔλ෼ੳج൫ߏங

Slide 2

Slide 2 text

͓࿩͢͠Δ͜ͱ wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳʹ࢖͑Δύλʔ ϯΛத৺ʹɺ࣮ࡍͷߏஙࣄྫΛަ͑ͳ͕Β͓࿩͠͠·͢ɻ wಛʹ4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔߏஙͰϋϚͬͨ ͱ͜ΖΛத৺ʹ͓࿩͠͠·͢ɻ

Slide 3

Slide 3 text

ࣗݾ঺հ w ּݪ޺ ೥݄δϣΠϯ w σʔλΞφϦςΟΫεࣄۀຊ෦ϏδωειϦϡʔγϣϯ෦Ϗοάσʔλ νʔϜ w ৽ׁݝ৽ׁࢢࡏॅ w "1/"845PQ&OHJOFFST "OBMZUJDT "1/"--"84 $FSUJ fi DBUJPOT&OHJOFFST w ޷͖ͳ"84αʔϏε4ɺ-BNCEBɺ4BHF.BLFS w +"846(৽ׁࢧ෦ʗ1ZUIPOػցֶशษڧձJO৽ׁʗ+B445৽ׁʗ 48"/**ʗFUD @kasacchiful @kasacchiful

Slide 4

Slide 4 text

ࠓճͷςʔϚ αʔόʔϨεύλʔϯ

Slide 5

Slide 5 text

αʔόʔϨεύλʔϯ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

Slide 6

Slide 6 text

Ϣʔεέʔεผʹύλʔϯ͕͋Δ IUUQTBXTBNB[PODPNKQTFSWFSMFTTQBUUFSOTTFSWFSMFTTQBUUFSO

Slide 7

Slide 7 text

αʔόʔϨεύλʔϯΛ࢖ͬͨσʔλ෼ੳج൫ wൺֱతن໛ͷখ͍͞σʔλ෼ੳج൫͔ΒɺαʔόʔϨεύ λʔϯΛ࢖ͬͯΈΔͱɺͲ͏ͳΔ͔ݟͯΈΔ wΑ͋͘Δύλʔϯ͔ΒϐοΫΞοϓͯ͠঺հ

Slide 8

Slide 8 text

ը૾ॲཧγϯϓϧͳσʔλՃ޻

Slide 9

Slide 9 text

·ͣ͸γϯϓϧͳσʔλՃ޻͔Β࢝ΊͯΈΔ w֨ೲ͢Δύε͸ɺ੔ܗલͱޙͰม ͑Δ͜ͱ wແݶϧʔϓΛճආ w-BNCEB಺ͷUNQྖҬʹҰ࣌อ ଘ wॲཧޙʹ໌ࣔతʹ࡟আ͢Δ͜ͱ w੔ܗޙ͸ผͷόέοτʹ֨ೲͯ͠ ΋ྑ͍

Slide 10

Slide 10 text

෼ࢄฒྻॲཧ

Slide 11

Slide 11 text

ྫ͑͹೥౓ຖʹॲཧΛ෼͚Δ wॲཧ͢Δσʔλຖʹ-BNCEBؔ ਺ىಈ wσʔλຖʹ෼ࢄॲཧͰ͖Δͷ Ͱɺτʔλϧͷ࣮ߦ͕࣌ؒ୹͘ ࡁΉ wσʔλຖʹॲཧ಺༰͕ҟͳΔ৔ ߹͸ɺ౎౓ؔ਺Λఆٛ͢Δඞཁ ͕͋Δέʔε΋

Slide 12

Slide 12 text

Πϕϯτۦಈͷۀ຿ॲཧ࿈ܞ

Slide 13

Slide 13 text

4/4ɾ424Λ૊߹ͤͯॲཧΛ࿈ܞ w4/4ͰϑΝϯΞ΢τ wฒྻ࣮ߦ w୯Ұ࣮ߦͰ΋σόοά༻్ʹ w424ͷΩϡʔʹ֨ೲ͢Δ͜ͱͰ ৴པੑ֬อ w'*'04/4'*'0424Ͱॱংੑ ΛߟྀՄೳ

Slide 14

Slide 14 text

424ͷલʹ4/4τϐοΫڬΜͰ͓͜͏ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTOTUPQJDTIPVMECFQMBDFECFIJOE TRTRVFVF

Slide 15

Slide 15 text

࣮ࡍʹߏஙͨ͠ج൫ wσʔλϑΝΠϧຖʹ4/4ɾ424 Λܦ༝ͯ͠-BNCEBͰॲཧ wॲཧͨ͠ϑΝΠϧΛ4ʹ֨ೲ͠ ͨΒɺޙଓͷॲཧ͕ىಈ w࠷ޙͷ-BNCEBॲཧͰϚʔδ

Slide 16

Slide 16 text

σʔλϑΝΠϧ͕ଟ͍৔߹ॲཧ͕ෳࡶͳ৔߹ Step Functionsͷ εςʔτϚγϯΛ׆༻

Slide 17

Slide 17 text

ΞϓϦέʔγϣϯॲཧ

Slide 18

Slide 18 text

"844UFQ'VODUJPOTͰϫʔΫϑϩʔ੍ޚ w4UFQ'VODUJPOTͷεςʔτϚγ ϯΛߏஙͯ͠ϫʔΫϑϩʔΛ੍ ޚ w1BSBMMFMλεΫɺ.BQλεΫΛ׆ ༻ͯ͠-BNCEBؔ਺Λฒྻ࣮ߦ wσʔλϑΝΠϧαΠζ͕খ͘͞ ͯେྔʹ͋ΔέʔεͰ༗ޮ

Slide 19

Slide 19 text

࣮ࡍʹߏஙͨ͠ج൫ w೔࣍ॲཧͰར༻ w֤छ%#΍ΦϒδΣΫτετϨʔ δʹΞΫηεͯ͠σʔλΛऔಘ wσʔλͷछྨ͕ଟ͍ͷͰɺछྨ ຖʹ1BSBMMFMλεΫͰॲཧ wσʔλϑΝΠϧͷ਺΋ଟ͍ͷ ͰɺϑΝΠϧຖʹ.BQλεΫͰ ॲཧ w࠷ޙʹ෼ੳ༻ͷσʔλΛੜ੒

Slide 20

Slide 20 text

࣮ࡍʹߏஙͨ͠ج൫ wຖ࣌ॲཧͰར༻ wϑΝΠϧ਺͕ඇৗʹଟ͍ͨ ΊɺͭͷεςʔτϚγϯͰ ͸ࡹ͖͖Εͳ͍ w࣮ߦཤྺ ݅ͷ ΫΥʔλʹҾ͔͔ͬΔ wεςʔτϚγϯΛೖΕࢠʹ ͢Δͱɺ-BNCEBͷಉ࣮࣌ ߦ਺΋૿Ճ

Slide 21

Slide 21 text

4UFQ'VODUJPOT৭ʑϋϚͬͨ ϋϚͬͨͱ͜ΖΛ 3ͭ঺հ

Slide 22

Slide 22 text

4UFQ'VODUJPOTͷΫΥʔλ ೰Έ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺͕ΫΥʔλʹҾ͔͔ͬΔ wຖ࣌ॲཧͷϑΝΠϧ਺͕ҟৗʹଟ͍ w࣌ؒܦͬͯҟৗऴྃɻ4UFQ'VODUJPOTͷΠϕϯτཤྺ਺ͷ্ݶ౸ୡ Π ϕϯτ w্ݶ؇࿨ෆՄͷ߲໨ { "error": "States.Runtime", "cause": "The execution reached the maximum number of history events (25000)." }

Slide 23

Slide 23 text

4UFQ'VODUJPOTͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

Slide 24

Slide 24 text

4UFQ'VODUJPOTͷΫΥʔλ ରॲ๏4UFQ'VODUJPOTͷεςʔτϚγϯΛೖΕࢠʹ wεςʔτϚγϯΛೖΕࢠʹ͢Δ͜ͱͰɺΠϕϯτཤྺ্ݶʹҾ͔͔ͬΒͳ͍Α͏ ʹͨ͠ w-BNCEBͷಉ࣮࣌ߦ਺͕͔ͳΓ૿͑ΔͷͰɺҎԼͷରԠΛ௥Ճ 㾎 -BNCEBͷಉ࣮࣌ߦ਺ͷ্ݶ؇࿨ਃ੥ 㾎 4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ

Slide 25

Slide 25 text

4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ

Slide 26

Slide 26 text

4UFQ'VODUJPOTͷΫΥʔλ มߋલ มߋޙ

Slide 27

Slide 27 text

೰Έ"UIFOBͷΫΤϦಉ࣮࣌ߦ਺ͷΫΥʔλʹҾ͔͔ͬΔ w σʔλҠߦ࣌ʹɺ೔࣍ॲཧͷ࠷ޙͷ-BNCEBͰΤϥʔʹͳΔ w લஈͰॲཧͨ͠ෳ਺σʔλΛ"UIFOB࢖ͬͯ42-ΫΤϦͰऔಘ͢Δͱ͜ΖͰ্ݶʹҾ͔͔ͬΔ w -BNCEBؔ਺ͭʹ͖ͭɺɹstart-query-executionɹ"1*Λճίʔϧ w Ұ࣌తʹόʔετͰ্ݶ·Ͱ૿͑Δ͚ͲɺσʔλҠߦ࣌ʹ͸Ͱ಄ଧͪ w ্ݶ؇࿨ਃ੥͢Ε͹্ݶ͋͛ΒΕΔ "UIFOBͷΫΥʔλ

Slide 28

Slide 28 text

"UIFOBͷΫΥʔλ IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHMJNJUTPWFSWJFXIUNM

Slide 29

Slide 29 text

"UIFOBͷΫΥʔλ ରॲ๏4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λઃఆ w.BQεςʔτ ഑ྻ౉͢ͱɺಉ࣮࣌ߦͰ഑ྻཁૉΛॲཧ͢ΔΠϝʔδ ͷ࠷େಉ࣌ ࣮ߦ਺Λઃఆ͠ɺ"UIFOBͷTUBSURVFSZFYFDVUJPO"1*ίʔϧΛ࠷େ·Ͱʹ ͓͑͞Δ

Slide 30

Slide 30 text

"UIFOBͷΫΥʔλ .BQεςʔτʹ͍ͭͯ͸ɺҎԼͷهࣄΛࢀߟʹ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTTUFQGVODUJPOTVQEBUFNBQTUBUF IUUQTEPDTBXTBNB[PODPNKB@KQTUFQGVODUJPOTMBUFTUEHBNB[POTUBUFTMBOHVBHFNBQTUBUFIUNM

Slide 31

Slide 31 text

-BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ ೰Έճ͚ͩ-BNCEBͷ3BUF-JNJUΤϥʔʹૺ۰ wಉ࣮࣌ߦ਺ͷΤϥʔͷΑ͏͚ͩͲʜ w͢Ͱʹಉ࣮࣌ߦ਺ͷ্ݶΛҾ্͖͍͛ͯΔ΋ͷͷɺ֤ؔ਺ͷϞχλϦϯάݟΔݶ Γɺಉ࣮࣌ߦ਺ʹ౸ୡ͍ͯ͠ͳ͍ { "error": "Lambda.TooManyRequestsException", "cause": "Rate Exceeded. (Service: Lambda, Status Code: 429, Request ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, Extended Request ID: null)" }

Slide 32

Slide 32 text

-BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ IUUQTEPDTBXTBNB[PODPNKB@KQMBNCEBMBUFTUEHJOWPDBUJPOTDBMJOHIUNM

Slide 33

Slide 33 text

-BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w4UFQ'VODUJPOTͷ.BQεςʔτͷ࠷େಉ࣮࣌ߦ਺Λݟ௚͠ w4UFQ'VODUJPOTͰఆٛ͢Δ-BNCEBͷ3FUSZઃఆΛݟ௚͠

Slide 34

Slide 34 text

-BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ ରॲ๏-BNCEBؔ਺ͷ3FUSZઃఆΛݟ௚͠ w3FUSZͷִؒʹ͍ͭͯ͸ҎԼͷهࣄ͕ৄ͍͠ $ node -e '((i,m,b)=>{for(let w=i,c=0;c

Slide 35

Slide 35 text

-BNCEBͷεέʔϧ͕௥͍͔ͭͳ͍ ରॲ๏-BNCEBؔ਺ͷ3FUSZ৚݅Λݟ௚͠ w-BNCEBͷ1SPWJTJPOFE$PODVSSFODZઃఆ͸ࠓճ࣮ࢪͯ͠ͳ͍ IUUQTEFWDMBTTNFUIPEKQBSUJDMFTMBNCEBQSPWJTJPOFEDPODVSSFODZDPMETUBSU

Slide 36

Slide 36 text

ن໛ͷେ͖͍σʔλॲཧʹ͸ GlueΛ࢖͏ύλʔϯ

Slide 37

Slide 37 text

σʔλϨΠΫपΓͷσʔλՃ޻

Slide 38

Slide 38 text

-BNCEBͰॲཧ͖͠Εͳ͍σʔλՃ޻͸(MVFͰ w-BNCEBؔ਺ͷ࠷େ࣮ߦ࣌ؒ ෼ Λ௒͑ΔσʔλՃ޻ͷ৔߹͸ (MVFΛ࢖༻ w4UFQ'VODUJPOT(MVF %BUBCSFXͷ૊Έ߹Θͤ΋ΞϦ

Slide 39

Slide 39 text

-BNCEBͱ(MVFɺͲͬͪ࢖͏ʁ wॲཧ͢ΔϑΝΠϧ਺͕ଟ͘ɺ֤ϑΝΠϧͷ༰ྔ͕খ͍͞৔߹ ͸-BNCEBͷىಈͷ଎͞ɺಉ࣮࣌ߦ਺ͷଟ͞ͷར఺͕׆͖Δ wϑΝΠϧ༰ྔ͕େ͖͘ɺॲཧ͕࣌ؒଟ͔͔͘ΔσʔλΛॲཧ ͢Δ৔߹͸(MVFΛ࢖ͬͨॲཧ͕ྑ͍ w-BNCEBͰ΋UNQͷ༰ྔ͕(#·Ͱ֦ுՄೳͳͷͰɺ ͋Δఔ౓ͷ༰ྔͳΒॲཧͰ͖ͳ͘΋ͳ͍

Slide 40

Slide 40 text

·ͱΊ wαʔόʔϨεύλʔϯͷத͔Βɺσʔλ෼ੳج൫ʹ࢖͑Δύ λʔϯΛத৺ʹ͝આ໌ w4UFQ'VODUJPOTΛ࢖ͬͨϫʔΫϑϩʔ੍ޚͰ৭ʑϋϚͬͨ ͷͰɺͦͷ಺༰Λ͝঺հ w·ͣ͸γϯϓϧͳσʔλՃ޻ͷج൫Λߏஙͯ͠Έ·͠ΐ͏ w༻్ʹԠͯ͡αʔϏεΛ͏·͘࢖͍·͠ΐ͏

Slide 41

Slide 41 text

No content