サーバーレスで構築するETL処理

Serverless Meetup Fukuoka #3で発表してきた資料です。

https://serverless.connpass.com/event/102585/
#serverlessfukuoka

https://dev.classmethod.jp/study_meeting/speaker_report_serverless_meetup-fukuoka_03

Transcript

1. サーόーレスͰ構築するETL処理 
   
   
   
   
   ۚ
   Serverless Meetup Fukuoka #3
2. "CPVU
   NF
3. ాத޹໌ @kongmingtrap wΫϥεϝιουגࣜձࣾ
   w෱ԬΦϑΟεॴଐ
   wϞόΠϧΞϓϦαʔϏε෦
   wϞόΠϧΞϓϦΤϯδχΞ
   wαʔόʔαΠυΞϓϦΤϯδχΞ
   wαʔόʔϨε։ൃ෦
   wΞϓϦέʔγϣϯΤϯδχΞ
5. .PUJWBUJPO
6. IUUQTHJUIVCDPNLPOHNJOHTUSBQ4FOTPS4USFBN αʔόʔϨεҊ݅ͱ"84
   *P5ͷ૬ੑ
7. wେྔͷηϯαʔ͔ΒͷσʔλΛϚωʔδυ αʔϏεͰߏஙͰ͖Δ
   w*P5
   3VMF
   &OHJOFͷ"DUJPOʹ,JOFTJTͳͲ Λઃఆ͢Δ͜ͱͰɺσʔλͷऔΓ͜΅͠Λ๷ ͙ αʔόʔϨεҊ݅ͱ"84
   *P5ͷ૬ੑ
8. w4ʹஔ͔ΕͨେྔͷηϯαʔσʔλΛՄࢹ Խ͍ͨ͠
   wσʔλͷத਎͔ΒҰ෦ΛՃ޻ͯ͠ՄࢹԽ͠ ͍ͨ σʔλΛཷΊͨ࣍ͷϑΣʔζ
9. w4ʹஔ͔ΕͨେྔͷηϯαʔσʔλΛՄ ࢹԽ͍ͨ͠
   wσʔλͷத਎͔ΒҰ෦ΛՃ޻ͯ͠ՄࢹԽ ͍ͨ͠ σʔλΛཷΊͨ࣍ͷϑΣʔζ
10. &5-
11. w&YUSBDU5SBOTGPSN-PBE
    w&YUSBDU
    wσʔλͷநग़
    w5SBOTGPSN
    wϏδωεཁ݅ʹԠͯ͡σʔλΛม׵ɾՃ޻
    w-PBE
    wՃ޻ࡁΈͷσʔλΛಡΈࠐΉ &5-
14. wநग़ɺม׵ɺϩʔυ
    &5-
    Λߦ͏׬શϚωʔ δυܕͷαʔϏε
    w"84
    (MVF
    ͸αʔόʔϨεͰ͋ΔͨΊɺΠ ϯϑϥετϥΫνϟͷߪೖɺઃఆɺ؅ཧ͸ෆ ཁ "84
    (MVF
15. wநग़ɺม׵ɺϩʔυ
    &5-
    Λߦ͏׬શϚωʔ δυܕͷαʔϏε
    w"84
    (MVF
    ͸αʔόʔϨεͰ͋Δͨ ΊɺΠϯϑϥετϥΫνϟͷߪೖɺઃఆɺ؅ ཧ͸ෆཁ "84
    (MVF
16. "84
    (MVFͰ&5-ॲཧ
17. w4ʹཷΊ͍ͯΔηϯαʔσʔλΛ$SBXMFS Λར༻ͯ͠σʔλϕʔεʹొ࿥͢Δ
    w+PCͷεΫϦϓτͰɺσʔλͷ஋Λݩʹɺ ม׵ॲཧΛߦ͍ɺ1BSRVFUܗࣜͰग़ྗ͢Δ
    w݁ՌͷσʔλΛ"UIFOBͰՄࢹԽ͢Δ ΍Γ͍ͨ͜ͱ
24. wσʔλΧλϩάΛߏங͢Δ
    wσʔλιʔεΛ࡞੒ͯ͠+40/ܗࣜͷϑΝΠϧΛಡ ΈࠐΉ
    wσʔλιʔε͔ΒಡΈࠐΜͩσʔλΛม׵͢Δ+PCͷ ࡞੒
    w1ZUIPO
    
    4DBMB
    w+PCͷεέδϡʔϦϯάͷ࡞੒ ࡞੒͢Δ΋ͷ
25. IUUQTBXTBNB[PODPNKQHMVF
26. αϯϓϧίʔυ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFS
27. $"65*0/ ՝ۚʹ஫ҙʂʂʂ
28. σʔλϕʔεˍ$SBXMFS IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFSCMPCNBTUFSUFNQMBUFT FUMZNM--
29. σʔλϕʔεˍ$SBXMFS
30. σʔλϕʔεˍ$SBXMFS IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFSCMPCNBTUFSUFNQMBUFT FUMZNM--
31. σʔλϕʔεˍ$SBXMFS
32. +PCͷ࡞੒ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFSCMPCNBTUFSUFNQMBUFT FUMZNM--
33. +PCͷ࡞੒
34. +PCͷεΫϦϓτ
35. +PCͷεΫϦϓτ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFSCMPCNBTUFSTSDEFWJDF IJTUPSZEFWJDF@IJTUPSZ@KPCQZ
36. w+PCͷॳظॲཧΛ࣮ࢪ͢Δ
    wσʔλιʔεΛಡΈࠐΉ
    wಡΈࠐΜͩσʔλιʔεΛϚοϐϯά͠ɺѻ͍΍ ͍͢ܗࣜʹ͢Δ
    wม׵͍ͨ͠ॲཧΛ࣮ࢪ͢Δ
    w݁ՌΛ1BSRVFUܗࣜͰग़ྗ͢Δ +PCͰ΍Δ͜ͱ
37. +PCͷεΫϦϓτ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFSCMPCNBTUFSTSDEFWJDF IJTUPSZEFWJDF@IJTUPSZ@KPCQZ--
38. +PCͷτϦΨʔ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC4BNQMFSCMPCNBTUFSUFNQMBUFT FUMZNM--
39. +PCͷτϦΨʔ
40. +PCͷτϦΨʔ
41. "UIFOBͰՄࢹԽ
43. αϯϓϧίʔυ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC"EE1BSUJUJPO
44. w"UIFOBͰՄࢹԽ͢Δσʔλͷςʔϒϧͷύʔ ςΟγϣϯΛ෼ׂ͢Δ
    w+PCͷεΫϦϓτ͕੒ޭͨ࣌͠ʹ-BNCEB ΛςʔϒϧͷύʔςΟγϣϯΛ௥Ճ͢Δ -BNCEBΛىಈ͢Δ ΍Γ͍ͨ͜ͱ IUUQTEPDTBXTBNB[PODPNKB@KQHMVFMBUFTUEHBVUPNBUJOHBXTHMVFXJUI DMPVEXBUDIFWFOUTIUNM
45. IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC"EE1BSUJUJPOCMPCNBTUFS TBNZNM-- &5-
    +PC͕੒ޭͨ͠ͱ͖ͷ
    -BNCEB
    ؔ਺ͷΞΫςΟϒԽ
46. IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC"EE1BSUJUJPOCMPCNBTUFS TBNZNM-- &5-
    +PC͕੒ޭͨ͠ͱ͖ͷ
    -BNCEB
    ؔ਺ͷΞΫςΟϒԽ
47. &5-
    +PC͕੒ޭͨ͠ͱ͖ͷ
    -BNCEB
    ؔ਺ͷΞΫςΟϒԽ IUUQTHJUIVCDPNLPOHNJOHTUSBQ(MVF+PC"EE1BSUJUJPOCMPCNBTUFSTSD IBOEMFSTEFWJDF@IJTUPSZ@BEE@QBSUJUJPOEFWJDF@IJTUPSZ@BEE@QBSUJUJPOQZ- -
48. &5-
    +PC͕੒ޭ࣌ʹύʔςΟγϣϯͷ෼ׂ
49. &5-
    +PC͕੒ޭ࣌ʹύʔςΟγϣϯͷ෼ׂ
50. &5-
    +PC͕੒ޭ࣌ʹύʔςΟγϣϯͷ෼ׂ
51. &5-
    +PC͕੒ޭ࣌ʹύʔςΟγϣϯͷ෼ׂ
52. ·ͱΊ
53. w-BNCEBͩͱɺ੍ݶʹҾ͔͔ͬΓͦ͏ͳॲ ཧ΋ؾʹͤͣߦ͏͜ͱ͕Ͱ͖Δ
    w+PCͷ੒ޭɾࣦഊΛϋϯυϦϯά͢Δ͜ͱ ͕Ͱ͖ΔͷͰɺ෮ؼͷॲཧ΍׬ྃॲཧ΋ผʹ ߏங͢Δ͜ͱ͕Ͱ͖Δ
    wඞཁʹԠͯ͡+PCͷεΫϦϓτΛ࣮ߦͰ͖ Δςετ؀ڥΛߏங͢Δ৔߹΋͋Δ ·ͱΊ
54. એ఻
56. ฐ͔ࣾΒ
    ໊͕
    ࢀՃ͠·͢
59. IUUQTDMBTTNFUIPEDPOOQBTTDPNFWFOU