Share
Embed
DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization
DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門
Masatoshi Shimada
January 19, 2019
Transcript
58 slides
-
ࣗݾհ w ౡాխ!TNENUT w σʔλΣΞϋεΞʔΩςΫτ 4DBMBc1ZUIPOc(PMBOHc+BWBc%%% w גࣜձࣾΤϒϦʔॴଐ w
%&-*4),*5$)&/σʔλੳܥશൠ୲ ओʹੳج൫ɺ৴໘࠷దԽͳͲΛ୲ -
ΞϓϦϦϦʔε μϯϩʔυɿສ Ϩγϐɿສ݅Ҏ্ ݄ؒಈը࠶ੜɿԯສճҎ্ "QQ4UPSF#&450' (PPHMF1MBZϕετΦϒ ࠃΞϓϦμϯϩʔυୈ̏Ґ ʢ̍Ґ-*/&ɺ̎Ґ:BIPP+"1"/˞ඇήʔϜɹ"QQ"OOJFௐʣ ɾຖ৽ணϨγϐಈը͕৴͞ΕΔ ɹɾྉཧϨγϐಈըʢௐཧݙཱखॱʣ
ɹɾࠂओλΠΞοϓಈը <13>%&-*4),*5$)&/ͷհʢ͔ΜͨΜʹʣ -
%BUBCSJDLT6OJpFE"OBMZUJDT1MBUGPSN ɾ4QBSLΛ࣮ߦج൫ͱͨ͠σʔλੳϓϥοτϑΥʔϜ 6OJpFE"OBMZUJDT1MBUGPSNΛࢦ͍ͯͯ͠࡞ΒΕ͍ͯΔ ɾ "84"[VSF ͷϢʔβΞΧϯτͰɺ71$࡞ͳͲॴఆͷΠϯετʔϧͰಋೖ ར༻ऀͷ "84"[VSF ΞΧϯτͰΫϥελ͕ىಈ͞ΕΔ ɾσʔλΤϯδχΞαΠΤϯςΟετ
ɹ/PUFCPPLͰεΫϦϓτΛॻ͍ͯ։ൃ͢Δ 4DBMB1ZUIPO34IFMM42-͕ར༻Մ &5-ॲཧ/PUFCPPLͰ࣮Մ -
Πϯϑϥߏྫʢ%&-*4),*5$)&/ͷ߹ʣ %BUBCSJDLT71$ ΞϓϦέʔγϣϯ71$ εέʔϥϒϧ 71/PS*OUFSOFU 71$ϐΞϦϯάଓ 4 &MBTUJ $BDIF 3%4
3FETIJGU %BUBCSJDLT $MPVE ʢཧը໘ʣ ىಈઃఆ /PUFCPPL -
ͦͦͲͷΑ͏ͳ͕͋ͬͨͷ͔ XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN
4&-&$5 WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4 */5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R ඵࢹௌ͞Εͨಈըͷ֤߹ܭճΛऔಘ͍ͨ͠߹ -
ͦͦͲͷΑ͏ͳ͕͋ͬͨͷ͔ ඵࢹௌ͞Εͨಈըͷ֤߹ܭճΛऔಘ͍ͨ͠߹ XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN
4&-&$5 WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4*/5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R ࣮ߦ݁Ռ SFDJQF@JESBUJOH -
ͦͦͲͷΑ͏ͳ͕͋ͬͨͷ͔ XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN
4&-&$5 WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4 */5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R Ϩγϐ*% J04CJUɺ4USJOHܕ ඵࢹௌ͞Εͨಈըͷ֤߹ܭճΛऔಘ͍ͨ͠߹ Ϩγϐ*% ͦΕҎ֎ͷɺ*/5ܕ -
XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN 4&-&$5
WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP ͦͦͲͷΑ͏ͳ͕͋ͬͨͷ͔ CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4 */5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R 42-্Ͱܕʹม 4USJOH*/5 ඵࢹௌ͞Εͨಈըͷ֤߹ܭճΛऔಘ͍ͨ͠߹ -
4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. WJFXFE@WJEFP 8)&3& FWFOU@EBUF
"/%TFDPOET (3061#: 4QBSLͰ&5-Λ͢Δ͜ͱͰؔ৺ࣄʹूதՄೳʹ w ܕมෆཁ w ໊લมෆཁ ඵࢹௌ͞Εͨಈըͷ֤߹ܭճΛऔಘ͍ͨ͠߹ ؔ৺ͷ -
$ % " # ' Ћ Ћ Ћ Ћ &
" # $ % & ' ܕมʢ&5-ʣΠϝʔδਤ Ϩίʔυ ςʔϒϧ" ຌྫ ςʔϒϧ# ςʔϒϧ$ #` # "730 1MB[NB%# 3FETIJGU -
ಉ͡ҙຯͰ໊લͱܕ͕ҟͳΔϨίʔυ͕ଘࡏ͢Δ 04ຖͰҟͳΔ։ൃऀʹΑΔ࣮ϛεͳͲ͕ݪҼ όʔδϣϯΞοϓͰमਖ਼ͯ͠چόʔδϣϯͰͦͷ··ͱͳΔʜ 3FDJQF@*% 3FDJQF@*E SFDJQF@JE SFDJQF*E 3FDJQF*E 3FDJQF*% 4USJOHܕ
*OUܕ ʷ εωʔΫέʔε Ωϟϝϧέʔε ʁʁʁέʔε SFDJQF@JE *OUܕ &5-ͱ %BUB'SBNF SFDJQF@JE -
$ % " # ' Ћ # "730 WBMSFDPSETQBSLSFBE GPSNBU
DPNEBUBCSJDLTTQBSLBWSP MPBE HTGPP BWSP DBDIF SFDPSEDSFBUF0S3FQMBDF5FNQ7JFX EBUB WBMEBUBTQBSLTRM 4&-&$5 UP@KTPO FWFOU@QBSBNT "4SFDPSE ུ '30. EBUB DBDIF 4QBSL42- %BUB'SBNF '30.Ͱݺͼग़͠Մೳʹ &YUSBDUϑΣʔζ -
\ FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF
JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ WBMEBUBTQBSLTRM 4&-&$5 UP@KTPO FWFOU@QBSBNT "4SFDPSE ུ '30. EBUB DBDIF 4QBSL42- 5SBOTGPSNϑΣʔζ +40/จࣈྻ -
\ FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF
JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ ܕใ͕+40/ͷΩʔͰఆٛ͞Ε͍ͯΔ *%ܥTUSJOHͱJOU͕ࠞࡏ͢Δ ɹCJU04ʹىҼ͢Δॾࣄ +40/શʹܕใΛอ࣋Ͱ͖ͳ͍ҝɺ 'JSFCBTFΩʔͰϝλใΛอ͍࣋ͯ͠Δɻ ͜Ε͕42-ΛෳࡶԽ͢ΔཁҼɻ 5SBOTGPSNϑΣʔζ -
\ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ 5SBOTGPSN \ FWFOU@OBNFGPP
FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ 5SBOTGPSNϑΣʔζ 'MBUUFO+40/ -
\ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ 5SBOTGPSNϑΣʔζ 5SBOTGPSN ԼهͷมΛ࣮ߦ͍ͯ͠Δ
LFZͷ 6TFS*%ʣΛεωʔΫέʔεԽͯ͠LFZԽ WBMVFTUSJOH@WBMVFͷΛ*OUԽͯ͠WBMVFԽ 5SBOTGPSN \ FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ 'MBUUFO+40/ -
\ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ 5SBOTGPSNϑΣʔζ 'MBUUFO+40/ FWFOU@OBNF4USJOH
VTFS@JE-POH OBNF4USJOH 4USVDU5ZQF %BUB'SBNF FWFOU@OBNF VTFS@JE OBNF GPP CBS -
5SBOTGPSNϑΣʔζ JNQPSUQBSTFS@ WBMVEG5ZQF1BSTFS 4USJOH4FR<5ZQF%FUFDUPS> KTPO4USJOH4USJOH \ 5ZQF1BSTFS KTPO4USJOH ղੳϩδοΫͷ࣮ଶ
^ TRM$POUFYUVEGSFHJTUFS UZQF@QBSTFS VEG5ZQF1BSTFS DBTFDMBTT5ZQF%FUFDUPS PSJHJO@LFZ4USJOH OPSNBMJ[F@LFZ4USJOH TOBLF@LFZ4USJOH DBNFM@LFZ4USJOH LFZ@UZQF4USJOH LFZ@UZQF@QSJPSJUZ4USJOH 4QBSL42- 6%' ܕใಉఆΫϥεͷ࡞ 4QBSL42-ͷ6%'Λར༻͢Δ ߏղੳ࣌ʹΩʔͷ໊લέʔεܕใΛղੳ͢Δ 1BSTFS෦Ͱ$JSDFΛར༻ͯ͠มΛߦ͍ͬͯΔ ίʔυൈਮ -
5SBOTGPSNϑΣʔζ WBMEFUFDUPS4FR<5ZQF%FUFDUPS>TQBSLTRM 8*5)B"4 4&-&$5 UP@KTPO QBSBNT "4DPM
FWFOU@OBNF '30. EBUB C"4 4&-&$5 FYQMPEF UZQF@QBSTFS DPM "4DPM '30. B 4&-&$5DPM'30.C BT<5ZQF%FUFDUPS> DPMMFDU 4QBSL42- 4QBSL42-ͰܕσʔλΛऔಘ͢Δ +40/4USJOHԽ ܕͷੜ 4DBMBͷܕʹม -
5SBOTGPSNϑΣʔζ JNQPSUQBSTFS@ WBMEFUFDUPS4FR<5ZQF%FUFDUPS>ུ WBMFWFOU1BSTFS 4USJOH4USJOH KTPO4USJOH4USJOH \ ύʔαʔͷ࣮ &WFOU1BSTFS
KTPO4USJOH 5ZQF%FUFDUPST EFUFDUPS ^ TRM$POUFYUVEGSFHJTUFS FWFOU@QBSTF FWFOU1BSTFS +40/4USJOHΛ'MBUUFOʢܕͱ໊ΛدͤΔʣ͢Δ ࣄલʹੜͨ͠ΠϯελϯεΛ6%'ʹ༩͑Δࣄ͕Մೳ 6%'ΛΈ߹ΘͤΔ͜ͱՄೳ WBMSFDPSETQBSLTRM TFMFDUFWFOU@QBSTF UP@KTPO QBSBNT BTSFDPSEGSPNEBUB ίʔυൈਮ -
+40/4USJOHΛ'MBUUFOʢܕͱ໊ΛدͤΔʣ͢Δ 5SBOTGPSNϑΣʔζ \ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ \
FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ 5SBOTGPSN ͜ͷ࣌Ͱ4USJOHͷͨΊɺ ͦͷ··ΫΤϦ͕࣮ߦͰ͖ͳ͍ ྻ໊SFDPSEɿ4USJOHܕ -
+40/4USJOHΛ%BUB'SBNFʹม͢Δ +40/4USJOH%BUB'SBNFͰΫΤϦՄೳʹ 5SBOTGPSNϑΣʔζ WBMEBUB4FU%BUB4FU<4USJOH>SFDPSE SEE NBQ @HFU4USJOH UP%4
+40/ͷ%BUB4FU͔Β%BUB'SBNFʹม WBMTUSVDUFE+TPO%BUB'SBNF TRM$POUFYUSFBEKTPO EBUB4FU TUSVDUFE+TPODSFBUF0S3FQMBDF5FNQ7JFX SFDPSET WBMSFHJTUFS%BUBTQBSLTRM TFMFDUOBNF VTFS@JE FWFOU@OBNFGSPNSFDPSET FWFOU@OBNF VTFS@JE OBNF GPP CBS %BUB'SBNF 3%%%BUB4FU%BUB'SBNF -
-PBEϑΣʔζ WBMSFHJTUFS%BUBTQBSLTRM TFMFDUOBNF VTFS@JE FWFOU@OBNFGSPNSFDPSET SFHJTUFS%BUB XSJUF GPSNBU
DPNEBUBCSJDLTTQBSLSFETIJGU PQUJPO VSM KECD6SM PQUJPO UFNQEJS UFNQ%JS PQUJPO ECUBCMF SFETIJGU5BCMF PQUJPO BXT@JBN@SPMF BSOBXTJBNGPPSPMFCBS TBWF 3FETIJGU -
XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN 4&-&$5
WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4*/5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R ࣮ߦ̨̡̦ͷൺֱ 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. WJFXFE@WJEFP 8)&3& FWFOU@EBUF "/%TFDPOET (3061#: ߦߦ #2Ͱ͘͢Δ͜ͱՄೳͰ͢ʢ೦ͷͨΊʣ -
·ͱΊ w σʔλͷՄࢹԽɺ σʔλऔΓग़࣌͠ͷ42-Λ؆ܿʹ͢Δࣄ͕ॏཁ ͦͷͨΊʹɺࣄલॲཧͰ&5-͢Δඞཁ͕͋Δ ߹ͷྑ͍ܕʹม͠ɺ؆ܿʹऔΓग़ͤΔूʹม͓ͯ͘͠ w 4QBSLʹΑΔ&5-Ͱ4DBMBʹΑΔॊೈͳίʔυ͕༗༻ 5SBOTGPSNϑΣʔζͰ$JSDFΛར༻͢Δͱ؆୯ͩͬͨ
w %BUBCSJDLTͷ߹4QBSLࣗମΛ༨Γҙࣝ͠ͳ͍ͰࡁΈɺ /PUFCPPLͷֶशίετ͕͘ಋೖ͍͢͠ͷͰ͓͢͢Ί