DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization

%BUBCSJDLTͱ4QBSLͰ͸͡ΊΔ <Ϗοάσʔλ&5-ॲཧσʔλՄࢹԽ> ࣮ફೖ໳ ౡాխ೥ 4DBMB෱Ԭ

ࣗݾ঺հ w ౡాխ೥!TNENUT w σʔλ΢ΣΞϋ΢εΞʔΩςΫτ 4DBMBc1ZUIPOc(PMBOHc+BWBc%%% w גࣜձࣾΤϒϦʔॴଐ w
%&-*4),*5$)&/σʔλ෼ੳܥશൠ୲౰ ओʹ෼ੳج൫ɺ഑৴໘࠷దԽͳͲΛ୲౰

ΞϓϦϦϦʔε μ΢ϯϩʔυɿສ Ϩγϐ਺ɿສ݅Ҏ্ ݄ؒಈը࠶ੜ਺ɿԯສճҎ্ "QQ4UPSF#&450' (PPHMF1MBZϕετΦϒ ೥ࠃ಺ΞϓϦμ΢ϯϩʔυୈ̏Ґ ʢ̍Ґ-*/&ɺ̎Ґ:BIPP+"1"/˞ඇήʔϜɹ"QQ"OOJFௐ΂ʣ ɾຖ೔৽ணϨγϐಈը͕഑৴͞ΕΔ ɹɾྉཧϨγϐಈըʢௐཧݙཱखॱʣ
ɹɾ޿ࠂओλΠΞοϓಈը <13>%&-*4),*5$)&/ͷ঺հʢ͔ΜͨΜʹʣ

ΞδΣϯμʢ֓ཁʣ %&-*4),*5$)&/Ͱ͸ɺ σʔλ෼ੳͷجװٕज़ͷҰͭͱͯ͠4QBSLΛར༻͍ͯ͠·͢ɹ ຊ೔͸αϯϓϧίʔυΛަ͑ͯɺ %BUBCSJDLTͷ֓ཁͱ&5-ʹ͓͚Δ4QBSLͷ࣮૷ςΫχοΫɺ σʔλՄࢹԽʹ͍ͭͯղઆͤͯ͞௖͖·͢

ΞδΣϯμ w %BUBCSJDLTͷղઆίϯηϓτಋೖํ๏ w 4QBSLʹΑΔ&5-ॲཧ w σʔλՄࢹԽ w ࢹௌҡ࣋཰ͷՄࢹԽ

%BUBCSJDLTͷղઆ ίϯηϓτಋೖํ๏

%BUBCSJDLT6OJpFE"OBMZUJDT1MBUGPSN

%BUBCSJDLT6OJpFE"OBMZUJDT1MBUGPSN ɾ4QBSLΛ࣮ߦج൫ͱͨ͠σʔλ෼ੳϓϥοτϑΥʔϜ 6OJpFE"OBMZUJDT1MBUGPSNΛ໨ࢦ͍ͯͯ͠࡞ΒΕ͍ͯΔ ɾ "84"[VSF ͷϢʔβΞΧ΢ϯτͰɺ71$࡞੒ͳͲॴఆͷΠϯετʔϧͰಋೖ ར༻ऀͷ "84"[VSF ΞΧ΢ϯτͰΫϥελ͕ىಈ͞ΕΔ ɾσʔλΤϯδχΞαΠΤϯςΟετ͸
ɹ/PUFCPPLͰεΫϦϓτΛॻ͍ͯ։ൃ͢Δ 4DBMB1ZUIPO34IFMM42-͕ར༻Մ &5-ॲཧ΋/PUFCPPLͰ࣮૷Մ

%BUBCSJDLT؅ཧը໘ᶃ ɾ։ൃऀʢσʔλΤϯδχΞσʔλαΠΤϯςΟετʣ͸ ɹ%BUBCSJDLT$MPVEͷ63-Λ։͖ɺ؅ཧը໘ʹ઀ଓͯ͠։ൃΛ࢝ΊΔ

%BUBCSJDLT؅ཧը໘ᶄ ɾ։ൃऀຖʹׂΓ౰ͯΒΕͨ8PSLTQBDF্ͷɺ/PUFCPPLͰ։ൃΛ͢͢ΊΔ ɹ4DBMB1ZUIPO342-͕બ୒ՄೳʢͲΕΛબΜͰ΋૬ޓݺͼग़͠Ͱ͖Δʣ

%BUBCSJDLT؅ཧը໘ᶅ ɾ4QBSLΫϥελͷΠϯελϯελΠϓઃఆը໘ ɹΦʔτεέʔϧىಈΫϥελ $16(16 4QBSLόʔδϣϯ౳͕ࢦఆՄೳ εϙοτೖࡳ ΠϯελϯελΠϓ ୆਺ׂ߹ Φʔτεέʔϧઃఆ Φʔτλʔϛωʔτઃఆ

Πϯϑϥߏ੒ྫʢ%&-*4),*5$)&/ͷ৔߹ʣ %BUBCSJDLT71$ ΞϓϦέʔγϣϯ71$ εέʔϥϒϧ 71/PS*OUFSOFU 71$ϐΞϦϯά઀ଓ 4 &MBTUJ $BDIF 3%4
3FETIJGU %BUBCSJDLT $MPVE ʢ؅ཧը໘ʣ ىಈઃఆ /PUFCPPL

ɾ/PUFCPPLΛ࡞੒ͯ͠ɺ4UFQCZ4UFQͰॲཧΛ࣮ߦ͢Δ ΠϯλϥΫςΟϒʢର࿩తʣʹίϚϯυ͕ஞ࣮࣍ߦ͞ΕΔ ɾίϚϯυؒͷσʔλ࿈ܞ͸ɺ4DBMBͷΠϯελϯε΍4QBSL42-Ͱߦ͏ ɾ42-%4-ͳͲͰ్தஅ໘ͷσʔλΛ֬ೝ͠ͳ͕Β։ൃΛਐΊΔ /PUFCPPLʹΑΔ։ൃ 5%4QBSL

/PUFCPPL࣮ߦ%&.0

4QBSLʹΑΔ&5-ॲཧʹ͍ͭͯ

&5-ͱ͸ &YUSBDU 5SBOTGPSN -PBE σʔλநग़ ܕม׵ ӬଓԽ ʨ+40/^

͜ͷ෦෼ͷ࿩ σʔλ΢ΣΞϋ΢εུ֓ਤ "1*αʔό ޿ࠂޮՌଌఆ Πϕϯτܭଌ ՄࢹԽ 3&45"1* %8)

w 'JSFCBTF"OBMZUJDT͸ແݶʹΠϕϯτϩάΛӬଓԽग़དྷ ͯศར͕ͩΫΤϦ͕ෳࡶԽ͢ΔͳͲͷ໰୊͕͋ͬͨ w 'JSFCBTFͷ#JH2VFSZ্ʹੜ੒͞ΕΔεΩʔϚ͸ΫΤϦ ͕ෳࡶԽ͠ΤϯδχΞҎ֎͸ΫΤϦΛॻ͘͜ͱ͕ࠔ೉ w ೥݄຤ʹεΩʔϚ͕ߋ৽͞Ε͕ͨͦΕͰ΋೉͍͠ w 6//&45จ͕େྔʹग़ݱ͢Δ෼ੳΫΤϦ͕ൃੜ͢Δ
ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN
4&-&$5 WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4 */5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R ඵࢹௌ͞Εͨಈըͷ֤߹ܭճ਺Λऔಘ͍ͨ͠৔߹

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ εΩʔϚఆ্ٛ͸ɺߦͰදݱ͞ΕΔ ໰୊͸3&$03%ܕͷ෼ղͰ 6//&45ߏจ͕ൃੜ͢Δ

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ 4&-&$5 '30.AGPPFWFOUT@A 3FDPSEܕ͕ωετ͞Ε͓ͯΓ ॎʹ௕͍Ϩίʔυ͕ग़ྗ͞ΕΔ Ωʔͷ਺͚ͩॎʹ৳ͼΔ ,74ͷߏ଄ͱྨࣅ

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ ඵࢹௌ͞Εͨಈըͷ֤߹ܭճ਺Λऔಘ͍ͨ͠৔߹ XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN
4&-&$5 WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4*/5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R ࣮ߦ݁Ռ SFDJQF@JESBUJOH

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ Կނ͜ͷΑ͏ʹ ௕͍ΫΤϦΛॻ͘ඞཁ͕͋Δͷ͔

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN
4&-&$5 WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4 */5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R Ϩγϐ*% J04CJU୺຤͸ɺ4USJOHܕ ඵࢹௌ͞Εͨಈըͷ֤߹ܭճ਺Λऔಘ͍ͨ͠৔߹ Ϩγϐ*% ͦΕҎ֎ͷ୺຤͸ɺ*/5ܕ

XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN 4&-&$5
WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4 */5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R 42-্Ͱ਺஋ܕʹม׵ 4USJOH*/5 ඵࢹௌ͞Εͨಈըͷ֤߹ܭճ਺Λऔಘ͍ͨ͠৔߹

ͦ΋ͦ΋ͲͷΑ͏ͳ໰୊͕͋ͬͨͷ͔ ͜ͷΑ͏ͳΫΤϦΛॻ͖ଓ͚Δࣄ͸ɺ ੜ࢈ੑʹ໰୊͕ग़Δࣄ͕༧૝͞Εͨ

4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. WJFXFE@WJEFP 8)&3& FWFOU@EBUF
"/%TFDPOET (3061#: 4QBSLͰ&5-Λ͢Δ͜ͱͰؔ৺ࣄʹूதՄೳʹ w ܕม׵ෆཁ w ໊લม׵ෆཁ ඵࢹௌ͞Εͨಈըͷ֤߹ܭճ਺Λऔಘ͍ͨ͠৔߹ ؔ৺ͷ෼཭

4QBSLʹΑΔ&5-ॲཧʹ͍ͭͯ

&5-ͱ͸ &YUSBDU 5SBOTGPSN -PBE σʔλநग़ ܕม׵ ӬଓԽ ʨ+40/^

$ % " # ' Ћ Ћ Ћ Ћ &
" # $ % & ' ܕม׵ʢ&5-ʣΠϝʔδਤ Ϩίʔυ ஋ ςʔϒϧ" ຌྫ ςʔϒϧ# ςʔϒϧ$ #` # "730 1MB[NB%# 3FETIJGU

ಉ͡ҙຯͰ΋໊લͱܕ͕ҟͳΔϨίʔυ͕ଘࡏ͢Δ 04ຖͰҟͳΔ։ൃऀʹΑΔ࣮૷ϛεͳͲ͕ݪҼ όʔδϣϯΞοϓͰमਖ਼ͯ͠΋چόʔδϣϯͰ͸ͦͷ··ͱͳΔʜ 3FDJQF@*% 3FDJQF@*E SFDJQF@JE SFDJQF*E 3FDJQF*E 3FDJQF*% 4USJOHܕ
*OUܕ ʷ εωʔΫέʔε Ωϟϝϧέʔε ʁʁʁέʔε SFDJQF@JE *OUܕ &5-ͱ͸ %BUB'SBNF SFDJQF@JE

1ZUIPOͰ"730ܗࣜͰ($4ʹμϯϓ͢Δ TQBSLCJHRVFSZ͸্ख͘ಈ͔ͳ͔ͬͨ EBUBCSJDLT͸1ZUIPOΛ௚઀ίʔϧͰ͖ͯศར &YUSBDUϑΣʔζ

$ % " # ' Ћ # "730 WBMSFDPSETQBSLSFBE GPSNBU
DPNEBUBCSJDLTTQBSLBWSP MPBE HTGPP BWSP DBDIF SFDPSEDSFBUF0S3FQMBDF5FNQ7JFX EBUB WBMEBUBTQBSLTRM 4&-&$5 UP@KTPO FWFOU@QBSBNT "4SFDPSE ུ '30. EBUB DBDIF 4QBSL42- %BUB'SBNF '30.Ͱݺͼग़͠Մೳʹ &YUSBDUϑΣʔζ

5SBOTGPSNϑΣʔζ %BUB'SBNFͷߏ଄Λ+40/4USJOHʹม׵ͯ͠ɺ ɹߏ଄ղੳ͠ܕ৘ใಉఆΫϥεΛ࡞੒͢Δ Ͱ࡞੒ͨ͠ܕ৘ใಉఆΫϥεΛར༻ͯ͠ɺ ɹ'MBUUFOʢܕͱ໊ΛدͤΔʣࡁΈͷ+40/4USJOHΛ࡞Δ ͷ+40/4USJOHΛ4QBSL͕ඪ४Ͱ༻ҙ͍ͯ͠Δɺ +40/ϦʔμʔʹಡΈࠐ·ͤͯ%BUB'SBNFʹม׵͢Δ

\ FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF
JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ WBMEBUBTQBSLTRM 4&-&$5 UP@KTPO FWFOU@QBSBNT "4SFDPSE ུ '30. EBUB DBDIF 4QBSL42- 5SBOTGPSNϑΣʔζ +40/จࣈྻ

\ FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF
JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ ܕ৘ใ͕+40/ͷΩʔͰఆٛ͞Ε͍ͯΔ *%ܥ͸TUSJOHͱJOU͕ࠞࡏ͢Δ ɹCJU04ʹىҼ͢Δॾࣄ৘ +40/͸׬શʹܕ৘ใΛอ࣋Ͱ͖ͳ͍ҝɺ 'JSFCBTF͸ΩʔͰϝλ৘ใΛอ͍࣋ͯ͠Δɻ ͜Ε͕42-ΛෳࡶԽ͢ΔཁҼɻ 5SBOTGPSNϑΣʔζ

\ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ 5SBOTGPSN \ FWFOU@OBNFGPP
FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ 5SBOTGPSNϑΣʔζ 'MBUUFO+40/

\ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ 5SBOTGPSNϑΣʔζ 5SBOTGPSN Լهͷม׵Λ࣮ߦ͍ͯ͠Δ
LFZͷ஋ 6TFS*%ʣΛεωʔΫέʔεԽͯ͠LFZԽ WBMVFTUSJOH@WBMVFͷ஋Λ*OUԽͯ͠WBMVFԽ 5SBOTGPSN \ FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ 'MBUUFO+40/

\ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ 5SBOTGPSNϑΣʔζ 'MBUUFO+40/ FWFOU@OBNF4USJOH
VTFS@JE-POH OBNF4USJOH 4USVDU5ZQF %BUB'SBNF FWFOU@OBNF VTFS@JE OBNF GPP CBS

5SBOTGPSNϑΣʔζ JNQPSUQBSTFS@ WBMVEG5ZQF1BSTFS 4USJOH4FR<5ZQF%FUFDUPS> KTPO4USJOH4USJOH \ 5ZQF1BSTFS KTPO4USJOH ղੳϩδοΫͷ࣮ଶ
^ TRM$POUFYUVEGSFHJTUFS UZQF@QBSTFS VEG5ZQF1BSTFS DBTFDMBTT5ZQF%FUFDUPS PSJHJO@LFZ4USJOH OPSNBMJ[F@LFZ4USJOH TOBLF@LFZ4USJOH DBNFM@LFZ4USJOH LFZ@UZQF4USJOH LFZ@UZQF@QSJPSJUZ4USJOH 4QBSL42- 6%' ܕ৘ใಉఆΫϥεͷ࡞੒ 4QBSL42-ͷ6%'Λར༻͢Δ ߏ଄ղੳ࣌ʹΩʔͷ໊લέʔεܕ৘ใΛղੳ͢Δ 1BSTFS಺෦Ͱ͸$JSDFΛར༻ͯ͠ม׵Λߦ͍ͬͯΔ ίʔυ͸ൈਮ

5SBOTGPSNϑΣʔζ WBMEFUFDUPS4FR<5ZQF%FUFDUPS>TQBSLTRM 8*5)B"4 4&-&$5 UP@KTPO QBSBNT "4DPM
FWFOU@OBNF '30. EBUB C"4 4&-&$5 FYQMPEF UZQF@QBSTFS DPM "4DPM '30. B 4&-&$5DPM'30.C BT<5ZQF%FUFDUPS> DPMMFDU 4QBSL42- 4QBSL42-ͰܕσʔλΛऔಘ͢Δ +40/4USJOHԽ ܕͷੜ੒ 4DBMBͷܕʹม׵

5SBOTGPSNϑΣʔζ JNQPSUQBSTFS@ WBMEFUFDUPS4FR<5ZQF%FUFDUPS>ུ WBMFWFOU1BSTFS 4USJOH4USJOH KTPO4USJOH4USJOH \ ύʔαʔͷ࣮૷ &WFOU1BSTFS
KTPO4USJOH 5ZQF%FUFDUPST EFUFDUPS ^ TRM$POUFYUVEGSFHJTUFS FWFOU@QBSTF FWFOU1BSTFS +40/4USJOHΛ'MBUUFOʢܕͱ໊ΛدͤΔʣ͢Δ ࣄલʹੜ੒ͨ͠ΠϯελϯεΛ6%'ʹ༩͑Δࣄ͕Մೳ 6%'Λ૊Έ߹ΘͤΔ͜ͱ΋Մೳ WBMSFDPSETQBSLTRM TFMFDUFWFOU@QBSTF UP@KTPO QBSBNT BTSFDPSEGSPNEBUB ίʔυ͸ൈਮ

+40/4USJOHΛ'MBUUFOʢܕͱ໊ΛدͤΔʣ͢Δ 5SBOTGPSNϑΣʔζ \ FWFOU@OBNFGPP VTFS@JE OBNFCBS ^ \
FWFOU@OBNFGPP FWFOU@QBSBNT\ FMFNFOU< \ LFZ6TFS*% WBMVF\ TUSJOH@WBMVF JOU@WBMVFOVMM qPBU@WBMVFOVMM EPVCMF@WBMVFOVMM ^ ^ লུ > ^ ^ 5SBOTGPSN ͜ͷ࣌఺Ͱ͸4USJOHͷͨΊɺ ͦͷ··ΫΤϦ͕࣮ߦͰ͖ͳ͍ ྻ໊SFDPSEɿ4USJOHܕ

+40/4USJOHΛ%BUB'SBNFʹม׵͢Δ +40/4USJOH%BUB'SBNFͰΫΤϦՄೳʹ 5SBOTGPSNϑΣʔζ WBMEBUB4FU%BUB4FU<4USJOH>SFDPSE SEE NBQ @HFU4USJOH UP%4
+40/ͷ%BUB4FU͔Β%BUB'SBNFʹม׵ WBMTUSVDUFE+TPO%BUB'SBNF TRM$POUFYUSFBEKTPO EBUB4FU TUSVDUFE+TPODSFBUF0S3FQMBDF5FNQ7JFX SFDPSET WBMSFHJTUFS%BUBTQBSLTRM TFMFDUOBNF VTFS@JE FWFOU@OBNFGSPNSFDPSET FWFOU@OBNF VTFS@JE OBNF GPP CBS %BUB'SBNF 3%%%BUB4FU%BUB'SBNF

-PBEϑΣʔζ WBMSFHJTUFS%BUBTQBSLTRM TFMFDUOBNF VTFS@JE FWFOU@OBNFGSPNSFDPSET SFHJTUFS%BUB XSJUF GPSNBU
DPNEBUBCSJDLTTQBSLSFETIJGU PQUJPO VSM KECD6SM PQUJPO UFNQEJS UFNQ%JS PQUJPO ECUBCMF SFETIJGU5BCMF PQUJPO BXT@JBN@SPMF BSOBXTJBNGPPSPMFCBS TBWF 3FETIJGU

-PBEϑΣʔζ JNQPSUDPNUSFBTVSFEBUBTQBSL@WBMUETQBSLUE SFHJTUFS%BUBJOTFSU*OUP5% GPPCBS "SN5SFBTVSF%BUB IUUQTUXJUUFSDPNUBSPMFPTUBUVTFT 5SFBTVSF%BUB͸4DIFNB0O3FBEͷҝɺ 8SJUF࣌ʹεΩʔϚΛҙࣝ͢Δඞཁ͕ແ͘ѻ͍΍͍͢ɻ

XJUIMPH"4 4&-&$5 VTFS@JE QMBUGPSN 4&-&$5
WBMVFTUSJOH@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@TUS 4&-&$5 WBMVFJOU@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ3FDJQF*% "4SFDJQF@JE@JOU 4&-&$5 WBMVFEPVCMF@WBMVF '30. 6//&45 FWFOU@QBSBNT 8)&3& LFZ4FDPOET "4TFDPOET 6/*9@4&$0/%4 5*.&45".1@.*$304 FWFOU@UJNFTUBNQ "4UJNF '30. AGPPBOBMZUJDT@FWFOUT@A"4EBUB 8)&3& FWFOU@OBNF7JFXFE7JEFP CBTF"4 4&-&$5 $"4& 8)&/QMBUGPSN*045)&/$"45 SFDJQF@JE@TUS"4*/5 &-4&SFDJQF@JE@JOU &/%"4SFDJQF@JE TFDPOET '30. MPH R"4 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. CBTF 8)&3& TFDPOET (3061#: 4&-&$5 DBTU SFDJQF@JEBTTUSJOH BTSFDJQF@JE SBUJOH '30. R ࣮ߦ̨̡̦ͷൺֱ 4&-&$5 SFDJQF@JE DPVOU BTSBUJOH '30. WJFXFE@WJEFP 8)&3& FWFOU@EBUF "/%TFDPOET (3061#: ߦߦ #2Ͱ΋୹͘͢Δ͜ͱ͸ՄೳͰ͢ʢ೦ͷͨΊʣ

σʔλՄࢹԽ

ࢹௌҡ࣋཰ ։࢝ ಈը͸ඵ਺͕͢͢ΉຖʹɺϢʔβ͕ٻΊΔ಺༰Ͱ͸ແ͍৔߹ʹɺ Ϣʔβ͸ࢹௌΛ΍Ίͯ཭୤͢Δ܏޲͕͋Δɻ ͜ΕΛՄࢹԽͨ͠άϥϑΛࢹௌҡ࣋཰ͱ͍͏ɻʢҎԼͷಈը͸೚ҙʣ ӈʹߦ͘΄ͲɺࢹௌϢʔβ਺͕ݮ͍ͬͯ͘

ࢹௌҡ࣋཰ ։࢝ ӈʹߦ͘΄Ͳɺ࢒ଘࢹௌϢʔβ͕ݮ͍ͬͯ͘

&YUSBDU લষͰ&5-ͰӬଓԽͨ͠ 5%͔ΒUETQBSLΛར༻ͯ͠ 1SFTUPʢ42-ʣͰ&YUSBDU σʔλՄࢹԽ

σʔλՄࢹԽ ਺஋͔Βάϥϑʹͯ͠ϓϩοτ %BUBCSJDLTͷػೳ 7JFXԽ ܭࢉ్தஅ໘ͷՄࢹԽ ओʹσόοάͰ໾ʹཱͭ

7JFXԽ

ࢹௌҡ࣋཰ ͜ͷಈըͰ͸લޙͰɺ ࢹௌ਺ҡ͍࣋ͯ͠ΔϢʔβ͕ଟ͍͜ͱ͕෼͔Δ ʢલޙͰ܏͖͕ͳͩΒ͔ʹͳΔʣ

มԽ཰ͷࠩ෼ ಈըईͷ໿લޙ͔ΒϢʔβ͕཭୤͍ͯ͠ͳ͍ ͭ·ΓईͷΛ௒͑ͨϢʔβʹ͸ɺ ͜ͷಈը΁ͷڵຯؔ৺͕ڧ͔ͬͨͱ൑அͰ͖Δ

·ͱΊ

·ͱΊ w σʔλͷՄࢹԽ͸ɺ σʔλऔΓग़࣌͠ͷ42-Λ؆ܿʹ͢Δࣄ͕ॏཁ ͦͷͨΊʹ͸ɺࣄલॲཧͰ&5-͢Δඞཁ͕͋Δ ౎߹ͷྑ͍ܕʹม׵͠ɺ؆ܿʹऔΓग़ͤΔू໿ʹม׵͓ͯ͘͠ w 4QBSLʹΑΔ&5-Ͱ͸4DBMBʹΑΔॊೈͳίʔυ͕༗༻ 5SBOTGPSNϑΣʔζͰ$JSDFΛར༻͢Δͱ؆୯ͩͬͨ
w %BUBCSJDLTͷ৔߹͸4QBSLࣗମΛ༨Γҙࣝ͠ͳ͍ͰࡁΈɺ /PUFCPPLͷֶशίετ͕௿͘ಋೖ͠΍͍͢ͷͰ͓͢͢Ί

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization

DatabricksとSparkではじめる [ビッグデータETL処理/データ可視化] 実践入門 / Databricks and Spark with ETL and Visualization

More Decks by Masatoshi Shimada

Other Decks in Technology

Featured

Transcript