2017/05/18 BigData-JAWS 勉強会での発表資料です。 Explanation of Columnar format such as Parquet and ORC in japanese.
ΧϥϜφϑΥʔϚοτͷ͖΄Μ!DIJF
View Slide
࣍• ࣗݾհ• Ϗοάσʔλج൫ʹ͓͚ΔΧϥϜφϑΥʔϚοτ• "QBDIF1BSRVFUͷ͖΄Μ– ωετ͞Εͨσʔλʹదͨ͠σʔλϨΠΞτ– Ϧιʔεར༻ͷ࠷దԽ– ૬ޓӡ༻ੑ• 03$ͱ1BSRVFUͷҧ͍• ·ͱΊ
ࣗݾհ$IJF)BZBTIJEB!DIJF3FUUZ*OD4PGUXBSF&OHJOFFS%BUB4DJFOUJTU"84ྺ΄Ͳɻ4QBSL;FQQFMJOͳͲʹίϯτϦϏϡʔτɻػցֶश)BEPPQ4QBSL4DBMB1ZUIPO%#WJNমϐΞϊςχεεϊϘ機械学習&分析基盤構築やってます！
ຊ࠷େڃɺ໊࣮ܕάϧϝαʔϏε
3FUUZͷੳج൫ͷհ
ൃද༰ʹ͍ͭͯ• Ϗοάσʔλج൫ͰɺσʔλͷϑΥʔϚοτʹΧϥϜφϑΥʔϚοτΛ͏ͱ͍͏ͷ͕Ұͭͷఆੴͱͳ͍ͬͯ·͢ɻ• ຊൃදͰɺΧϥϜφϑΥʔϚοτͷಛΛɺ1BSRVFUΛϕʔεʹղઆ͠·͢ɻ• ͳͥΧϥϜφϑΥʔϚοτΛ͏͖ͳͷ͔ɺͲͷΑ͏ʹ͏͖ͳͷ͔Λཧղͯ͠Β͑ͨΒΑ͍ͳɺͱࢥ͍·͢ɻAmazonEMRAmazonAthenaAmazonRedshi1
Ϗοάσʔλج൫ʹ͓͚ΔΧϥϜφϑΥʔϚοτ
• ςΩετϑΥʔϚοτ• ߦࢦϑΥʔϚοτ• ྻࢦʢΧϥϜφʣϑΥʔϚοτϏοάσʔλج൫ͰΘΕΔϑΥʔϚοτCSV { JSON }σʔλϕʔεʹΑ͘ΘΕΔϑΥʔϚοτ
• ߦࢦϑΥʔϚοτߦࢦϑΥʔϚοτͱྻࢦϑΥʔϚοτ• ྻࢦϑΥʔϚοτcolA colB colC1 aaa 2017-04-012 bbb 2017-04-023 ccc 2017-04-034 ddd 2017-04-045 eee 2017-04-05colA colB colC1 aaa 2017-04-012 bbb 2017-04-023 ccc 2017-04-034 aaa 2017-04-045 ccc 2017-04-05行方向に連続してデータを保持・ 行単位でデータを操作することの多いOLTP処理に有利列方向に連続してデータを保持・ 特定の列のみ取得する分析用途に有利・ 列のカーディナリティが低い場合、圧縮率が高くなるσʔλ֨ೲ୯Ґ一般的に分析用途のビッグデータ基盤はこっちが向いている
• 03$ "QBDIF)JWFͷͨΊʹ࡞ΒΕͨϑΥʔϚοτ "QBDIFͷτοϓϓϩδΣΫτ )JWF1SFTUP4QBSLͳͲ͕αϙʔτ• 1BSRVFU 5XJUUFS͕ࣾ։ൃɺ$MPVEFSB։ൃʹେ͖ؔ͘ΘΔ "QBDIFͷτοϓϓϩδΣΫτ 4QBSL1SFTUP%SJMM*NQBSB)JWFͳͲ͕αϙʔτओཁͳ044ͷྻࢦϑΥʔϚοτࠓճ1BSRVFUΛϕʔεʹઆ໌͠·͢
"QBDIF1BSRVFUͷ͖΄Μ
1BSRVFU• ʹ(PPHMF͕ൃදͨ͠%SFNFMͷจͷSFDPSETISFEEJOHBOEBTTFNCMZBMHPSJUINͷ༰Λϕʔεʹ։ൃ͞ΕͨҾ༻ɿIUUQTTUBUJDHPPHMFVTFSDPOUFOUDPNNFEJBSFTFBSDIHPPHMFDPNKBQVCTBSDIJWFQEG
• ωετ͞Εͨσʔλʹదͨ͠σʔλϨΠΞτ ඇϦϨʔγϣφϧͳσʔλΛఆ͢ΔͨΊɺԿઍͷϑΟʔϧυΛ࣋ͭૄͳωετ͞ΕͨσʔλΛޮతʹѻ͏͜ͱͷͰ͖ΔετϨʔδܗࣜͱΞϧΰϦζϜΛఏڙ͢Δ• Ϧιʔεར༻ͷ࠷దԽ ಡΈࠐΈॻ͖ࠐΈϥΠϒϥϦʹ͓͍ͯɺ*0ݮΤϯίʔυσίʔυ࣌ͷ$16ར༻ͷޮԽͷ͕͞Ε͍ͯΔ• ૬ޓӡ༻ੑ ༷ʑͳΫΤϦΤϯδϯσʔλϚωδϝϯτπʔϧʹରԠͰ͖ΔσʔλϞσϧͷಛ
ωετ͞Εͨσʔλʹదͨ͠σʔλϨΠΞτ
ωετ͞Εͨσʔλͷ࣋ͪํSFDPSEPSJFOUFE DPMVNOPSJFOUFES S"#$% &SSҰ෦ΛಡΈग़ͨ͢Ίʹωετ͞ΕͨσʔλશͯΛಡΈग़͢ඞཁ͕͋ΔඞཁͳσʔλʹͷΈΞΫηε͢Δ͜ͱ͕Ͱ͖Δ
ωετ͞Εͨσʔλͷ࣋ͪํSFDPSEPSJFOUFE DPMVNOPSJFOUFES S"#$% &SSωετ͞Εͨσʔλͯ͢ΛಡΈग़͢ඞཁ͕͋ΔඞཁͳσʔλͷΈΛಡΈग़͢͜ͱ͕Ͱ͖ΔσʔλߏͷใΛอ࣋ͭͭ͠αϒηοτ͝ͱʹσʔλΛอଘ͍ͨ͠
%SFNFMͷσʔλϞσϧϑΟʔϧυ͝ͱʹɺ෮ϨϕϧͱఆٛϨϕϧΛ࣋ͭ͜ͱͰɺσʔλߏͷใଛࣦͳ͠ʹσʔλΛอଘͰ͖Δ˞෮ϨϕϧͱఆٛϨϕϧͷ༰ෳࡶͳͷͰ͜͜Ͱઆ໌Λলུ͠·͢ɻ෮Ϩϕϧ ఆٛϨϕϧҾ༻ɿIUUQTTUBUJDHPPHMFVTFSDPOUFOUDPNNFEJBSFTFBSDIHPPHMFDPNKBQVCTBSDIJWFQEG
Ϧιʔεར༻ͷ࠷దԽ
Ϧιʔεར༻ͷ࠷దԽํ๏̍ɽܕεϖγϑΟοΫͳΤϯίʔυํࣜ̎ɽ$16ύΠϓϥΠϯͷޮԽ̏ɽσʔλϑΝΠϧͷϑΥʔϚοτ
ܕεϖγϑΟοΫͳΤϯίʔυํࣜ• 1MBJO– γϯϓϧʹΛ࿈ଓͯ͠อଘ͢Δํ๏• %JDUJPOBSZ&ODPEJOH– ࣙॻใΛ࡞ͯͦ͠ͷใΛ༻͍ͯΤϯίʔυ͢Δํ๏– ΧʔσΟφϦςΟ͕͍߹ʹ༗ޮ• 3VO-FOHUI&ODPEJOH– ܁Γฦ͕͠ଓ͘Λޮతʹอଘ͢Δํ๏• %FMUB&ODPEJOHT– ιʔτ͞ΕͨσʔλͷࠩΛه͢Δํ๏– 5JNF4UBNQͳͲͷΠϯΫϦϝϯλϧͳσʔλͳͲ• 1SFpY&ODPEJOH– %JDUJPOBSZ&ODPEJOH͕͑ͳ͍ͱ͖ʹར༻͢Δ
%JDUJPOBSZ&ODPEJOH*% 'SVJUT "QQMF 0SBOHF -FNPO 8BUFSNFMPO .FMPO #BOBOB"QQMF 0SBOHF 0SBOHF -FNPO "QQMF8BUFS.FMPO.FMPO #BOBOB "QQMF .FMPO &ODPEFE%BUB%JDUJPOBSZ
3VO-FOHUI&ODPEJOHB B B B B B C C C CB C &ODPEFE%BUB
%FMUB&ODPEJOH100 101 101 102 101 101 102 101 99 100 105 107 114 116 119 120 121101 101 102 101 101 102 101 99 100 105 107 114 116 119 120 121CMPDL CMPDLSFGFSFODF1 0 1 -1 0 1 -1 -2 1 5 2 7 2 3 1 1ͭલͷ͔Βͷࠩ3 2 3 1 2 3 1 0 0 4 1 6 1 2 0 0 NJOEFMUBNJOEFMUB11 10 11 01 10 11 01 00 000 100 001 110 001 010 000 000 ϏοτϏοτCJUCJU CJUCJUCJUσʔλྔ͕ଟ͘ͳΔͱߋʹѹॖ্͕Δ
$16ύΠϓϥΠϯ• ܭࢉϦιʔεΛޮతʹར༻Ͱ͖͍ͯΔঢ়ଶ• ܭࢉϦιʔε͕ޮతʹར༻Ͱ͖͍ͯͳ͍ঢ়ଶ 5JNF5JNF1*1&-*/&1*1&-*/&#VCCMF
1BSRVFUʹ͓͚Δ$16ޮԽ• $16ύΠϓϥΠϯʹ͓͚Δ#VCCMFͷݪҼʹͳΓ͍͢ذ໋ྩʢJGจʣΛۃྗݮΒ͢Ҿ༻ɿIUUQTXXXTMJEFTIBSFOFUDMPVEFSBIBEPPQTVNNJU SFGIUUQTQBSRVFUBQBDIFPSHQSFTFOUBUJPOT#JOBSZQBDLJOHͷྫ
1BSRVFUͷϑΝΠϧϑΥʔϚοτฒྻॲཧɿ'JMF3PX(SPVQ*0ɿ$PMVNO$IVOL&ODPEJOH$PNQSFTTJPOɿ1BHFͷ୯ҐͰॲཧΛฒྻԽ͍͢͠σʔλஔҾ༻ɿIUUQTQBSRVFUBQBDIFPSHEPDVNFOUBUJPOMBUFTU
1BSRVFUͷσʔλϨΠΞτNFUBEBUBϖʔδ͝ͱɺྻ͝ͱɺߦ͝ͱɺσʔλશମͱ͍ͬͨ୯ҐͰɺεΩʔϚใϨίʔυ֤ྻͷ࠷খɾ࠷େͱ͍ͬͨ౷ܭใΛ࣋ͭϝλσʔλͷใʹΑΓɺෆඞཁͳσʔλಡΈඈ͕͠ՄೳҾ༻ɿIUUQTQBSRVFUBQBDIFPSHEPDVNFOUBUJPOMBUFTU
ΦϓςΟϚΠβʹΑΔ࠷దԽ4&-&$5DPM'30.NZUBCMFXIFSFDPMcol1 col2 col3a1 0 c1a2 10 c2a3 10 c3a4 30 c4a5 50 c5• QSPKFDUJPOQVTIEPXOඞཁͳྻͷΈΛऔΓग़͢• QSFEJDBUFQVTIEPXOඞཁͳߦͷΈΛऔΓग़͢col1 col2 col3a1 0 c1a2 10 c2a3 10 c3a4 30 c4a5 50 c5
4QBSLʹ͓͚ΔΦϓςΟϚΠβʹΑΔ࠷దԽͷྫҾ༻ɿIUUQTEBUBCSJDLTDPNCMPHTQBSLTRMHSBEVBUFTGSPNBMQIBJOTQBSLIUNM
૬ޓӡ༻ੑ
1BSRVFUϓϩδΣΫτ$ʹΑΔ1BSRVFUϑΥʔϚοτͷ3FBE8SJUFͷͨΊͷϥΠϒϥϦ1BSRVFUͷσʔλϨΠΞτͳͲͷఆٛͷϥΠϒϥϦ+BWBʹΑΔ1BSRVFUϑΥʔϚοτͷ3FBE8SJUFͷͨΊͷϥΠϒϥϦ
ଞϓϩμΫτͱͷ૬ޓӡ༻ੑҾ༻ɿIUUQTXXXTMJEFTIBSFOFUDMPVEFSBIBEPPQTVNNJU SFGIUUQTQBSRVFUBQBDIFPSHQSFTFOUBUJPOT
03$WT1BSRVFU
03$WT1BSRVFU• ΞʔΩςΫνϟͷҧ͍ 03$ωετ͞ΕͨσʔλΛͦΕͧΕผͷྻͱͯ͠ѻ͏ 03$ϒϧʔϜϑΟϧλͷػೳ͕͋Δ 1BSRVFUʹ4DIFNBNFSHJOHͷػೳ͕͋Δ• ϓϩμΫτ͝ͱʹɺ֤ϑΥʔϚοτͷରԠͷҧ͍ʢϑΥʔϚοτ͋͘·Ͱ࠷దԽ͍͢͠σʔλܗࣜͰ͋ͬͯɺ࣮ࡍʹॲཧΛϓϩμΫτଆ͕ରԠ͍ͯ͠ͳ͍ͱҙຯ͕ͳ͍ʣ• σʔλͷಛੑར༻͢ΔϓϩμΫτɺ࣮ߦ͢ΔΫΤϦͷύλʔϯʹΑͬͯɺ͍͚Δͷ͕େ• ߹ʹΑͬͯ03$1BSRVFUΑΓ4FRVFODF'JMFͳͲͷ΄͏͕εϧʔϓοτ͕ྑ͍߹͋Δ• Ұ൪͍͍ͷɺݕূͰεϧʔϓοτσʔλαΠζΛ֬ೝ͢Δ͜ͱ
·ͱΊ• ੳ༻్ͷϏοάσʔλج൫ʹɺҰൠతʹΧϥϜφϑΥʔϚοτ͕ద͍ͯ͠Δ• 1BSRVFUωετ͞ΕͨσʔλͷରԠɺϦιʔεར༻ͷ࠷దԽɺ૬ޓӡ༻ੑʹؔͯ͠͞Ε͍ͯΔ• ϑΥʔϚοτσʔλͷಛੑར༻͢ΔϓϩμΫτɺ࣮ߦ͢ΔΫΤϦͷ༰ʹΑ͍͚ͬͯΔ
8F"SF)JSJOHʂؾܰʹΦϑΟεʹ༡ͼʹདྷ͍ͯͩ͘͞ʂ͋ͱ3FUUZ͔͍ͭͬͯͩ͘͞ʂ
ࢀߟࢿྉ• %SFNFMจ– IUUQTTUBUJDHPPHMFVTFSDPOUFOUDPNNFEJBSFTFBSDIHPPHMFDPNKBQVCTBSDIJWFQEG• "QBDIF1BSRVFUυΩϡϝϯτ– IUUQTQBSRVFUBQBDIFPSH• (JU)VC– IUUQTHJUIVCDPN1BSRVFU– IUUQTHJUIVCDPNBQBDIFPSD• "QBDIF03$υΩϡϝϯτ– IUUQTPSDBQBDIFPSH• &GpDJFOU%BUB4UPSBHFGPS"OBMZUJDTXJUI"QBDIF1BSRVFU– IUUQTXXXTMJEFTIBSFOFUDMPVEFSBIBEPPQTVNNJU SFGIUUQTQBSRVFUBQBDIFPSHQSFTFOUBUJPOT• +*3"– IUUQTJTTVFTBQBDIFPSHKJSBJTTVFT
chie8842
weaverryan
cyberarchitect
picardparis
eiki
pauli
tomzoh
prathamesh
line_developers
k_koheyi
.png){kind=icon}
pharma_x_tech
ikezoemakoto
whywaita
jasonvnalue
lara
palkan
gr2m
colly
roundedbygravity
addyosmani
malarkey
jrom
marcelosomers
swwweet
chrislema