FIT 2016 Tutorial 資料 スライド: http://goo.gl/MeMZyO Part2 資料: http://goo.gl/y6ZmKr Part3 資料: http://goo.gl/ATC9yv
1ZUIPOͱσʔλੳίϯςετͷ࣮ફ,PIFJ0[BLJ !TNMZ'*5νϡʔτϦΞϧεϥΠυ
View Slide
͜ͷνϡʔτϦΞϧʹ͍ͭͯσʔλੳίϯςετΛࡐͱͯ͠ɺ1ZUIPOʹΑΔύοέʔδར༻ྫΛհ͢Δɻ▶ 1BSU༧ଌϞσϧ࡞ϓϩηεͱؔ࿈ύοέʔδ NJO– ༧ଌϞσϧͷ࡞ϓϩηεͱؔ࿈ύοέʔδͷུ֓Λհ͢Δ– 1BOEBT 4DJLJU-FBSO▶ 1BSUσʔλੳίϯςετʹ͓͚Δ࣮ફ NJO– σʔλੳίϯςετΛࡐͱͯ͠۩ମతͳར༻ྫΛհ͢Δ– 1BOEBT TFBCPSO 9(#PPTU 1Z5BCMFT▶ 1BSUൃలతͳ NJO– తʢେنσʔλɾը૾σʔλʣ͝ͱʹಛԽͨ͠ར༻ྫΛհ͢Δ– 1BOEBT 3FETIJGU #JH2VFSZ 0QFO$7
ิɿࢿྉͱ%PDLFSʹΑΔ࠶ݱੑͷ֬อຊεϥΠυͰΘΕΔίʔυ+VQZUFSOPUFCPPLͱͯ͠ҎԼͰެ։͍ͯ͠Δɽ▶ εϥΠυIUUQHPPHM.F.;Z0▶ 1BSUࢿྉIUUQHPPHMZ;N,S▶ 1BSUࢿྉIUUQHPPHM"5$ZWɽຊࢿྉͰར༻ͨ͠+VQZUFSOPUFCPPLެ։͍ͯ͠Δ%PDLFSΠϝʔδ͔Β࡞ͨ͠ڥΛ͍ͬͯΔɽҎԼͷίϚϯυͰಉ༷ͷڥΛࢼ͢͜ͱ͕Ͱ͖Δɻ$ docker run --rm -ti \-p 8888:8888 \-v /path/to/data_directory:/mnt \-v /path/to/working_directory:/home/kohei/work \smly/notebook:0.4
1"35༧ଌϞσϧ࡞ϓϩηεͱؔ࿈ύοέʔδ
Ұൠతͳ༧ଌϞσϧ࡞ͷϓϩηε▶ ,BHHMF,%%$VQͳͲͷίϯςετͰɼz༩͑ΒΕͨσʔλͱλεΫzʹରͯ͠ɼzܾΊΒΕͨධՁࢦඪzʹΑΔείΞͰॱҐΛ͚Λߦ͏▶ ͦΕͧΕͷϓϩηεʹ͓͍ͯ༗༻ͳϞδϡʔϧʹ͍ͭͯհ͢ΔλεΫઃܭσʔλ࡞γεςϜΈࠐΈલॲཧಛநग़༧ଌϞσϧͷධՁ༧ଌϞσϧͷ࡞・ pandas・ psycopg2・ sklearn.feature_extraction・ sklearn.preprocessing・ pytables・ seaborn・ sklearn.linear_model・ sklearn.svm・ sklearn.ensemble・ xgboost・ stats-models・ sklearn.metrics・ sklearn.cross_validation・ ml_metrics
QBOEBTσʔλϑϨʔϜͰσʔλૢ࡞ ೖग़ྗ▶ /VN1ZͷOEBSSBZʹࣅͨσʔλΛૢ࡞͢ΔͨΊͷσʔλߏ▶ ΧϥϜͱΠϯσοΫε͕͋Γɺ໊લ͖ΧϥϜͰσʔλૢ࡞͕Մೳ▶ ΧϥϜ͝ͱʹҟͳΔܕΛ࣋ͭ͜ͱ͕Ͱ͖Δ▶ ߦͷૠೖྻͷআͳͲ͕ߦ͑ΔՄม NVUBCMFͳΦϒδΣΫτ▶ ๛ͳσʔλૢ࡞ػೳͱೖग़ྗΠϯλʔϑΣʔεલॲཧಛநग़NDArray Internaldimensionsdim countdtypestridesdata*23 35*12float320 1 2 3 4 5 6 7 8DataFrame InternalINDEXColumnsfloat string integer
ಛྔͷΤϯίʔυઢܗϞσϧͳͲΧςΰϦΧϧมΛѻ͏͜ͱ͕Ͱ͖ͳ͍ϞσϧͰɼΧςΰϦΧϧมΛมͱͯ͠දݱ͢Δඞཁ͕͋Δɽ༷ʑͳಛྔͷΤϯίʔυํ๏ͱ࣮͕TDJLJUMFBSOʹΑͬͯఏڙ͞Ε͍ͯΔɽ▶ TLMFBSOGFBUVSF@FYUSBDUJPO0OF)PU&ODPEFS▶ TLMFBSOGFBUVSF@FYUSBDUJPO-BCFM&ODPEFS▶ TLMFBSOGFBUVSF@FYUSBDUJPO%JDU7FDUPSJ[F▶ TLMFBSOGFBUVSF@FYUSBDUJPO'FBUVSF)BTIFS▶ TLMFBSOGFBUVSF@FYUSBDUJPOUFYU5pEG7FDUPSJ[FS▶ ʜදతͳΫϥεΛհ͢Δલॲཧಛநग़
0OF)PU&ODPEFSΧςΰϦΧϧมΛPG,දهʹΤϯίʔυ͢Δɽ͋ΔΧςΰϦΧϧมͷDBSEJOBMJUZΛ,ͱͨ͠ͱ͖ɼಛΛ,ྻͷͱͯ͠ѻ͏ɽͦΕͧΕͷྻʹΧςΰϦΧϧมͷಛఆͷ͕ରԠ͓ͯ͠ΓɼରԠ͢ΔྻͷΛɼͦΕҎ֎ΛͰදݱ͢ΔɽʢQEHFU@EVNNJFT ؔʹΑͬͯQBOEBTͰಉͷૢ࡞͕ՄೳͰ͋Δʣલॲཧಛநग़WeekDay0 Friday1 Monday2 Monday3 Sunday4 Tuesday5 Saturday6 Monday1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0ndarray orcsr_matrixK=5
-BCFM&ODPEFSΧςΰϦΧϧมΛʹΤϯίʔυ͢ΔɽΧςΰϦΧϧมͷDBSEJOBMJUZ͕,Ͱ͋Δ߹ɼมͷΛ< ,>ͷʹஔ͖͑ΔɽʢQEGBDUPSJ[F ؔʹΑͬͯQBOEBTͰಉͷૢ࡞͕ՄೳͰ͋Δʣલॲཧಛநग़WeekDay0 Friday1 Monday2 Monday3 Sunday4 Tuesday5 Saturday6 Monday0 1 1 2 3 4 1 ndarray
%JDU7FDUPSJ[FࣙॻΦϒδΣΫτͷϦετΛ4DJ1ZͷૄߦྻΦϒδΣΫτOEBSSBZΦϒδΣΫτʹม͢ΔɽࣙॻΦϒδΣΫτͷLFZ͕ߦྻͷಛఆͷྻʹҰରҰରԠ͢ΔɽWBMVF͕ಛʹରԠ͢Δʢ%BUB'SBNFΦϒδΣΫτͷίϯετϥΫλʹͯ͠pMMOB BT@NBUSJY ͰθϩϑΟϧˍม͢Δ͜ͱͰQBOEBTͰಉͷૢ࡞͕ՄೳͰ͋Δʣલॲཧಛநग़list of dict[{‘like’: 1, ‘rt’: 9},{‘like’: 2, ‘rt’: 2},{‘like’: 4},]1 9 2 2 4 0 ndarray
TDJLJUMFBSO9(#PPTUͰ༧ଌϞσϧΛ࡞TDJLJUMFBSOͰpU ͰσʔλͯΊɼQSFEJDU Ͱ༧ଌɼQSFEJDU@QSPCB Ͱ༧ଌ֬Λ͑Δڞ௨ͷΠϯλʔϑΣʔεΛ͍࣋ͬͯΔʢ˞ྨͷ߹ʣ>>> import xgboost as xgb>>> dtrain, dtest = xgb.DMatrix(), xgb.DMatrix()>>> watchlist = [(dtrain, 'train')]>>> booster = xgb.Booster()>>> gbtree = booster.train(dtrain, params)>>> y_pred = gbtree.predict(dtest)>>> from sklearn.linear_model import LogisticRegression>>> clf = LogisticRegression()>>> clf.fit(X_train, y_rain)>>> y_pred = clf.predict_proba(X_test)9(#PPTUTDJLJUMFBSOͱҟͳΔΠϯλʔϑΣʔεͰ͋Δ͕ɼΠςϨʔγϣϯ͝ͱͷFWBMVBUJPOɼFBSMZTUPQQJOHɼ࠷దԽؔͷΧελϚΠζͳͲͷ༷ʑͳػೳ͕͋Δɽ9(#PPTUϝϞϦޮͱ܇࿅ʹ࠷దԽ͞Εͨ෦σʔλߏ%.BUSJYͱͯ͠σʔλΛѻ͏ɽOEBSSBZ͔Β%.BUSJYΦϒδΣΫτΛ࡞Ͱ͖Δɽ༧ଌϞσϧͷ࡞
ϞσϧͷੑೳධՁTLMFBSODSPTT@WBMJEBUJPOϞδϡʔϧ༧ଌϞσϧͷੑೳΛධՁ͢ΔͨΊͷϔϧύʔؔΠςϨʔλʔΛఏڙ͢Δɻ༧ଌϞσϧͷධՁ>>> clf = LogisticRegression()>>> scores = cross_validation.cross_val_score(clf, X, y, cv=5)>>> scoresarray([ 0.92..., 1. ..., 0.92..., 1. ])༧ଌϞσϧ͕TLMFBSO$MBTTJpFS.JYJOΛܧঝͨ͠ΫϥεͰ͋ΕϔϧύʔؔΛ༻͍ͯަࠩ֬ೝΛ؆ܿʹهड़Ͱ͖ΔɻΠςϨʔλʔ֬ೝ༻ͷ܇࿅σʔλͱςετσʔλͷΠϯσοΫεϦετΛฦ͢>>> kf = KFold(n_samples, n_folds=5)>>> for idx_train, idx_test in kf:... y_train, y_test = y[idx_train], y[idx_test]... X_train, X_test = X[idx_train], X[idx_test]... clf.fit(X_train, y_train)... y_pred = clf.predict(X_test)֤'PMEͷ"DDVSBDZ
1"35σʔλੳίϯςετʹ͓͚Δ࣮ફ
σʔλͷ֬ೝ▶ l8BMNBSU3FDSVJUJOH5SJQ5ZQF$MBTTJpDBUJPOzίϯςετΛࡐͱͯ͠ѻ͏▶ ʮങ͍٬ʯΛʮങ͍ͷεΩϟϯཤྺʯ͔ΒଟΫϥεྨ͢ΔλεΫ▶ USBJODTW UFTUDTW TBNQMF4VCNJTTJPODTWͷ̏ͭͷϑΝΠϧ͕ఏڙ͞Ε͍ͯΔ܇࿅ࣄྫσʔλUSBJODTWʮങ͍ͷεΩϟϯཤྺʯͰ͋ΔɻϑΟʔϧυ໊ આ໌ 5SBJO 5FTU5SJQ5ZQF ඪม ʮങ͍٬ʯͷΧςΰϦΧϧͳ*% ✔ ✗7JTJU/VNCFS *OTUBODF*% ͋ΔҰਓͷސ٬ͷങ͍ʹରԠ͢Δ*% ✔ ✔8FFLEBZ ങ͍Λͨ͠ि ✔ ✔6QD ߪೖͷ61$൪߸ ✔ ✔4DBO$PVOU ߪೖ͞Εͨݸʢෛͷฦ٫͞Εͨʣ ✔ ✔%FQBSUNFOU%FTDSJQUJPO ߪೖͷδϟϯϧ ✔ ✔'JOFMJOF ߪೖΛߋʹࡉ͔͚͘Δδϟϯϧͷ*% ✔ ✔
σʔλͷ֬ೝ▶ ܇࿅σʔλͰങ͍٬ʢ7JTJU/VNCFSʣͷങ͍͔͝ཤྺͱɺ๚٬ʹରԠ͢ΔΧςΰϦʢ5SJQ5ZQFʣ͕༩͑ΒΕ͍ͯΔɻ▶ ςετσʔλͰΧςΰϦ͕༩͑ΒΕ͍ͯͳ͍ɻςετσʔλͷΧςΰϦΛ܇࿅σʔλʹج͍ͮͯ༧ଌ͢Δ͜ͱ͕༩͑ΒΕͨλεΫͰ͋Δɻ+VQZUFSOPUFCPPL্ͰΘ͔Γ͘͢ςʔϒϧͰදࣔ͞ΕΔ
σʔλͷ֬ೝ▶ ༧ଌ݁Ռͷఏग़ϑΥʔϚοτ๚٬൪߸ͱ֤ΧςΰϦΛྻͱͨ͠ςʔϒϧɻ๚٬൪߸͝ͱʹΧςΰϦʹଐ͢Δ֬Λղ͢Δɻ
TFBCPSOʹΑΔσʔλͷՄࢹԽTFBCPSOϋΠϨϕϧͳΠϯλʔϑΣʔεΛ࣋ͭՄࢹԽπʔϧɽσʔλϑϨʔϜΛೖྗͱͯ͠౷ܭάϥϑΛදࣔ͢Δ͜ͱ͕Ͱ͖Δɽඪม5SJQ5ZQF͝ͱʹภΓ͕͋Δ͜ͱ͕֬ೝͰ͖Δ7JTJU/VNCFS͕6OJRVFͱͳΔΑ͏ϨίʔυΛࣺͯΔ5SJQ5ZQFͷ͝ͱͷΧϯτ
TFBCPSOʹΑΔσʔλͷՄࢹԽ▶ ܇࿅ࣄྫͱςετࣄྫ࣌ܥྻʢ༵ʣ͕ಉ༷ͷͱͳΔΑ͏ʹαϯϓϧ͞Ε͍ͯΔ͜ͱ͕Θ͔Δ▶ Λ֬ೝ͢Δͱ༵ͷใΛֶशϞσϧʹಛྔͱͯ͠༻͍ͨ߹ʹɺ܇࿅ࣄྫʹରͯ͠աֶश͠ͳ͍ͱਪଌͰ͖Δ܇࿅σʔλ্ஈͷϓϩοτςετσʔλԼஈͷϓϩοτ8FFLEBZͷ͝ͱʹΧϯτ
Լ४උɿධՁࢦඪΛఆٛ͢ΔධՁࢦඪͷ࣮TLMFBSONFUSJDTϞδϡʔϧNM@NFUSJDTύοέʔδͳͲ͕ଘࡏ͢Δɻ௨ৗ͜ΕΒͷύοέʔδΛ͑ͳ͍ɻ▶ TLMFBSONFUSJDTMPH@MPTT ؔʮ܇࿅ࣄྫͷΫϥεʯͱʮςετࣄྫͷΫϥεʯ͕Ұக͠ͳ͍߹ΤϥʔͱͳΔɻଟΫϥεྨʹ͓͍ͯɺرগͳΫϥε͕ςετࣄྫʹग़ݱ͠ͳ͍ͱ͍͏͜ͱ͕ى͜ΓಘΔɻ▶ ࠓճ͜ͷέʔεʹͯ·ΔͨΊɺύοέʔδΛΘͣʹఆ͍ٛͯ͠Δɻ
Լ४උɿϞσϧධՁͷͨΊͷํ๏Λ༻ҙ͢Δ▶ ࠓճׂަࠩ֬ೝ TLMFBSODSPTT@WBMJEBUJPO,'PMEΛ༻͍Δ▶ ͯ͢ͷΧςΰϦʹରͯ֬͠Ͱ͋Δͱ͑ΔϕʔεϥΠϯઌʹఆٛͨ͠NVMUJMPHMPTTͰείΞϦϯά
Լ४උɿ1Z5BCMFTͰಛྔΛதؒϑΝΠϧʹѹॖͯ͠อଘ▶ 1Z5BCMFT)%'ܗࣜͷͨΊͷΠϯλʔϑΣʔεΛఏڙ͢Δύοέʔδ▶ σʔλͷѹॖʹCMPTDΛનΊΔɽCMPTD4*.%໋ྩΛαϙʔτͨ͠ߴͳγϦΞϥΠζɾσγϦΞϥΠζ͕Մೳɽɹ ˞σʔλϑϨʔϜΛγϦΞϥΠζɾσγϦΞϥΠζ͢Δ߹ɹQBOEBTͷ)%'4UPSF 1Z5BCMFTΛར༻Ϋϥε͕༗༻Ͱ͋Δѹॖํ๏ͱͯ͠CMPTDΛࢦఆQEGBDUPSJ[F ͰඪมΛJOEFYFEͳʹΤϯίʔυ
ಛྔΛ࡞͢Δྫʰ͋Δങ͍ʹ͓͍ͯɺͲͷΧςΰϦ͕ԿݸεΩϟϯ͞Ε͔ͨʱQEQJWPU@UBCMF ؔͰτϥϯβΫγϣϯΛू>>> df_long = pd.concat([df_train, df_test,]).fillna(“_NA_”)>>> df = pd.pivot_table(df_long,index=“VisitNumber”,columns=[“DepartmentDescription”],values=[“ScanCount”],aggfunc=[np.sum],)[‘sum’][‘ScanCount’]QEQJWPU@UBCMF EG@MPOH JOEFYl7JTJU/VNCFSz DPMVNOT WBMVFT BHHGVOD$PMVNO7BMVF͕λςํʹฒΜͰ͍Δ MPOHGPSNBU$PMVNO7BMVFͷରԠ͕ू͞ΕϤίํʹฒͿ XJEFGPSNBU
ಛྔͷอଘܗࣜΛܾΊΔ▶ ͜͜Ͱʮ࠷ॳͷߦΛ܇࿅ࣄྫʯʮଓ͘ߦΛςετࣄྫʯͱͯ͠ߦྻΛಛྔͷߦྻΛOEBSSBZͱͯ͠อଘ͢Δ͜ͱΛߟ͑Δ▶ EGͷΠϯσοΫε7JTJU/VNCFSͰ͋Δɽ܇࿅ࣄྫͷߦྻͱςετࣄྫͷߦྻ͕7JTJU/VNCFSॱʹͳΔΑ͏ʹMPDϝιουͰฒͼସ͑ΔPSEFSCZl7JTJU/VNCFSzPSEFSCZl7JTJU/VNCFSz>>> visit_number_order = (df_train.VisitNumber.drop_duplicates().append(df_test.VisitNumber.drop_duplicates())>>> df = df.loc[visit_order]>>> X = df.fillna(0).as_matrix()>>> X.shape(191348, 69)1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0܇࿅ࣄྫͷߦྻOEBSSBZPCKFDUςετࣄྫͷߦྻOEBSSBZPCKFDU7JTJU/VNCFS͕ΠϯσοΫε
༧ଌϞσϧΛ࡞Γަࠩ֬ೝʹΑͬͯධՁ TDJLJUMFBSO࡞ͨ͠ಛྔΛ͍TDJLJUMFBSOͷϩδεςΟοΫճؼϞσϧΛ࡞݁Ռɼަࠩ֬ೝʹΑΓϞσϧΛධՁ͠ͷείΞΛಘͨɽ܇࿅ࣄྫͷO@TBNQMFTͰަࠩ֬ೝΛ͢ΔʢςετσʔλΛΘͳ͍ʣ
DSPTT@WBM@TDPSF ؔͷ՝ͱϫʔΫΞϥϯυʲ՝ʳݱঢ়ͷDSPTT@WBM@TDPSF ϔϧύʔؔQSFEJDU@QSPCB ϝιουΛݺͼग़͢͜ͱ͕Ͱ͖ͣɺQSFEJDU ϝιουΛݺͿTDJLJUMFBSOEFW NBTUFS)&"%ͰTLMFBSONPEFM@TFMFDUJPODSPTT@WBM@TDPSF ϔϧύʔ͕ؔվम͞Ε͍ͯΔɽQSFEJDU@QSPCB ϝιουΛΦϓγϣϯͰࢦఆ͢Δ͜ͱͰݺͼग़͢͜ͱ͕Ͱ͖Δɻˠ<8PSLBSPVOE>ΫϥεΛܧঝͯ͠QSFEJDU ϝιουΛݺͿͱQSFEJDU@QSPCB ϝιου͕ݺΕΔΫϥεΛఆٛ͢Ε͏͜ͱ͕Ͱ͖Δɻ
༧ଌϞσϧΛ࡞Γަࠩ֬ೝʹΑͬͯධՁ 9(#PPTUXBUDIMJTUʹొ͞Εͨσʔληοτ3FHSFTTJPO5SFFΛ૿ͨ͢ͼʹධՁ͞Εͯλʔϛφϧʹग़ྗ͞ΕΔɽ%.BUSJYΦϒδΣΫτରԠ͢ΔඪมͱηοτͰఆٛ͢Δ͜ͱ͕Ͱ͖Δ9(#PPTU࣮ͷϒʔεςΟϯάϞσϧʹஔ͖͑ͯަࠩ֬ೝ4DPSFˠ
༧ଌ݁ՌΛ,BHHMFʹఏग़͢Δ༧ଌ݁Ռ͔ΒσʔλϑϨʔϜΛ࡞ͯ͠,BHHMFʹߘ͢ΔQEGBDUPSJ[F ͷೋ൪ͷฦΓඪมͷΤϯίʔυॱ൪UP@DTW Ͱ$47ϑΝΠϧग़ྗ
༧ଌ݁ՌΛ,BHHMFʹఏग़͢Δ༧ଌ݁Ռ͔ΒσʔλϑϨʔϜΛ࡞ͯ͠,BHHMFʹߘ͢ΔQEGBDUPSJ[F ͷೋ൪ͷฦΓඪมͷΤϯίʔυॱ൪UP@DTW Ͱ$47ϑΝΠϧग़ྗͨͩͪʹ)PMEPVUTFUͰͷධՁ͕͡·Γίϯςετʹ͓͚ΔॱҐ͕ࣔ͞ΕΔ
ಛྔΛ૿͢▶ ങ͍ͷ༵ˠQEGBDUPSJ[F ▶ τϥϯβΫγϣϯϨίʔυɺฦ٫ͷτϥϯβΫγϣϯϨίʔυɺങ͍ʹඥͮ͘%FQBSUNFOU%FTDSJQUJPOͷϢχʔΫˠHSPVQCZ ूͷͨΊͷؔΧελϚΠζͰ͖Δ
ಛྔΛ૿ͯ͠Ϟσϧͷަࠩ֬ೝதؒϑΝΠϧʹอଘͨͦ͠ΕͧΕͷಛྔߦྻΛOQITUBDL ؔͰ݁߹ͯ͠୯ҰͷOEBSSBZΛ࡞͢Δɽ༧ଌϞσϧઌʹఆٛͨ͠ύϥϝʔλΛͦͷ··ར༻ʢˠ$74DPSFʣ1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 2 3 1 1 1 1 2 1 0 0 0 1 0 0 0 1 1 0 0 0 0 2 1 0 0 0 1 0 0 0 3 1 0 0 0 1 0 0 0 1 2 0 0 0 0 1 0 0 1 1 1 1 OQITUBDL
ಛྔΛ૿ͯ͠Ϟσϧͷަࠩ֬ೝதؒϑΝΠϧʹอଘͨͦ͠ΕͧΕͷಛྔߦྻΛOQITUBDL ؔͰྻํʹܨ͛ͯ୯ҰͷOEBSSBZΛ࡞͢Δɽ༧ଌϞσϧઌʹఆٛͨ͠ύϥϝʔλΛͦͷ··ར༻ʢˠ$74DPSFʣ1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 2 3 1 1 1 1 2 1 0 0 0 1 0 0 0 1 1 0 0 0 0 2 1 0 0 0 1 0 0 0 3 1 0 0 0 1 0 0 0 1 2 0 0 0 0 1 0 0 1 1 1 1 OQITUBDL ˠલճͷ༧ଌϞσϧΑΓվળͯ͠ॱҐ্͕͕ͬͨ
1"35ൃలతͳ
ൃలతͳɿσʔλʹಛԽ༷ͨ͠ʑͳσʔλ͝ͱʹ༗༻ͱͳΔύοέʔδΛհ͢Δ▶ େنσʔλ ˞͜͜Ͱ%#શମͰ(#نͷ3%#Λఆ͠·͢– 3FETIJGU– #JH2VFSZ– ϕϯνϚʔΫ 3FETIJGUWT#JH2VFSZ▶ ը૾σʔλ– ಛͷαϯϓϦϯά– ࣹӨม
42-ΫΤϦ͔ΒQBOEBTσʔλϑϨʔϜΛ࡞͢ΔQBOEBT*0ͱͯ͠42-Λαϙʔτ͍ͯ͠ΔɽQBOEBTSFBE@TRM ؔΛ͏ͱΫΤϦൃߦͷ݁ՌΛσʔλϑϨʔϜͱͯ͠ड͚औΔ͜ͱ͕Ͱ͖Δ▶ ෳࡶͳ3%#ѻ͏߹ɼूॲཧͳͲΛ42-ʹΑͬͯ؆ܿʹهड़Ͱ͖Δ▶ େنͳσʔλΛѻ͏߹ɼͯ͢ͷσʔλΛखݩͷڥʹϩʔυ͢Δ͜ͱͳ͘42-αʔόʔͷܭࢉϦιʔεͰॲཧͰ͖Δ˞*NBHF4PVSDFIUUQTXXXLBHHMFDPNDDFSWJDBMDBODFSTDSFFOJOHEBUB
ΫϥυࢿݯΛར༻͠େنσʔλΛॲཧ 3FETIJGU3FETIJGU"84্ʹىಈͨ͠ߴີετϨʔδϊʔυ͋Δ͍ߴີίϯϐϡʔτϊʔυʹΑΔΫϥελ্Ͱಈ࡞͢Δɽ### 1. Amazon S3 σʔλΛΞοϓϩʔυ$ aws s3 sync data/input s3://kaggle-kohei/walmart_triptype/input### 2. ςʔϒϧΛ࡞ͯ͠S3 ͔Β Redshift σʔλΛϩʔυ͢Δ$ psql < schema.sql### 3. SQL ΫΤϦΛൃߦͯ͠ Pandas σʔλϑϨʔϜΛ࡞͢Δ>>> import psycopg2 as pg>>> import pandas as pdconn_string = ' '.join(["dbname='dwh'","port='5439'","user='kohei_ozaki'","password='{}'".format(os.environ['REDSHIFT_PWD']),"host='{}'".format(os.environ['REDSHIFT_HOST']), ])>>> conn = pg.connect(conn_string)>>> pd.read_sql("SELECT * FROM train WHERE TripType = 999", conn)ىಈͨ͠3FETIJGUͷଓઌΛࢦఆ42-͔ΒσʔλϑϨʔϜ࡞
ΫϥυࢿݯΛར༻͠େنσʔλΛॲཧ #JH2VFSZ### 1. Cloud Storage へデータをアップロードする$ gsutil rsync -r data/input gs://kaggle-kohei.appspot.com/walmart_triptype### 2. BigQuery テーブルとしてデータをインポート$ bq load --skip_leading_rows 1 \kaggle-kohei:walmart_triptype.train \gs://kaggle-kohei.appspot.com/walmart_triptype/train.csv \train.json### 3. Pandas からクエリを発行して，結果をデータフレームとして受け取る>>> df = pd.read_gbq("""SELECT t1.trip_type, COUNT(1) AS n_visitors FROM (SELECT FIRST(trip_type) AS trip_type FROM walmart_triptype.trainGROUP BY visit_number) t1 GROUP BY t1.trip_type""", "kaggle-kohei")1BOEBTΑΓQESFBE@HCR ͔ؔΒ#JH2VFSZʹΫΤϦΛൃߦͰ͖ΔɽϑϧϚωʔδυͳαʔϏεͰ͋ΔͨΊΫϥελΛҙࣝ͢Δඞཁ͕ͳ͍ར͕͋Δɽ+40/ܗࣜͰςʔϒϧεΩʔϚΛఆٛ͢Δ42-͔ΒσʔλϑϨʔϜ࡞
3FETIJGUͱ#JH2VFSZͷϕϯνϚʔΫ▶ (#ͷ$47ɼH[JQѹॖͰ(#ͷԯߦ ߦςʔϒϧσʔλΛϩʔυͯ͠ΫΤϦʢूؔʣͷ࣮ߦ࣌ؒΛൺֱ▶ ܭଌର#JH2VFSZ 3FETIJGU Y 3FETIJGU Y 3FETIJGU YP⒎UPQJDࠓճͷઃఆͰΛ૿ͤ3FETIJGUͷੑೳΛ্͛Δ͜ͱ͕Ͱ͖ͨɽ˞3FETIJGU OPEFT POEFNBOE EDMBSHFJOTUBODFͷίετ64%IPVS#JH2VFSZͷΫΤϦίετԁະຬ
ΫϥυࢿݯΛར༻͢ΔͨΊͷΞυόΠε▶ σʔλͷϩʔυʹ͕͔͔࣌ؒΔ͕ɼʮΞυϗοΫΫΤϦΛԿ࣮ߦ͢Δʯʮ֤ΞυϗοΫΫΤϦͷλʔϯΞϥϯυλΠϜΛ͍ͨ͘͠ʯͱ͍͏ڧ͍ཁ͕͋ΔͳΒඅ༻ରޮՌ͕ߴ͍▶ #JH2VFSZڊେͳѹॖϑΝΠϧΛҰʹόονͰϩʔυ͢Δ͜ͱ͕Ͱ͖ͳ͍ɽͦͷ߹ߦ୯ҐͰׂˍѹॖͯ͠$MPVE4UPSBHFʹΞοϓϩʔυͰ͖Δ$ zcat prescription_head.csv.gz | \split –d –C 1G --filter='gzip > $FILE.gz' – prescription_head.csv.partprescription_head.csv.gz prescription_head.csv.part01.gzprescription_head.csv.part02.gzprescription_head.csv.part03.gzzcat & split gsuIl rsyncP⒎UPQJD
1ZUIPOʹΑΔը૾σʔλͷॲཧ1ZUIPOͰը૾ॲཧΛ͢ΔͨΊͷύοέʔδͱͯ͠ɼTDJLJUJNBHF0QFO$7ͷ1ZUIPOόΠϯσΟϯά͕͋Δɽ֤ύοέʔδͰը૾ΛOEBSSBZΦϒδΣΫτͷߦྻͱͯ͠ѻ͏ɽ▶ มܗʢճసɼ֦େɼ΅͔͠ʣ▶ υϩʔΠϯά▶ ώετάϥϜฏୱԽ▶ ը૾ͷηάϝϯςʔγϣϯ▶ Τοδநग़ɼը૾ಛͷܭࢉ▶ ը૾ಛͷαϯϓϦϯάɼϚονϯά▶ FUDim[:, :, 2]im[:, :, 1]im[:, :, 0]#(3νϟϯωϧͷը૾දݱͷྫ
0QFO$7ʹΑΔը૾ಛͷܭࢉDW#3*4,@DSFBUF ؔ#3*4,ΞϧΰϦζϜͷ0QFO$7࣮ɽ#3*4,Λܭࢉ͢ΔͨΊͷEFUFDUPSΦϒδΣΫτΛ࡞͢Δ%3"1&34BUFMMJUF*NBHF$ISPOPMPHZίϯςετͷσʔλΛྫͱͯ͠հ͢ΔIUUQTXXXLBHHMFDPNDESBQFSTBUFMMJUFJNBHFDISPOPMPHZ
ಛͷϚονϯάͱ݁ՌΛࣹͬͨӨมDW%FTDSJQUPS.BUDIFS@DSFBUF ؔͰಛͷϚονϯά͢ΔɽϚονϯά݁ՌΛ͍ɼࡱӨ֯εέʔϧͷҧ͍Λࣸ૾มͰิਖ਼͢ΔɽDWQFSTQFDUJWF5SBOTGPSN ؔͰϗϞάϥϑΟߦྻΛجʹࣹӨมDWpOE)PNPHSBQIZ ؔͰϚονϯά݁ՌΛجʹϗϞάϥϑΟߦྻΛܭࢉ
ը૾ࠩͷՄࢹԽࣹӨมͯ͠ࡱӨ֯εέʔϧͷҟͳΔը૾ΛҰகͤͨ͞ɽೋԽͨ͠ը૾ΛϐΫηϧ୯ҐͰൺֱ͢Δ͜ͱͰɼࠩΛՄࢹԽͰ͖Δ3FG5VUPSJBMWJTVBMJ[JOHEJ⒎FSFODFCFUXFFOUISFFQJDUVSFTIUUQTXXXLBHHMFDPNDESBQFSTBUFMMJUFJNBHFDISPOPMPHZGPSVNTUUVUPSJBMWJTVBMJ[JOHEJ⒎FSFODFCFUXFFOUISFFQJDUVSFTը૾"ʹͷΈग़ݱը૾#ʹͷΈग़ݱ˞ίʔυͱৄࡉʹ͍ͭͯࢿྉͷ+VQZUFSOPUFCPPLΛࢀর͍ͯͩ͘͠͞ɽ˞ಛͷܭࢉͳͲ0QFO$7ͷ֦ுΛΠϯετʔϧ͢Δඞཁ͕͋Δɽ
͜ͷνϡʔτϦΞϧʹ͍ͭͯʢ࠶ܝʣσʔλੳίϯςετΛࡐͱͯ͠ɺ1ZUIPOʹΑΔύοέʔδར༻ྫΛհ͢Δɻ▶ 1BSU༧ଌϞσϧ࡞ϓϩηεͱؔ࿈ύοέʔδ NJO– ༧ଌϞσϧͷ࡞ϓϩηεͱؔ࿈ύοέʔδͷུ֓Λհ͢Δ– 1BOEBT 4DJLJU-FBSO▶ 1BSUσʔλੳίϯςετʹ͓͚Δ࣮ફ NJO– σʔλੳίϯςετΛࡐͱͯ͠۩ମతͳར༻ྫΛհ͢Δ– 1BOEBT TFBCPSO 9(#PPTU 1Z5BCMFT▶ 1BSUൃలతͳ NJO– తʢେنσʔλɾը૾σʔλʣ͝ͱʹಛԽͨ͠ར༻ྫΛհ͢Δ– 1BOEBT 3FETIJGU #JH2VFSZ 0QFO$7
ิɿࢿྉͱ%PDLFSʹΑΔ࠶ݱੑͷ֬อʢ࠶ܝʣຊεϥΠυͰΘΕΔίʔυ+VQZUFSOPUFCPPLͱͯ͠ҎԼͰެ։͍ͯ͠Δɽ▶ εϥΠυIUUQHPPHM.F.;Z0▶ 1BSUࢿྉIUUQHPPHMZ;N,S▶ 1BSUࢿྉIUUQHPPHM"5$ZWɽຊࢿྉͰར༻ͨ͠+VQZUFSOPUFCPPLެ։͍ͯ͠Δ%PDLFSΠϝʔδ͔Β࡞ͨ͠ڥΛ͍ͬͯΔɽҎԼͷίϚϯυͰಉ༷ͷڥΛࢼ͢͜ͱ͕Ͱ͖Δɻ$ docker run --rm -ti \-p 8888:8888 \-v /path/to/data_directory:/mnt \-v /path/to/working_directory:/home/kohei/work \smly/notebook:0.4
smly
62gerente
kneath
jonyablonski
pedronauck
bkeepers
morganepeng
eitanlees
shlominoach
aki_iinuma
davidbonilla
keathley