Share
Embed
Python とデータ分析コンテストの実践
FIT 2016 Tutorial 資料
スライド: http://goo.gl/MeMZyO
Part2 資料: http://goo.gl/y6ZmKr
Part3 資料: http://goo.gl/ATC9yv
@smly
September 08, 2016
Transcript
44 slides
-
͜ͷνϡʔτϦΞϧʹ͍ͭͯ σʔλੳίϯςετΛࡐͱͯ͠ɺ 1ZUIPOʹΑΔύοέʔδར༻ྫΛհ͢Δɻ ▶ 1BSU༧ଌϞσϧ࡞ϓϩηεͱؔ࿈ύοέʔδ NJO – ༧ଌϞσϧͷ࡞ϓϩηεͱؔ࿈ύοέʔδͷུ֓Λհ͢Δ –
1BOEBT 4DJLJU-FBSO ▶ 1BSUσʔλੳίϯςετʹ͓͚Δ࣮ફ NJO – σʔλੳίϯςετΛࡐͱͯ͠۩ମతͳར༻ྫΛհ͢Δ – 1BOEBT TFBCPSO 9(#PPTU 1Z5BCMFT ▶ 1BSUൃలతͳ NJO – తʢେنσʔλɾը૾σʔλʣ͝ͱʹಛԽͨ͠ར༻ྫΛհ͢Δ – 1BOEBT 3FETIJGU #JH2VFSZ 0QFO$7 -
ิɿࢿྉͱ%PDLFSʹΑΔ࠶ݱੑͷ֬อ ຊεϥΠυͰΘΕΔίʔυ+VQZUFSOPUFCPPLͱͯ͠ҎԼͰެ։͍ͯ͠Δɽ ▶ εϥΠυIUUQHPPHM.F.;Z0 ▶ 1BSUࢿྉIUUQHPPHMZ;N,S ▶ 1BSUࢿྉIUUQHPPHM"5$ZW ɽຊࢿྉͰར༻ͨ͠+VQZUFSOPUFCPPLެ։͍ͯ͠Δ%PDLFSΠϝʔδ͔Β
࡞ͨ͠ڥΛ͍ͬͯΔɽҎԼͷίϚϯυͰಉ༷ͷڥΛࢼ͢͜ͱ͕Ͱ͖Δɻ $ docker run --rm -ti \ -p 8888:8888 \ -v /path/to/data_directory:/mnt \ -v /path/to/working_directory:/home/kohei/work \ smly/notebook:0.4 -
Ұൠతͳ༧ଌϞσϧ࡞ͷϓϩηε ▶ ,BHHMF,%%$VQͳͲͷίϯςετͰɼz༩͑ΒΕͨσʔλͱλεΫz ʹରͯ͠ɼzܾΊΒΕͨධՁࢦඪzʹΑΔείΞͰॱҐΛ͚Λߦ͏ ▶ ͦΕͧΕͷϓϩηεʹ͓͍ͯ༗༻ͳϞδϡʔϧʹ͍ͭͯհ͢Δ λεΫઃܭ σʔλ࡞ γεςϜ ΈࠐΈ
લॲཧ ಛநग़ ༧ଌϞσϧ ͷධՁ ༧ଌϞσϧ ͷ࡞ ・ pandas ・ psycopg2 ・ sklearn.feature_extraction ・ sklearn.preprocessing ・ pytables ・ seaborn ・ sklearn.linear_model ・ sklearn.svm ・ sklearn.ensemble ・ xgboost ・ stats-models ・ sklearn.metrics ・ sklearn.cross_validation ・ ml_metrics -
QBOEBTσʔλϑϨʔϜͰσʔλૢ࡞ ೖग़ྗ ▶ /VN1ZͷOEBSSBZʹࣅͨσʔλΛૢ࡞͢ΔͨΊͷσʔλߏ ▶ ΧϥϜͱΠϯσοΫε͕͋Γɺ໊લ͖ΧϥϜͰσʔλૢ࡞͕Մೳ ▶ ΧϥϜ͝ͱʹҟͳΔܕΛ࣋ͭ͜ͱ͕Ͱ͖Δ ▶
ߦͷૠೖྻͷআͳͲ͕ߦ͑ΔՄม NVUBCMF ͳΦϒδΣΫτ ▶ ๛ͳσʔλૢ࡞ػೳͱೖग़ྗΠϯλʔϑΣʔε લॲཧ ಛநग़ NDArray Internal dimensions dim count dtype strides data * 2 3 3 5 * 12 float32 0 1 2 3 4 5 6 7 8 DataFrame Internal I N D E X Columns float string integer -
ಛྔͷΤϯίʔυ ઢܗϞσϧͳͲΧςΰϦΧϧมΛѻ͏͜ͱ͕Ͱ͖ͳ͍ϞσϧͰɼΧςΰ ϦΧϧมΛมͱͯ͠දݱ͢Δඞཁ͕͋Δɽ ༷ʑͳಛྔͷΤϯίʔυํ๏ͱ࣮͕TDJLJUMFBSOʹΑͬͯఏڙ͞Ε͍ͯΔɽ ▶ TLMFBSOGFBUVSF@FYUSBDUJPO0OF)PU&ODPEFS ▶ TLMFBSOGFBUVSF@FYUSBDUJPO-BCFM&ODPEFS
▶ TLMFBSOGFBUVSF@FYUSBDUJPO%JDU7FDUPSJ[F ▶ TLMFBSOGFBUVSF@FYUSBDUJPO'FBUVSF)BTIFS ▶ TLMFBSOGFBUVSF@FYUSBDUJPOUFYU5pEG7FDUPSJ[FS ▶ ʜ දతͳΫϥεΛհ͢Δ લॲཧ ಛநग़ -
0OF)PU&ODPEFS ΧςΰϦΧϧมΛPG,දهʹΤϯίʔυ͢Δɽ ͋ΔΧςΰϦΧϧมͷDBSEJOBMJUZΛ,ͱͨ͠ͱ͖ɼಛΛ,ྻͷͱ͠ ͯѻ͏ɽͦΕͧΕͷྻʹΧςΰϦΧϧมͷಛఆͷ͕ରԠ͓ͯ͠ΓɼରԠ͢ ΔྻͷΛɼͦΕҎ֎ΛͰදݱ͢Δɽ ʢQEHFU@EVNNJFT ؔʹΑͬͯQBOEBTͰಉͷૢ࡞͕ՄೳͰ͋Δʣ લॲཧ ಛநग़
WeekDay 0 Friday 1 Monday 2 Monday 3 Sunday 4 Tuesday 5 Saturday 6 Monday 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 ndarray or csr_matrix K=5 -
-BCFM&ODPEFS ΧςΰϦΧϧมΛʹΤϯίʔυ͢Δɽ ΧςΰϦΧϧมͷDBSEJOBMJUZ͕,Ͱ͋Δ߹ɼ มͷΛ< ,>ͷʹஔ͖͑Δɽ ʢQEGBDUPSJ[F ؔʹΑͬͯQBOEBTͰಉͷૢ࡞͕ՄೳͰ͋Δʣ લॲཧ
ಛநग़ WeekDay 0 Friday 1 Monday 2 Monday 3 Sunday 4 Tuesday 5 Saturday 6 Monday 0 1 1 2 3 4 1 ndarray -
%JDU7FDUPSJ[F ࣙॻΦϒδΣΫτͷϦετΛ4DJ1ZͷૄߦྻΦϒδΣΫτOEBSSBZΦϒδΣ Ϋτʹม͢Δɽ ࣙॻΦϒδΣΫτͷLFZ͕ߦྻͷಛఆͷྻʹҰରҰରԠ͢Δɽ WBMVF͕ಛʹରԠ͢Δʢ%BUB'SBNFΦϒδΣΫτͷίϯετϥΫλʹ͠ ͯpMMOB BT@NBUSJY ͰθϩϑΟϧˍม͢Δ͜ͱͰQBOEBTͰಉͷૢ࡞
͕ՄೳͰ͋Δʣ લॲཧ ಛநग़ list of dict [ {‘like’: 1, ‘rt’: 9}, {‘like’: 2, ‘rt’: 2}, {‘like’: 4}, ] 1 9 2 2 4 0 ndarray -
TDJLJUMFBSO9(#PPTUͰ༧ଌϞσϧΛ࡞ TDJLJUMFBSOͰpU ͰσʔλͯΊɼQSFEJDU Ͱ༧ଌɼQSFEJDU@QSPCB Ͱ༧ଌ֬Λ͑Δڞ௨ͷΠϯλʔϑΣʔεΛ͍࣋ͬͯΔʢ˞ྨͷ߹ʣ >>> import xgboost as
xgb >>> dtrain, dtest = xgb.DMatrix(), xgb.DMatrix() >>> watchlist = [(dtrain, 'train')] >>> booster = xgb.Booster() >>> gbtree = booster.train(dtrain, params) >>> y_pred = gbtree.predict(dtest) >>> from sklearn.linear_model import LogisticRegression >>> clf = LogisticRegression() >>> clf.fit(X_train, y_rain) >>> y_pred = clf.predict_proba(X_test) 9(#PPTUTDJLJUMFBSOͱҟͳΔΠϯλʔϑΣʔεͰ͋Δ͕ɼΠςϨʔγϣϯ͝ͱͷ FWBMVBUJPOɼFBSMZTUPQQJOHɼ࠷దԽؔͷΧελϚΠζͳͲͷ༷ʑͳػೳ͕͋Δɽ 9(#PPTUϝϞϦޮͱ܇࿅ʹ࠷దԽ͞Εͨ෦σʔλߏ%.BUSJYͱͯ͠ σʔλΛѻ͏ɽOEBSSBZ͔Β%.BUSJYΦϒδΣΫτΛ࡞Ͱ͖Δɽ ༧ଌϞσϧ ͷ࡞ -
ϞσϧͷੑೳධՁ TLMFBSODSPTT@WBMJEBUJPOϞδϡʔϧ༧ଌϞσϧͷੑೳΛධՁ͢ΔͨΊͷ ϔϧύʔؔΠςϨʔλʔΛఏڙ͢Δɻ ༧ଌϞσϧ ͷධՁ >>> clf = LogisticRegression() >>>
scores = cross_validation.cross_val_score(clf, X, y, cv=5) >>> scores array([ 0.92..., 1. ..., 0.92..., 1. ]) ༧ଌϞσϧ͕TLMFBSO$MBTTJpFS.JYJOΛܧঝͨ͠ΫϥεͰ͋ΕϔϧύʔؔΛ ༻͍ͯަࠩ֬ೝΛ؆ܿʹهड़Ͱ͖Δɻ ΠςϨʔλʔ֬ೝ༻ͷ܇࿅σʔλͱςετσʔλͷΠϯσοΫεϦετΛฦ͢ >>> kf = KFold(n_samples, n_folds=5) >>> for idx_train, idx_test in kf: ... y_train, y_test = y[idx_train], y[idx_test] ... X_train, X_test = X[idx_train], X[idx_test] ... clf.fit(X_train, y_train) ... y_pred = clf.predict(X_test) ֤'PMEͷ"DDVSBDZ -
σʔλͷ֬ೝ ▶ l8BMNBSU3FDSVJUJOH5SJQ5ZQF$MBTTJpDBUJPOzίϯςετΛࡐͱͯ͠ѻ͏ ▶ ʮങ͍٬ʯΛʮങ͍ͷεΩϟϯཤྺʯ͔ΒଟΫϥεྨ͢ΔλεΫ ▶ USBJODTW UFTUDTW TBNQMF4VCNJTTJPODTWͷ̏ͭͷϑΝΠϧ͕ఏڙ͞Ε͍ͯΔ
܇࿅ࣄྫσʔλUSBJODTWʮങ͍ͷεΩϟϯཤྺʯͰ͋Δɻ ϑΟʔϧυ໊ આ໌ 5SBJO 5FTU 5SJQ5ZQF ඪม ʮങ͍٬ʯͷΧςΰϦΧϧͳ*% ✔ ✗ 7JTJU/VNCFS *OTUBODF*% ͋ΔҰਓͷސ٬ͷങ͍ʹରԠ͢Δ*% ✔ ✔ 8FFLEBZ ങ͍Λͨ͠ि ✔ ✔ 6QD ߪೖͷ61$൪߸ ✔ ✔ 4DBO$PVOU ߪೖ͞Εͨݸʢෛͷฦ٫͞Εͨʣ ✔ ✔ %FQBSUNFOU%FTDSJQUJPO ߪೖͷδϟϯϧ ✔ ✔ 'JOFMJOF ߪೖΛߋʹࡉ͔͚͘Δδϟϯϧͷ*% ✔ ✔ -
Լ४උɿ1Z5BCMFTͰಛྔΛதؒϑΝΠϧʹѹॖͯ͠อଘ ▶ 1Z5BCMFT)%'ܗࣜͷͨΊͷΠϯλʔϑΣʔεΛఏڙ͢Δύοέʔδ ▶ σʔλͷѹॖʹCMPTDΛનΊΔɽCMPTD4*.%໋ྩΛαϙʔτͨ͠ߴ ͳγϦΞϥΠζɾσγϦΞϥΠζ͕Մೳɽ ɹ ˞σʔλϑϨʔϜΛγϦΞϥΠζɾσγϦΞϥΠζ͢Δ߹ ɹQBOEBTͷ)%'4UPSF 1Z5BCMFTΛར༻
Ϋϥε͕༗༻Ͱ͋Δ ѹॖํ๏ͱͯ͠ CMPTDΛࢦఆ QEGBDUPSJ[F ͰඪมΛ JOEFYFEͳʹΤϯίʔυ -
ಛྔΛ࡞͢Δ ྫʰ͋Δങ͍ʹ͓͍ͯɺ ͲͷΧςΰϦ͕Կݸ εΩϟϯ͞Ε͔ͨʱ QEQJWPU@UBCMF ؔͰ τϥϯβΫγϣϯΛू >>> df_long
= pd.concat([ df_train, df_test, ]).fillna(“_NA_”) >>> df = pd.pivot_table( df_long, index=“VisitNumber”, columns=[“DepartmentDescription”], values=[“ScanCount”], aggfunc=[np.sum], )[‘sum’][‘ScanCount’] QEQJWPU@UBCMF EG@MPOH JOEFYl7JTJU/VNCFSz DPMVNOT<l%FQBSUNFOU%FTDSJQUJPOz> WBMVFT<l4DBO$PVOUz> BHHGVOD<OQTVN> $PMVNO7BMVF͕λςํʹฒΜͰ͍Δ MPOHGPSNBU $PMVNO7BMVFͷରԠ͕ू͞ΕϤίํʹฒͿ XJEFGPSNBU -
ಛྔͷอଘܗࣜΛܾΊΔ ▶ ͜͜Ͱʮ࠷ॳͷߦΛ܇࿅ࣄྫʯʮଓ͘ߦΛςετࣄྫʯ ͱͯ͠ߦྻΛಛྔͷߦྻΛOEBSSBZͱͯ͠อଘ͢Δ͜ͱΛߟ͑Δ ▶ EGͷΠϯσοΫε7JTJU/VNCFSͰ͋Δɽ܇࿅ࣄྫͷߦྻͱςετࣄྫͷ ߦྻ͕7JTJU/VNCFSॱʹͳΔΑ͏ʹMPDϝιουͰฒͼସ͑Δ PSEFSCZ l7JTJU/VNCFSz PSEFSCZ
l7JTJU/VNCFSz >>> visit_number_order = ( df_train.VisitNumber.drop_duplicates().append( df_test.VisitNumber.drop_duplicates() ) >>> df = df.loc[visit_order] >>> X = df.fillna(0).as_matrix() >>> X.shape (191348, 69) 1 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 ܇࿅ࣄྫͷߦྻ OEBSSBZPCKFDU ςετࣄྫͷߦྻ OEBSSBZPCKFDU 7JTJU/VNCFS ͕ΠϯσοΫε -
DSPTT@WBM@TDPSF ؔͷ՝ͱϫʔΫΞϥϯυ ʲ՝ʳݱঢ়ͷDSPTT@WBM@TDPSF ϔϧύʔؔQSFEJDU@QSPCB ϝιου Λݺͼग़͢͜ͱ͕Ͱ͖ͣɺQSFEJDU ϝιουΛݺͿ TDJLJUMFBSOEFW NBTUFS)&"% Ͱ
TLMFBSONPEFM@TFMFDUJPODSPTT@WBM@TDPSF ϔϧύʔ͕ؔվम͞Ε͍ͯΔɽ QSFEJDU@QSPCB ϝιουΛΦϓγϣϯͰࢦఆ ͢Δ͜ͱͰݺͼग़͢͜ͱ͕Ͱ͖Δɻ ˠ<8PSLBSPVOE>ΫϥεΛܧঝͯ͠QSFEJDU ϝιουΛݺͿͱ QSFEJDU@QSPCB ϝιου͕ݺΕΔΫϥεΛఆٛ͢Ε͏͜ͱ͕Ͱ͖Δɻ -
ಛྔΛ૿ͯ͠Ϟσϧͷަࠩ֬ೝ தؒϑΝΠϧʹอଘͨͦ͠ΕͧΕͷಛྔߦྻΛOQITUBDL ؔͰ ݁߹ͯ͠୯ҰͷOEBSSBZΛ࡞͢Δɽ ༧ଌϞσϧઌʹఆٛͨ͠ύϥϝʔλΛͦͷ··ར༻ʢˠ$74DPSFʣ 1 0 0 0 0
0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 2 3 1 1 1 1 2 1 0 0 0 1 0 0 0 1 1 0 0 0 0 2 1 0 0 0 1 0 0 0 3 1 0 0 0 1 0 0 0 1 2 0 0 0 0 1 0 0 1 1 1 1 OQITUBDL -
ಛྔΛ૿ͯ͠Ϟσϧͷަࠩ֬ೝ தؒϑΝΠϧʹอଘͨͦ͠ΕͧΕͷಛྔߦྻΛOQITUBDL ؔͰ ྻํʹܨ͛ͯ୯ҰͷOEBSSBZΛ࡞͢Δɽ ༧ଌϞσϧઌʹఆٛͨ͠ύϥϝʔλΛͦͷ··ར༻ʢˠ$74DPSFʣ 1 0 0 0 0
0 1 0 0 0 0 1 0 0 0 0 0 1 0 0 2 3 1 1 1 1 2 1 0 0 0 1 0 0 0 1 1 0 0 0 0 2 1 0 0 0 1 0 0 0 3 1 0 0 0 1 0 0 0 1 2 0 0 0 0 1 0 0 1 1 1 1 OQITUBDL ˠ લճͷ༧ଌϞσϧΑΓվળͯ͠ॱҐ্͕͕ͬͨ -
ൃలతͳɿσʔλʹಛԽͨ͠ ༷ʑͳσʔλ͝ͱʹ༗༻ͱͳΔύοέʔδΛհ͢Δ ▶ େنσʔλ ˞͜͜Ͱ%#શମͰ(#نͷ3%#Λఆ͠·͢ – 3FETIJGU –
#JH2VFSZ – ϕϯνϚʔΫ 3FETIJGUWT#JH2VFSZ ▶ ը૾σʔλ – ಛͷαϯϓϦϯά – ࣹӨม -
42-ΫΤϦ͔ΒQBOEBTσʔλϑϨʔϜΛ࡞͢Δ QBOEBT*0ͱͯ͠42-Λαϙʔτ͍ͯ͠Δɽ QBOEBTSFBE@TRM ؔΛ͏ͱΫΤϦൃߦͷ ݁ՌΛσʔλϑϨʔϜͱͯ͠ड͚औΔ͜ͱ͕Ͱ͖Δ ▶ ෳࡶͳ3%#ѻ͏߹ɼूॲཧͳͲΛ42-ʹΑͬͯ؆ܿʹهड़Ͱ͖Δ ▶ େنͳσʔλΛѻ͏߹ɼͯ͢ͷσʔλΛखݩͷڥʹϩʔυ͢Δ͜ͱͳ ͘42-αʔόʔͷܭࢉϦιʔεͰॲཧͰ͖Δ
˞*NBHF4PVSDFIUUQTXXXLBHHMFDPNDDFSWJDBMDBODFSTDSFFOJOHEBUB -
ΫϥυࢿݯΛར༻͠େنσʔλΛॲཧ 3FETIJGU 3FETIJGU"84্ʹىಈͨ͠ߴີετϨʔδϊʔυ͋Δ͍ ߴີίϯϐϡʔτϊʔυʹΑΔΫϥελ্Ͱಈ࡞͢Δɽ ### 1. Amazon S3 σʔλΛΞοϓϩʔυ
$ aws s3 sync data/input s3://kaggle-kohei/walmart_triptype/input ### 2. ςʔϒϧΛ࡞ͯ͠S3 ͔Β Redshift σʔλΛϩʔυ͢Δ $ psql < schema.sql ### 3. SQL ΫΤϦΛൃߦͯ͠ Pandas σʔλϑϨʔϜΛ࡞͢Δ >>> import psycopg2 as pg >>> import pandas as pd conn_string = ' '.join([ "dbname='dwh'", "port='5439'", "user='kohei_ozaki'", "password='{}'".format(os.environ['REDSHIFT_PWD']), "host='{}'".format(os.environ['REDSHIFT_HOST']), ]) >>> conn = pg.connect(conn_string) >>> pd.read_sql("SELECT * FROM train WHERE TripType = 999", conn) ىಈͨ͠3FETIJGUͷଓઌΛࢦఆ 42-͔ΒσʔλϑϨʔϜ࡞ -
ΫϥυࢿݯΛར༻͠େنσʔλΛॲཧ #JH2VFSZ ### 1. Cloud Storage へデータをアップロードする $ gsutil
rsync -r data/input gs://kaggle-kohei.appspot.com/walmart_triptype ### 2. BigQuery テーブルとしてデータをインポート $ bq load --skip_leading_rows 1 \ kaggle-kohei:walmart_triptype.train \ gs://kaggle-kohei.appspot.com/walmart_triptype/train.csv \ train.json ### 3. Pandas からクエリを発行して,結果をデータフレームとして受け取る >>> df = pd.read_gbq(""" SELECT t1.trip_type, COUNT(1) AS n_visitors FROM ( SELECT FIRST(trip_type) AS trip_type FROM walmart_triptype.train GROUP BY visit_number ) t1 GROUP BY t1.trip_type""", "kaggle-kohei") 1BOEBTΑΓQESFBE@HCR ͔ؔΒ#JH2VFSZʹΫΤϦΛൃߦͰ͖Δɽ ϑϧϚωʔδυͳαʔϏεͰ͋ΔͨΊΫϥελΛҙࣝ͢Δඞཁ͕ͳ͍ར͕͋Δɽ +40/ܗࣜͰςʔϒϧεΩʔϚΛఆٛ͢Δ 42-͔ΒσʔλϑϨʔϜ࡞ -
3FETIJGUͱ#JH2VFSZͷϕϯνϚʔΫ ▶ (#ͷ$47ɼH[JQѹॖͰ(#ͷԯߦ ߦ ςʔϒϧσʔλΛϩʔυͯ͠ΫΤϦʢूؔʣͷ࣮ߦ࣌ؒΛൺֱ ▶ ܭଌର#JH2VFSZ
3FETIJGU Y 3FETIJGU Y 3FETIJGU Y P⒎UPQJD ࠓճͷઃఆͰΛ૿ͤ3FETIJGUͷੑೳΛ্͛Δ͜ͱ͕Ͱ͖ͨɽ ˞3FETIJGU OPEFT POEFNBOE EDMBSHFJOTUBODF ͷίετ64%IPVS#JH2VFSZͷΫΤϦίετԁະຬ -
ΫϥυࢿݯΛར༻͢ΔͨΊͷΞυόΠε ▶ σʔλͷϩʔυʹ͕͔͔࣌ؒΔ͕ɼʮΞυϗοΫΫΤϦΛԿ࣮ߦ͢Δʯ ʮ֤ΞυϗοΫΫΤϦͷλʔϯΞϥϯυλΠϜΛ͍ͨ͘͠ʯͱ͍͏ڧ͍ཁ ͕͋ΔͳΒඅ༻ରޮՌ͕ߴ͍ ▶ #JH2VFSZڊେͳѹॖϑΝΠϧΛҰʹόονͰϩʔυ͢Δ͜ͱ͕Ͱ͖ͳ͍ɽ ͦͷ߹ߦ୯ҐͰׂˍѹॖͯ͠$MPVE4UPSBHFʹΞοϓϩʔυͰ͖Δ $ zcat
prescription_head.csv.gz | \ split –d –C 1G --filter='gzip > $FILE.gz' – prescription_head.csv.part prescription_head.csv.gz prescription_head.csv.part01.gz prescription_head.csv.part02.gz prescription_head.csv.part03.gz zcat & split gsuIl rsync P⒎UPQJD -
1ZUIPOʹΑΔը૾σʔλͷॲཧ 1ZUIPOͰը૾ॲཧΛ͢ΔͨΊͷύοέʔδͱͯ͠ɼ TDJLJUJNBHF0QFO$7ͷ1ZUIPOόΠϯσΟϯά͕͋Δɽ ֤ύοέʔδͰը૾ΛOEBSSBZΦϒδΣΫτͷߦྻͱͯ͠ѻ͏ɽ ▶ มܗʢճసɼ֦େɼ΅͔͠ʣ ▶ υϩʔΠϯά ▶
ώετάϥϜฏୱԽ ▶ ը૾ͷηάϝϯςʔγϣϯ ▶ Τοδநग़ɼը૾ಛͷܭࢉ ▶ ը૾ಛͷαϯϓϦϯάɼϚονϯά ▶ FUD im[:, :, 2] im[:, :, 1] im[:, :, 0] #(3νϟϯωϧͷը૾දݱͷྫ -
ը૾ࠩͷՄࢹԽ ࣹӨมͯ͠ࡱӨ֯εέʔϧͷҟͳΔը૾ΛҰகͤͨ͞ɽ ೋԽͨ͠ը૾ΛϐΫηϧ୯ҐͰൺֱ͢Δ͜ͱͰɼࠩΛՄࢹԽͰ͖Δ 3FG5VUPSJBMWJTVBMJ[JOHEJ⒎FSFODFCFUXFFOUISFFQJDUVSFT IUUQTXXXLBHHMFDPNDESBQFSTBUFMMJUFJNBHFDISPOPMPHZGPSVNTUUVUPSJBMWJTVBMJ[JOHEJ⒎FSFODFCFUXFFOUISFFQJDUVSFT ը૾"ʹͷΈग़ݱ ը૾#ʹͷΈग़ݱ ˞ίʔυͱৄࡉʹ͍ͭͯࢿྉͷ +VQZUFSOPUFCPPLΛࢀর͍ͯͩ͘͠͞ɽ
˞ಛͷܭࢉͳͲ0QFO$7ͷ ֦ுΛΠϯετʔϧ͢Δඞཁ͕͋Δɽ -
͜ͷνϡʔτϦΞϧʹ͍ͭͯʢ࠶ܝʣ σʔλੳίϯςετΛࡐͱͯ͠ɺ 1ZUIPOʹΑΔύοέʔδར༻ྫΛհ͢Δɻ ▶ 1BSU༧ଌϞσϧ࡞ϓϩηεͱؔ࿈ύοέʔδ NJO – ༧ଌϞσϧͷ࡞ϓϩηεͱؔ࿈ύοέʔδͷུ֓Λհ͢Δ –
1BOEBT 4DJLJU-FBSO ▶ 1BSUσʔλੳίϯςετʹ͓͚Δ࣮ફ NJO – σʔλੳίϯςετΛࡐͱͯ͠۩ମతͳར༻ྫΛհ͢Δ – 1BOEBT TFBCPSO 9(#PPTU 1Z5BCMFT ▶ 1BSUൃలతͳ NJO – తʢେنσʔλɾը૾σʔλʣ͝ͱʹಛԽͨ͠ར༻ྫΛհ͢Δ – 1BOEBT 3FETIJGU #JH2VFSZ 0QFO$7 -
ิɿࢿྉͱ%PDLFSʹΑΔ࠶ݱੑͷ֬อʢ࠶ܝʣ ຊεϥΠυͰΘΕΔίʔυ+VQZUFSOPUFCPPLͱͯ͠ҎԼͰެ։͍ͯ͠Δɽ ▶ εϥΠυIUUQHPPHM.F.;Z0 ▶ 1BSUࢿྉIUUQHPPHMZ;N,S ▶ 1BSUࢿྉIUUQHPPHM"5$ZW ɽຊࢿྉͰར༻ͨ͠+VQZUFSOPUFCPPLެ։͍ͯ͠Δ%PDLFSΠϝʔδ͔Β
࡞ͨ͠ڥΛ͍ͬͯΔɽҎԼͷίϚϯυͰಉ༷ͷڥΛࢼ͢͜ͱ͕Ͱ͖Δɻ $ docker run --rm -ti \ -p 8888:8888 \ -v /path/to/data_directory:/mnt \ -v /path/to/working_directory:/home/kohei/work \ smly/notebook:0.4