How should we manage experiments?

Slide 1

Slide 1 text

)PXTIPVMEXFNBOBHFFYQFSJNFOUT .-LJUDIFO $IJF)BZBTIJEB

Slide 2

Slide 2 text

$IJF)BZBTIJEB !DIJF •  .BDIJOF-FBSOJOHBOE*OGSBTUSVDUVSF&OHJOFFSBU $PPLQBE*OD •  $POUSJCVUPSPG"QBDIF4QBSL )BEPPQ BOETPPO •  (SBEVBUFE0TBLB6OJWBOE/"*45 –  &OHBHFEJO7PJDFDPOWFSTJPOSFTFBSDI *DPOUSJCVUFEBSUJDMFTBCPVUTDJFOUJGJD DPNQVUJOHXJUI1ZUIPO *UbTPOTBMF"QSJM

Slide 3

Slide 3 text

5IJOHT*XBOUUPEP •  *OBEEJUJPOUPXPSLPONBDIJOFMFBSOJOH *`EMJLFUP JNQSPWFUIFNBDIJOFMFBSOJOHFYQFSJNFOU EFQMPZNFOU BOENPOJUPSJOHQSPDFTTGSPNUIFWJFXQPJOUPGTPGUXBSF FOHJOFFSJOH

Slide 4

Slide 4 text

1BTUXPSLFBTZ@OPUJGJFS •  4JNQMFEFDPSBUPSGVODUJPOUPOPUJGZPOTMBDL NBJM PS .BDEFTLUPQXIFOZPVSTDSJQUGJOJTIFE –  IUUQTHJUIVCDPNDIJFFBTZ@OPUJGJFS But Hyperdash is very convenient

Slide 5

Slide 5 text

5IFMJGFDZDMFPGUIFNBDIJOFMFBSOJOHFYQFSJNFOU •  8IFOXFXPSLPONBDIJOFMFBSOJOH XFMPPQGPMMPXJOH TUFQT –  %FDJEFFWBMVBUJPOJOEFY –  &YFDVUFFYQFSJNFOUT •  'FBUVSF&OHJOFFSJOH •  5SBJONPEFM •  &WBMVBUFNPEFM •  $IBOHFDPOEJUJPOT –  4VNNBSJ[FBOEWJTVBMJ[FUIFSFTVMUPGFYQFSJFODFTGPSSFQPSUTPS QBQFST Loop many times

Slide 6

Slide 6 text

"WBSJFUZPGQBUUFSOT •  $SFBUFGFBUVSFEBUBTFUGSPNIVHFEBUB –  )PXNVDIEBUBEPXFVTF –  8IBULJOEPGEBUBEPXFVTF BDUJPOMPH VTFSBOEJUFNDPOUFYUT JNBHFT FUD –  )PXUPJNQMFNFOUGFBUVSFFOHJOFFSJOH /PSNBMJ[BUJPO .JTTJOH WBMVFQSPDFTTJOH FUD •  5SZBWBSJFUZPGBMHPSJTNT •  1BSBNFUFS5VOJOH –  &HCBUDITJ[F MFBSOJOHSBUF FUD

Slide 7

Slide 7 text

%JGGJDVMUZJONBOBHJOHFYQFSJNFOUT •  8FIBWFUPNBOBHFTPNBOZJOGPSNBUJPO –  %P[FOTPGFYQFSJNFOUT –  .BOZQBSBNFUFST –  TPVSDFDPEFWFSTJPO –  &OWJSPONFOUBM*OGPSNBUJPO •  3FTPVSDFTJ[F •  6TF(16 •  'SBNFXPSLWFSTJPOT •  &YQFSJNFOUQJQFMJOFPGUFODPOUBJOTTFWFSBMOVNCFSPG TUFQT

Slide 8

Slide 8 text

*NQPSUBOUUIJOHTGPSFYQFSJNFOUNBOBHFNFOU •  4BWFBMMOFDFTTBSZJOGPSNBUJPOFGGJDJFOUMZ •  #FBCMFUPTFBSDIBOEBDDFTTQBTUJOGPSNBUJPOFBTJMZ •  #FBCMFUPTVNNBSJ[FTFWFSBMFYQFSJNFOUTFBTJMZ

Slide 9

Slide 9 text

&YJTUJOHXPSLT •  4BDSFE IUUQTHJUIVCDPN*%4*"TBDSFE •  '(-BC IUUQTHJUIVCDPN,BJYIJO'(-BC •  5IFZBSFUPPMTUPNBOBHFFYQFSJNFOUMPHT •  5IFZVTF.POHP%#GPSEBUBTUPDLFSBOEQSPWJEFTUIF SJDI6* •  *U`TWFSZDPOWFOJFOUGPSTDBMBCMFFYQFSJNFOUT –  4IBSFFYQFSJNFOUTXJUI.VMUJQMFFOWJSPONFOUT NVMUJQMF SFTFBSDIFS FUD •  #VUUIFZBSFPWFSTQFDGPSNFʜ –  /POFDFTTBSZPGSJDIQMBUGPSN –  *EPO`UXBOUUPNBJOUBJOB.POHP%# –  *EPO`UMJLFUPDIBOHFNZTDSJQUTGPSMPHNBOBHFNFOU

Slide 10

Slide 10 text

&YJTUJOHXPSLT •  $PNFUNM IUUQTXXXDPNFUNM –  $MPVE4FSWJDFGPSMPHUSBDLJOH –  4IBSFMPHT NFNPSBOEVNGVODUJPO FBTZUPVTF –  -PPLTHPPE #VUʜ –  8FDBO`UVQMPBEBEEJUJPOBMGJMFTTVDIBTUSBJOFENPEFMTPSEBUB –  /PDPMMBCPSBUJPOXJUIHJU –  8FIBWFUPUBLFDBSFOPUUPVQMPBEDPOGJEFOUJBMJOGPSNBUJPO

Slide 11

Slide 11 text

FYQTUPDL •  *DSFBUFEBUPPMXIJDITPMWFTUIFDPNQMFYJUZPGUIF FYQFSJNFOUNBOBHFNFOU –  IUUQTHJUIVCDPNDIJFFYQTUPDL

Slide 12

Slide 12 text

$PODFQUPGFYQTUPDL •  *UIBTWFSZTJNQMFBSDIJUFDUVSF –  /PEFQFOEFODZ –  /PNBJOUFOBODF •  8FEPO`UOFFEUPDIBOHFPVSTDSJQUT –  0OMZBEEJOHUXPPSUISFFDPNNBOET •  *UIBTHPPEMPH"DDFTTJCJMJUZXJUI7JTVBMJ[BUJPO –  8FDBODPNQBSFQBTUFYQFSJNFOUT –  8FDBOBDDFTTFBDIMPHGSPNBCSPXTFS

Slide 13

Slide 13 text

6TBHFPOGVODUJPOBMJ[FETDSJQUT *GZPVBMSFBEZIBWFGVODUJPOBMJ[FEUBSHFUQSPDFTT ZPVDBO VTFUIFGPMMPXJOHEFDPSBUPS from expstock import expstock a=1 b=2 e = expstock.ExpStock(exp_name=‘test_experiment’) @expstock.expstock(e) def run(a, b): return a + b e.append_param(a=a, b=b) run(a, b)

Slide 14

Slide 14 text

6TBHFXJUI+VQZUFS/PUFCPPL 8JUIJOUFSBDUJWFTDSJQUTFHFYQFSJNFOUTPO+VQZUFS /PUFCPPL ZPVDBOVTFUIFGPMMPXJOHDPNNBOE from expstock import expstock a=1 b=2 e = expstock.ExpStock(exp_name=‘test_experiment’) e.append_param(a=a, b=b) e.pre_stock() a + b e.post_stock()

Slide 15

Slide 15 text

%JSFDUPSZ4USVDUVSF -PHTBSFTUPDLFEBUFBDITVCEJSFDUPSZ experiments ᵓᴷᴷ _ ᴹ ᵓᴷᴷ exec_time.txt ᴹ ᵓᴷᴷ git_diff.txt ᴹ ᵓᴷᴷ git_head.txt ᴹ ᵓᴷᴷ machine_info.txt ᴹ ᵓᴷᴷ memo.txt ᴹ ᵓᴷᴷ params.txt ᴹ ᵓᴷᴷ report.txt ᴹ ᵓᴷᴷ result.txt ᴹ ᵓᴷᴷ stderr.txt ᴹ ᵋᴷᴷ stdout.txt

Slide 16

Slide 16 text

4BWFBEEJUJPOBMGJMFT •  :PVDBOTBWFBEEJUJPOBMGJMFTBUTBNFEJSFDUPSZ # save keras model from keras.models import load_model model_file = os.path.join(e.log_dirname, ‘model.h5’) model.save(model_file) # save matplotlib firure plt_png = os.path.join(e.log_dirname, ‘plt.png’) plt.savefig(plt_png)

Slide 17

Slide 17 text

FYQTUPDLTFSWFS •  &YQTUPDLIBTBTJNQMF6*JNQMFNFOUBUJPOXJUICPUUMF BOE42-JUF

Slide 18

Slide 18 text

'VUVSFXPSL •  *ODSFBTFMPHNFUSJDT –  .BDIJOF3FTPVSDFVUJMJ[BUJPO –  'SBNFXPSLWFSTJPO •  'VODUJPOUPFEJUNFUSJDTPO6* •  FUD