N o n -R es e ar c h Ti ps f or I nf or m ati o n S ci e nc e R es e ar c h ers C o n d u cti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms Y os hi a ki B a n d o S e ni or R es e arc h er AI R es e arc h C e nt er, AI S T, T o k y o

Y os hi a ki B a n d o, P h. D., S e ni or R es e ar c h er i n AI S T M ac hi n e list e ni n g f or r es c u e r o b ots, cr o w d r o b ots, a n d fi el d e x p eri m e nts • R es e ar c h f or a n al y zi n g n ot o nl y s p e e c h/ m usi c b ut als o f or e n vir o n m e nt al s o u n ds • Pr o b a bilisti c i nf er e nc e x d e e p l e ar ni n g  s elf-s u p er vis e d & e x pl ai n a bl e m o d els C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms R es c u e r o b ots Cr o w d r o b ots Fi el d e x p eri m e nts / 5 0 2

S o u n d S o ur c e S e p ar ati o n S o u n d s o ur c e s e p ar ati o n f or ms t h e b asis of m ac hi n e list e ni n g s yst e ms. • I n p ut: a mi xt ur e si g n al of m ulti pl e s o u n d s o ur c es • O ut p ut: i n di vi d u al s o ur c e si g n als o bs er v e d i n t h e i n p ut Dist a nt s p e e c h r e c o g niti o n ( D S R) [ W at a n a b e + 2 0 2 0, B a k er + 2 01 8] S o u n d e v e nt d et e cti o n ( S E D) [T ur p a ult + 2 0 2 0, D e nt o n + 2 0 2 2] C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 3

S elf -S u p er vis e d Tr ai ni n g of N e ur al S o ur c e S e p ar ati o n A m et h o d t o dis e nt a n gl e a mi xt ur e si g n al i nt o i n di vi d u al s o ur c e si g n als • W e tr ai n a v ari ati o n al a ut o e nc o d er ( V A E) f or m ulti c h a n n el mi xt ur e si g n als. T h e tr ai ni n g is p erf or m e d t o m a k e t h e r e c o nstr ucti o n cl os er t o t h e o bs er v ati o n. I nf er e nc e m o d el G e n er ati v e m o d el M ulti c h a n n el mi xt ur e M ulti c h a n n el r e c o nstr ucti o n ⋯ ⋯ L at e nt s o urc e f e at ur es ⋯ × × × ⋯ S C M S o ur c e P S D C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 4

D e m o: S e p ar ati n g S p e e c h Mi xt ur es I nt o I n di vi d u al S o ur c es T h e s p e e c h si g n als c a n b e e xtr act e d wit h o ut a n y m a n u al s u p er visi o n. • O ur st u di es w er e d o n e o n a n H P C s yst e m c all e d A B CI i n AI S T. • T o d a y, I will i ntr o d uc e h o w w e c o n d u ct o ur d ail y r es e ar c h. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms O bs er v e d mi xt ur e S e p ar at e d si g n al 1 S e p ar at e d si g n al 2 / 5 0 5

Y o u C a n A c c el er at e Y o ur R es e ar c h Wit h H P C S yst e ms A c c el er ati n g y o ur r es e ar c h c ycl e is ess e nti al i n y o ur ac a d e mi c lif e. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms Wit h o ut H P C s yst e ms D a y 1 D a y 2 D a y 3 D a y 4 ⋯ D e b u g Tr ai ni n g ( pr a yi n g ti m e) D e b u g Tr ai ni n g Wit h H P C s yst e ms D a y 1 D a y 2 D a y 3 D a y 4 ⋯ D e b u g Tr ai ni n g D e b u g D e b u g Tr ai ni n g Tr ai ni n g D e b u g Tr ai ni n g D e b u g Tr ai ni n g D e b u g … … / 5 0 6

T o d a y’s T o pi cs T hr e e t o pi cs f or c o n d u cti n g AI r es e ar c h o n H P C s yst e ms C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms P art 1: W h at is t h e H P C E n vir o n m e nt? P art 2: Writi n g F ast P yt h o n S cri pts P art 3: M a n a gi n g P yt h o n S cri pts f or R es e ar c h / 5 0 7

P art 1: W h at is t h e H P C E n vir o n m e nt? C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 3 3 8

H P C S yst e ms f or AI R es e ar c h C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms M a n y r es e ar c h i nstit ut es/ u ni v ersiti es o p er at e H P C s yst e ms f or AI R es e ar c h • T h e y t y pi c all y pr o vi d e n u m er o us G P Us s uit a bl e f or d e e p l e ar ni n g a n d its i nf er e nc e. A B CI 2. 0 i n AI S T T S U B A M E 4. 0 i n T o k y o T e c h Wist eri a/ B D E C -01 i n U T o k y o htt ps:// w w w.tit e c h. ac.j p/ n e ws/ 2 0 2 3/ 0 6 6 7 8 2 htt ps:// w w w.c c. u -t o k y o. ac.j p/ htt ps:// a b ci. ai/ • 9 6 0 N VI DI A A1 0 0 G P Us • 4, 3 5 2 N VI DI A V1 0 0 G P Us • 3 6 0 N VI DI A A1 0 0 G P Us • 9 6 0 N VI DI A H1 0 0 G P Us / 5 0 9

Ty pic al C o nfi g ur ati o n of AI H P C S yst e ms T h e y t y pi c all y c o nsist of a b u nc h of hi g h -p erf or m a nc e c o m p ut e n o d es. • W e us u all y S S H i nt o a n e ntr a nc e n o d e (s er v er) a n d r e q u est t o r u n a pr o gr a m (j o b). C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms Y o ur c o m p ut er C o m p ut e n o d es E ntr a nc e n o d e S S H S u b mit a j ot / 5 0 1 0

H o w t o R u n Y o ur J o bs T h er e will b e a j o b m a n a g e m e nt s er vi c e t o r e q u est y o ur j o b. • Y o u d o n’t h a v e t o c ar e a b o ut w h er e y o ur j o b is e x e c ut e d. • All n o d es ar e c o nfi g ur e d t o h a v e t h e s a m e b e h a vi or a n d s h ar e t h e s a m e fil es yst e m. M a n y s yst e ms h a v e t w o t y p es of j o bs: C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms 2. I nt er acti v e j o b (f or d e b u g gi n g) • A si mil ar b e h a vi or w h e n y o u S S H i nt o y o ur c o m p ut e s er v er. • A c o m p ut e r es o ur c e is all o c at e d wit h a n i nt er a cti v e c o m m a n d pr o m pt. 1. B at c h j o b ( us u al us a g e) • Y o u n e e d t o writ e a (s h ell) s cri pt t o b e e x e c ut e d. • T h e s cri pt is e x e c ut e d o n a c o m p ut e r es o ur c e y o u r e q u est e d. / 5 0 11

Ty pic al Diff er e n c e B et w e e n A B CI a n d C o m p ut ers i n Y o ur L a bs Al m ost t h e s a m e as y o ur c o m p ut ers, b ut t h er e ar e a f e w t hi n gs t o k e e p i n mi n d. • E ac h n o d e h as us u al C P U & R A M, b ut G P Us ar e li n k e d t o e a c h ot h er vi a f ast N V Li n k. • N o d es ar e c o n n e ct e d b y I nfi ni B a n d, w hi c h h a v e 8 0 0 ( 2 0 0) G b ps  Us u all y ≤ 1 0 G b ps T h e ar c hit e ct ur e is d esi g n e d t o p erf or m distri b ut e d tr ai ni n g v er y effici e ntl y. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms N o d e N o d e St or a g e H D D H D D H D D H D D H D D H D D H D D H D D St or a g e H D D H D D H D D H D D H D D H D D S S D S S D St or a g e S S D S S D S S D S S D S S D S S D S S D S S D I nfi ni B a n d ( E D R/ H D R) N et w or k C o m p ut e N o d e ( A/ V) C P U C P U R A M R A M G P U G P U G P U G P U N V Li n k ⋯ / 5 0 1 2

Distri b ut e d D at a P ar all el ( D D P) A m et h o d t o us e m ulti pl e G P Us/ n o d es f or tr ai ni n g a n e ur al n et w or k • W e c a n i m pr o v e t h e t hr o u g h p ut ( # of s a m pl es/ti m e) b y usi n g m ulti pl e G P Us. • M ost of t h e pr o c e d ur es e x c e pt f or a v er a gi n g t h e gr a di e nts c a n b e p ar all eli z e d. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms D at a l o a di n g F or w ar d B a c k w ar d M o d el u p d at e D at a l o a di n g F or w ar d B a c k w ar d M o d el u p d at e D at a l o a di n g F or w ar d B a c k w ar d M o d el u p d at e All -S c att er ( S y n c hr o ni z ati o n) G P U _1 G P U _ 2 G P U _ N ⋯ ⋯ ⋯ ⋯ / 5 0 1 3

H o w t o Us e D D P i n Y o ur ( P y T or c h) Tr ai ni n g S cri pt D D P c a n b e e asil y i ntr o d u c e d t o y o ur tr ai ni n g pi p eli n e writt e n wit h P y T or c h. • W h at y o u n e e d is j ust a d di n g f oll o wi n g li n es t o y o ur tr ai ni n g s cri pt. • It is m u c h e as y t o us e D D P i n wr a p p er li br ari es s u c h as P y T or c h Li g ht ni n g C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms # i n i t i a l i z e p r o c e s s g r o u p w o r l d _ s i z e = i n t ( o s . e n v i r o n [ " O M P I _ C O M M _ W O R L D _ S I Z E " ] ) r a n k = i n t ( o s . e n v i r o n [ " O M P I _ C O M M _ W O R L D _ R A N K " ] ) l o c a l _ r a n k = i n t ( o s . e n v i r o n [ ' O M P I _ C O M M _ W O R L D _ L O C A L _ R A N K ’ ] ) i n i t _ m e t h o d = f " t c p : / / { a r g s . m a s t e r _ a d d r } : { a r g s . m a s t e r _ p o r t } " d i s t . i n i t _ p r o c e s s _ g r o u p ( b a c k e n d = " n c c l " , i n i t _ m e t h o d = i n i t _ m e t h o d , w o r l d _ s i z e = w o r l d _ s i z e , r a n k = r a n k ) t o r c h . c u d a . s e t _ d e v i c e ( l o c a l _ r a n k ) # i n i t i a l i z e m o d e l o r i g _ m o d e l = X X X M o d e l ( ) m o d e l = D i s t r i b u t e d D a t a P a r a l l e l ( t o r c h . n n . S y n c B a t c h N o r m . c o n v e r t _ s y n c _ b a t c h n o r m ( o r i g _ m o d e l ) , d e v i c e _ i d s = [ l o c a l _ r a n k ] ) # i n i t i a l i z e d a t a l o a d e r t r a i n _ d a t a = t o r c h . u t i l s . d a t a . D a t a L o a d e r ( d a t a s e t = t r a i n _ d a t a s e t , . . . , s a m p l e r = D i s t r i b u t e d S a m p l e r ( t r a i n _ d a t a s e t ) ) A N VI DI A’s c o m m u ni c ati o n b a c k e n d, i. e., s u p er f ast b ac k e n d. B at c h N or m r e q uir es s y nc hr o ni z ati o n o v er G P Us. A s a m pl er t h at s plits d at a i nt o m ulti pl e n o d es * Y o u h a v e t o m a n u all y s y nc hr o ni z e tr ai ni n g m etrics if y o u n e e d e d ( e. g., v ali d ati o n s c or es, l oss v al u es t o m o nit or). / 5 0 1 4

B e A w ar e of T h e Hi er ar c hi c al Str u ct ur e of D at a St or a g e T h er e is a tr a d e off b et w e e n “s m all f ast ” vs. “l ar g e sl o w ” i n d at a st or a g e. • M ost m ac hi n e l e ar ni n g a p ps. sl o w d o w n d u e t o d at a a cc ess r at h er t h a n c o m p ut ati o n. T h er e is a diff er e n c e of m or e t h a n 1 0 ti m es b et w e e n R A Ms a n d st or a g es. Ty p e C a p a cit y S p e e d G P U R A M 1 6 or 4 0 G B/ G P U 1, 5 5 5 G B/s C P U R A M 3 6 0 or 4 8 0 G B/ n o d e 2 0 4 G B/s L oc al N V M e st or a g e 1. 6 - 2 (?) T B/ n o d e 3. 2 G B/s B e e O N D 1. 6 - 2 (?) T B/ n o d e # n o d e x 3. 2 G B/s H o m e st or a g e (/ h o m e) 2 0 0 G B * 4. 7 G B/s Gl o b al s cr at c h st or a g e (/s cr at c h) 1 0 T B * 4. 8 G B/s Gr o u p st or a g e (/ gr o u ps) -2 5 0 T B * 2. 5 G B/s C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms Cit e d fr o m htt ps://s p e a k er d e c k.c o m/ y os hi p o n/f ast est -d e e p -l e ar ni n g-o n -a bci, * m e a ns m e as ur e d v al u es / 5 0 1 5

Ti ps 1 -1: R e d u c e t h e St or a g e A c c ess as M u c h as P ossi bl e L et’s st or e all y o ur tr ai ni n g d at a o n R A Ms r at h er t h a n t h e st or a g e • M a n y H P C S yst e ms h a v e e n o u g h R A M t o st or e y o ur tr ai ni n g d at a. • T h e e asi est w a y is t o c o p y y o ur d at a o n t m pfs, a R A M fil es yst e m. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms u s e r @ y o u r _ s y s t e m : ~ $ d f - h F i l e s y s t e m S i z e U s e d A v a i l U s e % M o u n t e d o n d e v t m p f s 1 8 9 G 0 1 8 9 G 0 % / d e v t m p f s 1 8 9 G 0 1 8 9 G 0 % / d e v / s h m t m p f s 1 8 9 G 4 . 1 G 1 8 5 G 3 % / r u n t m p f s 1 8 9 G 0 1 8 9 G 0 % / s y s / f s / c g r o u p / d e v / s d a 4 2 2 G 1 7 G 5 . 8 G 7 5 % / / d e v / n v m e 0 n 1 1 . 5 T 1 1 G 1 . 5 T 1 % / l o c a l / d e v / s d a 1 5 1 1 M 5 . 8 M 5 0 6 M 2 % / b o o t / e f i / d e v / s d a 5 1 6 G 1 4 7 M 1 6 G 1 % / t m p / d e v / s d a 2 4 4 G 5 . 8 G 3 9 G 1 4 % / v a r / d e v/s h m is a dir ect or y t o s h ar e fil es a m o n g pr o c ess es. / 5 0 1 6

B e A w ar e of T h e C h ar a ct eristics of Fil e S yst e m M a n y H P C S yst e ms ar e b uilt wit h H P C -ori e nt e d fil e s yst e m c all e d L ustr e . • It c o nsists of m et a d at a s er v ers ( M D Ss) a n d o bj e ct st or a g e s er v ers ( O S Ss). • Us u all y # of M D Ss ar e m u c h s m all er t h a n # of O S Ss. A c c essi n g o nl y O S Ss as m u c h as p ossi bl e is i m p ort a nt f or hi g h t hr o u g h p ut. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms C o m p ut e n o d e C o m p ut e n o d e O S S O S S O S S O S S M D S M D S N et w or k • Fil e n a m es • L oc ati o ns of fil es st or e d • Dir e ct ori es • A ct u al fil e d at a / 5 0 1 7

Ti ps 1 -2: L et’s Us e Hi er ar c hic al D at a F or m at ( H D F 5) H D F 5 is a c o nt ai n er f or m at t o st or e s ets of m a n y ki n ds of d at a. • P utti n g tr ai ni n g s a m pl es i nt o a si n gl e fil e c a n r e d u c e t h e a cc ess t o M D S. • It is v er y e as y t o h a n dl e fr o m P yt h o n b y usi n g h 5 p y. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms It’s li k e cr e ati n g a dir e ct or y Cr e ati n g fil es i n “s a m pl e1 ” / 5 0 1 8

Ti ps 1 -3: E n a bl e t h e Stri pi n g of t h e St or a g e Writi n g t o/r e a di n g fr o m m ulti pl e dis k u nits, c all e d o bj e ct st or a g e t ar g ets. • S o m et hi n g li k e R AI D -0 o n t h e L ustr e fil e s yst e m. • It is q uit e i m p ort a nt f or d at a r e a d b y m ulti pl e n o d es ( e. g., c o nt ai n er i m a g e fil es) C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms u s e r @ y o u r _ s y s t e m : ~ $ m k d i r t m p u s e r @ y o u r _ s y s t e m : ~ $ t o u c h t m p / a a u s e r @ y o u r _ s y s t e m : ~ $ l f s g e t s t r i p e t m p / a a t m p / a a l m m _ s t r i p e _ c o u n t : 1 l m m _ s t r i p e _ s i z e : 1 0 4 8 5 7 6 l m m _ p a t t e r n : r a i d 0 l m m _ l a y o u t _ g e n : 0 l m m _ s t r i p e _ o f f s e t : 0 o b d i d x o b j i d o b j i d g r o u p 0 1 2 3 9 6 0 7 7 7 0 x 7 6 3 7 d c 9 0 x 3 8 0 0 0 0 4 0 0 u s e r @ y o u r _ s y s t e m : ~ $ m k d i r t m p u s e r @ y o u r _ s y s t e m : ~ $ l f s s e t s t r i p e - S 1 m - i 1 0 - c 4 s t r i p e - f i l e t m p u s e r @ y o u r _ s y s t e m : ~ $ t o u c h t m p / a a u s e r @ y o u r _ s y s t e m : ~ $ l f s g e t s t r i p e t m p / a a t m p / a a l m m _ s t r i p e _ c o u n t : 4 l m m _ s t r i p e _ s i z e : 1 0 4 8 5 7 6 l m m _ p a t t e r n : r a i d 0 l m m _ l a y o u t _ g e n : 0 l m m _ s t r i p e _ o f f s e t : 1 0 o b d i d x o b j i d o b j i d g r o u p 1 0 1 2 4 0 4 3 9 0 3 0 x 7 6 4 c 2 7 f 0 x 6 0 0 0 0 0 4 0 0 1 1 1 2 3 1 5 7 8 7 0 0 x 7 5 7 3 d 6 e 0 x 6 4 0 0 0 0 4 0 0 1 2 1 2 4 2 3 2 7 4 2 0 x 7 6 7 a 4 2 6 0 x 6 8 0 0 0 0 4 0 3 1 3 1 2 4 6 7 6 2 8 5 0 x 7 6 e 6 8 b d 0 x 6 c 0 0 0 0 4 0 0 Wit h o ut stri pi n g Wit h stri pi n g htt ps:// d ocs. a b ci. ai/j a/st or a g e/ # a d v a nc e d -o pti o n -fil e-stri pi n g / 5 0 1 9

S u m m ar y of P art 1: W h at is t h e H P C E n vir o n m e nt? T h e y t y pi c all y c o nsist of a b u nc h of hi g h -p erf or m a nc e c o m p ut e n o d es. • S p e ci ali z e d f or distri b ut e d tr ai ni n g  L et’s us e D D P tr ai ni n g • Hi er ar c hi c al Str uct ur e of D at a St or a g e  L et’s us e R A Ms as m u c h as p ossi bl e • S p e ci al fil e s yst e m c all e d L ustr e  L et’s us e H D F 5 or s o m e d at a c o nt ai n ers C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms N o d e N o d e St or a g e H D D H D D H D D H D D H D D H D D H D D H D D St or a g e H D D H D D H D D H D D H D D H D D S S D S S D St or a g e S S D S S D S S D S S D S S D S S D S S D S S D I nfi ni B a n d ( E D R/ H D R) N et w or k C o m p ut e N o d e ( A/ V) C P U C P U R A M R A M G P U G P U G P U G P U N V Li n k / 5 0 2 0

P art 2: Writi n g F ast P yt h o n S cri pts C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 3 3 21

P yt h o n vs. C/ C + +/ R ust W hi c h l a n g u a g e d o y o u pr ef er? P yt h o n is a gl u e f or r a pi d pr ot ot y pi n g b y c o n n e cti n g m o d ul es/c o m p o n e nts. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms P yt h o n is q uit e sl o w, l et’s writ e C/ C + +/ R ust pr o gr a ms! M e It’s tr u e. F or pr o d u cti o n, C/ C + +/ R ust ar e b est. B ut I t hi n k P yt h o n is b est f or r es e ar c h, w hi c h n e e ds a l ot of r a pi d tri al a n d err or. M a n y s u p er e n gi n e ers P yt h o n is a m o d er n/s ci e ntifi c s h ell s cri pt. / 5 0 2 2

M ost of t h e P yt h o n li br ari es ar e wr a p p ers of f ast C/ C + + li br ari es. • It is ess e nti al t o t a k e f ull a d v a nt a g e of b a c k e n d li br ari es writt e n i n C/ C + +/ F ortr a n. Ty pi c al Str u ct ur es of P yt h o n Li br ari es N VI DI A G P U c u D N N M A G M A c u B L A S At e n/c1 0 ( C/ C + +) P y T or c h ( P yt h o n) m ai n. p y I nt el C P U L A P A C K B L A S N u m P y ( P yt h o n / C / C + + / F ortr a n) m ai n. p y H ar d w ar e F ast li br ar y writt e n i n C/ C + +/ F ortr a n P yt h o n wr a p p er Us er pr o gr a m Pr o gr a ms r a n o n C P Us Pr o gr a ms r a n o n G P Us O S ・ Dri v ers ・ R u nti m es C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 2 3

A v oi d Writi n g P yt h o n as M u c h as P ossi bl e L et’s r e a d r ef er e n c e d oc u m e nts t o fi n d & us e f ast er r o uti n es. • P yt h o n is ess e nti all y sl o w l a n g u a g e b e c a us e it is e x e c ut e d o n a virt u al m a c hi n e ( P V M). • C o m m o n r o uti n es ar e i m pl e m e nt e d f ast er b y N u m P y a n d P y T or c h. If y o u n e e d t o writ e d o u bl e f or l o o p, l et’s r e a d a r ef er e nc e d oc u m e nt! F or J a p a n es e st u d e nts, pl e as e r e a d E n glis h o n e b e c a us e J a p a n es e o n es ar e oft e n ol d a n d i ncl u d e err ors. A = n p . r a n d o m . r a n d o m ( [ 1 0 0 , 1 0 0 ] ) B = n p . r a n d o m . r a n d o m ( [ 1 0 0 , 1 0 0 ] ) s t a r t _ t i m e = t i m e . t i m e ( ) Y = n p . z e r o s ( [ 1 0 0 , 1 0 0 ] ) f o r i , j i n i t e r t o o l s . p r o d u c t ( r a n g e ( 1 0 0 ) , r a n g e ( 1 0 0 ) ) : f o r k i n r a n g e ( 1 0 0 ) : Y [ i , j ] + = A [ i , k ] * B [ k , j ] d u r a t i o n = t i m e . t i m e ( ) - s t a r t _ t i m e p r i n t ( d u r a t i o n ) A = n p . r a n d o m . r a n d o m ( [ 1 0 0 , 1 0 0 ] ) B = n p . r a n d o m . r a n d o m ( [ 1 0 0 , 1 0 0 ] ) s t a r t _ t i m e = t i m e . t i m e ( ) Y = A @ B d u r a t i o n = t i m e . t i m e ( ) - s t a r t _ t i m e p r i n t ( d u r a t i o n ) El a ps e d ti m e  4 0 0 ms El a ps e d ti m e  3 ms !! * M e as ur e d o n D ell X P S 1 3 9 31 0 (i 7 -11 8 5 G 7) @ 2 0 2 2/ 0 4/ 0 6 C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 2 4

I nt el M K L f or C P Us a n d N VI DI A c u X X X f or G P U ar e t h e b est b ac k e n ds. • I n m a n y c as es, t h e f ast est li br ar y is o n e pr o vi d e d b y t h e H W m a n uf a ct ur er. • U ni v ersit y st u d e nts c a n us e A n a c o n d a, w h os e N u m P y/ S ci P y us es I nt el M K L b y d ef a ult. Ti ps 2 -1: Pl e as e C h ec k If Y o u Ar e Usi n g F ast er B a c k e n ds Pl e as e c h ec k if t h er e is “ m kl ” or n ot. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms N o n -ac a d e mic i nstit ut es n e e d p ai d lic e ns es / 5 0 2 5

Ti ps 2 -2: Utili z e G P G P U as M uc h as P ossi bl e H P C e n vir o n m e nt ( e. g., A B CI) us u all y h as m or e G P Us ( > 5 k) t h a n C P Us ( 2. 4 k). • Y o u c o ul d s p e e d u p y o ur pr o gr a m m or e t h a n 1 0 ti m es f ast er b y j ust r e writi n g f or G P Us. • Y o u c a n writ e al m ost t h e s a m e w a y as N u m P y b y usi n g C u P y f r o m o p t _ e i n s u m i m p o r t c o n t r a c t x = n p . r a n d o m . r a n d o m ( [ 1 0 0 0 0 , 5 0 ] ) # [ N , D ] z = n p . r a n d o m . r a n d o m ( [ 1 0 0 0 0 , 1 0 ] ) # [ N , K ] z / = z . s u m ( a x i s = - 1 , k e e p d i m s = T r u e ) e I = 1 e - 8 * n p . e y e ( x . s h a p e [ 1 ] ) f o r i i i n r a n g e ( 1 0 ) : z s u m = z . s u m ( a x i s = 0 ) . c l i p ( 1 e - 8 ) p i = z s u m / z . s h a p e [ 0 ] # [ K ] m u = c o n t r a c t ( " n k , n d - > k d " , z , x ) / z s u m [ : , N o n e ] # [ K , D ] x m u = x [ : , N o n e ] - m u # [ N , K , D ] V = c o n t r a c t ( " n k , n k d , n k e - > k d e " , z , x m u , x m u ) / z s u m [ : , N o n e , N o n e ] + e I V i , ( _ , l d V ) = n p . l i n a l g . i n v ( V ) , n p . l i n a l g . s l o g d e t ( V ) l o g z = n p . l o g ( p i ) - 0 . 5 * ( l d V + c o n t r a c t ( " n k d , k d e , n k e - > n k " , x m u , V i , x m u ) ) z = n p . e x p ( l o g z - l o g s u m e x p ( l o g z , a x i s = - 1 , k e e p d i m s = T r u e ) ) f r o m o p t _ e i n s u m i m p o r t c o n t r a c t x = c p . r a n d o m . r a n d o m ( [ 1 0 0 0 0 , 5 0 ] ) # [ N , D ] z = c p . r a n d o m . r a n d o m ( [ 1 0 0 0 0 , 1 0 ] ) # [ N , K ] z / = z . s u m ( a x i s = - 1 , k e e p d i m s = T r u e ) e I = 1 e - 8 * c p . e y e ( x . s h a p e [ 1 ] ) f o r i i i n r a n g e ( 1 0 ) : z s u m = z . s u m ( a x i s = 0 ) . c l i p ( 1 e - 8 ) p i = z s u m / z . s h a p e [ 0 ] # [ K ] m u = c o n t r a c t ( " n k , n d - > k d " , z , x ) / z s u m [ : , N o n e ] # [ K , D ] x m u = x [ : , N o n e ] - m u # [ N , K , D ] V = c o n t r a c t ( " n k , n k d , n k e - > k d e " , z , x m u , x m u ) / z s u m [ : , N o n e , N o n e ] + e I V i , ( _ , l d V ) = c p . l i n a l g . i n v ( V ) , c p . l i n a l g . s l o g d e t ( V ) l o g z = c p . l o g ( p i ) - 0 . 5 * ( l d V + c o n t r a c t ( " n k d , k d e , n k e - > n k " , x m u , V i , x m u ) ) z = c p . e x p ( l o g z - l o g s u m e x p ( l o g z , a x i s = - 1 , k e e p d i m s = T r u e ) )  E M -G M M writt e n wit h N u m P y  9, 9 5 0 ms  E M -G M M writt e n wit h C u P y  5 0 4 ms !! ※ M e as ur e d o n a V1 0 0 n o d e (rt _ F =1) of A B CI @ 2 0 2 2/ 0 4/ 0 6 C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 2 6

If Y o ur P yt h o n Pr o gr a m is Still Sl o w, L et’s Writ e B a c k e n d! T h er e ar e s e v er al m et h o ds t o writ e b ac k e n d li br ar y b y y o urs elf. • I us u all y writ e pr o gr a ms wit h P yt h o n a n d t h e n r e writ e t h e sl o w est p art as f oll o ws. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms F or C P U b ac k e n ds 1. N u m b a JI T: t h e e asi est w a y t o s p e e d u p P yt h o n s cri pts. 2. C yt h o n : a w a y t o s p e e d u p wit h m o d er at e m o difi c ati o n. 3. C + + wit h P y Bi n d11: t h e l ast o pti o n, w hi c h n e e ds r e as o n a bl e c ost. F or G P U ( C U D A) b ac k e n ds 1 . c u p y . E l e m e n t w i s e K e r n e l : t h e e asi est w a y t o writ e a C U D A k er n el 2. Writi n g C U D A k er n el i n C + +: t h e l ast a n d m ost e x p e nsi v e w a y t o writ e. / 5 0 2 7

N u m b a JI T A J ust -I n-Ti m e (JI T) C o m pil er f or P yt h o n s cri pts • Y o ur c o d e will s p e e d u p b y j ust a d di n g @ n u m b a . n j i t d e c or at or. • It is us ef ul f or s p e e di n g u p f or-l o o ps ( e. g., hi d d e n M ar k o v m o d els). C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms T h e first r u n n e e ds ti m e t o c o m pil e. htt ps:// n u m b a. p y d at a. or g/ n u m b a -d oc/l at est/ us er/ 5 mi n g ui d e. ht m / 5 0 2 8

C yt h o n A P yt h o n -li k e l a n g u a g e t h at c a n b e c o m pil e d i n a d v a nc e. • Y o u n e e d t o e x pli citl y d efi n e t h e t y p es of v ari a bl es s u c h as i nt, fl o at, str, … C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms c p d e f i n t f i b o n a c c i ( i n t n ) : i f n < = 0 : r e t u r n 0 e l i f n = = 1 : r e t u r n 1 c d e f i n t f i b _ 1 = 0 c d e f i n t f i b _ 2 = 1 c d e f i n t t m p f o r i i n r a n g e ( 2 , n + 1 ) : t m p = f i b _ 1 + f i b _ 2 f i b _ 1 = f i b _ 2 f i b _ 2 = t m p r e t u r n f i b _ 2 u s e r @ y o u r _ s y s t e m : ~ $ c y t h o n i z e - a - i f i b o n a c c i . p y x u s e r @ y o u r _ s y s t e m : ~ $ l s B u i l d f i b o n a c c i . c f i b o n a c c i . c p y t h o n - 3 1 0 - x 8 6 _ 6 4 - l i n u x - g n u . s o f i b o n a c c i . h t m l fi b o n a cci. p y x I n y o ur t er mi n al T h e c o m pil e d li br ar y c a n b e i m p ort e d i n P yt h o n / 5 0 2 9

c u p y. El e m e nt wis e K er n el A n e asi est w a y t o writ e a n el e m e nt -wis e k er n el (f u ncti o n) f or G P Us • J ust d efi ni n g i n p ut/ o ut p ut p ar a m et ers a n d t ar g et r o uti n e i n C/ C + + C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 3 0 i m p o r t c u p y a s c p a d d _ f u n c t i o n = c p . E l e m e n t w i s e K e r n e l ( i n _ p a r a m s = " f l o a t 3 2 x , f l o a t 3 2 y " , o u t _ p a r a m s = " f l o a t 3 2 z " , o p e r a t i o n = " z = x + y " ) d e f m a i n ( ) : x , y = c p . o n e s ( 1 0 , d t y p e = c p . f l o a t 3 2 ) , c p . a r a n g e ( 1 0 , d t y p e = c p . f l o a t 3 2 ) z = a d d _ f u n c t i o n ( x , y ) p r i n t ( " x : " , x ) p r i n t ( " y : " , y ) p r i n t ( " z : " , z ) i f _ _ n a m e _ _ = = " _ _ m a i n _ _ " : m a i n ( ) u s e r @ y o u r _ s y s t e m : ~ $ p y t h o n c u p y _ t e s t . p y x : [ 1 . 1 . 1 . 1 . 1 . 1 . 1 . 1 . 1 . 1 . ] y : [ 0 . 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . ] z : [ 1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 . 1 0 . ] I n y o ur t er mi n al c u p y _t est. p y

Ti ps 2 -3: Us e C u P y a n d P y T or c h i n a Si n gl e Pr o gr a m Y o u c a n writ e f or w ar d/ b ac k w ar d f u ncti o ns of P y T or c h b y usi n g C u P y • C u P y’s arr a ys c a n b e c o n v ert e d t o P y T or c h t e ns ors wit h z er o -c o p y. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms i m p o r t t o r c h f r o m p y t o r c h _ p f n _ e x t r a s i m p o r t f r o m _ n d a r r a y , a s _ n d a r r a y i m p o r t c u p y c l a s s C u p y I n v F u n c t i o n ( t o r c h . a u t o g r a d . F u n c t i o n ) : @ s t a t i c m e t h o d d e f f o r w a r d ( c t x , x _ : t o r c h . T e n s o r ) - > t o r c h . T e n s o r : x = a s _ n d a r r a y ( x _ ) x i = c u p y . l i n a l g . i n v ( x ) c t x . s a v e d _ c u p y _ a r r a y s = x i , r e t u r n f r o m _ n d a r r a y ( x i ) @ s t a t i c m e t h o d d e f b a c k w a r d ( c t x , g y _ : t o r c h . T e n s o r ) - > t u p l e [ t o r c h . T e n s o r ] : x i , = c t x . s a v e d _ c u p y _ a r r a y s g y = a s _ n d a r r a y ( g y _ ) g x = - c u p y . e i n s u m ( " . . . n m , . . . n o , . . . p o - > . . . m p " , x i . c o n j ( ) , g y , x i . c o n j ( ) ) # x i ^ H @ g y @ x i ^ H r e t u r n f r o m _ n d a r r a y ( g x ) , i n v = C u p y I n v F u n c t i o n . a p p l y / 5 0 31

Pr ofil ers f or Effi ci e ntl y S p e e di n g U p Y o ur Pr o gr a m A t o ol t o m e as ur e t h e el a ps e d ti m e f or e ac h p art of y o ur pr o gr a m. • T h e g ol d e n r ul e is t o s p e e d u p t h e h e a vi est p art first. • Y o ur pr e di cti o n is oft e n diff er e nt fr o m t h e o bs er v ati o n. I m a d e t h e m atri x i n v ersi o n 1 0 0 ti m es f ast er! M e A B CI M ost of t h e ti m e w as s p e nt b y I/ O … C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 3 2

Y o u c a n m e as ur e t h e el a ps e d ti m es li n e -b y -li n e wit h @ p r o f i l e d e c or at or • It c a n b e i nst all e d b y p i p i n s t a l l l i n e _ p r o f i l e r (htt ps:// git h u b. c o m/ p y utils/li n e _ pr ofil er ) Ti ps 2 -5: Us e li n e _ pr ofil er x = n p . r a n d o m . r a n d o m ( [ 1 0 0 0 0 , 5 0 ] ) # [ N , D ] @ p r o f i l e d e f t a r g e t ( ) : z = n p . r a n d o m . r a n d o m ( [ 1 0 0 0 0 , 1 0 ] ) # [ N , K ] z / = z . s u m ( a x i s = - 1 , k e e p d i m s = T r u e ) e I = 1 e - 8 * n p . e y e ( x . s h a p e [ 1 ] ) f o r i i i n r a n g e ( 1 0 ) : z s u m = z . s u m ( a x i s = 0 ) . c l i p ( 1 e - 8 ) p i = z s u m / z . s h a p e [ 0 ] # [ K ] m u = c o n t r a c t ( " n k , n d - > k d " , z , x ) / z s u m [ : , N o n e ] # [ K , D ] x m u = x [ : , N o n e ] - m u # [ N , K , D ] V = c o n t r a c t ( " n k , n k d , n k e - > k d e " , z , x m u , x m u ) / z s u m [ : , N o n e , N o n e ] + e I V i , ( _ , l d V ) = n p . l i n a l g . i n v ( V ) , n p . l i n a l g . s l o g d e t ( V ) l o g z = n p . l o g ( p i ) - 0 . 5 * ( l d V + c o n t r a c t ( " n k d , k d e , n k e - > n k " , x m u , V i , x m u ) ) z = n p . e x p ( l o g z - l o g s u m e x p ( l o g z , a x i s = - 1 , k e e p d i m s = T r u e ) ) t a r g e t ( ) 7 5 % of t h e ti m e w as s p e nt h er e. * M e as ur e d o n D ell X P S 1 3 9 31 0 (i 7 -11 8 5 G 7) @ 2 0 2 2/ 0 4/ 0 7 C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 3 3

Ti ps 2 -5: Us e t or c h. pr ofil er f or P y T or c h pr o gr a ms C ar e s h o ul d b e t a k e n i n m e as uri n g t h e el a ps e d ti m e f or G P U pr o gr a ms. • T h e C U D A ( G P U) a n d C P U pr o gr a ms ar e as y nc hr o n o usl y e x e c ut e d. • W e c a n n ot n ai v el y us e ti m e/ pri nt f u ncti o ns f or m e as uri n g t h e el a ps e d ti m e. E v e nts o n G P U E v e nts o n C P U F u ncti o n n a m e D et ails b y cli c ks o n t h e e v e nts htt ps:// p yt orc h. or g/t ut ori als/r e ci p es/r e ci p es/ pr ofil er _r e ci p e. ht ml C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 5 0 3 4

H y p er -P ar a m et er O pti mi z ati o n ( H P O) A t as k t o o pti mi z e a n o n -diff er e nti a bl e o bj e cti v e f u ncti o n. • M a n y s ol v ers s uit a bl e t o m a c hi n e l e ar ni n g m o d els h as b e e n pr o p os e d. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms B a y esi a n o pti mi z ati o n N el d er -M e a d (si m pl e x) m et h o d htt p:// o nis hi -l a b.j p/j oi n us/i nt er n/i n d e x. ht ml / 5 0 3 5

Ti ps 2 -6: Us e m pi 4 p y f or A cc el er ati n g H P O W e c a n e asil y p ar all eli z e i nf er e nc e b y usi n g m pi 4 p y • M PI P o ol E x e c ut or c a n b e us e d i n t h e s a m e w a y as i n T hr e a d P o ol E x e c ut or C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms f r o m m p i 4 p y . f u t u r e s i m p o r t M P I P o o l E x e c u t o r i m p o r t o p t u n a d e f i n f e r e n c e ( f i l e n a m e ) : . . . r e t u r n r e s u l t d e f m a i n ( a r g s ) : s t u d y = o p t u n a . c r e a t e _ s t u d y ( ) w i t h M P I P o o l E x e c u t o r ( ) a s p o o l : f o r n n i n r a n g e ( a r g s . n _ t r i a l s ) : t r i a l = s t u d y . a s k ( ) h p = { } h p [ k e y ] = t r i a l . s u g g e s t _ f l o a t ( k e y : = " p a r a m 1 " , 0 . 0 , 1 . 0 ) h p [ k e y ] = t r i a l . s u g g e s t _ f l o a t ( k e y : = " p a r a m 2 " , 0 . 0 , 1 . 0 ) h p [ k e y ] = t r i a l . s u g g e s t _ f l o a t ( k e y : = " p a r a m 3 " , 0 . 0 , 1 . 0 ) r e s u l t s = n p . s t a c k ( l i s t ( p o o l . m a p ( p a r t i a l ( i n f e r e n c e , h p = h p ) , f i l e n a m e _ l i s t ) ) , d i m = 0 ) s c o r e = e v a l u a t e _ s c o r e ( r e s u l t s ) f r o z e n _ t r i a l = s t u d y . t e l l ( t r i a l , s c o r e ) P ar all eli z e wit h m ulti -n ot e x m ulti -pr o c ess / 5 0 3 6

S u m m ar y of P art 2: Writi n g F ast P yt h o n S cri pts P yt h o n is a gl u e f or r a pi d pr ot ot y pi n g b y c o n n e cti n g m o d ul es/c o m p o n e nts. • L et’s a v oi d writi n g P yt h o n as m u c h as p ossi bl e • L et’s c h e c k If y o u ar e usi n g f ast er b a c k e n ds: I nt el M K L, c u B L A S , c u T E N S O R • L et’s us e pr ofil ers a n d H P O li br ari es f or o pti mi zi n g y o ur pr o gr a ms. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms N VI DI A G P U c u D N N M A G M A c u B L A S At e n/ c1 0 ( C/ C + +) P y T or c h ( P yt h o n) m ai n. p y I nt el C P U L A P A C K B L A S N u m P y ( P yt h o n / C / C + + / F ortr a n) m ai n. p y H ar d w ar e F ast li br ar y writt e n i nC / C + +/F ortr a n P yt h o n wr a p p er Us er pr o gr a m Pr o gr a ms r a n o n C P Us Pr o gr a ms r a n o n G P Us O S ・ Dri v ers ・ R u nti m es / 5 0 3 7

P art 3: M a n a gi n g P yt h o n S cri pts f or R es e ar c h C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms / 3 3 3 8

V ersi o n M a n a g e m e nt of O ur S o ur c e C o d e W h e n w e d o e x p eri m e nts i n p ar all el, w e li k el y f or g ot w hi c h c o d es w e r a n. • E x p eri m e nts of m ulti pl e pr oj e cts i n p ar all el, s e v er al m o nt h a g o, or d o n e i n mi d ni g ht … • It is ess e nti al t o usi n g a v ersi o n c o ntr ol s yst e m ( V C S) s u c h as Git. R e c or d hist ori es of y o ur e x p eri m e nts ( c o m mit I Ds) t o r e c or d r es e ar c h acti viti es C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms I di d e x p eri m e nt a l ot, b ut I’ m n o w s o c o nf us e d! D a y 1 D a y 2 D a y 3 D a y 4 ⋯ D e b u g Tr ai ni n g D e b u g D e b u g Tr ai ni n g Tr ai ni n g D e b u g Tr ai ni n g D e b u g Tr ai ni n g D e b u g … … / 5 0 3 9

V ersi o n M a n a g e m e nt f or R es e ar c h Wit h Git D o es “ Br a n c h ” f or s oft w ar e d e v el o p m e nt i n Git fit t o r es e ar c h acti vit y? • All tri als i n r es e ar c h s h o ul d b e s a v e d as r es e ar c h r e c or ds. • Git h as a “ br a nc h ” f u ncti o n t o s a v e br a nc hi n g hist ori es. I n r es e ar c h acti viti es, t h e # of br a nc h es will m o n ot o ni c all y i ncr e as e! C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms m ai n att e m pt _1 att e m pt _1 b 2. L et’s tr y diff er e nt o n e. 1. T his m o d el d o es n’t w or k 3. T h e n, t his m o d el als o. 4. A h, t h e pr e v. o n e w as b ett er 5. U g h h h , t h e first o n e w as b est. / 5 0 4 0

R es e ar c h A cti viti es Will Fit Dir ect or y -Fil e Str u ct ur es L et’s us e fil e str u ct ur es b e c a us e r es e ar c h acti viti es h a v e tr e e str u ct ur es. • Git is n ot a s oft w ar e t o s a v e diff er e nc es b ut o n e t o s a v e s n a ps h ots. It is ess e nti al t o i m pl e m e nt m o d ul es w hi c h d o n ot d e p e n d o n e ac h ot h er. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms ※ 「 コ ミ ッ ト は ス ナ ッ プ シ ョ ッ ト で あ り 差 分 で は な い 」 htt ps:// git h u b. bl o g/j p/ 2 0 21 -01 -0 6 -c o m mits -ar e -s n a ps h ots-n ot -diffs/ m y _ m et h o d/ m o d els/ c n n. p y tr ai n er. p y d at as et. p y … m y _ m et h o d/ m o d els/ c n n. p y c n n _ att m1. p y tr ai n er. p y d at as et. p y … m y _ m et h o d/ m o d els/ c n n. p y c n n _ att m1. p y c n n _ att m 2. p y tr ai n er. p y d at as et. p y … m y _ m et h o d/ m o d els/ c n n. p y vit. p y tr ai n er. p y d at as et. p y … S a v e tri als as diff er e nt fil es R e c or d t h e d el eti o n als o C o m mit 1 C o m mit 2 C o m mit 3 C o m mit 4 / 5 0 41

Ti ps 3 -1: I m pl e m e nt Y o ur M o d ul es Wit h P y T or c h Li g ht ni n g M a n y P y T or c h wr a p p ers ar e d esi g n e d t o s u p p ort t h e m o d ul ar c o di n g. • Y o u c a n writ e y o ur o w n wr a p p er, b ut Li g ht ni n g is b ett er f or its ri c h d o c u m e nts. • D o c u m e nt ati o n is m ost i m p ort a nt f or s h ari n g y o ur c o d e wit h y o ur c oll e a g u es. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms L oss & O pti mi z ers L i g h t n i n g M o d u l e N N m o d el M o d u l e D at al o a d ers D a t a M o d u l e D at as ets D a t a s e t L o g gi n g T e n s o r B o a r d L o g g e r Tr ai ni n g str at e g y D D P S t r a t e g y , F S D P S t r a t e g y Tr ai ni n g pi p eli n e T r a i n e r C all b a c ks (p l u gi ns f or e xt e n di n g y o ur pi p eli n e) Pr o gr ess B ars P r o g r e s s B a r M o d el s a vi n g M o d e l C h e c k p o i n t E arl y st o p pi n g E a r l y S t o p p i n g Vis u ali z e G P U St atisti cs D e v i c e S t a t s M o n i t o r Vis u ali z e i nf er e nc e r es ults V i s u a l i z e D a t a ( c u s t o m ) / 5 0 4 2

Ti ps 3 -2: Us e h y dr a. utils.i nst a nti at e t o I nt e gr at e Y o ur M o d ul es It i nst a nti at es a cl ass/f u ncti o n s p e cifi e d b y t h e _ t a r g e t _ f i e l d . • Y o u c a n a v oi d if -t h e n l o gi cs, w hi c h oft e n i ntr o d u c e b u gs o n y o ur c o d e. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms s a m p l e _ r a t e : 1 6 0 0 0 t r a i n e r : _ t a r g e t _ : l i g h t n i n g . T r a i n e r d e f a u l t _ r o o t _ d i r : $ { w o r k i n g _ d i r e c t o r y } m a x _ e p o c h s : 2 0 0 d a t a m o d u l e : _ t a r g e t _ : a u d i o _ t a g g i n g . d a t a s e t s . h d f 5 _ w a v . H D F 5 W a v D a t a M o d u l e t r a i n _ d a t a s e t _ p a t h : . / h d f 5 / a u d i o s e t - 1 6 k H z - t r . h d f 5 v a l _ d a t a s e t _ p a t h : . / h d f 5 / a u d i o s e t - 1 6 k H z - c v . h d f 5 b a t c h _ s i z e : 3 2 m o d e l : _ t a r g e t _ : a u d i o _ t a g g i n g . t a s k . A u d i o T a g g i n g T a s k s a m p l e _ r a t e : $ { s a m p l e _ r a t e } m o d e l : _ t a r g e t _ : a u d i o _ t a g g i n g . m o d e l s . C N N 1 4 c h a n n e l s : 2 5 6 o p t i m i z e r _ g e n e r a t o r : _ p a r t i a l _ : T r u e _ t a r g e t _ : t o r c h . o p t i m . A d a m l r : 1 . e - 3 f r o m o m e g a c o n f i m p o r t O m e g a C o n f a s o c f r o m h y d r a . u t i l s i m p o r t i n s t a n t i a t e p a r s e r = A r g u m e n t P a r s e r ( ) p a r s e r . a d d _ a r g u m e n t ( " c o n f i g " , t y p e = P a t h ) p a r s e r . a d d _ a r g u m e n t ( " - - w o r k i n g _ d i r e c t o r y " , t y p e = P a t h , d e f a u l t = P a t h . c w d ( ) ) a r g s , u n k _ a r g s = p a r s e r . p a r s e _ k n o w n _ a r g s ( ) # l o a d c o n f i g c o n f i g = o c . m e r g e ( o c . l o a d ( a r g s . c o n f i g ) , v a r s ( a r g s ) , o c . f r o m _ c l i ( u n k _ a r g s ) , ) # t r a i n t r a i n e r : l i g h t n i n g . T r a i n e r = i n s t a n t i a t e ( c o n f i g . t r a i n e r ) t r a i n e r . f i t ( m o d e l = i n s t a n t i a t e ( c o n f i g . t a s k ) , d a t a m o d u l e = i n s t a n t i a t e ( c o n f i g . d a t a m o d u l e ) , ) c o nfi g. y a ml tr ai n. p y / 5 0 4 3

L et’s M a k e Y o ur P yt h o n P a c k a g e A p ac k a g e c a n b e m a d e b y j ust c o nfi g uri n g s et u p. p y ( or p y pr oj e ct.t o ml ). • Y o u c a n i nst all y o ur p a c k a g e b y p i p i n s t a l l g i t + h t t p s : / / g i t h u b . c o m / u s e r / r e p o • P u blis hi n g y o ur c o d e f or y o ur p a p er will i ncr e as e its cit ati o n. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms htt ps:// git h u b.c o m/ A d a m S p a n n b a u er/ mi ni m al _ p yt h o n _ p ac k a g e / 5 0 4 4

L et’s Us e C o d e F or m att ers T o ols t o a ut o m ati c all y f or m at y o ur c o d e t o a st a n d ar d c o di n g st yl e. • P yt h o n h as a st a n d ar d c o di n g st yl e g ui d e c all e d P E P 8 • M ost e n gi n e ers & r es e ar c h ers ar e us e d t o r e a d P E P 8 -f oll o w d P yt h o n c o d es C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms d e f f o o _ f u n c t i o n ( a , b = 1 ) : i f ( a + b > 0 ) : r e t u r n a e l s e : r e t u r n b c l a s s B a r C l a s s : d e f _ _ i n i t _ _ ( m y c l a s s , p a r a m ) : m y c l a s s . p a r a m = p a r a m d e f h o g e ( s e l f ) : r e t u r n s e l f . p a r a m d e f f o o _ f u n c t i o n ( a , b = 1 ) : i f a + b > 0 : r e t u r n a e l s e : r e t u r n b c l a s s B a r C l a s s : d e f _ _ i n i t _ _ ( m y c l a s s , p a r a m ) : m y c l a s s . p a r a m = p a r a m d e f h o g e ( s e l f ) : r e t u r n s e l f . p a r a m B ef or e Aft er / 5 0 4 5 Bl a c k

L et’s Us e Li nt ers T o ols t o a n al y z e y o ur c o d e f or p ot e nti al err ors, b u gs, a n d st ylisti c err ors. • Li nt ers c a n als o s u g g est a n y i m pr o v e m e nts i n y o ur c o d e. • F or e x a m pl e, f l a k e 8 - s i m p l i f y c a n s u g g est m a n y w a ys t o si m plif y y o ur c o d e. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms i f i s i n s t a n c e ( o b j , i n t ) o r i s i n s t a n c e ( o b j , f l o a t ) : p a s s i f f o o = = " a a a " o r f o o = = " b b b " : p a s s i f k e y i n f o o . k e y s ( ) : p a s s i f i s i n s t a n c e ( o b j , ( i n t , f l o a t ) ) : p a s s i f f o o i n ( " a a a " , " b b b " ) : p a s s i f k e y i n f o o : p a s s B ef or e Aft er fl a k e 8-si m plif y / 5 0 4 6

Ti ps 3 -3: Us e R uff f or Li nti n g a n d C o d e F or m atti n g R uff: A f ast all -i n-o n e li nt er a n d c o d e f or m att er writt e n i n R ust. • P o p ul ar li nti n g a n d f or m atti n g pr o gr a ms ar e r ei m pl e m e nt e d b y R ust • It is v er y f ast t h a n ks t o t h e R ust i m pl e m e nt ati o n. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms htt ps:// d ocs. astr al.s h/r uff/ / 5 0 4 7

Ti ps 3 -4: Us e Pr e -c o m mit f or A ut o m ati c all y R u n ni n g R uff Pr e -c o m mit c a n m a k e a h o o k e x e c ut e d b ef or e e v er y c o m mits of Git. • Y o u c a n g u ar a nt e e t h at y o ur c o m mits p ass t h e R uff’s c h e c k a n d f or m at. • It us ef ul f or fi n di n g p ot e nti al b u gs i n y o ur s o ur c e c o d e. C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms r e p o s : - r e p o : h t t p s : / / g i t h u b . c o m / a s t r a l - s h / r u f f - p r e - c o m m i t r e v : v 0 . 1 . 9 h o o k s : - i d : r u f f a r g s : [ - - f i x ] - i d : r u f f - f o r m a t u s e r @ y o u r _ s y s t e m : ~ $ g i t a d d s e p a r a t e . p y u s e r @ y o u r _ s y s t e m : ~ $ g i t c m - m " u p d a t e s e p a r a t e . p y " r u f f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F a i l e d - h o o k i d : r u f f - e x i t c o d e : 1 - f i l e s w e r e m o d i f i e d b y t h i s h o o k s e p a r a t e . p y : 6 2 : 2 6 : F 8 2 1 U n d e f i n e d n a m e ` i s t f t ` F o u n d 2 e r r o r s ( 1 f i x e d , 1 r e m a i n i n g ) . r u f f - f o r m a t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . P a s s e d . pr e-c o m mit -c o nfi g. y a ml / 5 0 4 8 I n y o ur t er mi n al

P art 3: Writi n g P yt h o n S cri pts f or R es e ar c h It is i m p ort a nt t o r e c or d hist ori es of o ur e x p eri m e nts (c o m mit I Ds). • L et’s us e Git f or v ersi o n c o ntr olli n g • L et’s writ e m o d ul ar s o ur c e c o d e ( w/ P y T or c h Li g ht ni n g if y o u pr ef er) • L et’s us e R uff a n d pr e -c o m mit f or c o d e f or m atti n g a n d li nti n g C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms I di d e x p eri m e nt a l ot, b ut I’ m n o w s o c o nf us e d! D a y 1 D a y 2 D a y 3 D a y 4 ⋯ D e b u g Tr ai ni n g D e b u g D e b u g Tr ai ni n g Tr ai ni n g D e b u g Tr ai ni n g D e b u g Tr ai ni n g D e b u g … … / 5 0 4 9

T o d a y’s T o pi cs T hr e e t o pi cs f or c o n d u cti n g AI r es e ar c h o n H P C s yst e ms C o n d ucti n g AI R es e ar c h o n Hi g h -P erf or m a n c e C o m p uti n g ( H P C) S yst e ms P art 1: W h at is t h e H P C E n vir o n m e nt? G P U cl ust ers, hi er ar c hi c al str uct ur es, D D P, L ustr e , H D F 5 P art 2: Writi n g F ast P yt h o n S cri pts I nt el M K L, N VI DI A c u B L A S /c u D N N , G P G P U, n u m b a , C yt h o n , C u P y , li n e _ pr ofil er, t or c h. pr ofil er, H P O, m pi 4 p y P art 3: M a n a gi n g P yt h o n S cri pts f or R es e ar c h Git, P y T or c h Li g ht ni n g, h y dr a. utils.i nst a nti at e , P yt h o n p ac k a g es, f or m att ers, li nt ers, R uff / 5 0 5 0