医療ディープラーニング勉強会 DL勉強会 第3回 2020.4

https://deeplearning-b.connpass.com/event/169952/

Transcript

1. ҩྍσΟʔϓϥʔχϯάษڧձ

   ୈ3ճ DLษڧձ
   ᷂tech vein ழມ ॆԝ
   

   View Slide

2. ࣗݾ঺հ
   ழມ ॆԝ (͍ͷ·ͨ ΈͭͻΖ)
   גࣜձࣾ tech vein ୅දऔక໾ ݉ σϕϩούʔ
   twitter: @ino2222
   IUUQTXXXUFDIWFJODPN
   

   View Slide

3. ΞδΣϯμ
   Archive Sanity (arxiv-sanity.com) ͔ΒϐοΫΞο
   ϓͨ͠աڈ1ϲ݄ؒͷ࿦จ঺հɻ
   ɾtop recentͷ࿦จτοϓ10
   ɾtop hype ͷ࿦จτοϓ10
   ɾҰ൪ؾʹͳͬͨ࿦จ1ͭ
   

   View Slide

4. Archive Sanity?
   https://www.arxiv-sanity.com/top
   

   View Slide

5. Arxiv Sanity
   Top recent: Best10
   

   View Slide

6. ᶃA Primer in BERTology: What we know about how BERT works
   BERTֶͷೖ໳ॻɻBERT͕ͲͷΑ͏ʹػೳ͢Δ͔ʹ͍ͭͯ஌͍ͬͯΔ͜ͱ
   τϥϯεϑΥʔϚʔܕϞσϧ(Transformer-based models)͸ݱ
   ࡏɺNLPͰ޿͘࢖ΘΕ͍ͯ·͕͢ɺͦͷ಺෦ͷ࢓૊Έʹ͍ͭ
   ͯ͸·ͩ͋·Γཧղ͞Ε͍ͯ·ͤΜɻຊߘͰ͸ɺ༗໊ͳBERT
   Ϟσϧ(Devlin et al. 2019)ʹ͍ͭͯɺ40Ҏ্ͷղੳݚڀΛ߹
   ੒ͯ͠ɺ͜Ε·Ͱʹ஌ΒΕ͍ͯΔ͜ͱΛઆ໌͠·͢ɻ·ͨɺ
   ఏҊ͞Ε͍ͯΔϞσϧͷमਖ਼ͱͦͷ܇࿅ϨδʔϜͷ֓ཁΛઆ
   ໌͠·͢ɻͦͯ͠ɺ͞ΒͳΔݚڀͷํ޲ੑΛ֓આ͠·͢ɻ
   https://arxiv.org/abs/2002.12327v1
   

   View Slide

7. ᶄOn Feature Normalization and Data Augmentation
   ಛ௃ͷਖ਼نԽͱσʔλ֦ுʹ͍ͭͯ
   ݱ୅ͷχϡʔϥϧωοτϫʔΫ܇࿅͸ɺҰൠԽΛվળ͢ΔͨΊʹσʔλͷ૿
   େʹେ͖͘ґଘ͍ͯ͠·͢ɻϥϕϧอଘܕͷ૿େ๏͕࠷ॳʹ੒ޭͨ͠ޙɺ࠷ۙ
   Ͱ͸ɺֶश͞Εܾͨఆ໘Λ׈Β͔ʹ͢ΔͨΊʹɺֶशαϯϓϧશମͷಛ௃ͱ
   ϥϕϧΛ૊Έ߹ΘͤΔϥϕϧઁಈ๏΁ͷؔ৺͕ߴ·͍ͬͯ·͢ɻຊ࿦จͰ͸ɺ
   ಛ௃ͷਖ਼نԽʹΑͬͯநग़͞Εͨୈ1ͱୈ2ͷϞʔϝϯτΛར༻ͨ͠৽͍͠૿
   ڧ๏ΛఏҊ͠·͢ɻֶशͨ͠ಛ௃ྔͷϞʔϝϯτΛผͷֶशը૾ͷϞʔϝϯτ
   ʹஔ͖׵͑Δͱͱ΋ʹɺ໨ඪϥϕϧΛิؒ͢ΔɻզʑͷΞϓϩʔν͸ߴ଎Ͱ
   ͋Γɺಛ௃ۭؒશମͰಈ࡞͠ɺैདྷͷख๏ͱ͸ҟͳΔ৴߸Λࠞ߹͢ΔͨΊɺ
   طଘͷ૿ڧख๏ͱޮՌతʹ૊Έ߹ΘͤΔ͜ͱ͕Ͱ͖·͢ɻզʑ͸ɺίϯ
   ϐϡʔλϏδϣϯɺԻ੠ɺࣗવݴޠॲཧͷϕϯνϚʔΫσʔληοτʹ͓͍
   ͯɺͦͷ༗ޮੑΛ࣮ূ͠·ͨ͠ɻ
   https://arxiv.org/abs/2002.11102v2
   

   View Slide

8. View Slide

9. ᶅKnowledge Graphs
   φϨοδάϥϑ
   ຊ࿦จͰ͸ɺφϨοδάϥϑʹ͍ͭͯแׅతʹ঺հ͠·͢ɻφϨοδάϥϑ͸ɺଟ
   ༷Ͱಈతͳେن໛σʔλͷίϨΫγϣϯΛར༻͢Δ͜ͱΛඞཁͱ͢ΔγφϦΦʹ͓
   ͍ͯɺۙ೥ɺ࢈ۀքͱֶज़քͷ૒ํ͔Βେ͖ͳ஫໨ΛूΊ͍ͯ·͢ɻҰൠతͳ঺հ
   ͷޙɺφϨοδάϥϑʹ࢖༻͞ΕΔ༷ʑͳάϥϑϕʔεͷσʔλϞσϧͱΫΤϦݴ
   ޠͷಈػ෇͚ͱରൺΛߦ͍·͢ɻφϨοδάϥϑʹ͓͚ΔεΩʔϚɺಉҰੑɺίϯ
   ςΩετͷ໾ׂʹ͍ͭͯٞ࿦͠·͢ɻԋ៷తٕज़ͱؼೲతٕज़ͷ૊Έ߹ΘͤΛ༻͍
   ͯɺ஌͕ࣝͲͷΑ͏ʹදݱ͞Εɺநग़͞ΕΔ͔Λઆ໌͠·͢ɻφϨοδάϥϑͷ࡞
   ੒ɺॆ࣮ɺ඼࣭ධՁɺચ࿅ɺެ։ͷͨΊͷํ๏Λ·ͱΊ͍ͯ·͢ɻஶ໊ͳΦʔϓϯ
   φϨοδάϥϑͱΤϯλʔϓϥΠζφϨοδάϥϑͷ֓ཁɺͦΕΒͷΞϓϦέʔ
   γϣϯɺ͓ΑͼͦΕΒ্͕ड़ͷٕज़ΛͲͷΑ͏ʹ࢖༻͍ͯ͠Δ͔Λઆ໌͠·͢ɻ࠷
   ޙʹɺφϨοδάϥϑͷকདྷͷݚڀͷํ޲ੑʹ͍ͭͯड़΂·͢ɻ
   https://arxiv.org/abs/2003.02320v1
   

   View Slide

10. View Slide

11. ᶆBatch Normalization Biases Deep Residual Networks Towards
    Shallow Paths
    όονਖ਼نԽ͸ਂ͍࢒ࠩωοτϫʔΫΛઙ͍ܦ࿏ʹภΒͤΔ
    όονਖ਼نԽʹ͸ෳ਺ͷϝϦοτ͕͋Γ·͢ɻόονਖ਼نԽ͸ଛࣦϥϯυεέʔϓ
    ͷ৚݅෇͚Λվળ͠ɺڻ͘΄ͲޮՌతͳਖ਼ଇԽΛߦ͍·͢ɻ͔͠͠ɺόονਖ਼نԽ
    ͷ࠷΋ॏཁͳར఺͸࢒ࠩωοτϫʔΫ(Residual Network)ʹ͓͍ͯੜ͡·͢ɻॳظ
    Խͷࡍɺόονਖ਼نԽ͸ɺωοτϫʔΫͷਂ͞ͷฏํࠜʹൺྫͨ͠ਖ਼نԽ܎਺ʹ
    ΑͬͯɺεΩοϓ઀ଓʹର͢Δ࢒ࠩ෼ذΛμ΢ϯεέʔϧ͠·͢ɻ͜ΕʹΑΓɺτ
    Ϩʔχϯάͷॳظஈ֊Ͱ͸ɺਂ͍ਖ਼نԽ͞Εͨ࢒ࠩωοτϫʔΫʹΑͬͯܭࢉ͞Ε
    ͨؔ਺͸ɺྑ޷ͳޯ഑Λ࣋ͭઙ͍ύεʹΑͬͯࢧ഑͞ΕΔ͜ͱ͕อূ͞Ε·͢ɻ͜
    ͷಎ࡯Λ༻͍ͯɺਖ਼نԽͳ͠Ͱඇৗʹਂ͍࢒ࠩωοτϫʔΫΛ܇࿅Ͱ͖Δ؆୯ͳॳ
    ظԽεΩʔϜΛ։ൃͨ͠ɻ·ͨɺόονਖ਼نԽ͸ΑΓେ͖ͳֶश཰Ͱ҆ఆֶͨ͠श
    ΛՄೳʹ͠·͕͢ɺ͜ͷར఺͸େ͖ͳόοναΠζͷֶशΛฒྻԽ͍ͨ͠৔߹ʹͷ
    Έ༗༻Ͱ͋Δ͜ͱΛ໌Β͔ʹ͠·ͨ͠ɻզʑͷ݁Ռ͸ɺҟͳΔΞʔΩςΫνϟʹ͓
    ͚Δόονਖ਼نԽͷར఺Λ෼཭͢Δͷʹ໾ཱͪ·͢ɻ
    https://arxiv.org/abs/2002.10444v1
    

    View Slide

12. ᶇAutoML-Zero: Evolving Machine Learning Algorithms From Scratch.
    AutoML-Zero: εΫϥον͔ΒͷػցֶशΞϧΰϦζϜͷਐԽ
    ػցֶशͷݚڀ͸ɺϞσϧߏ଄΍ֶशํ๏ͳͲଟ໘తʹਐΜͰ͍·͢ɻAutoMLͱͯ͠஌ΒΕΔ͜
    ͷΑ͏ͳݚڀΛࣗಈԽ͠Α͏ͱ͢Δ౒ྗ΋·ͨɺେ͖ͳਐาΛ਱͖͛ͯ·ͨ͠ɻ͔͠͠ɺ͜ͷਐา
    ͸ओʹχϡʔϥϧωοτϫʔΫͷΞʔΩςΫνϟʹয఺Λ౰ͯͨ΋ͷͰ͋Γɺ͜͜Ͱ͸ɺϏϧσΟ
    ϯάϒϩοΫͱͯ͠ߴ౓ͳઐ໳Ո͕ઃܭͨ͠૚ʹґଘ͍ͯ͠·ͨ͠--͋Δ͍͸ಉ༷ʹ੍ݶͷ͋Δ୳
    ࡧۭؒʹґଘ͍ͯ͠·ͨ͠ɻࢲͨͪͷ໨ඪ͸ɺAutoML͕͞ΒʹਐԽͰ͖Δ͜ͱΛࣔ͢͜ͱͰ͋Γ
    ·͢ɻզʑ͸ɺҰൠతͳݕࡧۭؒΛ௨ͯ͠ਓؒͷόΠΞεΛେ෯ʹ௿ݮ͢Δ৽͍͠ϑϨʔϜϫʔΫ
    Λಋೖ͢Δ͜ͱʹΑͬͯɺ͜ΕΛ࣮ূ͠·͢ɻ͜ͷۭؒͷ޿େ͞ʹ΋͔͔ΘΒͣɺਐԽత୳ࡧ͸
    όοΫϓϩύήʔγϣϯʹΑͬͯ܇࿅͞Εͨ2૚ͷχϡʔϥϧωοτϫʔΫΛൃݟ͢Δ͜ͱ͕Ͱ͖
    ·͢ɻ͜ΕΒͷ୯७ͳχϡʔϥϧωοτϫʔΫ͸ɺͦͷޙɺؔ৺ͷ͋ΔλεΫɺྫ͑͹CIFAR-10ͷ
    มछͰ௚઀ਐԽͤ͞Δ͜ͱͰɺόΠϦχΞΠϯλϥΫγϣϯɺਖ਼نԽޯ഑ɺॏΈฏۉԽͳͲͷτο
    ϓΞϧΰϦζϜʹݱ୅తͳٕज़͕ݱΕΔ͜ͱͰ͙྇͜ͱ͕Ͱ͖·͢ɻ͞ΒʹɺਐԽ͸ΞϧΰϦζϜ
    ΛҟͳΔλεΫλΠϓʹదԠͤ͞·͢ɻθϩ͔ΒػցֶशΞϧΰϦζϜΛൃݟͨ͜͠ΕΒͷ༧උత
    ͳ੒ޭ͸ɺ͜ͷ෼໺ͷ༗๬ͳ৽͍͠ํ޲ੑΛ͍ࣔͯ͠Δͱ৴͍ͯ͡·͢ɻ
    https://arxiv.org/abs/2003.03384v1
    

    View Slide

13. View Slide

14. View Slide

15. ᶈHyper-Parameter Optimization: A Review of Algorithms and
    Applications.
    ϋΠύʔύϥϝʔλ࠷దԽ. ΞϧΰϦζϜͱΞϓϦέʔγϣϯͷϨϏϡʔ
    σΟʔϓχϡʔϥϧωοτϫʔΫ͕։ൃ͞ΕͯҎདྷɺ೔ৗੜ׆ʹଟେͳߩݙΛ͖ͯ͠·ͨ͠ɻػ
    ցֶश͸ɺ೔ৗੜ׆ͷ΄΅͢΂ͯͷଆ໘ʹ͓͍ͯɺਓ͕ؒͰ͖ΔҎ্ͷ߹ཧతͳΞυόΠεΛఏ
    ڙͯ͘͠Ε·͢ɻ͔͠͠ɺ͜ͷΑ͏ͳ੒Ռʹ΋͔͔ΘΒͣɺχϡʔϥϧωοτϫʔΫͷઃܭͱ܇
    ࿅͸ɺґવͱͯ͠ࠔ೉Ͱ༧ଌෆՄೳͳखॱͰ͢ɻҰൠతͳϢʔβʔͷٕज़తͳᮢ஋ΛԼ͛ΔͨΊ
    ʹɺࣗಈԽ͞ΕͨϋΠύʔύϥϝʔλ࠷దԽ(HPO)͸ɺֶज़తʹ΋࢈ۀతʹ΋ਓؾͷ͋Δτϐο
    Ϋͱͳ͍ͬͯ·͢ɻຊ࿦จͰ͸ɺϋΠύʔύϥϝʔλ࠷దԽʹؔ͢Δ࠷΋ॏཁͳτϐοΫͷϨ
    ϏϡʔΛߦ͍·͢ɻ࠷ॳʹɺϞσϧͷֶश΍ߏ଄ʹؔ࿈͢ΔओཁͳϋΠύʔύϥϝʔλΛ঺հ
    ͠ɺͦͷॏཁੑͱ஋ҬΛఆٛ͢Δํ๏Λ࿦͡·͢ɻ࣍ʹɺओཁͳ࠷దԽΞϧΰϦζϜͱͦͷద༻
    ੑʹয఺Λ౰ͯɺಛʹਂ૚ֶशωοτϫʔΫʹର͢Δޮ཰ͱਫ਼౓Λ໢ཏ͍ͯ͠·͢ɻ࣍ʹɺHPO
    ͷͨΊͷओཁͳαʔϏε΍πʔϧΩοτΛϨϏϡʔ͠ɺ࠷ઌ୺ͷݕࡧΞϧΰϦζϜ΁ͷରԠɺओ
    ཁͳਂ૚ֶशϑϨʔϜϫʔΫͰͷ࣮ݱੑɺϢʔβ͕ઃܭͨ͠৽͍͠Ϟδϡʔϧ΁ͷ֦ுੑΛൺֱ
    ͠·͢ɻ࠷ޙʹɺHPOΛਂ૚ֶशʹద༻ͨ͠৔߹ͷ໰୊఺ɺ࠷దԽΞϧΰϦζϜؒͷൺֱɺݶΒ
    ΕͨܭࢉࢿݯͰͷϞσϧධՁͷͨΊͷஶ໊ͳΞϓϩʔνΛ঺հ͠ɺ࿦จΛకΊ͘͘Γ·͢ɻ
    https://arxiv.org/abs/2003.05689v1
    

    View Slide

16. ᶉA Survey on Contextual Embeddings
    จ຺తΤϯϕοσΟϯάʹؔ͢Δௐࠪ
    ELMo΍BERTͳͲͷจ຺ʹج͍ͮͨΤϯϕοσΟϯά͸ɺ
    Word2VecͷΑ͏ͳάϩʔόϧͳ୯ޠදݱΛ௒͑ͯɺ෯޿͍ࣗવݴ
    ޠॲཧλεΫʹ͓͍ͯըظతͳύϑΥʔϚϯεΛ࣮ݱ͠·͢ɻจ຺
    ʹج͍ͮͨΤϯϕοσΟϯά͸ɺ֤୯ޠʹͦͷจ຺ʹج͍ͮͨදݱ
    ΛׂΓ౰ͯΔ͜ͱͰɺ༷ʑͳจ຺Ͱͷ୯ޠͷ࢖༻Λัଊ͠ɺݴޠؒ
    Ͱ఻ୡ͞ΕΔ஌ࣝΛූ߸Խ͠·͢ɻຊௐࠪͰ͸ɺطଘͷจ຺ʹجͮ
    ͘ຒΊࠐΈϞσϧɺݴޠԣஅతͳϙϦάϩοτͷࣄલ܇࿅ɺԼྲྀλ
    εΫʹ͓͚Δจ຺ʹجͮ͘ຒΊࠐΈͷԠ༻ɺϞσϧѹॖɺϞσϧղ
    ੳΛϨϏϡʔ͠·͢ɻ
    https://arxiv.org/abs/2003.07278v1
    

    View Slide

17. ᶊReZero is All You Need: Fast Convergence at Large Depth.
    ReZero͕͋Ε͹େৎ෉ɻେਂ౓Ͱͷߴ଎ऩଋ
    σΟʔϓωοτϫʔΫ͸ɺྖҬΛ௒͑ͯେ෯ͳੑೳ޲্ΛՄೳʹ͠·͕ͨ͠ɺଟ͘ͷ৔߹ɺফ
    ࣦ/രൃతͳޯ഑ʹ೰·͞Ε͍ͯ·͢ɻ͜Ε͸ಛʹτϥϯεϑΥʔϚʔΞʔΩςΫνϟʹ౰ͯ͸
    ·Γɺେن໛ͳσʔληοτ΍ܭࢉ༧ࢉ͕ͳ͍ͱ12૚Λ௒͑Δਂ͞ͷֶश͕ࠔ೉Ͱ͢ɻҰൠత
    ʹɺඇޮ཰ͳ৴߸఻೻͕σΟʔϓωοτϫʔΫͷֶशΛ્֐͢Δ͜ͱ͕Θ͔͍ͬͯ·͢ɻτϥ
    ϯεͰ͸ɺϚϧνϔουͷࣗݾ஫ҙ͕͜ͷѱ͍৴߸఻೻ͷओͳݪҼͱͳ͍ͬͯ·͢ɻਂ૚৴߸
    ఻೻Λଅਐ͢ΔͨΊʹɺզʑ͸ReZeroΛఏҊ͠·͢ɻ͜Ε͸ΞʔΩςΫνϟΛ؆୯ʹมߋͨ͠
    ΋ͷͰɺϨΠϠʔ͝ͱʹ1ͭͷ௥ՃֶशύϥϝʔλΛ࢖༻ͯ͠ɺ೚ҙͷϨΠϠʔΛಉҰੑϚο
    ϓͱͯ͠ॳظԽ͢Δ΋ͷͰ͢ɻզʑ͸͜ͷٕज़ΛݴޠϞσϦϯάʹద༻͠ɺ100૚Ҏ্ͷ
    ReZero-τϥϯεϑΥʔϚʔωοτϫʔΫΛ؆୯ʹ܇࿅Ͱ͖Δ͜ͱΛൃݟ͠·ͨ͠ɻ12૚ͷτ
    ϥϯεϑΥʔϚʔʹద༻͢Δͱɺenwiki8ͰReZero͸56%଎͘ऩଋ͠·͢ɻReZero͸
    TransformerΛ௒͑ͯଞͷ࢒ࠩωοτϫʔΫʹ΋ద༻͞Εɺਂ͍׬શʹ઀ଓ͞ΕͨωοτϫʔΫ
    Ͱ͸1,500%଎͘ऩଋ͠ɺCIFAR 10Ͱ܇࿅͞ΕͨResNet-56Ͱ͸32%଎͘ऩଋ͠·͢ɻ
    https://arxiv.org/abs/2003.04887v1
    

    View Slide

18. ᶋLagrangian Neural Networks
    ϥάϥϯδϡχϡʔϥϧωοτϫʔΫ
    ੈքͷਖ਼֬ͳϞσϧ͸ɺͦͷجૅͱͳΔରশੑͷ֓೦ʹج͍ͮͯߏங͞Ε͍ͯ·͢ɻ෺ཧֶ
    Ͱ͸ɺ͜ΕΒͷରশੑ͸ΤωϧΪʔ΍ӡಈྔͳͲͷอଘଇʹରԠ͍ͯ͠·͢ɻ͔͠͠ɺ
    χϡʔϥϧωοτϫʔΫϞσϧ͸෺ཧֶ෼໺Ͱͷར༻͕૿͍͑ͯΔʹ΋͔͔ΘΒͣɺ͜ΕΒ
    ͷରশੑΛֶश͢Δͷʹۤ࿑͍ͯ͠·͢ɻຊ࿦จͰ͸ɺχϡʔϥϧωοτϫʔΫΛ༻͍ͯ೚
    ҙͷϥάϥϯδΞϯΛύϥϝʔλԽͰ͖ΔϥάϥϯδΞϯχϡʔϥϧωοτϫʔΫ(LNN)Λ
    ఏҊ͠·͢ɻϋϛϧτχΞϯΛֶश͢ΔϞσϧͱ͸ରরతʹɺLNN͸ਖ਼४࠲ඪΛඞཁͱ͠ͳ
    ͍ͨΊɺਖ਼४ӡಈྔ͕ෆ໌Ͱ͋ͬͨΓɺܭࢉ͕ࠔ೉ͳ৔߹ʹ༗ޮͰ͢ɻ͜Ε·ͰͷΞϓϩʔ
    νͱ͸ҟͳΓɺզʑͷख๏͸ֶश͞ΕͨΤωϧΪʔͷؔ਺ܗࣜΛ੍ݶͤͣɺ༷ʑͳλεΫͷ
    ͨΊͷΤωϧΪʔอଘϞσϧΛੜ੒͠·͢ɻզʑ͸ɺೋॏৼΓࢠͱ૬ର࿦తཻࢠͰզʑͷΞ
    ϓϩʔνΛςετ͠ɺϕʔεϥΠϯΞϓϩʔνͰ͸ࢄҳ͕ൃੜ͢ΔΤωϧΪʔอଘΛ࣮ূ
    ͠ɺϋϛϧτχΞϯΞϓϩʔνͰ͸ࣦഊ͢Δਖ਼४࠲ඪͷͳ͍૬ରੑཧ࿦ΛϞσϧԽ͠·͢ɻ
    ࠷ޙʹɺϥάϥϯδϡάϥϑωοτϫʔΫΛ༻͍ͯɺ͜ͷϞσϧ͕ͲͷΑ͏ʹάϥϑ΍࿈ଓ
    ܥʹద༻Ͱ͖Δ͔Λࣔ͠ɺ1࣍ݩ೾ಈํఔ্ࣜͰ࣮ূ͠·͢ɻ
    https://arxiv.org/abs/2003.04630v1
    

    View Slide

19. View Slide

20. ᶌSet-Structured Latent Representations
    ू߹ߏ଄Խજࡏදݱ
    ߏ଄Խ͞Ε͍ͯͳ͍σʔλ͸ɺγʔϯͷΠϝʔδͷதͷΦϒδΣΫτͷΑ͏
    ʹɺજࡏతͳߏ੒ཁૉͷߏ଄Λ͍࣋ͬͯΔ͜ͱ͕ଟ͍Ͱ͢ɻ͜ͷΑ͏ͳঢ়گͰ
    ͸ɺແடংͳίϨΫγϣϯ΍ set ͕જࡏతͳߏ଄ͱͳΓ·͢ɻ͔͠͠ɼ͜ͷΑ
    ͏ͳදݱΛσʔλ͔Β௚઀ֶश͢Δ͜ͱ͸ɼ཭ࢄతͰແடংͳߏ଄ͷͨΊࠔ೉
    Ͱ͢ɻ ͜͜Ͱ͸ɼू߹ߏ଄Λ࣋ͭજࡏදݱΛඍ෼Մೳʹֶश͢ΔͨΊͷϑϨʔ
    ϜϫʔΫΛ։ൃ͠·͢ɻ͜ͷϑϨʔϜϫʔΫΛ༻͍ͯɺը૾ͳͲͷσʔλΛࣗ
    વʹղऍՄೳͰҙຯͷ͋Δ੒෼ͷू߹ʹ෼ղ͢Δํ๏Λࣔ͠ɺطଘͷख๏Ͱ͸
    ؔ࿈͢Δߏ଄Λద੾ʹ੾Γ཭͢͜ͱ͕Ͱ͖ͳ͍͜ͱΛࣔ͠·͢ɻ·ͨɺզʑͷ
    ํ๏࿦Λɺηοτݻ༗ͷૢ࡞Λ࢖༻͢ΔηοτϚονϯάͷΑ͏ͳԼྲྀͷλε
    Ϋʹ·Ͱ֦ு͢Δํ๏΋ࣔ͠·͢ɻզʑͷίʔυ͸ͪ͜Βͷhttps URL͔Βೖख
    ՄೳͰ͢ɻ
    https://arxiv.org/abs/2003.04448v1
    

    View Slide

21. View Slide

22. Arxiv Sanity
    Top hype: Best10
    

    View Slide

23. ᶃLearning to Simulate Complex Physics with Graph Networks
    άϥϑωοτϫʔΫΛ༻͍ͨෳࡶͳ෺ཧֶͷγϛϡϨʔγϣϯͷֶश
    ͜͜Ͱ͸ɺγϛϡϨʔγϣϯֶशͷͨΊͷҰൠతͳϑϨʔϜϫʔΫΛఏࣔ͠ɺྲྀମɺ߶ମɺ
    มܗՄೳͳ෺࣭͕૬ޓʹ࡞༻͍ͯ͠Δ༷ʑͳ෺ཧྖҬͰ࠷ઌ୺ͷੑೳΛൃش͢Δ୯ҰϞσϧ
    ͷ࣮૷Λఏڙ͠·͢ɻզʑͷϑϨʔϜϫʔΫʢզʑ͕ʮάϥϑωοτϫʔΫϕʔεγϛϡϨʔ
    λʯʢGNSʣͱݺͿʣ͸ɺ෺ཧγεςϜͷঢ়ଶΛཻࢠͰදݱ͠ɺάϥϑͷϊʔυͱͯ͠දݱ
    ͠ɺֶश͞ΕͨϝοηʔδύογϯάΛհͯ͠μΠφϛΫεΛܭࢉ͠·͢ɻͦͷ݁Ռɺզʑͷ
    Ϟσϧ͸ɺֶशதͷ਺ઍݸͷύʔςΟΫϧΛ༻͍ͨγϯάϧλΠϜεςοϓͷ༧ଌ͔Βɺҟͳ
    Δॳظ৚݅ɺ਺ઍݸͷλΠϜεςοϓɺࢼݧ࣌ʹ͸গͳ͘ͱ΋ҰܻҎ্ͷύʔςΟΫϧΛ༻͍
    ͨ༧ଌ΁ͱҰൠԽͰ͖Δ͜ͱ͕ࣔ͞Ε·ͨ͠ɻզʑͷϞσϧ͸ɺ༷ʑͳධՁࢦඪͷϋΠύʔ
    ύϥϝʔλͷબ୒ʹରͯ͠ϩόετͰͨ͠ɻ௕ظతͳੑೳͷओͳܾఆཁҼ͸ɺϝοηʔδ௨
    աεςοϓͷ਺ͱɺ܇࿅σʔλΛϊΠζͰഁյ͢Δ͜ͱʹΑΔΤϥʔͷ஝ੵΛܰݮ͢Δ͜ͱ
    Ͱͨ͠ɻզʑͷGNSϑϨʔϜϫʔΫ͸ɺ͜Ε·ͰͰ࠷΋ਖ਼֬ͳ൚༻ֶश෺ཧγϛϡϨʔλͰ
    ͋Γɺෳࡶͳॱํ޲͓Αͼٯํ޲ͷ໰୊Λ෯޿͘ղ͘͜ͱ͕ظ଴͞Ε͍ͯ·͢ɻ
    

    View Slide

24. View Slide

25. View Slide

26. ᶄA Primer in BERTology: What we know about how BERT works
    BERTֶ ͷೖ໳ॻɻBERT͕ͲͷΑ͏ʹػೳ͢Δ͔ʹ͍ͭͯ஌͍ͬͯΔ͜ͱ
    τϥϯεϑΥʔϚʔܕϞσϧ(Transformer-based models)͸ݱ
    ࡏɺNLPͰ޿͘࢖ΘΕ͍ͯ·͕͢ɺͦͷ಺෦ͷ࢓૊Έʹ͍ͭ
    ͯ͸·ͩ͋·Γཧղ͞Ε͍ͯ·ͤΜɻຊߘͰ͸ɺ༗໊ͳBERT
    Ϟσϧ(Devlin et al. 2019)ʹ͍ͭͯɺ40Ҏ্ͷղੳݚڀΛ߹੒
    ͯ͠ɺ͜Ε·Ͱʹ஌ΒΕ͍ͯΔ͜ͱΛઆ໌͠·͢ɻ·ͨɺఏ
    Ҋ͞Ε͍ͯΔϞσϧͷमਖ਼ͱͦͷ܇࿅ϨδʔϜͷ֓ཁΛઆ໌
    ͠·͢ɻͦͯ͠ɺ͞ΒͳΔݚڀͷํ޲ੑΛ֓આ͠·͢ɻ
    https://arxiv.org/abs/2002.12327v1
    ॏෳ
    

    View Slide

27. ᶅLearning to Shade Hand-drawn Sketches
    खඳ͖εέονͷӄӨΛֶͿ
    ઢըεέονͱর໌ํ޲ͷϖΞ͔Βɺৄࡉ͔ͭਖ਼֬ͳܳज़తͳӨΛੜ੒͢ΔͨΊͷશࣗಈ
    ख๏Λఏࣔ͠·͢ɻ·ͨɺઢըͱӨͷϖΞ͔Βɺর໌ํ޲ͱλά෇͚͞Εͨ1,000ྫͷ৽
    ͍͠σʔληοτΛఏڙ͠·͢ɻڻ͘΂͖͜ͱʹɼੜ੒͞ΕͨӨ͸ɼεέον͞Εͨγʔ
    ϯͷجૅͱͳΔ3Dߏ଄Λૉૣ͘఻͑·͢ɽͦͷ݁ՌɺզʑͷΞϓϩʔνʹΑͬͯੜ੒͞
    ΕͨӨ͸ɺ௚઀࢖༻͢Δ͜ͱ΋ɺΞʔςΟετͷͨΊͷ༏Εͨग़ൃ఺ͱͯ͠࢖༻͢Δ͜ͱ
    ΋Ͱ͖·͢ɻզʑ͕ఏҊ͢ΔσΟʔϓϥʔχϯάωοτϫʔΫ͕ɺखඳ͖ͷεέονΛड
    ͚औΓɺજࡏۭؒʹ3DϞσϧΛߏங͠ɺͦͷ݁Ռͱͯ͠ੜ੒͞ΕͨӨΛϨϯμϦϯά͢
    Δ͜ͱΛ࣮ূ͍ͯ͠·͢ɻੜ੒͞ΕͨӨ͸ɺखඳ͖ͷઢͱͦͷԼͷ3࣍ݩۭؒΛଚॏ͠ɺ
    ࣗӨޮՌͷΑ͏ͳચ࿅͞Εͨਖ਼֬ͳσΟςʔϧΛؚΜͰ͍·͢ɻ͞Βʹɺੜ੒͞Εͨγϟ
    υ΢ʹ͸ɺैདྷͷ3DϨϯμϦϯάख๏Ͱ͸࣮ݱͰ͖ͳ͔ͬͨɺϦϜϥΠςΟϯά΍όο
    ΫϥΠςΟϯά͔ΒݱΕΔϋϩʔͳͲͷܳज़తͳޮՌؚ͕·Ε͍ͯ·͢ɻ
    https://arxiv.org/abs/2002.11812v1
    

    View Slide

28. View Slide

29. View Slide

30. ᶆStyleGAN2 Distillation for Feed-forward Image Manipulation.
    StyleGAN2 ϑΟʔυϑΥϫʔυը૾ૢ࡞ͷͨΊͷৠཹ
    StyleGAN2͸ɺϦΞϧͳը૾Λੜ੒͢ΔͨΊͷ࠷ઌ୺ͷωοτϫʔΫͰ͢ɻStyleGAN2
    ͸ɺજࡏۭؒ಺Ͱͷํ޲ੑ͕ҟͳΔΑ͏ʹ໌ࣔతʹ܇࿅͞Ε͓ͯΓɺજࡏҼࢠΛมԽͤ͞
    ͯޮ཰తͳը૾ૢ࡞ΛՄೳʹ͠·͢ɻطଘͷը૾Λฤू͢Δʹ͸ɺ༩͑ΒΕͨը૾Λ
    StyleGAN2ͷજࡏۭؒʹຒΊࠐΉඞཁ͕͋Γ·͢ɻόοΫϓϩύήʔγϣϯΛ༻͍ͨજࡏ
    ίʔυ࠷దԽ͸ɺ࣮ੈքͷը૾ͷ࣭తຒΊࠐΈʹҰൠతʹ༻͍ΒΕ͍ͯ·͕͢ɺଟ͘ͷΞ
    ϓϦέʔγϣϯͰ͸๏֎ʹ͕͔͔࣌ؒΓ·͢ɻզʑ͸ɺStyleGAN2ͷಛఆͷը૾ૢ࡞Λɺ
    ରʹͳֶͬͯश͞Εͨը૾ରը૾ωοτϫʔΫʹৠཹ͢Δํ๏ΛఏҊ͢Δɻ݁Ռͱͯ͠ಘ
    ΒΕΔύΠϓϥΠϯ͸ɺطଘͷGANͷ୅ସͱͯ͠ɺରʹͳ͍ͬͯͳ͍σʔλΛ༻ֶ͍ͯश
    ͞Ε·͢ɻຊݚڀͰ͸ɺਓؒͷإͷม׵݁ՌΛఏڙ͠·͢ɿੑผަ׵ɺՃྸɾएฦΓɺε
    λΠϧม׵ɺը૾ϞʔϑΟϯάɻզʑͷख๏Λ༻͍ͨੜ੒ͷ඼࣭͸ɺ͜ΕΒͷಛఆͷλεΫ
    ʹ͓͍ͯɺStyleGAN2όοΫϓϩύήʔγϣϯ΍ݱࡏͷ࠷ઌ୺ͷख๏ͱಉ౳Ͱ͋Δ͜ͱΛ
    ࣔ͠·͢ɻ
    https://arxiv.org/abs/2003.03581v1
    

    View Slide

31. View Slide

32. View Slide

33. View Slide

34. ᶇAutoML-Zero: Evolving Machine Learning Algorithms From Scratch.
    AutoML-Zero: εΫϥον͔ΒͷػցֶशΞϧΰϦζϜͷਐԽ
    ػցֶशͷݚڀ͸ɺϞσϧߏ଄΍ֶशํ๏ͳͲଟ໘తʹਐΜͰ͍ΔɻAutoMLͱͯ͠஌ΒΕΔ͜ͷ
    Α͏ͳݚڀΛࣗಈԽ͠Α͏ͱ͢Δ౒ྗ΋·ͨɺେ͖ͳਐาΛ਱͖͛ͯ·ͨ͠ɻ͔͠͠ɺ͜ͷਐา͸
    ओʹχϡʔϥϧωοτϫʔΫͷΞʔΩςΫνϟʹয఺Λ౰ͯͨ΋ͷͰ͋Γɺ͜͜Ͱ͸ɺϏϧσΟϯ
    άϒϩοΫͱͯ͠ߴ౓ͳઐ໳Ո͕ઃܭͨ͠૚ʹґଘ͍ͯ͠·ͨ͠--͋Δ͍͸ಉ༷ʹ੍ݶͷ͋Δ୳ࡧ
    ۭؒʹґଘ͍ͯ͠·ͨ͠ɻࢲͨͪͷ໨ඪ͸ɺAutoML͕͞ΒʹਐԽͰ͖Δ͜ͱΛࣔ͢͜ͱͰ͋Γ·
    ͢ɻզʑ͸ɺҰൠతͳݕࡧۭؒΛ௨ͯ͠ਓؒͷόΠΞεΛେ෯ʹ௿ݮ͢Δ৽͍͠ϑϨʔϜϫʔΫΛ
    ಋೖ͢Δ͜ͱʹΑͬͯɺ͜ΕΛ࣮ূ͠·͢ɻ͜ͷۭؒͷ޿େ͞ʹ΋͔͔ΘΒͣɺਐԽత୳ࡧ͸όο
    ΫϓϩύήʔγϣϯʹΑͬͯ܇࿅͞Εͨ2૚ͷχϡʔϥϧωοτϫʔΫΛൃݟ͢Δ͜ͱ͕Ͱ͖·
    ͢ɻ͜ΕΒͷ୯७ͳχϡʔϥϧωοτϫʔΫ͸ɺͦͷޙɺؔ৺ͷ͋ΔλεΫɺྫ͑͹CIFAR-10ͷม
    छͰ௚઀ਐԽͤ͞Δ͜ͱͰɺόΠϦχΞΠϯλϥΫγϣϯɺਖ਼نԽޯ഑ɺॏΈฏۉԽͳͲͷτοϓ
    ΞϧΰϦζϜʹݱ୅తͳٕज़͕ݱΕΔ͜ͱͰ͙྇͜ͱ͕Ͱ͖·͢ɻ͞ΒʹɺਐԽ͸ΞϧΰϦζϜΛ
    ҟͳΔλεΫλΠϓʹదԠͤ͞·͢ɻθϩ͔ΒػցֶशΞϧΰϦζϜΛൃݟͨ͜͠ΕΒͷ༧උతͳ
    ੒ޭ͸ɺ͜ͷ෼໺ͷ༗๬ͳ৽͍͠ํ޲ੑΛ͍ࣔͯ͠Δͱ৴͍ͯ͡·͢ɻ
    https://arxiv.org/abs/2003.03384v1
    ॏෳ
    

    View Slide

35. ᶈLagrangian Neural Networks
    ϥάϥϯδϡχϡʔϥϧωοτϫʔΫ
    ੈքͷਖ਼֬ͳϞσϧ͸ɺͦͷجૅͱͳΔରশੑͷ֓೦ʹج͍ͮͯߏங͞Ε͍ͯ·͢ɻ෺ཧֶ
    Ͱ͸ɺ͜ΕΒͷରশੑ͸ΤωϧΪʔ΍ӡಈྔͳͲͷอଘଇʹରԠ͍ͯ͠·͢ɻ͔͠͠ɺ
    χϡʔϥϧωοτϫʔΫϞσϧ͸෺ཧֶ෼໺Ͱͷར༻͕૿͍͑ͯΔʹ΋͔͔ΘΒͣɺ͜ΕΒ
    ͷରশੑΛֶश͢Δͷʹۤ࿑͍ͯ͠·͢ɻຊ࿦จͰ͸ɺχϡʔϥϧωοτϫʔΫΛ༻͍ͯ೚
    ҙͷϥάϥϯδΞϯΛύϥϝʔλԽͰ͖ΔϥάϥϯδΞϯχϡʔϥϧωοτϫʔΫ(LNN)Λఏ
    Ҋ͠·͢ɻϋϛϧτχΞϯΛֶश͢ΔϞσϧͱ͸ରরతʹɺLNN͸ਖ਼४࠲ඪΛඞཁͱ͠ͳ͍
    ͨΊɺਖ਼४ӡಈྔ͕ෆ໌Ͱ͋ͬͨΓɺܭࢉ͕ࠔ೉ͳ৔߹ʹ༗ޮͰ͢ɻ͜Ε·ͰͷΞϓϩʔν
    ͱ͸ҟͳΓɺզʑͷख๏͸ֶश͞ΕͨΤωϧΪʔͷؔ਺ܗࣜΛ੍ݶͤͣɺ༷ʑͳλεΫͷͨ
    ΊͷΤωϧΪʔอଘϞσϧΛੜ੒͠·͢ɻզʑ͸ɺೋॏৼΓࢠͱ૬ର࿦తཻࢠͰզʑͷΞϓ
    ϩʔνΛςετ͠ɺϕʔεϥΠϯΞϓϩʔνͰ͸ࢄҳ͕ൃੜ͢ΔΤωϧΪʔอଘΛ࣮ূ͠ɺ
    ϋϛϧτχΞϯΞϓϩʔνͰ͸ࣦഊ͢Δਖ਼४࠲ඪͷͳ͍૬ରੑཧ࿦ΛϞσϧԽ͠·͢ɻ࠷ޙ
    ʹɺϥάϥϯδϡάϥϑωοτϫʔΫΛ༻͍ͯɺ͜ͷϞσϧ͕ͲͷΑ͏ʹάϥϑ΍࿈ଓܥʹ
    ద༻Ͱ͖Δ͔Λࣔ͠ɺ1࣍ݩ೾ಈํఔ্ࣜͰ࣮ূ͠·͢ɻ
    https://arxiv.org/abs/2003.04630v1
    ॏෳ
    

    View Slide

36. ᶉMLIR: A Compiler Infrastructure for the End of Moore's Law.
    MLIR: ϜʔΞͷ๏ଇͷऴᖼͷͨΊͷίϯύΠϥج൫
    ຊݚڀͰ͸ɺ࠶ར༻ՄೳͰ֦ுՄೳͳίϯύΠϥج൫Λߏங͢ΔͨΊͷ৽͍͠ΞϓϩʔνͰ͋Δ
    MLIRΛ঺հ͠·͢ɻMLIRͷ໨త͸ɺιϑτ΢ΣΞͷஅยԽʹରॲ͠ɺҟछϋʔυ΢ΣΞͷίϯύ
    ΠϧΛվળ͠ɺυϝΠϯݻ༗ͷίϯύΠϥΛߏங͢ΔͨΊͷίετΛେ෯ʹ࡟ݮ͠ɺطଘͷίϯύ
    ΠϥΛ઀ଓ͢ΔͷΛॿ͚Δ͜ͱͰ͢ɻMLIR͸ɺҟͳΔந৅౓ϨϕϧͰͷίʔυੜ੒ثɺτϥϯε
    ϨʔλɺΦϓςΟϚΠβͷઃܭͱ࣮૷Λ༰қʹ͠ɺ·ͨɺΞϓϦέʔγϣϯυϝΠϯɺϋʔυ΢Σ
    Ξλʔήοτɺ࣮ߦ؀ڥʹ·͕ͨΔίʔυੜ੒ثɺτϥϯεϨʔλɺΦϓςΟϚΠβͷઃܭͱ࣮૷
    Λ༰қʹ͠·͢ɻຊݚڀͰ͸ɼ(1)֦ுͱਐԽͷͨΊʹߏங͞Εͨݚڀ੒Ռ෺ͱͯ͠ͷMLIRʹ͍ͭ
    ͯٞ࿦͠ɼઃܭɼηϚϯςΟΫεɼ࠷దԽ࢓༷ɼγεςϜɼΤϯδχΞϦϯάʹ͓͍ͯɼ͜ͷ৽͠
    ͍ઃܭϙΠϯτ͕΋ͨΒ͢՝୊ͱػձΛ໌Β͔ʹ͠·͢ɽ(2) ίϯύΠϥߏஙίετΛ࡟ݮ͢ΔҰ
    ൠԽ͞ΕͨΠϯϑϥͱͯ͠ͷMLIRͷධՁ-ଟ༷ͳϢʔεέʔεΛ঺հ͠ɼকདྷͷϓϩάϥϛϯάݴ
    ޠɼίϯύΠϥɼ࣮ߦ؀ڥɼίϯϐϡʔλΞʔΩςΫνϟͷݚڀͱڭҭͷػձΛࣔ͠·͢ɽ·ͨɼ
    MLIRͷཧ࿦తࠜڌɼಠࣗͷઃܭݪཧɼߏ଄ɼҙຯ࿦ʹ͍ͭͯ΋঺հ͍ͯ͠·͢ɽ
    https://arxiv.org/abs/2002.11054v2
    લճͱॏෳ
    

    View Slide

37. ᶊSLIDE : In Defense of Smart Algorithms over Hardware Acceleration for
    Large-Scale Deep Learning Systems.
    SLIDE : େن໛σΟʔϓϥʔχϯάγεςϜͷͨΊͷϋʔυ΢ΣΞΞΫηϥϨʔ
    γϣϯΑΓ΋εϚʔτΞϧΰϦζϜΛकΔͨΊʹɻ
    σΟʔϓϥʔχϯάʢDLʣΞϧΰϦζϜ͸ɺݱ୅ͷػցֶशγεςϜͷத৺తͳয఺ͱͳ͍ͬͯ·͢ɻσʔ
    λྔͷ૿Ճʹ൐͍ɺ਺ԯݸͷύϥϝʔλΛ࣋ͭେن໛ͳχϡʔϥϧωοτϫʔΫΛ܇࿅ͯ͠ɺ͜ΕΒͷσʔ
    λྔΛهԱ͠ɺ࠷ઌ୺ͷਫ਼౓ΛಘΔͷʹे෼ͳ༰ྔΛҡ࣋͢Δ͜ͱ͕Ұൠతʹͳ͖͍ͬͯͯ·͢ɻେن໛ͳ
    Ϟσϧͱσʔλʹؔ࿈͢ΔߴֹͳܭࢉΛճආ͢ΔͨΊʹɺίϛϡχςΟͰ͸ϞσϧֶशͷͨΊͷઐ༻ϋʔυ
    ΢ΣΞ΁ͷ౤ࢿ͕૿Ճ͍ͯ͠·͢ɻ͔͠͠ɺಛघͳϋʔυ΢ΣΞ͸ߴՁͰ͋Γɺଟ͘ͷλεΫʹҰൠԽ͢Δ
    ͜ͱ͸ࠔ೉Ͱ͢ɻΞϧΰϦζϜͷਐา͸ɺNVIDIA-V100 GPUͷΑ͏ͳڧྗͳϋʔυ΢ΣΞʹରͯ͠௚઀త
    ͳ༏ҐੑΛࣔ͢͜ͱ͕Ͱ͖·ͤΜͰͨ͠ɻ͜ͷ࿦จ͸ྫ֎Λఏڙ͠·͢ɻզʑ͸ɺεϚʔτͳϥϯμϜԽΞ
    ϧΰϦζϜͱϚϧνίΞฒྻԽͱϫʔΫϩʔυ࠷దԽΛಠࣗʹ༥߹ͤͨ͞SLIDE (Sub-LInear Deep
    learning Engine)ΛఏҊ͠·͢ɻSLIDE͸CPUͷΈΛ࢖༻͢Δ͜ͱͰɺ࠷దԽ͞ΕͨTensorflow(TF)ͷ࣮૷
    ΛGPU্Ͱ࣮ߦͨ͠৔߹ʹൺ΂ͯɺֶशͱਪ࿦ͷ྆ํͷܭࢉྔΛେ෯ʹ࡟ݮ͢Δ͜ͱ͕Ͱ͖·͢ɻۀքن໛
    ͷਪ঑σʔληοτΛ༻͍ͨධՁͰ͸ɺ44ίΞͷCPUͰSLIDEΛ࢖༻ͨ͠৔߹ɺTesla V100ͰTFΛ࢖༻͠
    ֶͯशͨ͠৔߹ͱൺֱͯ͠ɺ೚ҙͷਫ਼౓ϨϕϧͰ3.5ഒҎ্(1࣌ؒର3.5࣌ؒ)ͷ଎౓Ͱֶश͕ՄೳͰ͋Δ͜
    ͱ͕ࣔ͞Ε͍ͯ·͢ɻಉ͡CPUͷϋʔυ΢ΣΞ্Ͱ͸ɺSLIDE͸TFΑΓ10ഒҎ্ߴ଎Ͱ͢ɻ࠶ݱੑͷͨΊͷ
    ίʔυͱεΫϦϓτΛఏڙ͠·͢ɻ
    https://arxiv.org/abs/1903.03129v2
    

    View Slide

38. ᶋAn Empirical Evaluation of Generic Convolutional and Recurrent
    Networks for Sequence Modeling.
    γʔέϯεϞσϦϯάͷͨΊͷ൚༻ίϯϘϦϡʔγϣϯɾϦΧϨϯτωοτϫʔ
    Ϋͷ࣮ূతධՁ
    ΄ͱΜͲͷਂ૚ֶशͷઐ໳ՈʹͱͬͯɺγʔέϯεϞσϦϯά͸ϦΧϨϯτωοτϫʔΫͱಉٛͰ
    ͢ɻ ͔͠͠ɺ࠷ۙͷ݁Ռ͸ɺԻ੠߹੒΍ػց຋༁ͳͲͷλεΫʹ͓͍ͯɺίϯϘϦϡʔγϣϯɾ
    ΞʔΩςΫνϟ͕ϦΧϨϯτɾωοτϫʔΫΑΓ΋༏Ε͍ͯΔ͜ͱΛ͍ࣔͯ͠·͢ɻ ৽͍͠γʔέ
    ϯεϞσϦϯάͷλεΫ΍σʔληοτ͕ൃੜͨ͠৔߹ɺͲͷΞʔΩςΫνϟΛ࢖༻͢΂͖͔ʁ
    զʑ͸ɺγʔέϯεϞσϦϯάͷͨΊͷҰൠతͳ৞ΈࠐΈ͓ΑͼϦΧϨϯτΞʔΩςΫνϟͷମܥ
    తͳධՁΛߦ͍ͬͯ·͢ɻ ͜ΕΒͷϞσϧ͸ɺϦΧϨϯτɾωοτϫʔΫͷϕϯνϚʔΫʹҰൠత
    ʹ࢖༻͞Ε͍ͯΔ෯޿͍ඪ४λεΫͰධՁ͞Ε͍ͯ·͢ɻ ͦͷ݁Ռɺ୯७ͳ৞ΈࠐΈΞʔΩςΫ
    νϟ͸ɺଟ༷ͳλεΫ΍σʔληοτʹ͓͍ͯɺLSTMͷΑ͏ͳਖ਼نͷϦΧϨϯτωοτϫʔΫΑ
    Γ΋༏Ε͍ͯΔ͜ͱ͕ࣔ͞ΕɺҰํͰɺΑΓ௕͍༗ޮϝϞϦΛࣔ͠·ͨ͠ɻ զʑ͸ɺγʔέϯε
    ϞσϦϯάͱϦΧϨϯτωοτϫʔΫͷؒͷڞ௨ͷؔ࿈ੑΛ࠶ߟ͢΂͖Ͱ͋Γɺ৞ΈࠐΈωοτ
    ϫʔΫ͸γʔέϯεϞσϦϯάλεΫͷࣗવͳग़ൃ఺ͱΈͳ͢΂͖Ͱ͋Δͱ݁࿦෇͚·ͨ͠ɻ ؔ
    ࿈͢Δ࡞ۀΛࢧԉ͢ΔͨΊʹɺզʑ͸͜ͷhttp URLͰίʔυΛར༻Ͱ͖ΔΑ͏ʹ͠·ͨ͠ɻ
    https://arxiv.org/abs/1803.01271v2
    

    View Slide

39. TCN
    Լ:ೖྗ=࣌ܥྻσʔλ(ӈ͔Βॱ൪ʹ0...nඵޙ)
    ্:ग़ྗ=nඵؒͷ݁Ռ(ӈ͔Βॱʹɺnඵલɺ2nඵલ…)
    https://github.com/philipperemy/keras-tcn
    

    View Slide

40. TCNΛॏͶͨྫ
    https://github.com/philipperemy/keras-tcn
    

    View Slide

41. CNN(TCN) vs RNN܈
    

    View Slide

42. ᶌSparse Orthogonal Variational Inference for Gaussian Processes.
    Ψ΢εաఔͷͨΊͷૄͳ௚ަมྔਪ࿦.
    ༠ಋ఺Λ༻͍ͨΨ΢εաఔͷεύʔεม෼ۙࣅͷ৽͍͠ղऍΛ঺հ͠
    ·͢ɻ͜Ε͸ɺΨ΢εաఔΛ2ͭͷಠཱͨ͠աఔͷ࿨ͱͯ͠෼ղ͢Δ
    ͜ͱʹج͍͍ͮͯ·͢ɻ1ͭ͸༠ಋ఺ͷ༗ݶجఈʹ·͕͓ͨͬͯΓɺ΋
    ͏1ͭ͸࢒ΓͷมಈΛัଊ͠·͢ɻ͜ͷఆࣜԽ͕طଘͷۙࣅ஋Λճ෮
    ͢Δͱಉ࣌ʹɺݶք໬౓ͷΑΓݫ͍͠Լݶ஋ͱ৽͍֬͠཰తม෼ਪ࿦
    ΞϧΰϦζϜΛಘΔ͜ͱ͕Ͱ͖Δ͜ͱΛࣔ͠·͢ɻඪ४ճؼ͔Βʢਂ
    ͍ʣ৞ΈࠐΈΨ΢εաఔΛ༻͍ͨଟΫϥε෼ྨ·Ͱɺ͍͔ͭ͘ͷΨ΢
    εաఔϞσϧʹ͓͍ͯ͜ΕΒͷΞϧΰϦζϜͷޮ཰ੑΛ࣮ূ͠ɺ७ਮ
    ʹGPϕʔεͷϞσϧͷதͰCIFAR-10Ͱͷ࠷৽ͷ݁ՌΛใࠂ͠·͢ɻ
    https://arxiv.org/abs/1910.10596v3
    

    View Slide

43. Ψ΢εաఔϞσϧͷ༧ଌྫ
    • ☓: ࣮ଌσʔλ
    • ೱ੨ઢ: ࠷΋֬཰ͷߴ͍༧૝஋
    • ബ੨ଳ:ى͜Γ͏ΔՄೳੑͷߴ͍ൣғ
    (=σ2)
    http://machine-learning.hatenablog.com/entry/2018/01/13/142612
    

    View Slide

44. My favorite
    

    View Slide

45. ᶆStyleGAN2 Distillation for Feed-forward Image Manipulation.
    StyleGAN2 ϑΟʔυϑΥϫʔυը૾ૢ࡞ͷͨΊͷৠཹ
    StyleGAN2͸ɺϦΞϧͳը૾Λੜ੒͢ΔͨΊͷ࠷ઌ୺ͷωοτϫʔΫͰ͢ɻStyleGAN2
    ͸ɺજࡏۭؒ಺Ͱͷํ޲ੑ͕ҟͳΔΑ͏ʹ໌ࣔతʹ܇࿅͞Ε͓ͯΓɺજࡏҼࢠΛมԽͤ͞
    ͯޮ཰తͳը૾ૢ࡞ΛՄೳʹ͠·͢ɻطଘͷը૾Λฤू͢Δʹ͸ɺ༩͑ΒΕͨը૾Λ
    StyleGAN2ͷજࡏۭؒʹຒΊࠐΉඞཁ͕͋Γ·͢ɻόοΫϓϩύήʔγϣϯΛ༻͍ͨજࡏ
    ίʔυ࠷దԽ͸ɺ࣮ੈքͷը૾ͷ࣭తຒΊࠐΈʹҰൠతʹ༻͍ΒΕ͍ͯ·͕͢ɺଟ͘ͷΞ
    ϓϦέʔγϣϯͰ͸๏֎ʹ͕͔͔࣌ؒΓ·͢ɻզʑ͸ɺStyleGAN2ͷಛఆͷը૾ૢ࡞Λɺ
    ରʹͳֶͬͯश͞Εͨը૾ରը૾ωοτϫʔΫʹৠཹ͢Δํ๏ΛఏҊ͢Δɻ݁Ռͱͯ͠ಘ
    ΒΕΔύΠϓϥΠϯ͸ɺطଘͷGANͷ୅ସͱͯ͠ɺରʹͳ͍ͬͯͳ͍σʔλΛ༻ֶ͍ͯश
    ͞Ε·͢ɻຊݚڀͰ͸ɺਓؒͷإͷม׵݁ՌΛఏڙ͠·͢ɿੑผަ׵ɺՃྸɾएฦΓɺε
    λΠϧม׵ɺը૾ϞʔϑΟϯάɻզʑͷख๏Λ༻͍ͨੜ੒ͷ඼࣭͸ɺ͜ΕΒͷಛఆͷλεΫ
    ʹ͓͍ͯɺStyleGAN2όοΫϓϩύήʔγϣϯ΍ݱࡏͷ࠷ઌ୺ͷख๏ͱಉ౳Ͱ͋Δ͜ͱΛ
    ࣔ͠·͢ɻ
    https://arxiv.org/abs/2003.03581v1
    

    View Slide

46. աڈ࿦จ: StyleGAN2 ͱ͸
    • ελΠϧΛϊΠζͱͯ͠ϥϯμϜͳը૾ੜ੒͢
    Δ࠷৽ͷGAN
    • ελΠϧʹը෩ɾྠֲɾ޲͖ɾ൅ͷ৭΍൅ܗɾ
    ໨ͷ৭΍ܗͳͲԿͰ΋
    

    View Slide

47. StyleGAN2ͷྫ1
    https://arxiv.org/abs/1912.04958
    

    View Slide

48. StyleGAN2ͷྫ2
    https://ai-scholar.tech/articles/others/stylegan-fashion-339
    

    View Slide

49. ࠓճͷ࿦จ:StyleGAN2ͷৠཹ
    • StyleGAN2 ͚ͩͰ΋͍͕͢͝ɺৠཹʢબผɾ࠶
    ֶशʣ͢Δ͜ͱͰ೥ྸΛएฦΒͤͨΓՃྸͨ͠
    ΓɺੑผΛΑΓࣗવʹม͑ͨإࣸਅΛੜ੒͢Δ͜
    ͱ͕Ͱ͖ΔΑ͏ʹͳͬͨͱ͍͏࿦จɻ
    • ࿝ԽɾएฦΓ΍ੑผͷೖΕସ͑ͷֶशσʔληο
    τΛ࡞Δͷ͸ݱ࣮తʹෆՄೳ͕ͩɺͲ͏΍ͬͯͦ
    ΕΛ৐Γӽ͑ͯσʔλΛ࡞͔ͬͨʁ͕ϙΠϯτɻ
    

    View Slide

50. StyleGAN2ৠཹͷ੒Ռྫ
    • ੑస׵

    • एฦΓɾ࿝Խ

    

    View Slide

51. StyleGAN2ͷৠཹɾֶश
    • 1.StyleGAN2͔Βֶशݩσʔλੜ੒
    • 2.ݩσʔλͷ೥ྸɾੑผม׵
    • 3.ผϞσϧͰ࠶ֶशͯ͠ར༻
    

    View Slide

52. step1 : ֶशݩσʔλੜ੒

    (StyleGAN2+إ෼ྨϞσϧ)
    

    View Slide

53. step1 : ֶशݩσʔλੜ੒

    (StyleGAN2+إ෼ྨϞσϧ)
    1.StyleGAN2Ͱ
    ϥϯμϜʹը૾Λ࡞Δ ੜ੒ը૾
    

    View Slide

54. step1 : ֶशݩσʔλੜ੒

    (StyleGAN2+إ෼ྨϞσϧ)
    إը૾෼ྨϞσϧͰ
    ը૾൑ఆ͢Δ
    ৴༻౓ ೥ྸ ੑผ
    

    View Slide

55. step1 : ֶशݩσʔλੜ੒

    (StyleGAN2+إ෼ྨϞσϧ)
    X4UZMF("/ͷ
    ը૾ੜ੒ͷॏΈσʔλ
    ੜ੒ը૾ͷಛ௃஋
    
    

    View Slide

56. step1 : ֶशݩσʔλੜ੒

    (StyleGAN2+إ෼ྨϞσϧ)
    • e
    ॏΈ ৴༻౓ ೥ྸ ੑผΛ̍૊ͱ͢Δ
    ˠ͜ΕΛͨ͘͞Μ࡞ͬͯɺ৴༻౓ͷߴ͍΋ͷΛσʔληοτʹɻ
    

    View Slide

57. ৠཹstep2:ݩσʔλͷ೥ྸɾੑผม׵
    (ྫ͸೥ྸ͕ͩɺੑผ΋ಉ͡)
    1. ೥ྸผʹॏΈσʔλ܈ͷதԝ஋Λܭࢉͯ͠ɺ
    ೥ྸ͕มΘͬͨͱ͖ͷॏΈͷࠩ෼ΛٻΊΔɻ
    2. ݩͷॏΈσʔλʹࠩ෼Λ଍ͨ͠ΓҾ͍ͨΓ͠
    ͯ࿝ԽɾएฦΓը૾Λ਺ύλʔϯ࡞Δ
    3. ΑΓࣗવͳ΋ͷΛϐοΫΞοϓͨ͠ΒಉҰਓ
    ෺ͷ࿝ԽɾएฦΓͷڭࢣσʔληοτ͕׬੒
    

    View Slide

58. ࡞ͬͨॏΈσʔλΛ
    Ճݮͯ͠ը૾ੜ੒
    ͦΕͬΆ͍΋ͷΛબͿ
    

    View Slide

59. step3: ࠶ֶशɾར༻
    pix2pixHD ͰɺಉҰਓ෺Ͱ೥ྸɾੑผΛม͑ͨ
    ΋ͷͷ૊Έ߹ΘͤΛͦΕͧΕݸผʹֶश͢Δ
    ͋ͱ͸Ճྸ͚ͨ͠Ε͹ɺՃྸֶशͨ͠pix2pixϞ
    σϧɺੑస׵͚ͨ͠Ε͹ੑస׵ͨ͠pix2pixϞσ
    ϧɺͱ͍ͬͨܗͰը૾ੜ੒͢Δ͚ͩɻ
    

    View Slide

60. View Slide

61. View Slide

62. View Slide

63. ͜ͷ࿦จΛબΜͩཧ༝
    γϯϓϧͳख๏Λ૊Έ߹Θͤͯطଘͷٕज़Λ௒
    ͍͑ͯΔͷ͕໘ന͍ɻ
    ৠཹͰݱ࣮తʹ༻ҙࠔ೉ͳσʔληοτΛ࡞ͬ
    ͍ͯΔɻଞͷͳʹ͔ʹԠ༻Ͱ͖Δ͔΋ɻ
    

    View Slide

64. Special Thanks
    

    View Slide

65. DeepL Translator (deepl.com)
    https://www.deepl.com/en/translator
    

    View Slide