Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
ICLR2017読み会@DeNA/iclr2017atDeNA_VLAE
Search
Masaki Kozuki
June 17, 2017
Research
29k
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
ICLR2017読み会@DeNA/iclr2017atDeNA_VLAE
Masaki Kozuki
June 17, 2017
Transcript
25 slides
Variational Lossy Autoencoder ICLR 2017 ಡΈձ @ DeNA @crcrpar 2017/6/17
1 / 25
จ • Variational Lossy Autoencoder • Xi Chen (UC Berkeley,
OpenAI), Diederik P. Kingma (OpenAI), Tim Salimans (OpenAI), et al. • දݱֶशͰજࡏมΛ׆༻͢Δ • Bits Back Coding Ͱ VAE ͷજࡏมʹ͍ͭͯͷߟ • જࡏมΛ lossy ʹ͢Δ • જࡏม z ͷ p(z), q(z|x) Λॊೈʹ • decoder ʹ PixelCNN 2 / 25
දهʹ͍ͭͯ • x ∈ Rd: σʔλ. x = ( x0
. . . xd )⊤ • x<i : x ͷ index ͕ i ະຬͷશཁૉ ( x0 . . . xi−1 )⊤ • z: જࡏม • pdata (x): σʔλΛੜ͢Δਅͷ • DKL (p∥q): p ͷ q ʹର͢Δ Kullback Leibler Divergence • θ: ϞσϧʢNNʣͷύϥϝʔλ • AR: PixelCNN ͳͲͷࣗݾճؼܕ NN • H, H: Τϯτϩϐʔ 3 / 25
VAE తؔ log p(X) = ∑ N i=1 log p(x(i))
࣮ࡍͷతؔ L(x; θ) = Eq(z|x) [log p(x|z) − DKL (q(z|x)∥p(z))] - ਖ਼نԽͨ͠ autoencoder ͱΈΕΔɻ VAE ͷ՝ɾऑ • දݱྗ͕ߴ͗͢Δ decoder જࡏมΛແࢹ • જࡏม͕ͭใΛཧͰ͖ͳ͍ 4 / 25
1 ͳͥʁ ײతʹ ཧʢBits Back Codingʣ 2 VLAE ֓ཁ Autoregressive
Flow decoder: PixelCNN 3 ࣮ݧɾ݁Ռ Lossy Comprssion Density Estimation 5 / 25
1 ͳͥʁ 2 VLAE 3 ࣮ݧɾ݁Ռ 6 / 25
ײతʹ... ͦͦɺRNN / AR ҙͷΛۙࣅͰ͖Δ 1 જࡏมʹใ͕΄ͱΜͲؚ·Εͳ͍ʢֶशॳظʣ 2 decoder σʔλΛ࠶ߏ͠Α͏ͱ͢Δ:
p(x|z) → pdecoder (x) 3 ࣄޙɾۙࣅࣄޙͱʹࣄલʹͳΔ p(z|x), q(z|x) → p(z) 7 / 25
গ͠ཧతʹ... VAE ≈ ූ߸Խ 1 σʔλͷຊ࣭ z Λූ߸Խ: p(z) 2
z ͷζϨΛූ߸Խ: p(x|z) ූ߸ͷ͞ʁ naive ʹ Cnaive (x) = Ex∼data,z∼q(z|x) [− log p(z) − log p(x|z)] Bits Back Coding ޮͷͨΊʹ encoder ͷ q(z|x) Λ༻͍Δ 8 / 25
Bits Back Coding q(z|x) ߴʑ H(q(z|x)) ϏοτͰใΛ͑ΒΕΔ ʢʣ ɿreceiver
q(z|x) ΛΈΕΔ߹ͷΈ Bits Back Coding ͷූ߸ Cnaive q(z|x) ͚ͩແବͰ L(x) = Eq(z|x) [log p(x|z) − log q(z|x)] ͳͷͰ CBitsBack (x) = Ex∼data [−L(x)] ≥ H(data) + Ex∼data [DKL (q(z|x)∥p(z|x))] 9 / 25
Bits Back Coding • ූ߸ͷ࠷খԽ = มԼքͷ࠷େԽ → z ͕ΘΕΔͷූ߸Խ͕ޮՌతͳ࣌
• ΑΓਖ਼֬ͳࣄޙʹΑΓมਪߴਫ਼ʹͳ Δ͕ɺݱ࣌Ͱଘࡏ͠ͳ͍ → DKL (≥ 0) ແࢹͰ͖ͳ͍ 10 / 25
Information Preference z ͕ແࢹ͞ΕΔͷ... p(x|z) ͕ pdata (x) Λz ͷใͳ͠ʹϞσϧԽͰ͖Δ߹
1 ࣄޙ pz|x) ͕ p(z) ʹͳΓɺ 2 ۙࣅࣄޙ q(z|x) p(z) ʹͳΔ ∵ KL ߲Λখ͘͢͞ΔͨΊ Information Preference • z ͳ͠ͰہॴతʹϞσϧԽͰ͖Δใہॴతʹ ූ߸Խ • ͦΕҎ֎ͷใ z Λͬͯ෮߸Խ જࡏมΛ hack ͢Δํ๏ɿ free bits, annealing the relative weight of DKL 11 / 25
1 ͳͥʁ 2 VLAE 3 ࣮ݧɾ݁Ռ 12 / 25
Ϟσϧͷ֓ཁ 1 ॊೈͳࣄલ 2 දݱྗͷ͋Δ decoder 13 / 25
ࣄલͷվળ • ٿ໘ΨεɾҰ༷͕ద͔ٙ • જࡏมͷ׆༻ʹෆՄܽ • → autoregressive flow 14
/ 25
Autoregressive Flow normalizing flows ʹ͍ͭͯ • ୯७ͳ͔ΒॊೈͳͷՄٯͳม • general normalizing
flow • volume preserving flow • Jacobian ͷѻ͍ʹҧ͍ AF ͷಛ IAF ͱಉ͡ܭࢉྔ͕ͩϞσϧ͕ΑΓਂ͍ 15 / 25
Inverse Autoregressive Flow zt = µt + σt ⊙ zt−1
log q(zT |x) = − D ∑ i=1 1 2 ϵ2 i + 1 2 log(2π) + T ∑ t=0 log σt,i ਤ 1: IAF ͷ֓ཁ 16 / 25
IAF posterior ॊೈͳࣄޙΛ֫ಘ͍ͯ͠Δʂ ਤ 2: IAF ͷࣄޙ 17 / 25
AF prior ≡ IAF posterior L(x; θ) = Ez∼q(z|x) [log
p(x|z) + log p(z) − log q(z|x)] = Ez∼q(z|x),ϵ=f−1(z) [ log p(x|f(ϵ)) + log u(ϵ) + log det dϵ dz − log q(z|x) ] = Ez∼q(z|x),ϵ=f−1(z) log p(x|f(ϵ)) + log u(ϵ) − ( log q(z|x) − log det dϵ dz ) IAF posterior 18 / 25
1 ͳͥʁ 2 VLAE 3 ࣮ݧɾ݁Ռ 19 / 25
࣮ݧ֓ཁ • త • જࡏม͕େҬతͳใΛ֫ಘ͍ͯ͠Δ͔ • AF prior ͕ IAF
posterior ΑΓ༏Ε͍ͯΔ͔ • AR decoder ʹΑΓີਪఆͷਫ਼্͕͕Δ͔ • ݕূϞσϧ: AF prior & PixelCNN decoder • σʔληοτ: 2 ͷ 28×28 ը૾ • MNIST, OMNIGLOT, Caltech - 101 Silhouettes • ΞʔΩςΫνϟɾજࡏมͷ࣍ݩ౷Ұ 20 / 25
Lossy Compression - MNIST ࠨɿೖྗɺӈɿग़ྗ • Ͳͷࣈ͔Θ͔Δ • ͨͩͷ࠶ߏͰͳ͍ ਤ
3: original & decompressed MNIST 21 / 25
Lossy Compression - OMNIGLOT ࠨɿೖྗɺӈɿग़ྗ • semantics ͕อଘ͞Ε ͍ͯͳ͍ •
λεΫɾσʔληοτ ͝ͱʹใΛಛఆ͢Δ ඞཁ ਤ 4: original & decompressed OMNIGLOT 22 / 25
જࡏม͔ΒͷαϯϓϦϯά • Սۭͷࣈ • େҬతͳಛ ਤ 5: VLAE ͔Βͷαϯϓϧ 23
/ 25
Density Estimation Unconditional Decoder γϯϓϧͳ PixelCNN 24 / 25
AF priorͷޮՌ • ີਪఆ͕վળ • AR ʹΑͬͯજࡏม ͷ࣋ͭใ͕૿Ճ ਤ 6:
AF prior ͷޮՌ 25 / 25
More Decks by Masaki Kozuki
See All by Masaki Kozuki
Sanity Checks for Saliency Maps explained in Japanese language
crcrpar
0
2.7k
Deep Learning for clothes and changing pose
crcrpar
0
950
夏のトップカンファレンス論文読み会 / InnovationMeetup20170918csn_cvpr2k17
crcrpar
3
1.5k
iclr読み会 / iclrjp2017vlae
crcrpar
3
1.1k
Other Decks in Research
See All in Research
SAM3を用いたコマ・吹き出しの 領域検出と分割構造からの読み順推定
kzmssk
0
230
Hitting the buffers
beale
0
410
[Research Paper Analysis] Deep Learning - SigLIP2
auranightsky
0
180
幾何学的深層学習入門 ~3次元構造と対称性を学習するニューラルネットワーク~
taniai
3
1.1k
[webinaire du Ciirpo] Agnelages plein air : des éleveurs partagent leurs pratiques
institutdelelevage
PRO
0
340
[最先端NLP勉強会2026] Checklists Are Better Than Reward Models For Aligning Language Models
nzw0301
1
360
Santé et fertilité des sols : ce que nous disent les pratiques dans les systèmes de polyculture-élevage
institutdelelevage
PRO
0
160
Des solutions pour pâturer davantage en hiver
institutdelelevage
PRO
0
140
Être éleveur dans un monde en transition : Comprendre les changements pour mieux les accompagner
institutdelelevage
PRO
0
160
Pesée et pointage par imagerie 3D : PHENO3D entre en phase de test terrains
institutdelelevage
PRO
0
160
Volatilité des marchés et sécheresse historique, l’élevage bovin viande sous contraintes en 2026
institutdelelevage
PRO
0
240
[最先端NLP勉強会2026] Agentic Rubrics as Contextual Verifiers for SWE Agents
rfujii
1
430
Featured
See All Featured
Redefining SEO in the New Era of Traffic Generation
szymonslowik
1
530
sira's awesome portfolio website redesign presentation
elsirapls
0
510
Designing Powerful Visuals for Engaging Learning
tmiket
1
660
brightonSEO & MeasureFest 2025 - Winning Strategies for Black Friday CRO & PPC - Christian Goodrich
cargoodrich
3
900
Speed Design
sergeychernyshev
33
2.2k
For a Future-Friendly Web
brad_frost
184
11k
How People are Using Generative and Agentic AI to Supercharge Their Products, Projects, Services and Value Streams Today
helenjbeal
2
400
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
123
22k
Paper Plane (Part 1)
katiecoart
PRO
3
12k
Neural Spatial Audio Processing for Sound Field Analysis and Control
skoyamalab
0
630
Future Trends and Review - Lecture 12 - Web Technologies (1019888BNR)
signer
PRO
0
3.8k
The SEO Collaboration Effect
kristinabergwall1
1
620