Share
Embed
Transcript
30 slides
-
Variational Lossy Autoencoder ICLR 2017 ಡΈձ @ DeNA Masaki Kozuki
2017/6/17 Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 1 / 30 -
จ • Variational Lossy Autoencoder • Xi Chen (UC Berkeley,
OpenAI), Diederik P. Kingma (OpenAI), Tim Salimans (OpenAI), et al. • ߩݙ: જࡏมΛ Lossy ʹ͢Δ 1 Bits Back Coding Ͱ VAE ͷજࡏมʹ͍ͭͯͷߟ 2 VLAE • ֶशՄೳͳࣄલɿAutoregressive Flow • ੍ݶΛ՝ͨ͠ PixelCNN Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 2 / 30 -
දهʹ͍ͭͯ • x ∈ Rd: σʔλ. x = ( x0
. . . xd )⊤ • x<i : x ͷ index ͕ i ະຬͷશཁૉ ( x0 . . . xi−1 )⊤ • z: જࡏม • pdata (x): σʔλΛੜ͢Δਅͷ • DKL (p∥q): KL divergence • θ: ϞσϧʢNNʣͷύϥϝʔλ • AR: PixelCNN ͳͲͷࣗݾճؼܕ NN • H, H: Τϯτϩϐʔ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 3 / 30 -
VAE తؔ L(x; θ) = Eq(z|x) [log p(x|z) − DKL
(q(z|x)∥p(z))] VAE ͷ՝ɾऑ • autoencoding Ͱ͖Δ͕݅ෆ໌ྎ • decoder ͷදݱ͕ߴ͗͢Δͱજࡏมແࢹ͞ ΕΔ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 4 / 30 -
1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 5 / 30 -
1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 6 / 30 -
ײతʹ... ͦͦɺRNN / AR ҙͷΛۙࣅͰ͖Δ 1 જࡏมʹใ͕΄ͱΜͲؚ·Εͳ͍ʢֶशॳظʣ 2 decoder σʔλΛ࠶ߏ͠Α͏ͱ͢Δ:
p(x|z) → pdecoder (x) 3 ࣄޙɾۙࣅࣄޙͱʹࣄલʹͳΔ p(z|x), q(z|x) → p(z) Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 7 / 30 -
গ͠ཧతʹ... VAE ≈ ූ߸Խ 1 σʔλͷຊ࣭ z Λූ߸Խ: p(z) 2
z ͷζϨΛූ߸Խ: p(x|z) ූ߸ͷ͞ʁ naive ʹ Cnaive (x) = Ex∼data,z∼q(z|x) [− log p(z) − log p(x|z)] Bits Back Coding ޮͷͨΊʹ encoder ͷ q(z|x) Λ༻͍Δ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 8 / 30 -
Bits Back Coding q(z|x) ߴʑ H(q(z|x)) ϏοτͰใΛ͑ΒΕΔ ʢʣ ɿड͚औΓख q(z|x)
ΛΈΕΔ߹ͷΈ Bits Back Coding ͷූ߸ Cnaive q(z|x) ͚ͩແବͰ L(x) = Eq(z|x) [log p(x|z) − log q(z|x)] ͳͷͰ CBitsBack (x) = Ex∼data [−L(x)] ≥ H(data) + Ex∼data [DKL (q(z|x)∥p(z|x))] Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 9 / 30 -
Bits Back Coding • ූ߸ͷ࠷খԽ = มԼքͷ࠷େԽ → z ͕ΘΕΔͷූ߸Խ͕ޮՌతͳ࣌
• ΑΓਖ਼֬ͳࣄޙʹΑΓมਪߴਫ਼ʹͳ Δ͕ɺݱ࣌Ͱଘࡏ͠ͳ͍ → DKL (≥ 0) ແࢹͰ͖ͳ͍ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 10 / 30 -
Information Preference z ͕ແࢹ͞ΕΔͷ... p(x|z) ͕ pdata (x) Λz ͷใͳ͠ʹϞσϧԽͰ͖Δ߹
1 ࣄޙ p(z|x) ͕ p(z) ʹͳΓɺ 2 ۙࣅࣄޙ q(z|x) p(z) ʹͳΔ ∵ KL ߲Λখ͘͢͞ΔͨΊ Information Preference • z ͳ͠ͰہॴతʹϞσϧԽͰ͖Δใہॴతʹ ෮߸Խ • ͦΕҎ֎ͷใ z Λͬͯ෮߸Խ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 11 / 30 -
1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 12 / 30 -
Ϟσϧͷ֓ཁ 1 දݱྗͷ͋Δ decoder: LOSSY CODE VIA EXPLICIT INFORMATION PLACEMENT
2 ॊೈͳࣄલ: LEARNED PRIOR WITH AUTOREGRESSIVE FLOW Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 13 / 30 -
੍ݶ͖PixelCNN Ϟνϕʔγϣϯ • decoder ʹදݱྗཉ͍͠ • xi ͷ context Λ
x<i ʹ͢Δͱ z ͕ແࢹ͞ΕΔ ղܾࡦɿ੍ݶΛ՝͢ WindowAround(i) < x<i Λຬͨ͢ WindowAround(i) Λ ͏ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 14 / 30 -
ࣄલͷվળ: Autoregressive Flow Ϟνϕʔγϣϯ • աʹ୯७ͳ q(z|x) ֶशΛ͛Δ • q(z|x)
Λ expressive ʹ͢Δํ๏ e.g. Inverse Autoregressive Flow (IAF) ఏҊख๏: Autoregressive Flow (AF) p(z|x) ֶश͢Δ IAF ͷۙࣅࣄޙͱՁͰදݱྗউΔ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 15 / 30 -
IAF posterior ਤ 1: IAF Masaki Kozuki Variational Lossy Autoencoder
2017/6/17 16 / 30 -
IAF posterior ਤ 2: IAF ͷࣄޙ Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 17 / 30 -
ܭࢉϑϩʔ ਤ 3: outline of Inverse Autoregressive Flow Masaki Kozuki
Variational Lossy Autoencoder 2017/6/17 18 / 30 -
ܭࢉϑϩʔ ਤ 4: outline of VLAE Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 19 / 30 -
AF prior ͱ IAF posterior L(x; θ) = Ez∼q(z|x) [log
p(x|z) + log p(z) − log q(z|x)] = Ez∼q(z|x),ϵ=f−1(z) [ log p(x|f(ϵ)) + log u(ϵ) + log det dϵ dz − log q(z|x) ] = Ez∼q(z|x),ϵ=f−1(z) log p(x|f(ϵ)) + log u(ϵ) − ( log q(z|x) − log det dϵ dz ) IAF posterior Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 20 / 30 -
1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 21 / 30 -
࣮ݧ֓ཁ • త • જࡏม͕େҬతͳใΛ֫ಘ͍ͯ͠Δ͔ • AF prior ͕ IAF
posterior ΑΓ༏Ε͍ͯΔ͔ • AR decoder ʹΑΓີਪఆͷਫ਼্͕͕Δ͔ • ݕূϞσϧ: AF prior & PixelCNN decoder • σʔληοτ: 2 ͷ 28×28 ը૾ • MNIST, OMNIGLOT, Caltech - 101 Silhouettes • ΞʔΩςΫνϟɾજࡏมͷ࣍ݩ౷Ұ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 22 / 30 -
Lossy Compression - MNIST ࠨɿೖྗɺӈɿग़ྗ • Ͳͷࣈ͔Θ͔Δ • ͨͩͷ࠶ߏͰͳ͍ ਤ
5: original & decompressed MNIST Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 23 / 30 -
Lossy Compression - OMNIGLOT ࠨɿೖྗɺӈɿग़ྗ • semantics ͕อଘ͞Ε ͍ͯͳ͍ •
λεΫɾσʔληοτ ͝ͱʹใΛಛఆ͢Δ ඞཁ ਤ 6: original & decompressed OMNIGLOT Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 24 / 30 -
જࡏม͔ΒͷαϯϓϦϯά • Սۭͷࣈ • େҬతͳಛ ਤ 7: VLAE ͔Βͷαϯϓϧ Masaki
Kozuki Variational Lossy Autoencoder 2017/6/17 25 / 30 -
Density Estimation Unconditional Decoder γϯϓϧͳ PixelCNN Masaki Kozuki Variational Lossy
Autoencoder 2017/6/17 26 / 30 -
AF prior/ AR decoderͷޮՌ ਤ 8: AR decoder ͷޮՌ ਤ
9: AF prior ͷޮՌ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 27 / 30 -
cifar-10 ਤ 11: cifar-10 NLL • PixelCNN++ʹΘ͔ͣ ʹྼΔ • (a)-(c):
৭ใ͕མͪ ͍ͯΔ • (d): p(xi |z, GrayScale(xWindowAround(i) )) Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 29 / 30 -
Reviewൈਮ • interesting • Bits Back Coding • Autoregressive Flow
• cifar-10 ͳͲͰ࣮ݧ͢Δ͖ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 30 / 30