iclr読み会 / iclrjp2017vlae

いろいろ変更しました。

Transcript

1. Variational Lossy Autoencoder ICLR 2017 ಡΈձ @ DeNA Masaki Kozuki 2017/6/17 Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 1 / 30
2. ࿦จ • Variational Lossy Autoencoder • Xi Chen (UC Berkeley, OpenAI), Diederik P. Kingma (OpenAI), Tim Salimans (OpenAI), et al. • ߩݙ: જࡏม਺Λ Lossy ʹ͢Δ 1 Bits Back Coding Ͱ VAE ͷજࡏม਺ʹ͍ͭͯͷߟ࡯ 2 VLAE • ֶशՄೳͳࣄલ෼෍ɿAutoregressive Flow • ੍ݶΛ՝ͨ͠ PixelCNN Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 2 / 30
3. දهʹ͍ͭͯ • x ∈ Rd: σʔλ. x = ( x0 . . . xd )⊤ • x<i : x ͷ index ͕ i ະຬͷશཁૉ ( x0 . . . xi−1 )⊤ • z: જࡏม਺ • pdata (x): σʔλΛੜ੒͢Δਅͷ෼෍ • DKL (p∥q): KL divergence • θ: ϞσϧʢNNʣͷύϥϝʔλ • AR: PixelCNN ͳͲͷࣗݾճؼܕ NN • H, H: Τϯτϩϐʔ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 3 / 30
4. VAE ໨తؔ਺ L(x; θ) = Eq(z|x) [log p(x|z) − DKL (q(z|x)∥p(z))] VAE ͷ՝୊ɾऑ఺ • autoencoding Ͱ͖Δ৚͕݅ෆ໌ྎ • decoder ͷදݱ͕ߴ͗͢Δͱજࡏม਺͸ແࢹ͞ ΕΔ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 4 / 30
5. 1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 5 / 30
6. 1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 6 / 30
7. ௚ײతʹ... ͦ΋ͦ΋ɺRNN / AR ͸೚ҙͷ෼෍ΛۙࣅͰ͖Δ 1 જࡏม਺ʹ৘ใ͕΄ͱΜͲؚ·Εͳ͍ʢֶशॳظʣ 2 decoder ͸௚઀σʔλΛ࠶ߏ੒͠Α͏ͱ͢Δ: p(x|z) → pdecoder (x) 3 ࣄޙ෼෍ɾۙࣅࣄޙ෼෍ͱ΋ʹࣄલ෼෍ʹͳΔ p(z|x), q(z|x) → p(z) Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 7 / 30
8. গ͠ཧ࿦తʹ... VAE ≈ ූ߸Խ 1 σʔλͷຊ࣭ z Λූ߸Խ: p(z) 2 z ͷζϨΛූ߸Խ: p(x|z) ූ߸ͷ௕͞͸ʁ naive ʹ Cnaive (x) = Ex∼data,z∼q(z|x) [− log p(z) − log p(x|z)] Bits Back Coding ޮ཰ͷͨΊʹ encoder ͷ෼෍ q(z|x) Λ༻͍Δ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 8 / 30
9. Bits Back Coding q(z|x) ߴʑ H(q(z|x)) ϏοτͰ৘ใΛ఻͑ΒΕΔ ʢ஫ʣ ɿड͚औΓख΋ q(z|x) ΛΈΕΔ৔߹ͷΈ Bits Back Coding ͷූ߸௕ Cnaive ͸ q(z|x) ͚ͩແବͰ L(x) = Eq(z|x) [log p(x|z) − log q(z|x)] ͳͷͰ CBitsBack (x) = Ex∼data [−L(x)] ≥ H(data) + Ex∼data [DKL (q(z|x)∥p(z|x))] Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 9 / 30
10. Bits Back Coding • ූ߸௕ͷ࠷খԽ = ม෼Լքͷ࠷େԽ → z ͕࢖ΘΕΔͷ͸ූ߸Խ͕ޮՌతͳ࣌ • ΑΓਖ਼֬ͳࣄޙ෼෍ʹΑΓม෼ਪ࿦͸ߴਫ਼౓ʹͳ Δ͕ɺݱ࣌఺Ͱ͸ଘࡏ͠ͳ͍ → DKL (≥ 0) ͸ແࢹͰ͖ͳ͍ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 10 / 30
11. Information Preference z ͕ແࢹ͞ΕΔͷ͸... p(x|z) ͕ pdata (x) Λz ͷ৘ใͳ͠ʹϞσϧԽͰ͖Δ৔߹ 1 ࣄޙ෼෍ p(z|x) ͕ p(z) ʹͳΓɺ 2 ۙࣅࣄޙ෼෍ q(z|x) ΋ p(z) ʹͳΔ ∵ KL ߲Λখ͘͢͞ΔͨΊ Information Preference • z ͳ͠ͰہॴతʹϞσϧԽͰ͖Δ৘ใ͸ہॴతʹ ෮߸Խ • ͦΕҎ֎ͷ৘ใ͸ z Λ࢖ͬͯ෮߸Խ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 11 / 30
12. 1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 12 / 30
13. Ϟσϧͷ֓ཁ 1 දݱྗͷ͋Δ decoder: LOSSY CODE VIA EXPLICIT INFORMATION PLACEMENT 2 ॊೈͳࣄલ෼෍: LEARNED PRIOR WITH AUTOREGRESSIVE FLOW Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 13 / 30
14. ੍ݶ෇͖PixelCNN Ϟνϕʔγϣϯ • decoder ʹදݱྗ͸ཉ͍͠ • xi ͷ context Λ x<i ʹ͢Δͱ z ͕ແࢹ͞ΕΔ ղܾࡦɿ੍ݶΛ՝͢ WindowAround(i) < x<i Λຬͨ͢ WindowAround(i) Λ ࢖͏ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 14 / 30
15. ࣄલ෼෍ͷվળ: Autoregressive Flow Ϟνϕʔγϣϯ • ա౓ʹ୯७ͳ q(z|x) ͸ֶशΛ๦͛Δ • q(z|x) Λ expressive ʹ͢Δํ๏ e.g. Inverse Autoregressive Flow (IAF) ఏҊख๏: Autoregressive Flow (AF) p(z|x) ΋ֶश͢Δ IAF ͷۙࣅࣄޙ෼෍ͱ౳ՁͰදݱྗ͸উΔ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 15 / 30
16. IAF posterior ਤ 1: IAF Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 16 / 30
17. IAF posterior ਤ 2: IAF ͷࣄޙ෼෍ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 17 / 30
18. ܭࢉϑϩʔ ਤ 3: outline of Inverse Autoregressive Flow Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 18 / 30
19. ܭࢉϑϩʔ ਤ 4: outline of VLAE Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 19 / 30
20. AF prior ͱ IAF posterior L(x; θ) = Ez∼q(z|x) [log p(x|z) + log p(z) − log q(z|x)] = Ez∼q(z|x),ϵ=f−1(z) [ log p(x|f(ϵ)) + log u(ϵ) + log det dϵ dz − log q(z|x) ] = Ez∼q(z|x),ϵ=f−1(z)                   log p(x|f(ϵ)) + log u(ϵ) − ( log q(z|x) − log det dϵ dz ) IAF posterior                   Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 20 / 30
21. 1 ݪҼ 2 VLAE 3 ࣮ݧɾ݁Ռ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 21 / 30
22. ࣮ݧ֓ཁ • ໨త • જࡏม਺͕େҬతͳ৘ใΛ֫ಘ͍ͯ͠Δ͔ • AF prior ͕ IAF posterior ΑΓ༏Ε͍ͯΔ͔ • AR decoder ʹΑΓີ౓ਪఆͷਫ਼౓্͕͕Δ͔ • ݕূϞσϧ: AF prior & PixelCNN decoder • σʔληοτ: 2 ஋ͷ 28×28 ը૾ • MNIST, OMNIGLOT, Caltech - 101 Silhouettes • ΞʔΩςΫνϟɾજࡏม਺ͷ࣍ݩ਺͸౷Ұ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 22 / 30
23. Lossy Compression - MNIST ࠨɿೖྗɺӈɿग़ྗ • Ͳͷ਺ࣈ͔͸Θ͔Δ • ͨͩͷ࠶ߏ੒Ͱ͸ͳ͍ ਤ 5: original & decompressed MNIST Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 23 / 30
24. Lossy Compression - OMNIGLOT ࠨɿೖྗɺӈɿग़ྗ • semantics ͕อଘ͞Ε ͍ͯͳ͍ • λεΫɾσʔληοτ ͝ͱʹ৘ใΛಛఆ͢Δ ඞཁ ਤ 6: original & decompressed OMNIGLOT Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 24 / 30
25. જࡏม਺͔ΒͷαϯϓϦϯά • Սۭͷ਺ࣈ • େҬతͳಛ௃ ਤ 7: VLAE ͔Βͷαϯϓϧ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 25 / 30
26. Density Estimation Unconditional Decoder ͸γϯϓϧͳ PixelCNN Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 26 / 30
27. AF prior/ AR decoderͷޮՌ ਤ 8: AR decoder ͷޮՌ ਤ 9: AF prior ͷޮՌ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 27 / 30
28. cifar-10 Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 28 / 30
29. cifar-10 ਤ 11: cifar-10 NLL • PixelCNN++ʹΘ͔ͣ ʹྼΔ • (a)-(c): ৭৘ใ͕མͪ ͍ͯΔ • (d): p(xi |z, GrayScale(xWindowAround(i) )) Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 29 / 30
30. Reviewൈਮ • interesting • Bits Back Coding • Autoregressive Flow • cifar-10 ͳͲͰ΋࣮ݧ͢Δ΂͖ Masaki Kozuki Variational Lossy Autoencoder 2017/6/17 30 / 30