【ICML読み会】Unsupervised Deep Embedding for Clustering Analysis

Transcript

1. ICML2016࿦จ঺հ Unsupervised Deep Embedding for Clustering Analysis Ross Girshick Jungian

   Xie Ali Farhadi University of Washington Facebook AI Research University of Washington ൃදऀ ਅ໦༐ਓ ಸྑઌ୺Պֶٕज़େֶӃେֶ ৘ใՊֶݚڀՊ ത࢜ޙظ՝ఔ ஌ೳίϛϡχέʔγϣϯݚڀࣨ 2016/07/16 @NAIST

2. 3ߦͰཁ໿ • ର৅ɿݹయతͳΫϥελϦϯά໰୊ • ख๏ɿਂ૚ֶशΛར༻ͨ࣍͠ݩ࡟ݮύ
 ϥϝλͱΫϥελϦϯάͷಉ࣌
 ࠷దԽ • ݁Ռɿैདྷख๏ΑΓ΋ߴ͍ਫ਼౓ɼ୹͍
 ܭࢉ࣌ؒΛ࣮ݱ

3. ΫϥελϦϯάͷؔ࿈ݚڀ • k-means ٴͼ ࠞ߹ਖ਼ن෼෍Ϟσϧ(GMM) • ೖྗͷ࣍ݩ͕ߴ͍ͱࣦഊ͠΍͍͢ • ࣍ݩ࡟ݮͱΫϥελϦϯάΛಉ࣌ʹߦ͏ख๏ •

   ௿࣍ݩۭؒʹࣸ૾ɼࣸ૾ͨ͠ઌͰΫϥελϦϯά • ैདྷख๏͸ઢܗࣸ૾ͷΈ • εϖΫτϥϧɾΫϥελϦϯά • σʔλͷάϥϑߏ଄Λར༻͢Δख๏ • k-meansΑΓྑ͍݁ՌʹͳΔ͜ͱ͕ଟ͍ • ܭࢉྔ͕αϯϓϧ਺ͷ̎৐·ͨ͸̐৐ʹൺྫ

4. ه߸ • σʔλ਺ɿ • σʔλɿ • Ϋϥελͷ਺ʢࣄલʹܾఆʣɿ • ࣸ૾ɿ •

   ɹ ͷ࣍ݩ <<< ͷ࣍ݩ • ࣸ૾ͷύϥϝλ ΛDNNͰֶश • ࣸ૾ઌͷσʔλɿ • ηϯτϩΠυʢΫϥελΛ୅ද͢Δ఺ʣɿ n { xi 2 X }n i=1 k zi = f✓( xi) ✓ {zi 2 Z}n i=1 {µj 2 Z}k i=1 zi = f✓( xi) zi = f✓( xi)

5. ఏҊ๏ͷྲྀΕ ॳظԽ ࣍ݩ࡟ݮ ΫϥελׂΓ౰ͯ KL divergenceܭࢉ ύϥϝλߋ৽

6. ࣍ݩ࡟ݮ • ਂ૚ֶशΛར༻ͨ͠ඇઢܗͳ௿࣍ݩԽࣸ૾ f✓ : X ! Z { xi

   2 X }n i=1 zi = • ڭࢣͳֶ͠शͷͨΊɼަ ࠩݕূ๏ʹΑΔϋΠύʔύ ϥϝλͷௐ੔͸Ͱ͖ͳ͍ • ͦͷͨΊɼΑ͘࢖ΘΕΔ ωοτϫʔΫߏ੒Λ࢖༻ • ֤૚ͷ࣍ݩ͸
 (input)-500-500-2000-10 • શ݁߹ [van der Maaten, 09]

7. ΫϥελׂΓ౰ͯ Soft Asignment • ࣸ૾͞Εͨσʔλ ͱηϯτϩΠυ ͷྨࣅ ౓ई౓ (soft assignment)


   
 
 
 
 
 
 ͸ɼ ͕̹൪໨ͷΫϥελʹೖΔ֬཰ͱͯ͠ ղऍͰ͖Δɽ qij = 1 + kzi µj0 k2/↵ (↵+1)/2 P j0 (1 + kzi µj0 k2/↵) (↵+1)/2 ↵ = 1 {zi 2 Z}n i=1 µj [van der Maaten & Hinton, 08] qij = 1 + kzi µj0 k2/↵ (↵+1)/2 P j0 (1 + kzi µj0 k2/↵) (↵+1)/2 {zi 2 Z}n i=1 • ڭࢣͳֶ͠शʹ͓͍ͯ͸ɼަࠩݕূ๏͸࢖͑ͳ͍ ͨΊɼ ʹݻఆɽ

8. KLμΠόʔδΣϯεʹΑΔDNNֶश • ࢑ఆతͳׂΓ౰ͯ෼෍ • ໨ඪ෼෍ʢཧ૝తͳΫϥελϦϯάΛߦ͏ͱߟ͑Β ΕΔ෼෍ʣ • PͱQͷKLμΠόʔδΣϯεΛ࠷খԽ͢ΔΑ͏ʹDNN Λֶश •

   Pͷઃఆ͕ຊख๏ͷΩϞ

9. ύϥϝλߋ৽ • DNNͷύϥϝλθ ͱ
 ηϯτϩΠυ μj Λߋ৽ • SGDͰߋ৽ (θ͸όοΫϓϩύήʔγϣϯ)

10. ॳظԽ • DNNͷॳظԽɿ Stacked Auto Encoder Λར༻ • ηϯτϩΠυͷॳظԽɿॳظԽDNNΛར༻ͯ࣍͠ݩ࡟ ݮ͠ɼࣸ૾ઌͰk-means

11. ࣮ݧ • σʔληοτ • ൺֱख๏ • k-means • LDGMI (εϖΫτϥϧɾΫϥελϦϯά)

    • SEC (εϖΫτϥϧɾΫϥελϦϯά) • Without back propagation

12. ࣮૷ • Stacked Auto EncoderͷॳظԽ • ฏۉ0ɼඪ४ภࠩ0.01ͷਖ਼ن෼෍Λ࢖ͬͨ ཚ਺ͰॏΈΛॳظԽ • ֤૚͝ͱʹ50000ճ൓෮ʢ20%Dropoutʣ

    • Auto EncoderશମͰ100000ճ൓෮ͯ͠ fine tuning (Dropoutແ͠) • ϛχόοναΠζ=256 • ֶश཰=0.1 ←20000൓෮ຖʹ1/10

13. ࣮૷ • ηϯτϩΠυͷॳظԽ • ҟͳΔॳظ஋Ͱ20ճ࣮ߦͯ͠ϕετͳ΋ ͷΛબ୒ • KLμΠόʔδΣϯεͷ࠷খԽ • ֶश཰=0.01

    (ݻఆ) • ऩଋ൑ఆ • ΫϥελͷׂΓ౰͕ͯมԽ͢Δσʔλ͕ 0.1%ҎԼʹͳΔ·Ͱ

14. ධՁج४ • Unsupervised Clustering Accuracy (ACC) • pi ɿਅͷϥϕϧ •

    qi ɿΞϧΰϦζϜ͕ग़ྗͨ͠ϥϕϧ • map()ɿϥϕϧ͔ΒΫϥελ΁ͷ࠷దͳϚοϐϯά

15. ݁Ռ • ఏҊ๏͕ϕετͷੑೳ • REUTERSʹ͍ͭͯ • ఏҊ๏ͷֶश࣌ؒ͸30෼ఔ౓ • LDMGIͱSEC͸਺ϲ݄Ҏ্ͷܭࢉ࣌ؒͱςϥ ୯ҐͷϝϞϦ͕ඞཁ

16. ϋΠύʔύϥϝλʹର͢Δؤ݈ੑ • ҟͳΔ9ͭͷϋΠύʔύϥϝλʢΞχʔϦϯάʣ
 ͰੑೳΛൺֱ • ఏҊ๏͸ϋΠύʔύϥϝλͷมಈʹରͯ͠ؤ݈ɼ
 ͔ͭσʔληοτʹඇґଘ • ڭࢣͳֶ͠शʹ͓͍ͯ͸ॏཁͳੑ࣭

17. ໨ඪ෼෍Pͷੑ࣭ • qij ͕େ͖΄Ͳ(֬৴౓͕େ͖͍΄Ͳ)ɼޯ഑͕ େ͖͘ͳΔ
 ˠP͸๬·͍͠ੑ࣭Λ͍࣋ͬͯΔ

18. ࣍ݩ࡟ݮͷ൓෮࠷దԽʹΑΔޮՌ • t-SNEΛར༻ͨ͠ՄࢹԽ [van der Maaten & Hinton, 08] •

    ߋ৽͕ਐΉ΄ͲΫϥελʔͷ෼཭͕໌֬ʹ

19. Auto EncoderʹΑΔಛ௃நग़ͷޮՌ • Auto EncoderͰಛ௃நग़ˠ֤ΞϧΰϦζϜͰॲཧ • Auto EncoderʹΑΔߩݙ͕େ͖͍

20. ෆۉҰͳσʔληοτʹର͢Δؤ݈ੑ • αϯϓϧ਺͕࠷খͷΫϥεͷαϯϓϧ਺Λɼ ࠷େͷαϯϓϧ਺ͷΫϥεͷrmin ഒʹઃఆɼ ͦͷଞͷΫϥε͸0.1ͣͭ૿΍͍ͯ͘͠ • ఏҊ๏͸αϯϓϧ਺ͷෆۉҰੑʹରͯ͠ؤ݈

21. ݁࿦ • ఏҊ๏͸ɼਂ૚ֶशΛར༻ͨ࣍͠ݩ࡟ݮʹΑΓΫϥ ελϦϯάͷੑೳΛ޲্ • ࣍ݩ࡟ݮͷύϥϝλͱΫϥελϦϯάͷ݁ՌΛಉ ࣌ʹ࠷దԽ • ΫϥελϦϯά࢑ఆతͳग़ྗͱ໨ඪ෼෍ͱͷKL μΠόʔδΣϯεΛଛࣦؔ਺ͱͯ͠όοΫϓϩύ

    ήʔγϣϯ • ैདྷ๏ΑΓߴਫ਼౓͔ͭߴ଎ʢܭࢉ࣌ؒ͸αϯϓϧ਺ ʹରͯ͠ઢܗʹൺྫʣɼσʔληοτඇґଘɼϋΠύʔ ύϥϝλඇґଘɼαϯϓϧ਺ͷෆۉҰੑʹରͯ͠ؤ݈

22. Ҏ্