Upgrade to Pro — share decks privately, control downloads, hide ads and more …

【ICML読み会】Unsupervised Deep Embedding for Clustering Analysis

【ICML読み会】Unsupervised Deep Embedding for Clustering Analysis

18d356eace1ae9fb3c1188871990a482?s=128

Hayato Maki

July 16, 2016
Tweet

Transcript

  1. ICML2016࿦จ঺հ Unsupervised Deep Embedding for Clustering Analysis Ross Girshick Jungian

    Xie Ali Farhadi University of Washington Facebook AI Research University of Washington ൃදऀ ਅ໦༐ਓ ಸྑઌ୺Պֶٕज़େֶӃେֶ ৘ใՊֶݚڀՊ ത࢜ޙظ՝ఔ ஌ೳίϛϡχέʔγϣϯݚڀࣨ 2016/07/16 @NAIST
  2. 3ߦͰཁ໿ • ର৅ɿݹయతͳΫϥελϦϯά໰୊ • ख๏ɿਂ૚ֶशΛར༻ͨ࣍͠ݩ࡟ݮύ
 ϥϝλͱΫϥελϦϯάͷಉ࣌
 ࠷దԽ • ݁Ռɿैདྷख๏ΑΓ΋ߴ͍ਫ਼౓ɼ୹͍
 ܭࢉ࣌ؒΛ࣮ݱ

  3. ΫϥελϦϯάͷؔ࿈ݚڀ • k-means ٴͼ ࠞ߹ਖ਼ن෼෍Ϟσϧ(GMM) • ೖྗͷ࣍ݩ͕ߴ͍ͱࣦഊ͠΍͍͢ • ࣍ݩ࡟ݮͱΫϥελϦϯάΛಉ࣌ʹߦ͏ख๏ •

    ௿࣍ݩۭؒʹࣸ૾ɼࣸ૾ͨ͠ઌͰΫϥελϦϯά • ैདྷख๏͸ઢܗࣸ૾ͷΈ • εϖΫτϥϧɾΫϥελϦϯά • σʔλͷάϥϑߏ଄Λར༻͢Δख๏ • k-meansΑΓྑ͍݁ՌʹͳΔ͜ͱ͕ଟ͍ • ܭࢉྔ͕αϯϓϧ਺ͷ̎৐·ͨ͸̐৐ʹൺྫ
  4. ه߸ • σʔλ਺ɿ • σʔλɿ • Ϋϥελͷ਺ʢࣄલʹܾఆʣɿ • ࣸ૾ɿ •

    ɹ ͷ࣍ݩ <<< ͷ࣍ݩ • ࣸ૾ͷύϥϝλ ΛDNNͰֶश • ࣸ૾ઌͷσʔλɿ • ηϯτϩΠυʢΫϥελΛ୅ද͢Δ఺ʣɿ n { xi 2 X }n i=1 k zi = f✓( xi) ✓ {zi 2 Z}n i=1 {µj 2 Z}k i=1 zi = f✓( xi) zi = f✓( xi)
  5. ఏҊ๏ͷྲྀΕ ॳظԽ ࣍ݩ࡟ݮ ΫϥελׂΓ౰ͯ KL divergenceܭࢉ ύϥϝλߋ৽

  6. ࣍ݩ࡟ݮ • ਂ૚ֶशΛར༻ͨ͠ඇઢܗͳ௿࣍ݩԽࣸ૾ f✓ : X ! Z { xi

    2 X }n i=1 zi = • ڭࢣͳֶ͠शͷͨΊɼަ ࠩݕূ๏ʹΑΔϋΠύʔύ ϥϝλͷௐ੔͸Ͱ͖ͳ͍ • ͦͷͨΊɼΑ͘࢖ΘΕΔ ωοτϫʔΫߏ੒Λ࢖༻ • ֤૚ͷ࣍ݩ͸
 (input)-500-500-2000-10 • શ݁߹ [van der Maaten, 09]
  7. ΫϥελׂΓ౰ͯ Soft Asignment • ࣸ૾͞Εͨσʔλ ͱηϯτϩΠυ ͷྨࣅ ౓ई౓ (soft assignment)


    
 
 
 
 
 
 ͸ɼ ͕̹൪໨ͷΫϥελʹೖΔ֬཰ͱͯ͠ ղऍͰ͖Δɽ qij = 1 + kzi µj0 k2/↵ (↵+1)/2 P j0 (1 + kzi µj0 k2/↵) (↵+1)/2 ↵ = 1 {zi 2 Z}n i=1 µj [van der Maaten & Hinton, 08] qij = 1 + kzi µj0 k2/↵ (↵+1)/2 P j0 (1 + kzi µj0 k2/↵) (↵+1)/2 {zi 2 Z}n i=1 • ڭࢣͳֶ͠शʹ͓͍ͯ͸ɼަࠩݕূ๏͸࢖͑ͳ͍ ͨΊɼ ʹݻఆɽ
  8. KLμΠόʔδΣϯεʹΑΔDNNֶश • ࢑ఆతͳׂΓ౰ͯ෼෍ • ໨ඪ෼෍ʢཧ૝తͳΫϥελϦϯάΛߦ͏ͱߟ͑Β ΕΔ෼෍ʣ • PͱQͷKLμΠόʔδΣϯεΛ࠷খԽ͢ΔΑ͏ʹDNN Λֶश •

    Pͷઃఆ͕ຊख๏ͷΩϞ
  9. ύϥϝλߋ৽ • DNNͷύϥϝλθ ͱ
 ηϯτϩΠυ μj Λߋ৽ • SGDͰߋ৽ (θ͸όοΫϓϩύήʔγϣϯ)

  10. ॳظԽ • DNNͷॳظԽɿ Stacked Auto Encoder Λར༻ • ηϯτϩΠυͷॳظԽɿॳظԽDNNΛར༻ͯ࣍͠ݩ࡟ ݮ͠ɼࣸ૾ઌͰk-means

  11. ࣮ݧ • σʔληοτ • ൺֱख๏ • k-means • LDGMI (εϖΫτϥϧɾΫϥελϦϯά)

    • SEC (εϖΫτϥϧɾΫϥελϦϯά) • Without back propagation
  12. ࣮૷ • Stacked Auto EncoderͷॳظԽ • ฏۉ0ɼඪ४ภࠩ0.01ͷਖ਼ن෼෍Λ࢖ͬͨ ཚ਺ͰॏΈΛॳظԽ • ֤૚͝ͱʹ50000ճ൓෮ʢ20%Dropoutʣ

    • Auto EncoderશମͰ100000ճ൓෮ͯ͠ fine tuning (Dropoutແ͠) • ϛχόοναΠζ=256 • ֶश཰=0.1 ←20000൓෮ຖʹ1/10
  13. ࣮૷ • ηϯτϩΠυͷॳظԽ • ҟͳΔॳظ஋Ͱ20ճ࣮ߦͯ͠ϕετͳ΋ ͷΛબ୒ • KLμΠόʔδΣϯεͷ࠷খԽ • ֶश཰=0.01

    (ݻఆ) • ऩଋ൑ఆ • ΫϥελͷׂΓ౰͕ͯมԽ͢Δσʔλ͕ 0.1%ҎԼʹͳΔ·Ͱ
  14. ධՁج४ • Unsupervised Clustering Accuracy (ACC) • pi ɿਅͷϥϕϧ •

    qi ɿΞϧΰϦζϜ͕ग़ྗͨ͠ϥϕϧ • map()ɿϥϕϧ͔ΒΫϥελ΁ͷ࠷దͳϚοϐϯά
  15. ݁Ռ • ఏҊ๏͕ϕετͷੑೳ • REUTERSʹ͍ͭͯ • ఏҊ๏ͷֶश࣌ؒ͸30෼ఔ౓ • LDMGIͱSEC͸਺ϲ݄Ҏ্ͷܭࢉ࣌ؒͱςϥ ୯ҐͷϝϞϦ͕ඞཁ

  16. ϋΠύʔύϥϝλʹର͢Δؤ݈ੑ • ҟͳΔ9ͭͷϋΠύʔύϥϝλʢΞχʔϦϯάʣ
 ͰੑೳΛൺֱ • ఏҊ๏͸ϋΠύʔύϥϝλͷมಈʹରͯ͠ؤ݈ɼ
 ͔ͭσʔληοτʹඇґଘ • ڭࢣͳֶ͠शʹ͓͍ͯ͸ॏཁͳੑ࣭

  17. ໨ඪ෼෍Pͷੑ࣭ • qij ͕େ͖΄Ͳ(֬৴౓͕େ͖͍΄Ͳ)ɼޯ഑͕ େ͖͘ͳΔ
 ˠP͸๬·͍͠ੑ࣭Λ͍࣋ͬͯΔ

  18. ࣍ݩ࡟ݮͷ൓෮࠷దԽʹΑΔޮՌ • t-SNEΛར༻ͨ͠ՄࢹԽ [van der Maaten & Hinton, 08] •

    ߋ৽͕ਐΉ΄ͲΫϥελʔͷ෼཭͕໌֬ʹ
  19. Auto EncoderʹΑΔಛ௃நग़ͷޮՌ • Auto EncoderͰಛ௃நग़ˠ֤ΞϧΰϦζϜͰॲཧ • Auto EncoderʹΑΔߩݙ͕େ͖͍

  20. ෆۉҰͳσʔληοτʹର͢Δؤ݈ੑ • αϯϓϧ਺͕࠷খͷΫϥεͷαϯϓϧ਺Λɼ ࠷େͷαϯϓϧ਺ͷΫϥεͷrmin ഒʹઃఆɼ ͦͷଞͷΫϥε͸0.1ͣͭ૿΍͍ͯ͘͠ • ఏҊ๏͸αϯϓϧ਺ͷෆۉҰੑʹରͯ͠ؤ݈

  21. ݁࿦ • ఏҊ๏͸ɼਂ૚ֶशΛར༻ͨ࣍͠ݩ࡟ݮʹΑΓΫϥ ελϦϯάͷੑೳΛ޲্ • ࣍ݩ࡟ݮͷύϥϝλͱΫϥελϦϯάͷ݁ՌΛಉ ࣌ʹ࠷దԽ • ΫϥελϦϯά࢑ఆతͳग़ྗͱ໨ඪ෼෍ͱͷKL μΠόʔδΣϯεΛଛࣦؔ਺ͱͯ͠όοΫϓϩύ

    ήʔγϣϯ • ैདྷ๏ΑΓߴਫ਼౓͔ͭߴ଎ʢܭࢉ࣌ؒ͸αϯϓϧ਺ ʹରͯ͠ઢܗʹൺྫʣɼσʔληοτඇґଘɼϋΠύʔ ύϥϝλඇґଘɼαϯϓϧ਺ͷෆۉҰੑʹରͯ͠ؤ݈
  22. Ҏ্