Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hivemallmtup 20160908 factorizations

Avatar for fan-k-oomori fan-k-oomori
September 08, 2016

Hivemallmtup 20160908 factorizations

Explains mathematical models of matrix factorization and factorization machines (in Japanese)

Avatar for fan-k-oomori

fan-k-oomori

September 08, 2016

Other Decks in Programming

Transcript

  1. ࠓճ༻͍Δςετσʔλ λΠλχοΫ (TI) ϊ ο ς Ο ϯ άώϧͷ࿀ਓ (NH)

    ελʔ΢Υʔ ζ (SW) ελʔτϨο Ϋ (ST) Alice (A) 5 3 1 2010-1 2010-2 2010-4 Bob (B) 4 5 2009-5 2009-8 Charlie (C) 1 5 2009-9 2009-12 Factorization MachinesΛఏҊͨ͠Rendle͞Μͷݪ࿦จΑ Γഈआɻ ্ஈ=֤өըʹର͢ΔධՁ஋ Լஈ=ධՁΛߦͬͨ೥݄ ʲ໰୊ઃఆʳۭཝͷະධՁ஋Λ༧ଌ͠ɺϨίϝϯυ͍ͨ͠ 2
  2. Matrix Factorization (MF) ઌ΄ͲͷσʔλΛߦྻ(matrix)ͷܗͰදݱ͢Δͱ V =   5 3

    1 4 5 1 5   (1) ͜ΕΛ2ͭͷߦྻͷੵʹ෼ղ͢Δ(factorize)   5 3 1 4 5 1 5   =   ∗ ∗ ∗ ∗ ∗ ∗   ( ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ) (2) 3
  3. 2 × 2ߦྻͷੵ͸ ( u1 v1 u2 v2 ) (

    x1 x2 y1 y2 ) = ( u1 x1 + v1 y1 u1 x2 + v1 y2 u2 x1 + v2 y1 u2 x2 + v2 y2 ) ͜͜Ͱ ⃗ wi = ( ui vi ) , ⃗ zi = ( xi yi ) (i = 1, 2)ͱ͓͘ͱ্ࣜ͸ ( ⃗ wT 1 ⃗ wT 2 ) ( ⃗ z1 ⃗ z2 ) = ( ⃗ w1 · ⃗ z1 ⃗ w1 · ⃗ z2 ⃗ w2 · ⃗ z1 ⃗ w2 · ⃗ z2 ) ͱॻ͖௚ͤΔɻ ஫) ߦྻه๏ͷ ⃗ wT ⃗ z ͱϕΫτϧ಺ੵ ⃗ w · ⃗ z ͸౳Ձɻ 4
  4. (2)ࣜΛಉ༷ʹϕΫτϧͰॻ͖ද͢ͱ   5 3 1 4 5 1 5

      =   ∗ ∗ ∗ ∗ ∗ ∗   ( ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ) =   ⃗ vT A ⃗ vT B ⃗ vT C   ( ⃗ vT I ⃗ vNH ⃗ vSW ⃗ vST ) =   ⃗ vA · ⃗ vT I ⃗ vA · ⃗ vNH ⃗ vA · ⃗ vSW ⃗ vA · ⃗ vST ⃗ vB · ⃗ vT I ⃗ vB · ⃗ vNH ⃗ vB · ⃗ vSW ⃗ vB · ⃗ vST ⃗ vC · ⃗ vT I ⃗ vC · ⃗ vNH ⃗ vC · ⃗ vSW ⃗ vC · ⃗ vST   ղऍɿ⃗ vA ͸ Aliceͷಛ௃Λද͢ϕΫτϧྔɺ⃗ vB ͸ Bobͷಛ௃Λද͢ ϕΫτϧྔɺʜɺ⃗ vST ͸ελʔτϨοΫͷಛ௃Λද͢ϕΫτϧྔͱߟ ͑ΔɻධՁ஋͸ɺਓͱ࡞඼ʹରԠ͢ΔϕΫτϧͷ಺ੵͱͯ͠ܭࢉ͢Δ 5
  5. ༩͑ΒΕͨσʔλ͔Β͸ϕΫτϧͷ࣍ݩʢཁૉ਺ʣ͸ܾ·Β ͳ͍ ˠ ϞσϧͷܗΛܾΊΔϋΠύʔύϥϝʔλɻ ͜͜Ͱ͸r࣍ݩͱ͢Δɻ ݩͷߦྻͷαΠζʹൺ΂ͯrΛখ͘͞Ͱ͖Ε͹࣍ݩ࡟ݮʹͭ ͳ͕Δɻ n × m

    vs. (n + m) × r ਓͱөը͕r࣍ݩͷಛ௃ۭؒ಺ͷϕΫτϧͱͯ͠දݱ͞Εɺ ͦͷ಺ੵΛܭࢉ͢Δ͜ͱʹΑͬͯ૬ੑʢධՁͷߴ௿ʣ͕Θ͔ Δͱ͍͏ߏ଄ʹͳ͍ͬͯΔɻ • ௚ަˠ௿ධՁ • ฏߦˠߴධՁ 6
  6. ͳͥະධՁͷ஋ΛਪఆͰ͖Δͷ͔ʁ • ͢ͰʹධՁ͕ͳ͞Εͨ৘ใʢྫ: ⃗ vA · ⃗ vT I =

    5ʣΛ଍ֻ͔Γʹ͠ ͯਓɾ࡞඼ͷϕΫτϧΛٻΊΔ͜ͱ͕Ͱ͖Ε͹ɺະධՁͷ૊Έ߹Θ ͤʢྫ: Bob×λΠλχοΫʣ΋⃗ vB · ⃗ vT I ͱͯ͠ܭࢉ͢Δ͜ͱ͕ Ͱ͖Δ͔Βɻ • εύʔεʢૄʣͳσʔλʹରͯ͠΋͏·͘ಇ͘ɻ   5 3 1 4 5 1 5   =   ⃗ vA · ⃗ vT I ⃗ vA · ⃗ vNH ⃗ vA · ⃗ vSW ⃗ vA · ⃗ vST ⃗ vB · ⃗ vT I ⃗ vB · ⃗ vNH ⃗ vB · ⃗ vSW ⃗ vB · ⃗ vST ⃗ vC · ⃗ vT I ⃗ vC · ⃗ vNH ⃗ vC · ⃗ vSW ⃗ vC · ⃗ vST   7
  7. Ͳ͏͢Ε͹ಛ௃ϕΫτϧΛٻΊΒΕΔͷ͔ʁ ʢ=Ͳ͏΍Ε͹ߦྻΛ෼ղͰ͖Δͷ͔ʁʣ Ͳͷ͘Β͍͏·͘෼ղͰ͖͍ͯΔ͔ͱ͍͏ࢦඪ͕ඞཁ ˠ ߦྻͷ“େ͖͞”ΛଌΔߦྻϊϧϜ (matrix norm)|| · ||Λಋೖ V

    ≈ W H ͱ෼ղͨ͠ͱ͖ɺ W H ͕V ΛΑ͘࠶ݱ(͏·͘෼ղ) ⇔ ||V − W H||͕খ͍͞ ࠷దԽ໰୊(optimization problem)ʹؼண argmin W,H ||V − W H|| 8
  8. m × nߦྻA = (aij )ʹରͯ͠ɺϑϩϕχ΢εϊϧϜ (Frobenius norm)͸ ||A||F =

    m ∑ i=1 n ∑ j=1 |aij |2 (3) ֤ཁૉͷ2৐࿨ͷฏํࠜͳͷͰɺϕΫτϧͷϊϧϜͷࣗવͳ֦ுʹ ͳ͍ͬͯΔɻίετؔ਺ f(W, H) = 1 2 ||V − W H||2 F = 1 2 m ∑ i=1 n ∑ j=1 ( Vij − r ∑ k=1 WikHkj ) 2 = m ∑ i=1 n ∑ j=1 fij (W, H) Λ࠷খԽ͢Δ͜ͱΛߟ͑Δɻ 9
  9. • ALS (Alternating Least Squaresɺަޓ࠷খೋ৐๏) ඍ෼܎਺͕0ʹͳΔ఺Λ୳͢: ∂f ∂Wik = 0,

    ∂f ∂Hkj = 0˰ V HT − W HHT = 0 (4) W T V − W T W H = 0 (5) 1. H ΛϥϯμϜʹॳظԽɻ 2. (4)ࣜΛW ʹ͍ͭͯղ͘ɻ 3. (5)ࣜΛH ʹ͍ͭͯղ͘ɻ 4. ऩଋ͢Δ·Ͱ2ͱ3Λަޓʹ(Alternating)܁Γฦ͢ɻ ϑϩϕχ΢εϊϧϜ =࢒ࠩͷ 2৐࿨Λ࠷খԽ (Least Squares) 10
  10. - Stochastic Gradient Descent (SGD, ֬཰తޯ഑߱Լ๏) (i, j)ΛϥϯμϜʹʢ֬཰తʹʣબ୒͠ɺޯ഑ͷ૯࿨ΛऔΔͷͰ͸ͳ͘ (i, j)ʹؔ͢Δޯ഑ͷΈΛܭࢉ͢Δɻ

    Wik ← Wik − η ∂ ∂Wik fij (W, H) Hkj ← Hkj − η ∂ ∂Hkj fij (W, H) - AdaGrad ֶश཰Λ η √∑ (∇f)2 ͷΑ͏ʹௐ੔ʢͩΜͩΜখ͘͞ʣ͍ͯ͘͠ɻ HivemallͰ͸ SGDͱ AdaGradΛαϙʔτɻ 12
  11. MFͷൃలܗ • όΠΞε߲෇͖Matrix Factorization ධՁ͕ਏޱ(؁ޱ)ͳਓɺධՁ͕ߴ͍(௿͍)࡞඼ ˠ όΠΞεͱͯ͠ൈ ͖ग़͢ ߦྻ෼ղ͸ɺͦͷฏۉ͔Βͷҳ୤ͷهड़ʹઐ೦͢Δɻ BobʷTitanicͷධՁ஋͸

    ˆ VB−T I = µ + wB + wT I + ⃗ vB · ⃗ vT I (6) HivemallͰ࠾༻͞Ε͍ͯΔख๏Ͱ͢ɻ HivemallͰͷMFͷ࣮ߦํ๏ʹ͍ͭͯ͸ҎԼͷ༉Ҫ͞ΜͷهࣄΛࢀর͍ͩ͘͞ɻ http://qiita.com/myui/items/dccb4f58799f080e24ab 13
  12. • Non-negative Matrix Factorization (NMF) ෼ղલͷV ͷཁૉ͕શͯ0Ҏ্ʢඇෛʣͷ஋ͷͱ͖ɺ෼ղޙͷW ͱ H ͷ֤ཁૉ΋ඇෛʹͳΔΑ͏੍໿Λஔ͍ͨ΋ͷɻ

    V   ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗   = W   ∗ ∗ ∗ ∗ ∗ ∗   H ( ∗ ∗ ∗ ∗ ∗ ∗ ) Ԡ༻ྫ: إը૾ͷύʔπ΁ͷ෼ղɺจॻʷ୯ޠߦྻ͔Βͷಛ௃நग़ 14
  13. MFͷͱ͖user uΛද͢ϕΫτϧ͕⃗ vuɺitem iΛද͢ϕΫτϧ͕⃗ vi ͷͱ͖ɺධՁ஋͸ ˆ Vui = ⃗

    vu · ⃗ vi = r ∑ f=1 (vu)f (vi)f ͱܭࢉ͞ΕͨɻTensor FactorizationͰ͸ɺϢʔβu(⃗ vu), αΠτ s(⃗ vs), Ωϟϯϖʔϯc(⃗ vc)ʹରͯ͠ ˆ Vusc = r ∑ f=1 (vu)f (vs)f (vc)f ͱͯ͠ܭࢉ͢Δɻ4࣠Ҏ্ͷ࣌΋ಉ༷ɻ όΠΞε߲෇͖ͷ Tensor Factorization΍ɺNon-negative Tensor Factorization (NTF)ͱ͍֦ͬͨு΋Մೳ 17
  14. Factorization Machines (FM) MFͰͷσʔλͷ࣋ͪํ   5 3 1 4

    5 1 5   ʹରͯ͠ɺFMͰ͸࣍ͷ Α͏ʹ࣋ͪ·͢ɻ A B C TI NH SW ST ධՁ஋ ⃗ x(1) = (1 0 0 1 0 0 0), y(1) = 5 ⃗ x(2) = (1 0 0 0 1 0 0), y(2) = 3 ⃗ x(3) = (1 0 0 0 0 1 0), y(3) = 1 ⃗ x(4) = (0 1 0 0 0 1 0), y(4) = 4 ⃗ x(5) = (0 1 0 0 0 0 1), y(5) = 5 ⃗ x(6) = (0 0 1 1 0 0 0), y(6) = 1 ⃗ x(7) = (0 0 1 0 0 1 0), y(7) = 5 ⃗ x(i): ೖྗͷಛ௃ϕΫτϧɺ࣍ݩΛ nͱ͓͘ʢ͜͜Ͱ͸ 7ʣ 18
  15. FMͷ༧ଌϞσϧ͸ ˆ y(⃗ x) = w0 + n ∑ i=1

    wixi + n ∑ i=1 n ∑ j=i+1 (⃗ vi · ⃗ vj )xixj (7) ʢύϥϝʔλ͕ٻ·ͬͨͱͯ͠ʣBobʷTitanicͷධՁ஋Λਪఆ͢Δɻ ⃗ xB−T I = (0 1 0 1 0 0 0) Λ(7)ࣜʹೖΕΔͱ: ˆ y(⃗ xB−T I ) = w0 + w2 + w4 + ⃗ v2 · ⃗ v4 ͜ΕΛBiased MFͷ(6)ࣜ: ˆ VB−T I = µ + wB + wT I +⃗ vB ·⃗ vT I ͱൺֱ͢ΔͱͽͬͨΓҰக͍ͯ͠Δʂ ఏএऀͷRendle͞Μ΋࿦จதͰ A FM using this feature vector x is identical to the matrix factorization model because xj is only non-zero for u and i ͱॻ͍͍ͯ·͢ɻ 19
  16. ͞Βʹ n ∑ i=1 n ∑ j=i+1 n ∑ k=j+1

      r ∑ f=1 (vi)f (vj )f (vk)f   xixj xk ͱ͍͏߲Λ௥Ճ͢Ε͹ 3֊ͷ Tensor Factorizationͷ FM൛ (3-way FM)ʹͳΓ·͢ɻ • MFΛ࠶ݱ͢Δ͜ͱ͸Θ͔͚ͬͨͲɺΘ͟Θ͟ Factoization MachinesΛߟ͑Δཧ༝͸ʁ ˠಛ௃ϕΫτϧ⃗ xΛԣʹ৳͹͢͜ͱʹΑͬͯʮuserʯ ʮitemʯҎ֎ͷଐ ੑ, จ຺৘ใ (context information) ΛՃ͑Δ͜ͱ͕Ͱ͖Δɻ 20
  17. Rendle͞Μͷ࿦จ͔ΒҾ༻ Time: 2008೥ 12݄͔Β਺͑ͯԿϱ݄ޙʹධՁ͔ͨ͠ A B C TI NH SW

    ST TI NH SW ST Time ධՁ஋ ⃗ x(1) = (1 0 0 1 0 0 0 1/3 1/3 1/3 0 13) y(1) = 5 ⃗ x(2) = (1 0 0 0 1 0 0 1/3 1/3 1/3 0 14) y(2) = 3 ⃗ x(3) = (1 0 0 0 0 1 0 1/3 1/3 1/3 0 16) y(3) = 1 ⃗ x(4) = (0 1 0 0 0 1 0 0 0 1/2 1/2 5) y(4) = 4 ⃗ x(5) = (0 1 0 0 0 0 1 0 0 1/2 1/2 8) y(5) = 5 ⃗ x(6) = (0 0 1 1 0 0 0 1/2 0 1/2 0 9) y(6) = 1 ⃗ x(7) = (0 0 1 0 0 1 0 1/2 0 1/2 0 12) y(7) = 5 21
  18. FMͷ৔߹΋ίετؔ਺ ∑ i ( ˆ y(⃗ x(i)) − y(i) )

    2 ˆ y(⃗ x) = w0 + n ∑ i=1 wixi + n ∑ i=1 n ∑ j=i+1 (⃗ vi · ⃗ vj )xixj ΛSGDͳͲΛ༻͍ͯ࠷খԽ͢Δ͜ͱʹΑΓύϥϝʔλwi, ⃗ vi ΛܾΊ Δ͜ͱ͕Ͱ͖Δɻ(ೖྗϕΫτϧͷ੒෼͕૿͑ͨ෼ɺύϥϝʔλ΋૿͑ ͍ͯΔ) BobʷTitanicͷ༧ଌ஋ (ධՁ࣌ظ͸2010೥6݄) ⃗ xB−T I = (0 1 0 1 0 0 0 1 3 0 1 3 1 3 18) ˆ y(⃗ xB−T I ) = w0 + wB + wT I + 1 3 w T I + 1 3 w SW + 1 3 w ST + 18wT +⃗ vB · ⃗ vT I + 1 3 ⃗ vB · ⃗ v T I + 1 3 ⃗ vB · ⃗ v SW + 1 3 ⃗ vB · ⃗ v ST + 18⃗ vB · ⃗ vT + 1 3 ⃗ vT I · ⃗ v T I + 1 3 ⃗ vT I · ⃗ v SW + 1 3 ⃗ vT I · ⃗ v ST +18⃗ vT I · ⃗ vT + 1 9 ⃗ v T I · ⃗ v SW + 1 9 ⃗ v T I · ⃗ v ST + 6⃗ v T I · ⃗ vT + 1 9 ⃗ v SW · ⃗ v ST + 6⃗ v SW · ⃗ vT + 6⃗ v ST · ⃗ vT 22
  19. ͜ΕΒͷ਺ֶతͳཪଆΛ஌Βͳͯ͘΋ɺϓϩάϥϛϯά͠ͳͯ͘΋ɺ Factorization MachinesʹΑΔ෼ੳΛߦ͏͜ͱ͕Ͱ͖Δɻ ͦ͏ɺHivemallͳΒͶɻ ֶश INSERT OVERWRITE TABLE fm model

    SELECT feature, avg(Wi) AS Wi, array_avg(Vif) AS Vif FROM ( SELECT train fm(features, rating, ’-factor 2 -iters 50 -min 1 -max 5’) AS (feature, Wi, Vif) FROM observed_features ) t GROUP BY feature ༧ଌ SELECT t1.rowid, fm predict(p1.Wi, p1.Vif, t1.Xi) as predicted FROM features_to_be_predicted_exploded t1 LEFT OUTER JOIN fm model p1 ON (t1.feature = p1.feature) GROUP BY t1.rowid 24
  20. Field-aware Factorization Machines (FFM) ͜Ε·Ͱ͸ • ϢʔβΛද͢ಛ௃ϕΫτϧ • ࡞඼Λද͢ಛ௃ϕΫτϧ •

    FMͷcontext informationΛද͢ಛ௃ϕΫτϧ ͸Ұͭͱ͍ͯͨ͠ɻ(ex: AliceΛද͢ͷ͸⃗ vA) ˠFFMͰ͸ҎԼͷΑ͏ͳfieldΛߟ͑Δɻ • user field f(A) = f(B) = f(C) = fu • movie field f(TI) = f(NH) = f(SW) = f(ST) = fm • context informationͷछྨ͝ͱͷfield f(TI) = f(NH) = f(SW) = f(ST) = fn, f(Time) = ft 25
  21. ಺ੵΛऔΔࡍɺ૬खͷfieldʹ߹ͬͨϕΫτϧΛ༻͍Δɻ n ∑ i=1 n ∑ j=i+1 (⃗ vi ·

    ⃗ vj )xixj → n ∑ i=1 n ∑ j=i+1 (⃗ vi,f(j) · ⃗ vj,f(i) )xixj ۩ମతʹ͸BobϕΫτϧΛ3ͭ(⃗ vB,fm , ⃗ vB,fn , ⃗ vB,ft )༻ҙͯ͠ ⃗ vB,fm · ⃗ vT I,fu , ⃗ vB,fn · ⃗ v T I,fu , ⃗ vB,ft · ⃗ vT ,fu ⃗ vB,fm · ⃗ vSW ,fu , ⃗ vB,fn · ⃗ v SW ,fu ͱ͢Δɻύϥϝʔλ͕૿͑Δ෼ɺ͏·͘ௐ੔Ͱ͖Ε͹ਫ਼౓ΛߴΊΔ͜ ͱ͕Ͱ͖Δɻ ˠ CTR༧ଌίϯϖͰ޷੒੷ 26
  22. ·ͱΊ Matrix Factorization ਐԽ ↓↑ context information Λͳ͘͠ user ʷ

    movie ͷΈʹݶΔ Factorization Machines ਐԽ ↓↑ field ͷ۠ผΛͳ͘͠ಉҰͷϕΫτϧΛ༻͍Δ Field-aware Factorization Machines جຊઓུɿϢʔβ΍࡞඼ͳͲͷಛ௃Λදݱ͢ΔϕΫτϧ΍όΠΞεύ ϥϝʔλΛֶशʹΑͬͯٻΊɺϕΫτϧͷ಺ੵͱͯ͠ධՁ஋Λ༧ଌ ͢Δɻ જࡏతͳಛ௃ͷநग़ɺ࣍ݩ࡟ݮͱ͍ͬͨޮՌ΋ɻ 28