Share
Embed
Hivemallmtup 20160908 factorizations
Explains mathematical models of matrix factorization and factorization machines (in Japanese)
fan-k-oomori
September 08, 2016
Transcript
31 slides
-
ࣗݾհ • nex8(μΠφϛοΫϦλʔήςΟϯάࠂ৴ϓϥοτϑΥʔϜ) ͷ։ൃͱ͔σʔλੳۀΛߦ͍ͬͯ·͢ɻ • όοΫάϥϯυૉཻࢠ, ݭཧɻۀͰߦྻͱ͔Λ͏ػձ ͕͋Δͱςϯγϣϯ্͕͕ΔλΠϓɻ • ฐࣾͷnend(ΞυωοτϫʔΫ),
nex8ͷ։ൃΛ୲͢Δ෦ॺͰٕ ज़ϒϩάΛ͍ͬͯ·͢ʂ http://tech-blog.fancs.com/ 1 -
ࠓճ༻͍Δςετσʔλ λΠλχοΫ (TI) ϊ ο ς Ο ϯ άώϧͷ࿀ਓ (NH)
ελʔΥʔ ζ (SW) ελʔτϨο Ϋ (ST) Alice (A) 5 3 1 2010-1 2010-2 2010-4 Bob (B) 4 5 2009-5 2009-8 Charlie (C) 1 5 2009-9 2009-12 Factorization MachinesΛఏҊͨ͠Rendle͞ΜͷݪจΑ Γഈआɻ ্ஈ=֤өըʹର͢ΔධՁ Լஈ=ධՁΛߦ݄ͬͨ ʲઃఆʳۭཝͷະධՁΛ༧ଌ͠ɺϨίϝϯυ͍ͨ͠ 2 -
Matrix Factorization (MF) ઌ΄ͲͷσʔλΛߦྻ(matrix)ͷܗͰදݱ͢Δͱ V = 5 3
1 4 5 1 5 (1) ͜ΕΛ2ͭͷߦྻͷੵʹղ͢Δ(factorize) 5 3 1 4 5 1 5 = ∗ ∗ ∗ ∗ ∗ ∗ ( ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ) (2) 3 -
2 × 2ߦྻͷੵ ( u1 v1 u2 v2 ) (
x1 x2 y1 y2 ) = ( u1 x1 + v1 y1 u1 x2 + v1 y2 u2 x1 + v2 y1 u2 x2 + v2 y2 ) ͜͜Ͱ ⃗ wi = ( ui vi ) , ⃗ zi = ( xi yi ) (i = 1, 2)ͱ͓͘ͱ্ࣜ ( ⃗ wT 1 ⃗ wT 2 ) ( ⃗ z1 ⃗ z2 ) = ( ⃗ w1 · ⃗ z1 ⃗ w1 · ⃗ z2 ⃗ w2 · ⃗ z1 ⃗ w2 · ⃗ z2 ) ͱॻ͖ͤΔɻ ) ߦྻه๏ͷ ⃗ wT ⃗ z ͱϕΫτϧੵ ⃗ w · ⃗ z Ձɻ 4 -
(2)ࣜΛಉ༷ʹϕΫτϧͰॻ͖ද͢ͱ 5 3 1 4 5 1 5
= ∗ ∗ ∗ ∗ ∗ ∗ ( ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ) = ⃗ vT A ⃗ vT B ⃗ vT C ( ⃗ vT I ⃗ vNH ⃗ vSW ⃗ vST ) = ⃗ vA · ⃗ vT I ⃗ vA · ⃗ vNH ⃗ vA · ⃗ vSW ⃗ vA · ⃗ vST ⃗ vB · ⃗ vT I ⃗ vB · ⃗ vNH ⃗ vB · ⃗ vSW ⃗ vB · ⃗ vST ⃗ vC · ⃗ vT I ⃗ vC · ⃗ vNH ⃗ vC · ⃗ vSW ⃗ vC · ⃗ vST ղऍɿ⃗ vA AliceͷಛΛද͢ϕΫτϧྔɺ⃗ vB BobͷಛΛද͢ ϕΫτϧྔɺʜɺ⃗ vST ελʔτϨοΫͷಛΛද͢ϕΫτϧྔͱߟ ͑ΔɻධՁɺਓͱ࡞ʹରԠ͢ΔϕΫτϧͷੵͱͯ͠ܭࢉ͢Δ 5 -
༩͑ΒΕͨσʔλ͔ΒϕΫτϧͷ࣍ݩʢཁૉʣܾ·Β ͳ͍ ˠ ϞσϧͷܗΛܾΊΔϋΠύʔύϥϝʔλɻ ͜͜Ͱr࣍ݩͱ͢Δɻ ݩͷߦྻͷαΠζʹൺͯrΛখ͘͞Ͱ͖Ε࣍ݩݮʹͭ ͳ͕Δɻ n × m
vs. (n + m) × r ਓͱөը͕r࣍ݩͷಛۭؒͷϕΫτϧͱͯ͠දݱ͞Εɺ ͦͷੵΛܭࢉ͢Δ͜ͱʹΑͬͯ૬ੑʢධՁͷߴʣ͕Θ͔ Δͱ͍͏ߏʹͳ͍ͬͯΔɻ • ަˠධՁ • ฏߦˠߴධՁ 6 -
ͳͥະධՁͷΛਪఆͰ͖Δͷ͔ʁ • ͢ͰʹධՁ͕ͳ͞Εͨใʢྫ: ⃗ vA · ⃗ vT I =
5ʣΛֻ͔Γʹ͠ ͯਓɾ࡞ͷϕΫτϧΛٻΊΔ͜ͱ͕Ͱ͖ΕɺະධՁͷΈ߹Θ ͤʢྫ: Bob×λΠλχοΫʣ⃗ vB · ⃗ vT I ͱͯ͠ܭࢉ͢Δ͜ͱ͕ Ͱ͖Δ͔Βɻ • εύʔεʢૄʣͳσʔλʹରͯ͠͏·͘ಇ͘ɻ 5 3 1 4 5 1 5 = ⃗ vA · ⃗ vT I ⃗ vA · ⃗ vNH ⃗ vA · ⃗ vSW ⃗ vA · ⃗ vST ⃗ vB · ⃗ vT I ⃗ vB · ⃗ vNH ⃗ vB · ⃗ vSW ⃗ vB · ⃗ vST ⃗ vC · ⃗ vT I ⃗ vC · ⃗ vNH ⃗ vC · ⃗ vSW ⃗ vC · ⃗ vST 7 -
Ͳ͏͢ΕಛϕΫτϧΛٻΊΒΕΔͷ͔ʁ ʢ=Ͳ͏ΕߦྻΛղͰ͖Δͷ͔ʁʣ Ͳͷ͘Β͍͏·͘ղͰ͖͍ͯΔ͔ͱ͍͏ࢦඪ͕ඞཁ ˠ ߦྻͷ“େ͖͞”ΛଌΔߦྻϊϧϜ (matrix norm)|| · ||Λಋೖ V
≈ W H ͱղͨ͠ͱ͖ɺ W H ͕V ΛΑ͘࠶ݱ(͏·͘ղ) ⇔ ||V − W H||͕খ͍͞ ࠷దԽ(optimization problem)ʹؼண argmin W,H ||V − W H|| 8 -
m × nߦྻA = (aij )ʹରͯ͠ɺϑϩϕχεϊϧϜ (Frobenius norm) ||A||F =
m ∑ i=1 n ∑ j=1 |aij |2 (3) ֤ཁૉͷ2ͷฏํࠜͳͷͰɺϕΫτϧͷϊϧϜͷࣗવͳ֦ுʹ ͳ͍ͬͯΔɻίετؔ f(W, H) = 1 2 ||V − W H||2 F = 1 2 m ∑ i=1 n ∑ j=1 ( Vij − r ∑ k=1 WikHkj ) 2 = m ∑ i=1 n ∑ j=1 fij (W, H) Λ࠷খԽ͢Δ͜ͱΛߟ͑Δɻ 9 -
• ALS (Alternating Least Squaresɺަޓ࠷খೋ๏) ඍ͕0ʹͳΔΛ୳͢: ∂f ∂Wik = 0,
∂f ∂Hkj = 0˰ V HT − W HHT = 0 (4) W T V − W T W H = 0 (5) 1. H ΛϥϯμϜʹॳظԽɻ 2. (4)ࣜΛW ʹ͍ͭͯղ͘ɻ 3. (5)ࣜΛH ʹ͍ͭͯղ͘ɻ 4. ऩଋ͢Δ·Ͱ2ͱ3Λަޓʹ(Alternating)܁Γฦ͢ɻ ϑϩϕχεϊϧϜ =ࠩͷ 2Λ࠷খԽ (Least Squares) 10 -
• Gradient Descent (GD, ޯ߱Լ๏) ؔΛඍͯ͠ූ߸Λม͑ͨͷ(−⃗ ∇f)ؔͷ͕ݮগ͢Δํ Λද͢ɻैͬͯͦͷํʹਐΜͰ͍͚࠷খʹ౸ୡͰ͖Δͣɻ Wik ←
Wik − η ∂ ∂Wik ∑ a,b fab(W, H) Hkj ← Hkj − η ∂ ∂Hkj ∑ a,b fab(W, H) k = 1, . . . , r, ηֶश 11 -
- Stochastic Gradient Descent (SGD, ֬తޯ߱Լ๏) (i, j)ΛϥϯμϜʹʢ֬తʹʣબ͠ɺޯͷ૯ΛऔΔͷͰͳ͘ (i, j)ʹؔ͢ΔޯͷΈΛܭࢉ͢Δɻ
Wik ← Wik − η ∂ ∂Wik fij (W, H) Hkj ← Hkj − η ∂ ∂Hkj fij (W, H) - AdaGrad ֶशΛ η √∑ (∇f)2 ͷΑ͏ʹௐʢͩΜͩΜখ͘͞ʣ͍ͯ͘͠ɻ HivemallͰ SGDͱ AdaGradΛαϙʔτɻ 12 -
MFͷൃలܗ • όΠΞε߲͖Matrix Factorization ධՁ͕ਏޱ(ޱ)ͳਓɺධՁ͕ߴ͍(͍)࡞ ˠ όΠΞεͱͯ͠ൈ ͖ग़͢ ߦྻղɺͦͷฏۉ͔Βͷҳͷهड़ʹઐ೦͢Δɻ BobʷTitanicͷධՁ
ˆ VB−T I = µ + wB + wT I + ⃗ vB · ⃗ vT I (6) HivemallͰ࠾༻͞Ε͍ͯΔख๏Ͱ͢ɻ HivemallͰͷMFͷ࣮ߦํ๏ʹ͍ͭͯҎԼͷ༉Ҫ͞ΜͷهࣄΛࢀর͍ͩ͘͞ɻ http://qiita.com/myui/items/dccb4f58799f080e24ab 13 -
• Non-negative Matrix Factorization (NMF) ղલͷV ͷཁૉ͕શͯ0Ҏ্ʢඇෛʣͷͷͱ͖ɺղޙͷW ͱ H ͷ֤ཁૉඇෛʹͳΔΑ͏੍Λஔ͍ͨͷɻ
V ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ = W ∗ ∗ ∗ ∗ ∗ ∗ H ( ∗ ∗ ∗ ∗ ∗ ∗ ) Ԡ༻ྫ: إը૾ͷύʔπͷղɺจॻʷ୯ޠߦྻ͔Βͷಛநग़ 14 -
MFͷͱ͖user uΛද͢ϕΫτϧ͕⃗ vuɺitem iΛද͢ϕΫτϧ͕⃗ vi ͷͱ͖ɺධՁ ˆ Vui = ⃗
vu · ⃗ vi = r ∑ f=1 (vu)f (vi)f ͱܭࢉ͞ΕͨɻTensor FactorizationͰɺϢʔβu(⃗ vu), αΠτ s(⃗ vs), Ωϟϯϖʔϯc(⃗ vc)ʹରͯ͠ ˆ Vusc = r ∑ f=1 (vu)f (vs)f (vc)f ͱͯ͠ܭࢉ͢Δɻ4࣠Ҏ্ͷ࣌ಉ༷ɻ όΠΞε߲͖ͷ Tensor FactorizationɺNon-negative Tensor Factorization (NTF)ͱ͍֦ͬͨுՄೳ 17 -
Factorization Machines (FM) MFͰͷσʔλͷ࣋ͪํ 5 3 1 4
5 1 5 ʹରͯ͠ɺFMͰ࣍ͷ Α͏ʹ࣋ͪ·͢ɻ A B C TI NH SW ST ධՁ ⃗ x(1) = (1 0 0 1 0 0 0), y(1) = 5 ⃗ x(2) = (1 0 0 0 1 0 0), y(2) = 3 ⃗ x(3) = (1 0 0 0 0 1 0), y(3) = 1 ⃗ x(4) = (0 1 0 0 0 1 0), y(4) = 4 ⃗ x(5) = (0 1 0 0 0 0 1), y(5) = 5 ⃗ x(6) = (0 0 1 1 0 0 0), y(6) = 1 ⃗ x(7) = (0 0 1 0 0 1 0), y(7) = 5 ⃗ x(i): ೖྗͷಛϕΫτϧɺ࣍ݩΛ nͱ͓͘ʢ͜͜Ͱ 7ʣ 18 -
FMͷ༧ଌϞσϧ ˆ y(⃗ x) = w0 + n ∑ i=1
wixi + n ∑ i=1 n ∑ j=i+1 (⃗ vi · ⃗ vj )xixj (7) ʢύϥϝʔλ͕ٻ·ͬͨͱͯ͠ʣBobʷTitanicͷධՁΛਪఆ͢Δɻ ⃗ xB−T I = (0 1 0 1 0 0 0) Λ(7)ࣜʹೖΕΔͱ: ˆ y(⃗ xB−T I ) = w0 + w2 + w4 + ⃗ v2 · ⃗ v4 ͜ΕΛBiased MFͷ(6)ࣜ: ˆ VB−T I = µ + wB + wT I +⃗ vB ·⃗ vT I ͱൺֱ͢ΔͱͽͬͨΓҰக͍ͯ͠Δʂ ఏএऀͷRendle͞ΜจதͰ A FM using this feature vector x is identical to the matrix factorization model because xj is only non-zero for u and i ͱॻ͍͍ͯ·͢ɻ 19 -
͞Βʹ n ∑ i=1 n ∑ j=i+1 n ∑ k=j+1
r ∑ f=1 (vi)f (vj )f (vk)f xixj xk ͱ͍͏߲ΛՃ͢Ε 3֊ͷ Tensor Factorizationͷ FM൛ (3-way FM)ʹͳΓ·͢ɻ • MFΛ࠶ݱ͢Δ͜ͱΘ͔͚ͬͨͲɺΘ͟Θ͟ Factoization MachinesΛߟ͑Δཧ༝ʁ ˠಛϕΫτϧ⃗ xΛԣʹ৳͢͜ͱʹΑͬͯʮuserʯ ʮitemʯҎ֎ͷଐ ੑ, จ຺ใ (context information) ΛՃ͑Δ͜ͱ͕Ͱ͖Δɻ 20 -
Rendle͞Μͷจ͔ΒҾ༻ Time: 2008 12݄͔Β͑ͯԿϱ݄ޙʹධՁ͔ͨ͠ A B C TI NH SW
ST TI NH SW ST Time ධՁ ⃗ x(1) = (1 0 0 1 0 0 0 1/3 1/3 1/3 0 13) y(1) = 5 ⃗ x(2) = (1 0 0 0 1 0 0 1/3 1/3 1/3 0 14) y(2) = 3 ⃗ x(3) = (1 0 0 0 0 1 0 1/3 1/3 1/3 0 16) y(3) = 1 ⃗ x(4) = (0 1 0 0 0 1 0 0 0 1/2 1/2 5) y(4) = 4 ⃗ x(5) = (0 1 0 0 0 0 1 0 0 1/2 1/2 8) y(5) = 5 ⃗ x(6) = (0 0 1 1 0 0 0 1/2 0 1/2 0 9) y(6) = 1 ⃗ x(7) = (0 0 1 0 0 1 0 1/2 0 1/2 0 12) y(7) = 5 21 -
FMͷ߹ίετؔ ∑ i ( ˆ y(⃗ x(i)) − y(i) )
2 ˆ y(⃗ x) = w0 + n ∑ i=1 wixi + n ∑ i=1 n ∑ j=i+1 (⃗ vi · ⃗ vj )xixj ΛSGDͳͲΛ༻͍ͯ࠷খԽ͢Δ͜ͱʹΑΓύϥϝʔλwi, ⃗ vi ΛܾΊ Δ͜ͱ͕Ͱ͖Δɻ(ೖྗϕΫτϧͷ͕૿͑ͨɺύϥϝʔλ૿͑ ͍ͯΔ) BobʷTitanicͷ༧ଌ (ධՁ࣌ظ20106݄) ⃗ xB−T I = (0 1 0 1 0 0 0 1 3 0 1 3 1 3 18) ˆ y(⃗ xB−T I ) = w0 + wB + wT I + 1 3 w T I + 1 3 w SW + 1 3 w ST + 18wT +⃗ vB · ⃗ vT I + 1 3 ⃗ vB · ⃗ v T I + 1 3 ⃗ vB · ⃗ v SW + 1 3 ⃗ vB · ⃗ v ST + 18⃗ vB · ⃗ vT + 1 3 ⃗ vT I · ⃗ v T I + 1 3 ⃗ vT I · ⃗ v SW + 1 3 ⃗ vT I · ⃗ v ST +18⃗ vT I · ⃗ vT + 1 9 ⃗ v T I · ⃗ v SW + 1 9 ⃗ v T I · ⃗ v ST + 6⃗ v T I · ⃗ vT + 1 9 ⃗ v SW · ⃗ v ST + 6⃗ v SW · ⃗ vT + 6⃗ v ST · ⃗ vT 22 -
͜ΕΒͷֶతͳཪଆΛΒͳͯ͘ɺϓϩάϥϛϯά͠ͳͯ͘ɺ Factorization MachinesʹΑΔੳΛߦ͏͜ͱ͕Ͱ͖Δɻ ͦ͏ɺHivemallͳΒͶɻ ֶश INSERT OVERWRITE TABLE fm model
SELECT feature, avg(Wi) AS Wi, array_avg(Vif) AS Vif FROM ( SELECT train fm(features, rating, ’-factor 2 -iters 50 -min 1 -max 5’) AS (feature, Wi, Vif) FROM observed_features ) t GROUP BY feature ༧ଌ SELECT t1.rowid, fm predict(p1.Wi, p1.Vif, t1.Xi) as predicted FROM features_to_be_predicted_exploded t1 LEFT OUTER JOIN fm model p1 ON (t1.feature = p1.feature) GROUP BY t1.rowid 24 -
Field-aware Factorization Machines (FFM) ͜Ε·Ͱ • ϢʔβΛද͢ಛϕΫτϧ • ࡞Λද͢ಛϕΫτϧ •
FMͷcontext informationΛද͢ಛϕΫτϧ Ұͭͱ͍ͯͨ͠ɻ(ex: AliceΛද͢ͷ⃗ vA) ˠFFMͰҎԼͷΑ͏ͳfieldΛߟ͑Δɻ • user field f(A) = f(B) = f(C) = fu • movie field f(TI) = f(NH) = f(SW) = f(ST) = fm • context informationͷछྨ͝ͱͷfield f(TI) = f(NH) = f(SW) = f(ST) = fn, f(Time) = ft 25 -
ੵΛऔΔࡍɺ૬खͷfieldʹ߹ͬͨϕΫτϧΛ༻͍Δɻ n ∑ i=1 n ∑ j=i+1 (⃗ vi ·
⃗ vj )xixj → n ∑ i=1 n ∑ j=i+1 (⃗ vi,f(j) · ⃗ vj,f(i) )xixj ۩ମతʹBobϕΫτϧΛ3ͭ(⃗ vB,fm , ⃗ vB,fn , ⃗ vB,ft )༻ҙͯ͠ ⃗ vB,fm · ⃗ vT I,fu , ⃗ vB,fn · ⃗ v T I,fu , ⃗ vB,ft · ⃗ vT ,fu ⃗ vB,fm · ⃗ vSW ,fu , ⃗ vB,fn · ⃗ v SW ,fu ͱ͢Δɻύϥϝʔλ͕૿͑Δɺ͏·͘ௐͰ͖Εਫ਼ΛߴΊΔ͜ ͱ͕Ͱ͖Δɻ ˠ CTR༧ଌίϯϖͰ 26 -
·ͱΊ Matrix Factorization ਐԽ ↓↑ context information Λͳ͘͠ user ʷ
movie ͷΈʹݶΔ Factorization Machines ਐԽ ↓↑ field ͷ۠ผΛͳ͘͠ಉҰͷϕΫτϧΛ༻͍Δ Field-aware Factorization Machines جຊઓུɿϢʔβ࡞ͳͲͷಛΛදݱ͢ΔϕΫτϧόΠΞεύ ϥϝʔλΛֶशʹΑͬͯٻΊɺϕΫτϧͷੵͱͯ͠ධՁΛ༧ଌ ͢Δɻ જࡏతͳಛͷநग़ɺ࣍ݩݮͱ͍ͬͨޮՌɻ 28