実践機械学習 — MahoutとSolrを活用したレコメンデーションにおけるイノベーション

® © 2014 MapR Technologies 1 ® © 2014 MapR
Technologies áǎìǅéƓ— MahoutSSolrlįæDL ¬~£¯3¨¯V<Ahq3¨¯ ƪǐƞƝ MapR Technologies

® © 2014 MapR Technologies 3 ´°W{ •  ¬~£¯3¨¯SX? • 
¬~£¯3¨¯l§33VģŨFhVX? •  ƅſ¬~£¯3¨¯Wũ •  ege9¬~£¯3¨¯VFhVX?

® © 2014 MapR Technologies 4 áǎìǅéƓ ¬~£¯3¨¯V<Ahq3¨¯ =; 32
ষɹࣄྫɿԻָϨίϝϯμʔ 6.3ɹେن໛ͳϨίϝϯσʔγϣϯ ࢹௌཤྺΛγϛϡϨʔτ͢Δ͜ͱͰɺେྔͷσʔλΛੜ੒͢Δ͜ͱ͕Ͱ͖·͕͢ɺ ਺ेԯͷࢹௌه࿥Λੜ੒͢ΔͨΊʹ͸ɺϏοάσʔλͷॲཧ͸࣮ࡍͷͱ͜ΖඞཁͰ͸ ͋Γ·ͤΜɻͨͩ͠ɺ͜ͷن໛Ͱੜ੒͞Εͨϩάͷ෼ੳΛߦ͏ʹ͋ͨͬͯ͸ɺγες ϜͷεέʔϥϏϦςΟʹ͍ͭͯ஫ҙΛ෷͍࢝ΊΔඞཁ͕͋Γ·͢ɻ Music Machine ͷϨίϝϯμʔͷγεςϜ͸ɺMahout ͷ RowSimilarityJob Λ ࢖ͬͯϢʔβʔͷࢹௌϩάΛॲཧ͠ɺΞʔςΟετͷΠϯδέʔλʔΛؚΉϨίϝϯ σʔγϣϯϞσϧΛߏங͠·͢ɻ͜ͷԋࢉॲཧ͸ɺγεςϜશମͷҰ෦ʹա͗·ͤ Μɻݱ࣮ʹ࢖ΘΕΔγεςϜ͕ৗʹͦ͏Ͱ͋ΔΑ͏ʹɺطଘͷγεςϜ܈ͱͷϑΥʔ Ϛοτม׵ͷͨΊʹɺେྔͷ࿈ܞͷͨΊͷίʔυ͕ඞཁͰͨ͠ɻͦͷ࣮૷ͷ֓ཁΛɺ ਤ 6-3 ʹࣔ͠·͢ɻ͜͜Ͱ͸ MapR Λ࢖͍ͬͯΔͷͰɺPythonɺPostgreSQLɺ LucidWorks ΛؚΉ΄΅͢΂ͯͷίϯϙʔωϯτ͸ɺΫϥελ಺Ͱಈ࡞͠·͢ɻ ਤ 6-3ɹϨίϝϯμʔͷΦϑϥΠϯύʔτʹ͓͚Δɺैདྷܕ͓Αͼ Hadoop ʹΑΔίϯϐϡʔςΟ ϯάɻփ৭ͷ෦෼͸ɺMusic Machine ͷϨίϝϯμʔͷ Hadoop ϕʔεͷΫϥελ্Ͱಈ ࡞͢Δ෦෼Λ͍ࣔͯ͠Δɻ͜͜Ͱ͸ɺϦΞϧλΠϜͷ෼ࢄϑΝΠϧγεςϜΛ࣋ͭɺMapR distribution for Apache Hadoop Λ࢖༻͍ͯ͠Δɻଞͷ Hadoop σΟετϦϏϡʔγϣϯ Ͱ࢖ΘΕ͍ͯΔϑΝΠϧγεςϜʢHDFSʣΛ࢖͏ͨΊʹ͸ɺ͞Βʹ͍͔ͭ͘ͷεςοϓ͕ ඞཁ Document ϢʔβʔʵΞΠςϜ ϩάϑΝΠϧ ϢʔβʔʗΞΠςϜ*%ͷ ੔਺஋΁ͷϚοϐϯά .BIPVUͷ 3PX4JNJMBSJUZ+PC ϢʔβʔʗΞΠςϜͷ ੔਺஋ͷ*%΁ͷ Ϛοϐϯάͱɺ 4PMS+40/ ϑΥʔϚο τ΁ͷม׵ ϢʔβʔʗΞΠςϜ *%ͷࣙॻ 4PMSͷ+40/ ϑΥʔϚο τ΁ͷม׵ Document ΞΠςϜͷ ϝλσʔλ .BQSFEVDF 1ZUIPO 1PTUHSF42- 17 17 ɹ"QBDIF.BIPVU ʹΑΔϞσϧߏஙͷํ๏ 4.1ɹApache Mahout ʹΑΔϞσϧߏஙͷํ๏ ࣮ફతͳϨίϝϯμʔͷͨΊͷϨίϝϯσʔγϣϯϞσϧΛߏங͢Δʹ͋ͨͬͯ ͸ɺΦʔϓϯιʔεͷεέʔϥϒϧͳػցֶशϥΠϒϥϦͷ Apache Mahout ͷ࣋ͬ ͍ͯΔΞϧΰϦζϜΛ࢖͏͜ͱʹ͠·͠ΐ͏ɻ΍Γ͍ͨ͜ͱ͸ɺMahout ͷߦྻԋࢉ Λ࢖ͬͯɺϢʔβʔͷߦಈཤྺ͔ΒɺϨίϝϯσʔγϣϯͷͨΊͷ༗༻ͳΠϯδέʔ λʔΛಘΔ͜ͱͰ͢ɻͦͷͨΊʹɺ3 ͭͷߦྻΛߏங͍͖ͯ͠·͢ɻ ཤྺϚτϦΫε ϢʔβʔͱΞΠςϜؒͷؔ࿈ΛɺϢʔβʔͱΞΠςϜͷߦྻͱͯ͠ه࿥͢Δɻ ڞىϚτϦΫε ཤྺϚτϦΫεΛɺΞΠςϜͱΞΠςϜͷߦྻʹม׵͠ɺϢʔβʔͷཤྺ಺ͰͲ ͏͍ͬͨΞΠςϜಉ͕࢜Ұॹʹొ৔͍ͯ͠Δ͔Λه࿥͢Δɻ ਤ 4-3ɹཤྺˠڞىˠΠϯδέʔλʔϚτϦΫεɻΠϯδέʔλʔϚτϦΫεͰදݱ͞Ε͍ͯΔ͜ͷϞ σϧʹ͸ɺ Ϧϯΰ͕ ʮࢠݘʯ ΛϨίϝϯυ͢΂͖ΠϯδέʔλʔͰ͋Δ͜ͱ͕ූ߸Խ͞Ε͍ͯΔɻ ཤྺϚ τϦΫε ڞىϚ τϦΫε Πϯδέʔλʔ Ϛ τϦΫε "MJDF #PC $IBSMFT *$$!$""$'!#$ Ƹ o3«0{3ĩĻ³½ƪǐƞƝ Ňƀ ĩĻ³½¬q¯ŤÂżŦ.ÛÂƿ±.đÂƠƝ Żîǂƃ ƾ r¯3X & %*)S%"'lįæDL. ¯«U>faôÙU¬~£¯3¨ ¯t¯¯Wq¯SáƖlĜś

Technologies ƋŕVP9Q

® © 2014 MapR Technologies 7 ìǅéƓo«ª¢SJWáƖ A 1 A
2 ! " # $ T A 1 A 2 ! " # $= A 1 T A 2 T ! " % % # $ & & A 1 A 2 ! " # $ = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & r 1 r 2 ! " % % # $ & & = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & h 1 h 2 ! " % % # $ & & r 1 = A 1 T A 1 A 1 T A 2 ! " % # $ & h 1 h 2 ! " % % # $ & & O(κ k d + k3 d) = O(k2 d log n + k3 d) for small k, high quality O(κ d log k) or O(d log κ log k) for larger k, looser quality B<GA (;29DH >C I

® © 2014 MapR Technologies 8 ¬~£¯3¨¯ •  3: µ2W»ê4§335Soq¢SWq¯©{¨¯
–  3X¬~£¯3¨¯¤«WéƓVĬkih •  ťĂWƊīUaD@XŴ]D9q¯©{¨¯lģĘFhBS>àÐ •  %-0'(.2!>: ƚĸ.ƅſ.ÇŸ3W¬©¯ßƮ.e oc¬¬3RWŪāŉĶWģĘ

Technologies ¬~£¯3¨¯WD@^ (]LX6I want a pony7)

® © 2014 MapR Technologies 12 I want a pony!
<5"34$:=@ 79$#?$ AB68#,1*

® © 2014 MapR Technologies 15 ¬~£¯3¨¯ oªXª¯SÒǄlƲÉ ¥3«XÀŃĕlƲÉ oª
¥3«

® © 2014 MapR Technologies 16 ¬~£¯3¨¯ oªXª¯SÒǄlƲÉ ¥3«XÀŃĕlƲÉ Xª¯lƲÉ
oª ¥3«

® © 2014 MapR Technologies 17 ¬~£¯3¨¯ XƌVţlưD>h? oª
¥3« ?

® © 2014 MapR Technologies 18 ¬~£¯3¨¯ ÒǄ.aNjm! oª
¥3«

® © 2014 MapR Technologies 21 ¬~£¯3¨¯ çÝ>3lƲÉDLfT:U h? oª
¥3« o£ªo ƌVţlo£ªoV<FF`FiY9 9? ?

® © 2014 MapR Technologies 22 ¬~£¯3¨¯ oª ¥3«
o£ªo aDçÝ>3lƲÉDLf. 3XƌWaWlčơFhWV ŌV×LU9... ?

® © 2014 MapR Technologies 23 ƘWĺŒ3Wòą •  ƈœVµøWoq¢XF[QSĺŒFh (ǁ>3>ưD9
S9:ĞĆXŌV×LU9…) –  ũrs«w¢y¦£¯.t¬33Wƅſ •  ĤƷƟU¸úXĨŲ^>U9 –  =^JgWǌS=3We:Uƥ;GŴ]ihoq¢lģŨD L9WRU9őg •  ưD9WXũóÐUĺŒ –  BiBJ>žźų9ū^WċƂWĞĆƑR8g.¬~£¯3¨¯ WėǀSUh

® © 2014 MapR Technologies 24 »ê=fŌV×PċƂlĚhVX •  |nq«lĬOQ§33 x
oq¢WǊƢ ª{l Ģö –  ŝBWq¯©{¨¯WǊƢXçQWǇŀÐUĉ^¿kHSŊ[h SǍU3R8h •  oq¢ x oq¢WĺŒ ª{VŁƁ •  ũóÐUĺŒlƭD.ŌV×Pq¯}33lÅPAQq¯ }33 ª{lĢö –  ÊþǑÜŊLLRXTWĺŒ>ū^Wq¯}33SDQĬ;h= lıülĈOQďňFhWVŬAVUh –  Apache Mahout WRowSimilarityJobXLLRlĬ:

® © 2014 MapR Technologies 25 Apache Mahout •  v3¯3
Apache s{http://mahout.apache.org •  Mahout version 0.9 X 2014²2ÁVªª3 •  ìǅéƓWL`W}3©«Uo«ª¢W©q©ª –  Apache Hadoop Rê@aWa8iY.Hadoop ëñWaWa8h –  äC9}3«RW^ê@aWa8h –  Ƈƣá»FhaWa8iY.3}¯¥«UaWa8h •  ÛğWaWlŭ_ –  {©ª¯| & ÖãŋŠ –  ÃƆ –  ¬~£¯3¨¯ –  Mahout Math ©q©ª

® © 2014 MapR Technologies 26 |nq« oª ¥3«
oª ¥3« oª

® © 2014 MapR Technologies 27 |nq« u1 u3 u2
u1 u3 u2 u1 t1# t4# t3# t2# t3# t3# t1#

® © 2014 MapR Technologies 28 ǊƢ ª{: §33/oq¢ oª
¥3« ✔ ✔ ✔ ✔ ✔ ✔ ✔

® © 2014 MapR Technologies 29 ĺŒ ª{: oq¢/oq¢ -
1# 2# 1# 1# 1# 1# 2# 1# SWV@ ?. 0# 0# 0# 0#

® © 2014 MapR Technologies 30 ĺŒ ª{: oq¢/oq¢ LLR
L?PE SW";)&+U[… - 1# 2# 1# 1# 1# 1# 2# 1# 0# 0# 0# 0#

® © 2014 MapR Technologies 31 ĺŒqª ª{ 1# 1#
not# not# 1#

® © 2014 MapR Technologies 32 ũólÅPAh A not A
B 13 1000 not B 1000 100,000 A not A B 1 0 not B 0 2 A not A B 1 0 not B 0 10,000 A not A B 10 0 not B 0 100,000 JiKiWōŷ=fTmUĖł>ŐAh?

® © 2014 MapR Technologies 33 ũólÅPAh •  «3 LLR
X<<eJƂļǉŧVġ9 •  Apache Mahout RXRowSimilarityJob > LLR lăæ A not A B 1 0 not B 0 2 A not A B 1 0 not B 0 10,000 A not A B 10 0 not B 0 100,000 1.95 4.52 14.3 JiKiWōŷ=fTmUĖł>ŐAh? A not A B 13 1000 not B 1000 100,000 0.90

® © 2014 MapR Technologies 34 ĺŒ ª{ : LLR
L?PE SW";)&+U[ - 1# 2# 1# 1# 1# 1# 2# 1# 0# 0# 0# 0#

® © 2014 MapR Technologies 35 q¯}33 ª{: ũóÐUĺŒ ✔#
✔# MQ: 3% = "-40$65;/";)&+1! 90YJ …

® © 2014 MapR Technologies 39 Apache Solr/Apache Lucene • 
Apache Solr/Lucene: v3¯3W®«UĒƼt¯¯ –  ƻǃ=PƩƵVq¯{CiL3ZW{tª –  «y.Çå3.ŎČÐVĊ^ķACiL3 •  Lucene –  ĒƼW~oìĪlģŨ –  Ŕ¬«©q©ª •  Solr –  *$lƶ_3W©3 –  q¯3s3RŶûR?hL`.ãŹ>ħƗ

® © 2014 MapR Technologies 40 LucidWorks •  Solr/Lucene l3VDLt¯3©qĀA©
u3¢ –  %"'lƶ_©3 –  ª3W3¨¯>&VķƧ •  LucidWorks X Solr WìĪXJW]]V.Solr WĦæřålħƗ VFhBSR.Lucene lƳƍVăæR?he:VDQ9h

® © 2014 MapR Technologies 42 y¦£¯Sp3«VÊFhĒƼ •  3Xy¦£¯WĽ¿ • 
y¦£¯X¹ƜÐVq«.Ƹ¾.y3®3.´İW p3«lĈP •  ¹ƜÐU¯{ –  field:(୯ޠ1 ୯ޠ2) •  ]LX –  field:(୯ޠ1 OR ୯ޠ2) field:(“୯ޠ1 ୯ޠ2” ~ 5) field:(୯ޠ1 AND ୯ޠ 2)

® © 2014 MapR Technologies 43 q¯}33 ª{ ✔ id:
t4 title: puppy desc: The sweetest little puppy ever. keywords: puppy, dog, pet indicators: (t1) ";)&+3/8%*1= Solr 0$65;/";)&+1 !90:'5;.(7;#;);D\KE X: ";)&+1!90 Solr ";.,%*0$65;/5+.+ OZYJ ";)&+R";.,%*NTF

® © 2014 MapR Technologies 46 §33W»ê: ¥WĔƴ6Take the A
Train7l ƨ@

® © 2014 MapR Technologies 49 ĒƼy¦£¯W3SUhſƴW£ 3 •  MusicBrainz
W3 •  3VXo3p ID.MusicBrainz ID.o3pĔ. |«3/Ŝµ.¸Ŗ4ÇŢĔ5.õĵWe:UĞĆ>ŭ]ih

® © 2014 MapR Technologies 50 §33»êǊƢWũ: ŗƨ|nq« q©qDQ9h ÓÃR§33119
XţlDQ9]F=? ¥¡¦3¥¯ Duke EllingtonW o3pID ÌƦ q¯ q §33ID o3pID ©{ID

® © 2014 MapR Technologies 53 ƅſ¬~£¯3¨¯W Lucene y¦£¯ • 
¬3¯|CiL Mahout ¬~£¯3¤«Wq¯}3 3 ª{W3>.o3p3W y¦£¯¶W indicator p3«VťĂCiQ9h id 541902 mbid 983d4f8f-473e-4091-8394-415c105c4656 ame Charlie Winston area United Kingdom gender None Indicator_artists 997727,815,830794,59588,900,2591,1344,696268 id 1710 mbid 592a3b6d-c42b-4567-99c9-ecf63bd66499 ame Chck Berry area United States gender Male Indicator_artists 386685,875994,637954,3418,1344,789739,1460

® © 2014 MapR Technologies 54 v©q¯éƓ oq¢ £3 §33ǊƢ
|nq« Mahout l ĬOLÃƹ ĒƼ {3 q¯}33

® © 2014 MapR Technologies 55 MapR lĬOLªo«q¢¬~£¯3¨¯ oq¢ £3
|nq« Mahout l ĬOLÃƹ Web po ĒƼ {3 Pig Python ÔD9§33ǊƢ ¬~£¯3¨¯ èĿW3ĄÉ èĿ èĿR+) %$ lĬæDQĄÉ &/)+

® © 2014 MapR Technologies 56 ¯«V •  h l»OL§33(§33>cgígDL¼ŘW{«)
•  Ah AT Ah ( ) AT A ( )h §33¶ĝW¬~£¯3¨¯ (ŃĹX¼ŘZSƛFŁƁ) oq¢¶ĝW¬~£¯3¨¯ (ƯĢWƙšlŁƔ) A X¼Řl§33ZŁƁ

® © 2014 MapR Technologies 57 o3y{¥·WĳĐõ AT Ah (
) AT A ( )h §33¶ĝW¬~£¯3¨¯ oq¢¶ĝW¬~£¯3¨¯ 1PàW÷ČRX.]Gªo«q¢WČĴ>ľñ4ƺǒW¶5/ ¼ÎČĴXR?U9L`.ƈŮŅRŔű/ 2PàW÷ČRX.v©q¯R4ƉÏV5¼ÎČĴ>ņĪ/ ÔD9§33W{«(h)WäC9ČĴW^lªo«q¢R »;Ye9/BWL`ńŽ>ƈœVe@Uh/

Technologies ege9¬~£¯3¨¯VFhVX

® © 2014 MapR Technologies 61 ũ •  §33>{tª (A)
lÉÙ –  (»êðÞ §33oq¢ {tª •  §33>v (B) lŗƨ –  (»êðÞ §33oq¢ v •  TA X{tªW¬~£¯3¨¯lģĥ –  ,BNfl<ƭDMOLWRXU9RDd:=- •  TB XvW¬~£¯3¨¯lģĥ –  ,BifWêĸ>ū?RXU9RF=-

® © 2014 MapR Technologies 63 áēWũ •  {tª: ,Paco
de Lucia- (q¯Wzª) •  ƄùW£3ĒƼĖĭ: –  , %#'(&%- q¯W~£pšĉě –  ƌX\SmTUD •  ¬~£¯3¨¯3ĒƼ: –  ©£¯~z3S¯3 –  q¯ş.{©{z3 –  $"$>{©{U ©£¯~WªlƏǈ

® © 2014 MapR Technologies 68 A 1 A 2
! " # $ T A 1 A 2 ! " # $= A 1 T A 2 T ! " % % # $ & & A 1 A 2 ! " # $ = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & r 1 r 2 ! " % % # $ & & = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & h 1 h 2 ! " % % # $ & & r 1 = A 1 T A 1 A 1 T A 2 ! " % # $ & h 1 h 2 ! " % % # $ & &

® © 2014 MapR Technologies 70 ïćWáƖÍWúáWòą •  ǋǆWƷƟ →
§33X3D=ÅU9 •  ĎŰõ → ƫÕÈEĖĭMSǏ?fiQD]: •  éƓűÜ → Jiÿ·éƓDU@UOQD]: •  ØDQïÍWǓƕUòąRXU9

® © 2014 MapR Technologies 71 ĖĭWpª¯| •  ¬~£¯3¨¯ĖĭWƙšlÉiƐ;hL`Vpª¯ |>Ĭkih
–  ƙšWÉiƁ;X©¯¢V»: •  pª¯|Xv©q¯õĪlŵâCHh •  pª¯|XÈÌVáēWõĪlegŏ@Fh3s{ Vġ9áƎ>8h

® © 2014 MapR Technologies 72 ĖĭWpª¯| •  ¬~£¯3¨¯ĖĭWƙšlÉiƐ;hL`Vpª¯ |>Ĭkih
–  ƙšWÉiƁ;X©¯¢V»: •  pª¯|Xv©q¯õĪlŵâCHh •  pª¯|XÈÌVáēWõĪlegŏ@Fh3s{ Vġ9áƎ>8h “ƌWTmUŁƔega±?UŞ9>¸h”

® © 2014 MapR Technologies 74 ¯«Upª¯|o«ª¢ •  ©¯{WÊþVxrÃƤlĂ;L¿ö~o • 
ƊīU¬«WÉiƐ;l»:qW}3«lßƮ •  ƬÚXBW@f9 •  ]L … floor(t/T) l3SDQĬ: s = logr + N(0,logε) ε ∈ 1.5,3 [ ] Δr r ∝ε

® © 2014 MapR Technologies 75 ũ;Y… ε = 2
1 2 8 3 9 15 7 6 1 8 14 15 3 2 22 10 1 3 8 2 10 5 7 4 1 2 10 7 3 8 6 14 1 5 33 15 2 9 11 29 1 2 7 3 5 4 19 6 1 3 5 23 9 7 4 2 2 4 11 8 3 1 44 9 2 3 1 4 6 7 8 33 3 4 1 2 10 11 15 14 11 1 2 4 5 7 3 14 1 8 7 3 22 11 2 33

実践機械学習 — MahoutとSolrを活用したレコメンデーションにおけるイノベーション

実践機械学習 — MahoutとSolrを活用したレコメンデーションにおけるイノベーション

More Decks by 草薙昭彦

Other Decks in Technology

Featured

Transcript