Share
Embed
実践機械学習 — MahoutとSolrを活用したレコメンデーションにおけるイノベーション
機械学習は、増え続けるデータをもとに、事業戦略の判断やより正確な予測、関連性の推定を行うための、重要なツールです。機械学習の中でも、最も幅広く活用されているアプリケーションはレコメンデーションエンジンです。スケーラブルな機械学習ライブラリであるMahoutは、レコメンデーションの生成とデータの扱いをシンプルなものにしてくれます。本講演では、より構築が簡単なレコメンデーションエンジンのデザインと、そのイノベーティブな実装方法を活用した場合の利点を紹介します。2014年7月8日に開催されたHadoop Conference Japan 2014での講演資料です。
草薙昭彦
July 09, 2014
Transcript
76 slides
-
® © 2014 MapR Technologies 1 ® © 2014 MapR
Technologies áǎìDžéƓ— MahoutSSolrlįæDL ¬~£¯3¨¯V<Ahq3¨¯ ƪǐƞƝ MapR Technologies -
® © 2014 MapR Technologies 3 ´°W{ • ¬~£¯3¨¯SX? •
¬~£¯3¨¯l§33VģŨFhVX? • ƅſ¬~£¯3¨¯Wũ • ege9¬~£¯3¨¯VFhVX? -
® © 2014 MapR Technologies 4 áǎìDžéƓ ¬~£¯3¨¯V<Ahq3¨¯ =; 32
ষɹࣄྫɿԻָϨίϝϯμʔ 6.3ɹେنͳϨίϝϯσʔγϣϯ ࢹௌཤྺΛγϛϡϨʔτ͢Δ͜ͱͰɺେྔͷσʔλΛੜ͢Δ͜ͱ͕Ͱ͖·͕͢ɺ ेԯͷࢹௌهΛੜ͢ΔͨΊʹɺϏοάσʔλͷॲཧ࣮ࡍͷͱ͜ΖඞཁͰ ͋Γ·ͤΜɻͨͩ͠ɺ͜ͷنͰੜ͞ΕͨϩάͷੳΛߦ͏ʹ͋ͨͬͯɺγες ϜͷεέʔϥϏϦςΟʹ͍ͭͯҙΛ͍࢝ΊΔඞཁ͕͋Γ·͢ɻ Music Machine ͷϨίϝϯμʔͷγεςϜɺMahout ͷ RowSimilarityJob Λ ͬͯϢʔβʔͷࢹௌϩάΛॲཧ͠ɺΞʔςΟετͷΠϯδέʔλʔΛؚΉϨίϝϯ σʔγϣϯϞσϧΛߏங͠·͢ɻ͜ͷԋࢉॲཧɺγεςϜશମͷҰ෦ʹա͗·ͤ Μɻݱ࣮ʹΘΕΔγεςϜ͕ৗʹͦ͏Ͱ͋ΔΑ͏ʹɺطଘͷγεςϜ܈ͱͷϑΥʔ ϚοτมͷͨΊʹɺେྔͷ࿈ܞͷͨΊͷίʔυ͕ඞཁͰͨ͠ɻͦͷ࣮ͷ֓ཁΛɺ ਤ 6-3 ʹࣔ͠·͢ɻ͜͜Ͱ MapR Λ͍ͬͯΔͷͰɺPythonɺPostgreSQLɺ LucidWorks ΛؚΉ΄΅ͯ͢ͷίϯϙʔωϯτɺΫϥελͰಈ࡞͠·͢ɻ ਤ 6-3ɹϨίϝϯμʔͷΦϑϥΠϯύʔτʹ͓͚Δɺैདྷܕ͓Αͼ Hadoop ʹΑΔίϯϐϡʔςΟ ϯάɻփ৭ͷ෦ɺMusic Machine ͷϨίϝϯμʔͷ Hadoop ϕʔεͷΫϥελ্Ͱಈ ࡞͢Δ෦Λ͍ࣔͯ͠Δɻ͜͜ͰɺϦΞϧλΠϜͷࢄϑΝΠϧγεςϜΛ࣋ͭɺMapR distribution for Apache Hadoop Λ༻͍ͯ͠Δɻଞͷ Hadoop σΟετϦϏϡʔγϣϯ ͰΘΕ͍ͯΔϑΝΠϧγεςϜʢHDFSʣΛ͏ͨΊʹɺ͞Βʹ͍͔ͭ͘ͷεςοϓ͕ ඞཁ Document ϢʔβʔʵΞΠςϜ ϩάϑΝΠϧ ϢʔβʔʗΞΠςϜ*%ͷ ͷϚοϐϯά .BIPVUͷ 3PX4JNJMBSJUZ+PC ϢʔβʔʗΞΠςϜͷ ͷ*%ͷ Ϛοϐϯάͱɺ 4PMS+40/ ϑΥʔϚο τͷม ϢʔβʔʗΞΠςϜ *%ͷࣙॻ 4PMSͷ+40/ ϑΥʔϚο τͷม Document ΞΠςϜͷ ϝλσʔλ .BQSFEVDF 1ZUIPO 1PTUHSF42- 17 17 ɹ"QBDIF.BIPVU ʹΑΔϞσϧߏஙͷํ๏ 4.1ɹApache Mahout ʹΑΔϞσϧߏஙͷํ๏ ࣮ફతͳϨίϝϯμʔͷͨΊͷϨίϝϯσʔγϣϯϞσϧΛߏங͢Δʹ͋ͨͬͯ ɺΦʔϓϯιʔεͷεέʔϥϒϧͳػցֶशϥΠϒϥϦͷ Apache Mahout ͷ࣋ͬ ͍ͯΔΞϧΰϦζϜΛ͏͜ͱʹ͠·͠ΐ͏ɻΓ͍ͨ͜ͱɺMahout ͷߦྻԋࢉ ΛͬͯɺϢʔβʔͷߦಈཤྺ͔ΒɺϨίϝϯσʔγϣϯͷͨΊͷ༗༻ͳΠϯδέʔ λʔΛಘΔ͜ͱͰ͢ɻͦͷͨΊʹɺ3 ͭͷߦྻΛߏங͍͖ͯ͠·͢ɻ ཤྺϚτϦΫε ϢʔβʔͱΞΠςϜؒͷؔ࿈ΛɺϢʔβʔͱΞΠςϜͷߦྻͱͯ͠ه͢Δɻ ڞىϚτϦΫε ཤྺϚτϦΫεΛɺΞΠςϜͱΞΠςϜͷߦྻʹม͠ɺϢʔβʔͷཤྺͰͲ ͏͍ͬͨΞΠςϜಉ͕࢜Ұॹʹొ͍ͯ͠Δ͔Λه͢Δɻ ਤ 4-3ɹཤྺˠڞىˠΠϯδέʔλʔϚτϦΫεɻΠϯδέʔλʔϚτϦΫεͰදݱ͞Ε͍ͯΔ͜ͷϞ σϧʹɺ Ϧϯΰ͕ ʮࢠݘʯ ΛϨίϝϯυ͖͢ΠϯδέʔλʔͰ͋Δ͜ͱ͕ූ߸Խ͞Ε͍ͯΔɻ ཤྺϚ τϦΫε ڞىϚ τϦΫε Πϯδέʔλʔ Ϛ τϦΫε "MJDF #PC $IBSMFT *$$!$""$'!#$ Ƹ o3«0{3 ĩϳ½ƪǐƞƝ Ňƀ ĩϳ½¬q¯ŤÂżŦ.ÛÂƿ±.đÂƠƝ Żîǂƃ ƾ r¯3X & %*)S%"'lįæDL. ¯«U>faôÙU¬~£¯3¨ ¯t¯¯Wq¯SáƖlĜś -
® © 2014 MapR Technologies 5 ® © 2014 MapR
Technologies ƋŕVP9Q -
® © 2014 MapR Technologies 7 ìDžéƓo«ª ¢SJWáƖ A 1 A
2 ! " # $ T A 1 A 2 ! " # $= A 1 T A 2 T ! " % % # $ & & A 1 A 2 ! " # $ = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & r 1 r 2 ! " % % # $ & & = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & h 1 h 2 ! " % % # $ & & r 1 = A 1 T A 1 A 1 T A 2 ! " % # $ & h 1 h 2 ! " % % # $ & & O(κ k d + k3 d) = O(k2 d log n + k3 d) for small k, high quality O(κ d log k) or O(d log κ log k) for larger k, looser quality B<GA (;29DH >C I -
® © 2014 MapR Technologies 8 ¬~£¯3¨¯ • 3: µ2W»ê4§335Soq¢SWq¯©{¨¯
– 3X¬~£¯3¨¯¤«WéƓVĬkih • ťĂWƊīUaD@XŴ]D9q¯©{¨¯lģĘFhBS>àÐ • %-0'(.2!>: ƚĸ.ƅſ.ÇŸ3W¬©¯ßƮ.e oc¬¬3RWŪāʼnĶWģĘ -
® © 2014 MapR Technologies 11 ® © 2014 MapR
Technologies ¬~£¯3¨¯WD@^ (]LX6I want a pony7) -
® © 2014 MapR Technologies 12 I want a pony!
<5"34$:=@ 79$#?$ AB68#,1* -
® © 2014 MapR Technologies 21 ¬~£¯3¨¯ çÝ>3lƲÉDLfT:U h? oª
¥3« o£ªo ƌVţlo£ªoV<FF`FiY9 9? ? -
® © 2014 MapR Technologies 22 ¬~£¯3¨¯ oª ¥3«
o£ªo aDçÝ>3lƲÉDLf. 3XƌWaWlčơFhWV ŌV×LU9... ? -
® © 2014 MapR Technologies 23 ƘWĺŒ3Wòą • ƈœVµøWoq¢XF[QSĺŒFh (ǁ>3>ưD9
S9:ĞĆXŌV×LU9…) – ũrs«w¢y¦£¯.t¬33Wƅſ • ĤƷƟU¸úXĨŲ^>U9 – =^JgWnjS=3We:Uƥ;GŴ]ihoq¢lģŨD L9WRU9őg • ưD9WXũóÐUĺŒ – BiBJ>žźų9ū^WċƂWĞĆƑR8g.¬~£¯3¨¯ WėǀSUh -
® © 2014 MapR Technologies 24 »ê=fŌV×PċƂlĚhVX • |nq«lĬOQ§33 x
oq¢WNJƢ ª{l Ģö – ŝBWq¯©{¨¯WNJƢXçQWLJŀÐUĉ^¿kHSŊ[h SǍU3R8h • oq¢ x oq¢WĺŒ ª{VŁƁ • ũóÐUĺŒlƭD.ŌV×Pq¯}33lÅPAQq¯ }33 ª{lĢö – ÊþǑÜŊLLRXTWĺŒ>ū^Wq¯}33SDQĬ;h= lıülĈOQďňFhWVŬAVUh – Apache Mahout WRowSimilarityJobXLLRlĬ: -
® © 2014 MapR Technologies 25 Apache Mahout • v3¯3
Apache s{http://mahout.apache.org • Mahout version 0.9 X 2014²2ÁVªª3 • ìDžéƓWL`W}3©«Uo«ª ¢W©q©ª – Apache Hadoop Rê@aWa8iY.Hadoop ëñWaWa8h – äC9}3«RW^ê@aWa8h – Ƈƣá»FhaWa8iY.3}¯¥«UaWa8h • ÛğWaWlŭ_ – {©ª¯| & ÖãŋŠ – ÃƆ – ¬~£¯3¨¯ – Mahout Math ©q©ª -
® © 2014 MapR Technologies 26 |nq« oª ¥3«
oª ¥3« oª -
® © 2014 MapR Technologies 27 |nq« u1 u3 u2
u1 u3 u2 u1 t1# t4# t3# t2# t3# t3# t1# -
® © 2014 MapR Technologies 28 NJƢ ª{: §33/oq¢ oª
¥3« ✔ ✔ ✔ ✔ ✔ ✔ ✔ -
® © 2014 MapR Technologies 29 ĺŒ ª{: oq¢/oq¢ -
1# 2# 1# 1# 1# 1# 2# 1# SWV@ ?. 0# 0# 0# 0# -
® © 2014 MapR Technologies 30 ĺŒ ª{: oq¢/oq¢ LLR
L?PE SW";)&+U[… - 1# 2# 1# 1# 1# 1# 2# 1# 0# 0# 0# 0# -
® © 2014 MapR Technologies 31 ĺŒqª ª{ 1# 1#
not# not# 1# -
® © 2014 MapR Technologies 32 ũólÅPAh A not A
B 13 1000 not B 1000 100,000 A not A B 1 0 not B 0 2 A not A B 1 0 not B 0 10,000 A not A B 10 0 not B 0 100,000 JiKiWōŷ=fTmUĖł>ŐAh? -
® © 2014 MapR Technologies 33 ũólÅPAh • «3 LLR
X<<eJƂļljŧVġ9 • Apache Mahout RXRowSimilarityJob > LLR lăæ A not A B 1 0 not B 0 2 A not A B 1 0 not B 0 10,000 A not A B 10 0 not B 0 100,000 1.95 4.52 14.3 JiKiWōŷ=fTmUĖł>ŐAh? A not A B 13 1000 not B 1000 100,000 0.90 -
® © 2014 MapR Technologies 34 ĺŒ ª{ : LLR
L?PE SW";)&+U[ - 1# 2# 1# 1# 1# 1# 2# 1# 0# 0# 0# 0# -
® © 2014 MapR Technologies 35 q¯}33 ª{: ũóÐUĺŒ ✔#
✔# MQ: 3% = "-40$65;/";)&+1! 90YJ … -
® © 2014 MapR Technologies 39 Apache Solr/Apache Lucene •
Apache Solr/Lucene: v3¯3W®«UĒƼt¯¯ – ƻǃ=PƩƵVq¯{CiL3ZW{tª – «y.Çå3.ŎČÐVĊ^ķACiL3 • Lucene – ĒƼW~oìĪlģŨ – Ŕ¬«©q©ª • Solr – *$lƶ_3W©3 – q¯3s3RŶûR?hL`.ãŹ>ħƗ -
® © 2014 MapR Technologies 40 LucidWorks • Solr/Lucene l3VDLt¯3©q ĀA©
u3¢ – %"'lƶ_©3 – ª3W3¨¯>&VķƧ • LucidWorks X Solr WìĪXJW]]V.Solr WĦæřålħƗ VFhBSR.Lucene lƳƍVăæR?he:VDQ9h -
® © 2014 MapR Technologies 42 y¦£¯Sp3«VÊFhĒƼ • 3Xy¦£¯WĽ¿ •
y¦£¯X¹ƜÐVq«.Ƹ¾.y3®3.´İW p3«lĈP • ¹ƜÐU¯{ – field:(୯ޠ1 ୯ޠ2) • ]LX – field:(୯ޠ1 OR ୯ޠ2) field:(“୯ޠ1 ୯ޠ2” ~ 5) field:(୯ޠ1 AND ୯ޠ 2) -
® © 2014 MapR Technologies 43 q¯}33 ª{ ✔ id:
t4 title: puppy desc: The sweetest little puppy ever. keywords: puppy, dog, pet indicators: (t1) ";)&+3/8%*1= Solr 0$65;/";)&+1 !90:'5;.(7;#;);D\KE X: ";)&+1!90 Solr ";.,%*0$65;/5+.+ OZYJ ";)&+R";.,%*NTF -
® © 2014 MapR Technologies 49 ĒƼy¦£¯W3SUhſƴW£ 3 • MusicBrainz
W3 • 3VXo3p ID.MusicBrainz ID.o3pĔ. |«3/Ŝµ.¸Ŗ4ÇŢĔ5.õĵWe:UĞĆ>ŭ]ih -
® © 2014 MapR Technologies 50 §33»êNJƢWũ: ŗƨ|nq« q©qDQ9h ÓÃR§33119
XţlDQ9]F=? ¥ ¡¦3¥¯ Duke EllingtonW o3pID ÌƦ q¯ q §33ID o3pID ©{ID -
® © 2014 MapR Technologies 53 ƅſ¬~£¯3¨¯W Lucene y¦£¯ •
¬3¯|CiL Mahout ¬~£¯3¤«Wq¯}3 3 ª{W3>.o3p3W y¦£¯¶W indicator p3«VťĂCiQ9h id 541902 mbid 983d4f8f-473e-4091-8394-415c105c4656 ame Charlie Winston area United Kingdom gender None Indicator_artists 997727,815,830794,59588,900,2591,1344,696268 id 1710 mbid 592a3b6d-c42b-4567-99c9-ecf63bd66499 ame Chck Berry area United States gender Male Indicator_artists 386685,875994,637954,3418,1344,789739,1460 -
® © 2014 MapR Technologies 54 v©q¯éƓ oq¢ £3 §33NJƢ
|nq« Mahout l ĬOLÃƹ ĒƼ {3 q¯}33 -
® © 2014 MapR Technologies 55 MapR lĬOLªo«q¢¬~£¯3¨¯ oq¢ £3
|nq« Mahout l ĬOLÃƹ Web po ĒƼ {3 Pig Python ÔD9§33NJƢ ¬~£¯3¨¯ èĿW3ĄÉ èĿ èĿR+) %$ lĬæDQĄÉ &/)+ -
® © 2014 MapR Technologies 56 ¯«V • h l»OL§33(§33>cgígDL¼ŘW{«)
• Ah AT Ah ( ) AT A ( )h §33¶ĝW¬~£¯3¨¯ (ŃĹX¼ŘZSƛFŁƁ) oq¢¶ĝW¬~£¯3¨¯ (ƯĢWƙšlŁƔ) A X¼Řl§33ZŁƁ -
® © 2014 MapR Technologies 57 o3y{¥·WijĐõ AT Ah (
) AT A ( )h §33¶ĝW¬~£¯3¨¯ oq¢¶ĝW¬~£¯3¨¯ 1PàW÷ČRX.]Gªo«q¢WČĴ>ľñ4ƺǒW¶5/ ¼ÎČĴXR?U9L`.ƈŮŅRŔű/ 2PàW÷ČRX.v©q¯R4ƉÏV5¼ÎČĴ>ņĪ/ ÔD9§33W{«(h)WäC9ČĴW^lªo«q¢R »;Ye9/BWL`ńŽ>ƈœVe@Uh/ -
® © 2014 MapR Technologies 58 ® © 2014 MapR
Technologies ege9¬~£¯3¨¯VFhVX -
® © 2014 MapR Technologies 61 ũ • §33>{tª (A)
lÉÙ – (»êðÞ §33oq¢ {tª • §33>v (B) lŗƨ – (»êðÞ §33oq¢ v • TA X{tªW¬~£¯3¨¯lģĥ – ,BNfl<ƭDMOLWRXU9RDd:=- • TB XvW¬~£¯3¨¯lģĥ – ,BifWêĸ>ū?RXU9RF=- -
® © 2014 MapR Technologies 62 ±¼USBj • BTA X{tªVÊFhêĸlª~£¯
– (8i.BiĒƼt¯¯EbU=OLOA – 9cŞ:U.~¯¯c£3XÅQU9D -
® © 2014 MapR Technologies 63 áēWũ • {tª: ,Paco
de Lucia- (q¯Wzª) • ƄùW£3ĒƼĖĭ: – , %#'(&%- q¯W~£pšĉě – ƌX\SmTUD • ¬~£¯3¨¯3ĒƼ: – ©£¯~z3S¯3 – q¯ş.{©{z3 – $"$>{©{U ©£¯~WªlƏLj -
® © 2014 MapR Technologies 65 ÊƽõVeg{¬~£¯3¨¯lÆö AT A (
)h BT A ( )h v©q¯©3¯|lăæDL ƄùůW¬~£¯3¨¯ {¬~£¯3¨¯ -
® © 2014 MapR Technologies 67 A 1 A 2
! " # $ §33 ¼Ř q1 ¼Ř q2 -
® © 2014 MapR Technologies 68 A 1 A 2
! " # $ T A 1 A 2 ! " # $= A 1 T A 2 T ! " % % # $ & & A 1 A 2 ! " # $ = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & r 1 r 2 ! " % % # $ & & = A 1 T A 1 A 1 T A 2 AT 2 A 1 AT 2 A 2 ! " % % # $ & & h 1 h 2 ! " % % # $ & & r 1 = A 1 T A 1 A 1 T A 2 ! " % # $ & h 1 h 2 ! " % % # $ & & -
® © 2014 MapR Technologies 70 ïćWáƖÍWúáWòą • NjdžWƷƟ →
§33X3D=ÅU9 • ĎŰõ → ƫÕÈEĖĭMSǏ?fiQD]: • éƓűÜ → Jiÿ·éƓDU@UOQD]: • ØDQïÍWǓƕUòąRXU9 -
® © 2014 MapR Technologies 71 ĖĭWpª¯| • ¬~£¯3¨¯ĖĭWƙšlÉiƐ;hL`Vpª¯ |>Ĭkih
– ƙšWÉiƁ;X©¯¢V»: • pª¯|Xv©q¯õĪlŵâCHh • pª¯|XÈÌVáēWõĪlegŏ@Fh3s{ Vġ9áƎ>8h -
® © 2014 MapR Technologies 72 ĖĭWpª¯| • ¬~£¯3¨¯ĖĭWƙšlÉiƐ;hL`Vpª¯ |>Ĭkih
– ƙšWÉiƁ;X©¯¢V»: • pª¯|Xv©q¯õĪlŵâCHh • pª¯|XÈÌVáēWõĪlegŏ@Fh3s{ Vġ9áƎ>8h “ƌWTmUŁƔega±?UŞ9>¸h” -
® © 2014 MapR Technologies 74 ¯«Upª¯|o«ª ¢ • ©¯{WÊþVxrÃƤlĂ;L¿ö~o •
ƊīU¬«WÉiƐ;l»:q W}3«lßƮ • ƬÚXBW@f9 • ]L … floor(t/T) l3SDQĬ: s = logr + N(0,logε) ε ∈ 1.5,3 [ ] Δr r ∝ε -
® © 2014 MapR Technologies 75 ũ;Y… ε = 2
1 2 8 3 9 15 7 6 1 8 14 15 3 2 22 10 1 3 8 2 10 5 7 4 1 2 10 7 3 8 6 14 1 5 33 15 2 9 11 29 1 2 7 3 5 4 19 6 1 3 5 23 9 7 4 2 2 4 11 8 3 1 44 9 2 3 1 4 6 7 8 33 3 4 1 2 10 11 15 14 11 1 2 4 5 7 3 14 1 8 7 3 22 11 2 33 -
® © 2014 MapR Technologies 77 Q & A @mapr_japan
maprjapan [email protected] <ò9¿kHXBNf]R MapR maprtech mapr-technologies