PySparkを使った レコメンドアルゴリズムの改良

"PyData Tokyo Meetup #3 - 分散型機械学習 with PySpark"で発表して来ました！
@nashibao
株式会社プレイド
http://plaid.co.jp

1. PySparkΛ࢖ͬͨ ϨίϝϯυΞϧΰϦζϜͷվྑ (ג)ϓϨΠυ @nashibao 2015/01/16 http://plaid.co.jp PyData Tokyo Meetup #3 - ෼ࢄܕػցֶश with PySpark
2. ஫ҙɿ PyData Tokyo༻ʹ࡞ͬͯΈͨ࿩Ͱɺϓϩ μΫγϣϯͰ࢖͍ͬͯΔ࿩Ͱ͸͋Γ·ͤΜɽ
3. MLlib͸ɺϞσϧɺΞϧΰϦ ζϜ͕·ͩ·ͩগͳ͍ https://github.com/apache/spark/tree/ master/mllib/src/main/scala/org/apache/ spark/mllib
4. MLlib͸ɺϞσϧɺΞϧΰϦ ζϜ͕·ͩ·ͩগͳ͍ Classification Logistic Regression Naive Bayes SVM Clustering Gaussian Mixture K-means Regression Linear Regression Ridge Regression Lasso Regression Recommendation Matrix Factorization Tree Decision Tree Gradient Boosted Tree Random Forest
5. MLlib͸ɺϞσϧɺΞϧΰϦ ζϜ͕·ͩ·ͩগͳ͍ Classification Logistic Regression Naive Bayes SVM Clustering Gaussian Mixture K-means Regression Linear Regression Ridge Regression Lasso Regression Recommendation Matrix Factorization Alternating Least Square Tree Decision Tree Gradient Boosted Tree Random Forest
6. ࢼ͠ʹMatrix Factorization ͷΞϧΰϦζϜΛվྑͯ͠Έ ͨ
7. Matrix Factorization ෼ࢄग़དྷΔΞϧΰϦζϜ΋ΘΓͱ஌ΒΕͯΔ Alternating Least Square (MLlibͷ࣮૷) ऩଋ଎͍͕อূ͞Εͳ͍ɽղ෇ۙͰ஗͍ Gaussian Multiplicative Update ऩଋ஗͍͕อূ͞ΕͯΔ GMUͰऩଋੑΛվળͯ͠ΈΔ
8. https://www.youtube.com/watch?v=3LBgiFch4_g spotifyͷਓʹΑΔMLlibͷMF࣮૷ղઆ http://meyer.math.ncsu.edu/Meyer/Talks/ SAS_6_9_05_NmfWorkshop.pdf ALS,GMU౳ͷղઆ http://research.microsoft.com/pubs/119077/DNMF.pdf MicrosoftͰͷ෼ࢄGMUͷ࿦จ ࢀߟʹͨ͠΋ͷ
9. def update_als(x, W, H, V): w = W[x, :] v = V[x, :] m = H.shape[0] k = H.shape[1] HtH = H.T * H HtVt = H.T * v.T for i in range(k): HtH[i, i] += LAMBDA * m return np.linalg.solve(HtH, HtVt) def update_gmu(x, W, H, V): w = W[x, :] v = V[x, :] return multiply(w, (v * H) / (w * (H.T * H) + 10**-9)).T ٖࣅత
10. &OFSHZ      *UFSBUJPO "-4 "-4 (.6 ݁Ռ ALS GMU 収束 EMR m1.xlarge x 10 ີཚ਺ߦྻ 10000 x 10000
11. ஫ҙɿ broadcastΛ࢖࣮ͬͨ૷Ͱ໛ٖతɽ͞Βʹେ͖͍ૄߦྻΛѻ͑Δ Α͏ʹϑΝΠϧϕʔεʹมߋ͢Δඞཁ͋Γɽ Ͱ΋෼ࢄΞϧΰϦζϜͷςετʹ͸ͳΔ
12. 1. PySparkͷAPI៉ྷ/࢖͍΍͍͢ʂࣧౄ͢Β֮͑Δʂ 2. Interactiveʹग़དྷͯɺฦΓ΋σʔλ͕খ͚͞Ε͹଎͍ͷͰα ΫαΫ։ൃՄೳɽnumpy࢖͑ΔʂϑΥʔʂ 3. MLlibʹͪΐͬͱײ͡ΔMahoutͱಉ͡Α͏ͳҋ - [ҋ] ݸʑͷ࣮૷ͰΠϯλʔϑΣʔε͕͋·Γ౷Ұ͞Ε͍ͯͳ͍ɽΞϧΰϦζϜͱ ͦΕҎ֎͕੾Γ཭͞Ε͍ͯͳ͍ɽΞϧΰϦζϜ͕͋·Γ૿͍͑ͯͳ͍ɽ - اۀ಺ར༻ʹ͸ɺMLlibΛࢀߟఔ౓ʹ࣮૷ͨ͠ํ͕਎ܰʁ - ݸਓతʹ͸ScalaΑΓPythonଆʹ࣮૷ஔ͖͍ͨؾ͕͢Δ ײ૝
13. (ג)ϓϨΠυ http://plaid.co.jp EC޲͚ղੳαʔϏε΍ͬͯ·͢ɽ ࠷ۙ17໊͘Β͍ʹ૿͑ͨΑʂ