疎行列と Jaccard 類似度の高速計算

Transcript

1. ૄߦྻ ͋Δ͍͸ Jaccard ྨࣅ౓Λߴ଎Ͱܭࢉ͢Δํ๏ @na_o_ys
2. Agenda 1. ૄߦྻͷσʔλߏ଄ 2. Python ͱ਺஋ܭࢉ 3. Python ͱૄߦྻ 4. Jaccard ྨࣅ౓
3. 1. ૄߦྻͷσʔλߏ଄
4. ૄߦྻͱ͸ ΄ͱΜͲͷཁૉ͕ 0 Ͱ͋Δߦྻ 
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
5. 1. ૄߦྻͷσʔλߏ଄ (1) • ௨ৗͷߦྻ͸ Array • ૄߦྻΛ Array Ͱѻ͏ͱϝϞϦ΋ԋࢉ΋ແବ • 0 ϕΫτϧಉ࢜ͷ৐ࢉͱ͔໌Β͔ʹແବ
6. 1. ૄߦྻͷσʔλߏ଄ (2) • Compressed Sparse Row (CSR) • CSR ಉ࢜ͷՃࢉ, ߦྻੵ͕ߴ଎ • ߦϕΫτϧͷऔΓग़͕͠ߴ଎ • ྻϕΫτϧͷऔΓग़͕͠௿଎ • (wikipedia)
7. 2. Python ͱ਺஋ܭࢉ
8. 2. Python ͱ਺஋ܭࢉ • ੒ख़ͨ͠਺஋ܭࢉϥΠϒϥϦ • NumPy, SciPy • Scikit-learn
9. ͱ͜ΖͰɺPython ͸஗͍ (DEMO)
10. Python ͸஗͍ • 5000 ഒ ࣮ߦ࣌ؒ 1ZUIPO 
    NT Ұ෦
    /VN1Z 
    NT શ෦
    /VN1Z 
    NT
11. Python-loop is Evil • ߦྻϧʔϓ͸ઈରʹॻ͍ͯ͸͍͚ͳ͍ • 1 ඵͰऴΘΔ͸ͣͷॲཧʹ 2 ͔͔࣌ؒΔ • ߦϧʔϓ/ྻϧʔϓ΋ॻ͔ͳ͍ํ͕ྑ͍ • 1 ඵͰऴΘΔ͸ͣͷॲཧʹ 1 ෼͔͔Δ
12. 3. Python ͱૄߦྻ
13. 3. Python ͱૄߦྻ • scipy.sparse.csr_matrix
14. ޮ཰తͳߦྻॲཧ • ߦϕΫτϧͷऔΓग़͠ • Ճࢉ৐ࢉ, ߦྻੵ • ಺෦දݱΛ numpy.ndarray ͱͯ͠อ࣋ • औΓग़ͯ͠௚઀ૢ࡞Ͱ͖Δ (NumPy ͷੈք Ͱ)
15. 4. Jaccard ྨࣅ౓
16. 4. Jaccard ྨࣅ౓ • ϕΫτϧಉ࢜ͷྨࣅ౓ • ڠௐϑΟϧλϦϯάͱ͔Ͱ࢖͏ • ϢʔβAͱϢʔβB͸ͲΕ͘Β͍ࣅ͍ͯΔ͔ Jaccard(a, b) = a・b / (a・a + b・b - a・b)
17. ࢓ࣄͰඞཁʹͳͬͨ͜ͱ • ૄߦྻͷߦϕΫτϧಉ࢜ͷ Jaccard ྨࣅ౓Λ ܭࢉ͍ͨ͠ 
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
18. DEMO
19. ·ͱΊ
20. ·ͱΊ • Python ͸஗͍ • ϥΠϒϥϦΛ͏·͘࢖͏ඞཁ͕͋Δ • ϒϩάΛॻ͍ͨ • http://na-o-ys.github.io/others/ 2015-11-07-sparse-vector- similarities.html