Elasticsearch での類似文書検索と More Like This Query API 詳解

Elasticsearch 勉強会 in 京都での発表スライド
（Elasticsearch 勉強会 in 大阪での発表と同じ内容です）

Transcript

1. Elasticsearch Ͱͷྨࣅจॻݕࡧͱ
   More Like This Query API ৄղ
   Elasticsearch ษڧձ in ژ౎
   @ ͸ͯͳژ౎ΦϑΟε
   ઙ໺ ୎໵
   id:takuya-a

   @takuya_a
   

   View Slide

2. id:takuya-a
   ϓϥοτϑΥʔϜˍΞυςΫνʔϜ
   • 2015 ೥ 4 ݄ʹೖࣾ
   • ͸ͯͳϒοΫϚʔΫશจݕࡧͷվળΛ୲౰
   ڵຯ
   • ৘ใݕࡧ
   • ࣗવݴޠॲཧ
   • ػցֶश
   OSS ׆ಈ
   • kuromoji.js ͳͲͷ JavaScript ϥΠϒϥϦΛ։ൃ
   

   View Slide

3. ͸ͯͳϒοΫϚʔΫ

   શจݕࡧͷਫ਼౓վળ
   ͸ͯϒશจݕࡧͷ͘͠Έʹ͍ͭͯ Hatena Engineer Seminar #5 Ͱൃද͠·ͨ͠
   https://speakerdeck.com/takuyaa/hatenabutukumakuquan-wen-jian-suo-falsejing-du-gai-shan
   

   View Slide

4. ྨࣅจॻݕࡧ (More Like This) ͱ͸
   ೖྗɿจॻ ग़ྗɿจॻू߹

   ʢྨࣅจॻʣ
   ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ
   ʮElasticsearch CheatSheetʯ
   ʮ[຋༁] From Solr to elasticsearchʯ
   ʮelasticsearchษڧձ | Doorkeeperʯ
   

   View Slide

5. ຊ೔ͷτʔΫͷ֓ཁ
   1. More Like This Query API ʹΑΔྨࣅจॻݕࡧ
   • Elasticsearch ʹඪ४ͰඋΘ͍ͬͯΔྨࣅจॻݕࡧ API
   • More Like This Query API ʹ͍ͭͯৄࡉͳΞϧΰϦζϜΛ঺հ

   ʢElasticsearch ͕ྨࣅจॻΛͲ͏΍ͬͯܭࢉ͍ͯ͠Δ͔ʣ
   2. Term Vectors API ʹΑΔྨࣅจॻݕࡧ
   • υΩϡϝϯτʹೖ͍ͬͯΔޠͷ౷ܭ৘ใΛऔಘ͢Δ API
   • ΑΓॊೈʹΧελϚΠζ͕Մೳ

   ʢޠͷ౷ܭ৘ใ͔Βࣗ෼Ͱܭࢉ͢Δʣ
   

   View Slide

6. ྨࣅจॻݕࡧͱ

   More Like This Query API
   

   View Slide

7. More Like This Query API
   https://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-mlt-query.html
   ೖྗɿจॻ ग़ྗɿจॻू߹

   ʢྨࣅจॻʣ
   ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ
   ʮElasticsearch CheatSheetʯ
   ʮ[຋༁] From Solr to elasticsearchʯ
   ʮelasticsearchษڧձ | Doorkeeperʯ
   ͜ΕΛ࣮ݱͰ͖Δ API
   

   View Slide

8. More Like This Query API
   https://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-mlt-query.html
   ϑΟʔϧυΛࢦఆ
   จॻ ID Λࢦఆ
   ύϥϝʔλΛࢦఆ
   ໊લ͕ࣅ͍ͯΔͷͰฆΒΘ͍͕͠ Search API ͷ More Like This API ͸ 1.6.0 Ͱ Deprecated ɺ 2.0 Ͱഇࢭ༧ఆ
   https://www.elastic.co/guide/en/elasticsearch/reference/current/search-more-like-this.html
   

   View Slide

9. ྨࣅจॻݕࡧͷཧ࿦తഎܠ

   - Bag of words Ϟσϧ -
   • จॻʢυΩϡϝϯτʣΛ୯ޠͷू߹ (Set) ͩͱΈͳ͢

   (୯ޠͷॱং͸ؾʹ͠ͳ͍)
   • Bag of words (BOW) ʹΑΔϕΫτϧۭؒϞσϧΛ

   جຊతͳߟ͑ํͱ͢Δ
   ಉ͡ޠΛଟؚ͘Ήจॻ = ྨࣅจॻͱఆٛ
   

   View Slide

10. Bag of Words ϞσϧʹΑΔ

    ྨࣅจॻͷྫ
    ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ
    Elasticsearch Elasticsearch
    ϦϦʔε ϦϦʔε
    Lucene Lucene
    synced-flush
    Ϋϥελ Ϋϥελ
    …
    …
    Pipeline
    ಉ͡ޠ͕ͲΕ͚ͩೖͬͯΔ͔ʁ
    Λܭࢉ
    

    View Slide

11. ྨࣅจॻݕࡧ

    - จॻͷಛ௃நग़ -
    1ͭͷจॻͷதʹ͸ɺͨ͘͞ΜͷޠʢλʔϜʣؚ͕·Ε͍ͯΔ
    • ͢΂ͯͷλʔϜɾ͢΂ͯͷจॻʹରͯ͠ɺྨࣅ౓ͷܭࢉΛ͢Δͷ͸ݱ࣮తͰ͸ͳ͍
    • ͍͔ͭ͘ͷॏཁޠΛબ୒͠ɺͦͷจॻͷಛ௃ྔͱ͢Δ

    ʢBag of Words Ͱͷྨࣅ౓ܭࢉΛ ॏཁͳλʔϜʹ͍͚ͭͯͩܭࢉ͢Δ͜ͱͰۙࣅ
    ͍ͯ͠Δʣ
    ʮॏཁޠΛͲͷΑ͏ʹબͿ͔ʯ͕ྨࣅจॻݕࡧͷϙΠϯτ
    ౷ܭ৘ใ͔ΒλʔϜͷॏཁ౓ʢείΞʣΛܭࢉͯ͠ɺ্Ґ Top-K ͷλʔϜΛબ୒͢Δ
    ʢλʔϜͷείΞؔ਺ͷྫɿ TF-IDF, IDF, RIDF, Gain ͳͲʣ
    

    View Slide

12. ྨࣅจॻݕࡧ
    - ΫΤϦ࡞੒ˍݕࡧ -
    நग़ͨ͠ॏཁޠΛ OR ΫΤϦʹઃఆͯ͠ݕࡧ͢Δ
    • ॏཁޠͷ͏ͪɺͲΕؚ͔͕·ΕΕ͹Α͍
    • ॏཁޠΛͨ͘͞ΜؚΉจॻͷείΞ͕ߴ͘ͳΔ
    

    View Slide

13. ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    

    View Slide

14. ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    More Like This Query API ΋ಉ͡ྲྀΕ
    

    View Slide

15. ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    More Like This Query API ΋ಉ͡ྲྀΕ
    

    View Slide

16. More Like This Query API ͷΞϧΰϦζϜ

    - λʔϜͷ౷ܭ৘ใͷऔಘ -
    IndexReader Ϋϥε Λ࢖͏
    • Lucene ͷసஔΠϯσοΫεΛಡΉͨΊͷΫϥε

    ʢ௨ৗͷݕࡧͰ΋࢖ΘΕΔʣ
    • λʔϜͷ౷ܭ৘ใ΋औಘͰ͖Δ
    • IndexReader#getTermVectors() ʹจॻ ID Λ༩͑ͯ

    term vector Λऔಘ
    

    View Slide

17. More Like This Query API ͷΞϧΰϦζϜ
    - ॲཧͷྲྀΕ -
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    

    View Slide

18. More Like This Query API ͷΞϧΰϦζϜ

    - λʔϜͷείΞܭࢉ -
    More Like This ͸ɺλʔϜͷείΞؔ਺ʹ TF-IDF Λ࢖͍ͬͯΔ
    1. term vector ͔ΒɺTF ΛΧ΢ϯτ
    2. IDF ͷܭࢉʹ͸TFIDFSimilarity Ϋϥε Λ࢖͏
    3. TF * IDF ΛɺͦͷλʔϜͷείΞͱ͢Δ
    TF : λʔϜස౓ fi,j → จॻதʹԿճ΋ग़ͯ͘ΔλʔϜ΄Ͳߴ͘ͳΔ
    IDF : จॻස౓ ni
    ͷٯ਺ʢN ͸จॻ਺ʣ→ ϨΞͳλʔϜ΄Ͳߴ͘ͳΔ
    fi,j : ΤϯτϦʔ j ʹݱΕΔλʔϜͷग़ݱճ਺ʢස౓ʣ = term_freq
    N : ΠϯσοΫεʹೖ͍ͬͯΔΤϯτϦʔͷ૯਺ = doc_count
    ni : ΠϯσοΫεશମͰλʔϜ͕ݱΕΔΤϯτϦʔͷ਺ = doc_freq
    

    View Slide

19. More Like This Query API ͷΞϧΰϦζϜ
    - ॲཧͷྲྀΕ -
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    

    View Slide

20. More Like This Query API ͷΞϧΰϦζϜ

    - λʔϜΛείΞॱͰฒͼସ͑ -
    λʔϜͷ਺͸͔ͳΓଟ͍
    ͢΂ͯͷλʔϜΛϝϞϦʹ͓͍ͯιʔτ͢Δͷ͸

    ۭؒతʹ΋࣌ؒతʹ΋ܭࢉίετ͕ߴ͍
    Top-K ͕ཉ͍͠৔߹
    ্Ґ K ݸ͚ͩΛอ͓͍࣋ͯͯ͠

    ͋ͱ͸ࣺͯͳ͕Βιʔτ
    

    View Slide

21. ༏ઌ౓͖ͭΩϡʔ (PriorityQueue)
    ߴ଎ʹ Top-K ΛٻΊΒΕΔσʔλߏ଄
    • Java Ͱ͸ඪ४ϥΠϒϥϦʹೖ͍ͬͯΔ
    • όΠφϦώʔϓʢ̎෼ώʔϓʣͰ࣮૷͢Δ͜ͱ͕ଟ͍
    • Elasticsearch Ͱ͸ɺ Lucene ͷ util ύοέʔδʹ͋Δ

    org.apache.lucene.util.PriorityQueue ΫϥεΛ࢖͍ͬͯΔ
    https://github.com/apache/lucene-solr/blob/trunk/lucene/core/src/java/org/apache/lucene/util/PriorityQueue.java
    

    View Slide

22. More Like This Query API ͷΞϧΰϦζϜ
    - ॲཧͷྲྀΕ -
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    লུ
    

    View Slide

23. More Like This Query API ͷΞϧΰϦζϜ

    - ·ͱΊ -
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ (retrieveTerms())
    IndexReader ͷϝιου getTermVectors(docNum) Ͱ term vectors ʢλʔϜͷ౷ܭ৘ใʣΛಡΈग़͠
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͰείΞϦϯάʣ
    1. term vector ͔Β TF ΛΧ΢ϯτ
    2. TFIDFSimilarity Ϋϥεʹ DF ͱ numDocs(શυΩϡϝϯτ਺) Λ౉ͯ͠ IDF Λܭࢉ
    3. TF * IDF Λܭࢉʢ͜Ε͕λʔϜͷείΞʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    PriorityQueue ΫϥεͰ Top-K ͷλʔϜʢॏཁޠʣΛબ୒
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    XMoreLikeThis Ϋϥε

    ʹ࣮૷͞Ε͍ͯΔ
    

    View Slide

24. Term Vectors API Λ࢖ͬͨ

    ྨࣅจॻݕࡧ
    

    View Slide

25. ྨࣅจॻݕࡧͷΞϧΰϦζϜʢ࠶ܝʣ
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    

    View Slide

26. Term Vectors API Λ࢖ͬͨྨࣅจॻݕࡧ
    λʔϜͷ֤छ౷ܭ৘ใ

    ΛऔಘͰ͖Δ
    ͜ΕΒͷ౷ܭ৘ใ͔Β

    λʔϜͷείΞΛܭࢉ͠

    ಛ௃ޠΛબ୒͢Δ
    

    View Slide

27. Term Vectors API Λ࢖ͬͨ

    ྨࣅจॻݕࡧͷΞϧΰϦζϜ
    1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ
    • Term Vectors API
    2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ
    • ౷ܭ৘ใ͔ΒಠࣗͷείΞؔ਺Ͱܭࢉ
    3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़
    • ॻ͖͍ͨݴޠͰϥΠϒϥϦ͕ͳ͔ͬͨΒࣗલͰ࣮૷
    4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒
    • Boolean Query ͷ should અʹλʔϜʢॏཁޠʣΛฒ΂Δ
    5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
    • Boolean Query Λ Elasticsearch ʹ౤͛Δ
    

    View Slide

28. ·ͱΊ
    2 छྨͷྨࣅจॻݕࡧͷ΍ΓํΛ঺հ
    1. More Like This Query API ʹΑΔྨࣅจॻݕࡧ
    ࠷ॳ͔Β૊Έࠐ·Ε͓ͯΓ API ͚ͩͰ؆୯ʹ࢖͑Δ
    ΫΤϦ֦ுͷܭࢉ͕ Elasticsearch ಺Ͱ׬݁ʢແବͳ IO ͕ൃੜ͠ͳ͍ʣ
    νϡʔχϯά͕͍͔ͭ͘ͷύϥϝʔλͷௐ੔ʹݶΒΕΔ
    2. Term Vectors API ʹΑΔྨࣅจॻݕࡧ
    ΑΓॊೈͳΫΤϦ֦ு͕ߦ͑Δʢࣗવݴޠॲཧ΍ػցֶशͱͷ૊Έ߹Θ͕ͤ༰қʣ
    Term Vector API Ͱฦͬͯ͘ΔϨεϙϯεαΠζ͕େ͖͍ʢωοτϫʔΫෛՙ͕ߴ͍ʣ
    ΞϓϦέʔγϣϯαʔόͰେྔͷλʔϜΛॲཧ͢Δඞཁ͕༗ΔʢCPUෛՙ͕ߴ͍ʣ
    

    View Slide

29. We are hiring !
    ৘ใݕࡧɾࣗવݴޠॲཧɾػցֶशʹڵຯͷ͋Δ͔ͨ

    ϩάղੳج൫ߏஙɾϩά෼ੳʹڵຯͷ͋Δ͔ͨ
    ͝Ԡื͓଴͍ͪͯ͠·͢ʂ
    

    View Slide