Elasticsearch での類似文書検索と More Like This Query API 詳解

Elasticsearch での類似文書検索と More Like This Query API 詳解

Elasticsearch 勉強会 in 京都での発表スライド
(Elasticsearch 勉強会 in 大阪での発表と同じ内容です)

13f3313ae1ec1d9b3ed76ccbd746291b?s=128

Takuya Asano

July 14, 2015
Tweet

Transcript

  1. Elasticsearch Ͱͷྨࣅจॻݕࡧͱ More Like This Query API ৄղ Elasticsearch ษڧձ

    in ژ౎ @ ͸ͯͳژ౎ΦϑΟε ઙ໺ ୎໵ id:takuya-a
 @takuya_a
  2. id:takuya-a ϓϥοτϑΥʔϜˍΞυςΫνʔϜ • 2015 ೥ 4 ݄ʹೖࣾ • ͸ͯͳϒοΫϚʔΫશจݕࡧͷվળΛ୲౰ ڵຯ

    • ৘ใݕࡧ • ࣗવݴޠॲཧ • ػցֶश OSS ׆ಈ • kuromoji.js ͳͲͷ JavaScript ϥΠϒϥϦΛ։ൃ
  3. ͸ͯͳϒοΫϚʔΫ
 શจݕࡧͷਫ਼౓վળ ͸ͯϒશจݕࡧͷ͘͠Έʹ͍ͭͯ Hatena Engineer Seminar #5 Ͱൃද͠·ͨ͠ https://speakerdeck.com/takuyaa/hatenabutukumakuquan-wen-jian-suo-falsejing-du-gai-shan

  4. ྨࣅจॻݕࡧ (More Like This) ͱ͸ ೖྗɿจॻ ग़ྗɿจॻू߹
 ʢྨࣅจॻʣ ʮElasticsearch 1.6.0ϦϦʔεʯ

    ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ ʮElasticsearch CheatSheetʯ ʮ[຋༁] From Solr to elasticsearchʯ ʮelasticsearchษڧձ | Doorkeeperʯ
  5. ຊ೔ͷτʔΫͷ֓ཁ 1. More Like This Query API ʹΑΔྨࣅจॻݕࡧ • Elasticsearch

    ʹඪ४ͰඋΘ͍ͬͯΔྨࣅจॻݕࡧ API • More Like This Query API ʹ͍ͭͯৄࡉͳΞϧΰϦζϜΛ঺հ
 ʢElasticsearch ͕ྨࣅจॻΛͲ͏΍ͬͯܭࢉ͍ͯ͠Δ͔ʣ 2. Term Vectors API ʹΑΔྨࣅจॻݕࡧ • υΩϡϝϯτʹೖ͍ͬͯΔޠͷ౷ܭ৘ใΛऔಘ͢Δ API • ΑΓॊೈʹΧελϚΠζ͕Մೳ
 ʢޠͷ౷ܭ৘ใ͔Βࣗ෼Ͱܭࢉ͢Δʣ
  6. ྨࣅจॻݕࡧͱ
 More Like This Query API

  7. More Like This Query API https://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-mlt-query.html ೖྗɿจॻ ग़ྗɿจॻू߹
 ʢྨࣅจॻʣ ʮElasticsearch

    1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ ʮElasticsearch CheatSheetʯ ʮ[຋༁] From Solr to elasticsearchʯ ʮelasticsearchษڧձ | Doorkeeperʯ ͜ΕΛ࣮ݱͰ͖Δ API
  8. More Like This Query API https://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-mlt-query.html ϑΟʔϧυΛࢦఆ จॻ ID Λࢦఆ

    ύϥϝʔλΛࢦఆ ໊લ͕ࣅ͍ͯΔͷͰฆΒΘ͍͕͠ Search API ͷ More Like This API ͸ 1.6.0 Ͱ Deprecated ɺ 2.0 Ͱഇࢭ༧ఆ https://www.elastic.co/guide/en/elasticsearch/reference/current/search-more-like-this.html
  9. ྨࣅจॻݕࡧͷཧ࿦తഎܠ
 - Bag of words Ϟσϧ - • จॻʢυΩϡϝϯτʣΛ୯ޠͷू߹ (Set)

    ͩͱΈͳ͢
 (୯ޠͷॱং͸ؾʹ͠ͳ͍) • Bag of words (BOW) ʹΑΔϕΫτϧۭؒϞσϧΛ
 جຊతͳߟ͑ํͱ͢Δ ಉ͡ޠΛଟؚ͘Ήจॻ = ྨࣅจॻͱఆٛ
  10. Bag of Words ϞσϧʹΑΔ
 ྨࣅจॻͷྫ ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ Elasticsearch

    Elasticsearch ϦϦʔε ϦϦʔε Lucene Lucene synced-flush Ϋϥελ Ϋϥελ … … Pipeline ಉ͡ޠ͕ͲΕ͚ͩೖͬͯΔ͔ʁ Λܭࢉ
  11. ྨࣅจॻݕࡧ
 - จॻͷಛ௃நग़ - 1ͭͷจॻͷதʹ͸ɺͨ͘͞ΜͷޠʢλʔϜʣؚ͕·Ε͍ͯΔ • ͢΂ͯͷλʔϜɾ͢΂ͯͷจॻʹରͯ͠ɺྨࣅ౓ͷܭࢉΛ͢Δͷ͸ݱ࣮తͰ͸ͳ͍ • ͍͔ͭ͘ͷॏཁޠΛબ୒͠ɺͦͷจॻͷಛ௃ྔͱ͢Δ
 ʢBag

    of Words Ͱͷྨࣅ౓ܭࢉΛ ॏཁͳλʔϜʹ͍͚ͭͯͩܭࢉ͢Δ͜ͱͰۙࣅ ͍ͯ͠Δʣ ʮॏཁޠΛͲͷΑ͏ʹબͿ͔ʯ͕ྨࣅจॻݕࡧͷϙΠϯτ ౷ܭ৘ใ͔ΒλʔϜͷॏཁ౓ʢείΞʣΛܭࢉͯ͠ɺ্Ґ Top-K ͷλʔϜΛબ୒͢Δ ʢλʔϜͷείΞؔ਺ͷྫɿ TF-IDF, IDF, RIDF, Gain ͳͲʣ
  12. ྨࣅจॻݕࡧ - ΫΤϦ࡞੒ˍݕࡧ - நग़ͨ͠ॏཁޠΛ OR ΫΤϦʹઃఆͯ͠ݕࡧ͢Δ • ॏཁޠͷ͏ͪɺͲΕؚ͔͕·ΕΕ͹Α͍ •

    ॏཁޠΛͨ͘͞ΜؚΉจॻͷείΞ͕ߴ͘ͳΔ
  13. ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ

    OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
  14. ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ

    OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ More Like This Query API ΋ಉ͡ྲྀΕ
  15. ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ

    OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ More Like This Query API ΋ಉ͡ྲྀΕ
  16. More Like This Query API ͷΞϧΰϦζϜ
 - λʔϜͷ౷ܭ৘ใͷऔಘ - IndexReader

    Ϋϥε Λ࢖͏ • Lucene ͷసஔΠϯσοΫεΛಡΉͨΊͷΫϥε
 ʢ௨ৗͷݕࡧͰ΋࢖ΘΕΔʣ • λʔϜͷ౷ܭ৘ใ΋औಘͰ͖Δ • IndexReader#getTermVectors() ʹจॻ ID Λ༩͑ͯ
 term vector Λऔಘ
  17. More Like This Query API ͷΞϧΰϦζϜ - ॲཧͷྲྀΕ - 1.

    จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
  18. More Like This Query API ͷΞϧΰϦζϜ
 - λʔϜͷείΞܭࢉ - More

    Like This ͸ɺλʔϜͷείΞؔ਺ʹ TF-IDF Λ࢖͍ͬͯΔ 1. term vector ͔ΒɺTF ΛΧ΢ϯτ 2. IDF ͷܭࢉʹ͸TFIDFSimilarity Ϋϥε Λ࢖͏ 3. TF * IDF ΛɺͦͷλʔϜͷείΞͱ͢Δ TF : λʔϜස౓ fi,j → จॻதʹԿճ΋ग़ͯ͘ΔλʔϜ΄Ͳߴ͘ͳΔ IDF : จॻස౓ ni ͷٯ਺ʢN ͸จॻ਺ʣ→ ϨΞͳλʔϜ΄Ͳߴ͘ͳΔ fi,j : ΤϯτϦʔ j ʹݱΕΔλʔϜͷग़ݱճ਺ʢස౓ʣ = term_freq N : ΠϯσοΫεʹೖ͍ͬͯΔΤϯτϦʔͷ૯਺ = doc_count ni : ΠϯσοΫεશମͰλʔϜ͕ݱΕΔΤϯτϦʔͷ਺ = doc_freq
  19. More Like This Query API ͷΞϧΰϦζϜ - ॲཧͷྲྀΕ - 1.

    จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
  20. More Like This Query API ͷΞϧΰϦζϜ
 - λʔϜΛείΞॱͰฒͼସ͑ - λʔϜͷ਺͸͔ͳΓଟ͍

    ͢΂ͯͷλʔϜΛϝϞϦʹ͓͍ͯιʔτ͢Δͷ͸
 ۭؒతʹ΋࣌ؒతʹ΋ܭࢉίετ͕ߴ͍ Top-K ͕ཉ͍͠৔߹ ্Ґ K ݸ͚ͩΛอ͓͍࣋ͯͯ͠
 ͋ͱ͸ࣺͯͳ͕Βιʔτ
  21. ༏ઌ౓͖ͭΩϡʔ (PriorityQueue) ߴ଎ʹ Top-K ΛٻΊΒΕΔσʔλߏ଄ • Java Ͱ͸ඪ४ϥΠϒϥϦʹೖ͍ͬͯΔ • όΠφϦώʔϓʢ̎෼ώʔϓʣͰ࣮૷͢Δ͜ͱ͕ଟ͍

    • Elasticsearch Ͱ͸ɺ Lucene ͷ util ύοέʔδʹ͋Δ
 org.apache.lucene.util.PriorityQueue ΫϥεΛ࢖͍ͬͯΔ https://github.com/apache/lucene-solr/blob/trunk/lucene/core/src/java/org/apache/lucene/util/PriorityQueue.java
  22. More Like This Query API ͷΞϧΰϦζϜ - ॲཧͷྲྀΕ - 1.

    จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ লུ
  23. More Like This Query API ͷΞϧΰϦζϜ
 - ·ͱΊ - 1.

    จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ (retrieveTerms()) IndexReader ͷϝιου getTermVectors(docNum) Ͱ term vectors ʢλʔϜͷ౷ܭ৘ใʣΛಡΈग़͠ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͰείΞϦϯάʣ 1. term vector ͔Β TF ΛΧ΢ϯτ 2. TFIDFSimilarity Ϋϥεʹ DF ͱ numDocs(શυΩϡϝϯτ਺) Λ౉ͯ͠ IDF Λܭࢉ 3. TF * IDF Λܭࢉʢ͜Ε͕λʔϜͷείΞʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ PriorityQueue ΫϥεͰ Top-K ͷλʔϜʢॏཁޠʣΛબ୒ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ XMoreLikeThis Ϋϥε
 ʹ࣮૷͞Ε͍ͯΔ
  24. Term Vectors API Λ࢖ͬͨ
 ྨࣅจॻݕࡧ

  25. ྨࣅจॻݕࡧͷΞϧΰϦζϜʢ࠶ܝʣ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ

    OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢
  26. Term Vectors API Λ࢖ͬͨྨࣅจॻݕࡧ λʔϜͷ֤छ౷ܭ৘ใ
 ΛऔಘͰ͖Δ ͜ΕΒͷ౷ܭ৘ใ͔Β
 λʔϜͷείΞΛܭࢉ͠
 ಛ௃ޠΛબ୒͢Δ

  27. Term Vectors API Λ࢖ͬͨ
 ྨࣅจॻݕࡧͷΞϧΰϦζϜ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ • Term

    Vectors API 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ • ౷ܭ৘ใ͔ΒಠࣗͷείΞؔ਺Ͱܭࢉ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ • ॻ͖͍ͨݴޠͰϥΠϒϥϦ͕ͳ͔ͬͨΒࣗલͰ࣮૷ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ • Boolean Query ͷ should અʹλʔϜʢॏཁޠʣΛฒ΂Δ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ • Boolean Query Λ Elasticsearch ʹ౤͛Δ
  28. ·ͱΊ 2 छྨͷྨࣅจॻݕࡧͷ΍ΓํΛ঺հ 1. More Like This Query API ʹΑΔྨࣅจॻݕࡧ

    ࠷ॳ͔Β૊Έࠐ·Ε͓ͯΓ API ͚ͩͰ؆୯ʹ࢖͑Δ ΫΤϦ֦ுͷܭࢉ͕ Elasticsearch ಺Ͱ׬݁ʢແବͳ IO ͕ൃੜ͠ͳ͍ʣ νϡʔχϯά͕͍͔ͭ͘ͷύϥϝʔλͷௐ੔ʹݶΒΕΔ 2. Term Vectors API ʹΑΔྨࣅจॻݕࡧ ΑΓॊೈͳΫΤϦ֦ு͕ߦ͑Δʢࣗવݴޠॲཧ΍ػցֶशͱͷ૊Έ߹Θ͕ͤ༰қʣ Term Vector API Ͱฦͬͯ͘ΔϨεϙϯεαΠζ͕େ͖͍ʢωοτϫʔΫෛՙ͕ߴ͍ʣ ΞϓϦέʔγϣϯαʔόͰେྔͷλʔϜΛॲཧ͢Δඞཁ͕༗ΔʢCPUෛՙ͕ߴ͍ʣ
  29. We are hiring ! ৘ใݕࡧɾࣗવݴޠॲཧɾػցֶशʹڵຯͷ͋Δ͔ͨ
 ϩάղੳج൫ߏஙɾϩά෼ੳʹڵຯͷ͋Δ͔ͨ ͝Ԡื͓଴͍ͪͯ͠·͢ʂ