Elasticsearch での類似文書検索と More Like This Query API 詳解

Slide 1

Slide 1 text

Elasticsearch Ͱͷྨࣅจॻݕࡧͱ More Like This Query API ৄղ Elasticsearch ษڧձ in ژ౎ @ ͸ͯͳژ౎ΦϑΟε ઙ໺ ୎໵ id:takuya-a  @takuya_a

Slide 2

Slide 2 text

id:takuya-a ϓϥοτϑΥʔϜˍΞυςΫνʔϜ • 2015 ೥ 4 ݄ʹೖࣾ • ͸ͯͳϒοΫϚʔΫશจݕࡧͷվળΛ୲౰ ڵຯ • ৘ใݕࡧ • ࣗવݴޠॲཧ • ػցֶश OSS ׆ಈ • kuromoji.js ͳͲͷ JavaScript ϥΠϒϥϦΛ։ൃ

Slide 3

Slide 3 text

͸ͯͳϒοΫϚʔΫ  શจݕࡧͷਫ਼౓վળ ͸ͯϒશจݕࡧͷ͘͠Έʹ͍ͭͯ Hatena Engineer Seminar #5 Ͱൃද͠·ͨ͠ https://speakerdeck.com/takuyaa/hatenabutukumakuquan-wen-jian-suo-falsejing-du-gai-shan

Slide 4

Slide 4 text

ྨࣅจॻݕࡧ (More Like This) ͱ͸ ೖྗɿจॻ ग़ྗɿจॻू߹  ʢྨࣅจॻʣ ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ ʮElasticsearch CheatSheetʯ ʮ[຋༁] From Solr to elasticsearchʯ ʮelasticsearchษڧձ | Doorkeeperʯ

Slide 5

Slide 5 text

ຊ೔ͷτʔΫͷ֓ཁ 1. More Like This Query API ʹΑΔྨࣅจॻݕࡧ • Elasticsearch ʹඪ४ͰඋΘ͍ͬͯΔྨࣅจॻݕࡧ API • More Like This Query API ʹ͍ͭͯৄࡉͳΞϧΰϦζϜΛ঺հ  ʢElasticsearch ͕ྨࣅจॻΛͲ͏΍ͬͯܭࢉ͍ͯ͠Δ͔ʣ 2. Term Vectors API ʹΑΔྨࣅจॻݕࡧ • υΩϡϝϯτʹೖ͍ͬͯΔޠͷ౷ܭ৘ใΛऔಘ͢Δ API • ΑΓॊೈʹΧελϚΠζ͕Մೳ  ʢޠͷ౷ܭ৘ใ͔Βࣗ෼Ͱܭࢉ͢Δʣ

Slide 6

Slide 6 text

ྨࣅจॻݕࡧͱ  More Like This Query API

Slide 7

Slide 7 text

More Like This Query API https://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-mlt-query.html ೖྗɿจॻ ग़ྗɿจॻू߹  ʢྨࣅจॻʣ ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ ʮElasticsearch CheatSheetʯ ʮ[຋༁] From Solr to elasticsearchʯ ʮelasticsearchษڧձ | Doorkeeperʯ ͜ΕΛ࣮ݱͰ͖Δ API

Slide 8

Slide 8 text

More Like This Query API https://www.elastic.co/guide/en/elasticsearch/reference/1.x/query-dsl-mlt-query.html ϑΟʔϧυΛࢦఆ จॻ ID Λࢦఆ ύϥϝʔλΛࢦఆ ໊લ͕ࣅ͍ͯΔͷͰฆΒΘ͍͕͠ Search API ͷ More Like This API ͸ 1.6.0 Ͱ Deprecated ɺ 2.0 Ͱഇࢭ༧ఆ https://www.elastic.co/guide/en/elasticsearch/reference/current/search-more-like-this.html

Slide 9

Slide 9 text

ྨࣅจॻݕࡧͷཧ࿦తഎܠ  - Bag of words Ϟσϧ - • จॻʢυΩϡϝϯτʣΛ୯ޠͷू߹ (Set) ͩͱΈͳ͢  (୯ޠͷॱং͸ؾʹ͠ͳ͍) • Bag of words (BOW) ʹΑΔϕΫτϧۭؒϞσϧΛ  جຊతͳߟ͑ํͱ͢Δ ಉ͡ޠΛଟؚ͘Ήจॻ = ྨࣅจॻͱఆٛ

Slide 10

Slide 10 text

Bag of Words ϞσϧʹΑΔ  ྨࣅจॻͷྫ ʮElasticsearch 1.6.0ϦϦʔεʯ ʮElasticsearch 2.0.0.beta1ϦϦʔεؒۙʯ Elasticsearch Elasticsearch ϦϦʔε ϦϦʔε Lucene Lucene synced-ﬂush Ϋϥελ Ϋϥελ … … Pipeline ಉ͡ޠ͕ͲΕ͚ͩೖͬͯΔ͔ʁ Λܭࢉ

Slide 11

Slide 11 text

ྨࣅจॻݕࡧ  - จॻͷಛ௃நग़ - 1ͭͷจॻͷதʹ͸ɺͨ͘͞ΜͷޠʢλʔϜʣؚ͕·Ε͍ͯΔ • ͢΂ͯͷλʔϜɾ͢΂ͯͷจॻʹରͯ͠ɺྨࣅ౓ͷܭࢉΛ͢Δͷ͸ݱ࣮తͰ͸ͳ͍ • ͍͔ͭ͘ͷॏཁޠΛબ୒͠ɺͦͷจॻͷಛ௃ྔͱ͢Δ  ʢBag of Words Ͱͷྨࣅ౓ܭࢉΛ ॏཁͳλʔϜʹ͍͚ͭͯͩܭࢉ͢Δ͜ͱͰۙࣅ ͍ͯ͠Δʣ ʮॏཁޠΛͲͷΑ͏ʹબͿ͔ʯ͕ྨࣅจॻݕࡧͷϙΠϯτ ౷ܭ৘ใ͔ΒλʔϜͷॏཁ౓ʢείΞʣΛܭࢉͯ͠ɺ্Ґ Top-K ͷλʔϜΛબ୒͢Δ ʢλʔϜͷείΞؔ਺ͷྫɿ TF-IDF, IDF, RIDF, Gain ͳͲʣ

Slide 12

Slide 12 text

ྨࣅจॻݕࡧ - ΫΤϦ࡞੒ˍݕࡧ - நग़ͨ͠ॏཁޠΛ OR ΫΤϦʹઃఆͯ͠ݕࡧ͢Δ • ॏཁޠͷ͏ͪɺͲΕؚ͔͕·ΕΕ͹Α͍ • ॏཁޠΛͨ͘͞ΜؚΉจॻͷείΞ͕ߴ͘ͳΔ

Slide 13

Slide 13 text

ྨࣅจॻݕࡧͷΞϧΰϦζϜ·ͱΊ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

More Like This Query API ͷΞϧΰϦζϜ  - λʔϜͷ౷ܭ৘ใͷऔಘ - IndexReader Ϋϥε Λ࢖͏ • Lucene ͷసஔΠϯσοΫεΛಡΉͨΊͷΫϥε  ʢ௨ৗͷݕࡧͰ΋࢖ΘΕΔʣ • λʔϜͷ౷ܭ৘ใ΋औಘͰ͖Δ • IndexReader#getTermVectors() ʹจॻ ID Λ༩͑ͯ  term vector Λऔಘ

Slide 17

Slide 17 text

More Like This Query API ͷΞϧΰϦζϜ - ॲཧͷྲྀΕ - 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢

Slide 18

Slide 18 text

More Like This Query API ͷΞϧΰϦζϜ  - λʔϜͷείΞܭࢉ - More Like This ͸ɺλʔϜͷείΞؔ਺ʹ TF-IDF Λ࢖͍ͬͯΔ 1. term vector ͔ΒɺTF ΛΧ΢ϯτ 2. IDF ͷܭࢉʹ͸TFIDFSimilarity Ϋϥε Λ࢖͏ 3. TF * IDF ΛɺͦͷλʔϜͷείΞͱ͢Δ TF : λʔϜස౓ fi,j → จॻதʹԿճ΋ग़ͯ͘ΔλʔϜ΄Ͳߴ͘ͳΔ IDF : จॻස౓ ni ͷٯ਺ʢN ͸จॻ਺ʣ→ ϨΞͳλʔϜ΄Ͳߴ͘ͳΔ fi,j : ΤϯτϦʔ j ʹݱΕΔλʔϜͷग़ݱճ਺ʢස౓ʣ = term_freq N : ΠϯσοΫεʹೖ͍ͬͯΔΤϯτϦʔͷ૯਺ = doc_count ni : ΠϯσοΫεશମͰλʔϜ͕ݱΕΔΤϯτϦʔͷ਺ = doc_freq

Slide 19

Slide 19 text

Slide 20

Slide 20 text

More Like This Query API ͷΞϧΰϦζϜ  - λʔϜΛείΞॱͰฒͼସ͑ - λʔϜͷ਺͸͔ͳΓଟ͍ ͢΂ͯͷλʔϜΛϝϞϦʹ͓͍ͯιʔτ͢Δͷ͸  ۭؒతʹ΋࣌ؒతʹ΋ܭࢉίετ͕ߴ͍ Top-K ͕ཉ͍͠৔߹ ্Ґ K ݸ͚ͩΛอ͓͍࣋ͯͯ͠  ͋ͱ͸ࣺͯͳ͕Βιʔτ

Slide 21

Slide 21 text

༏ઌ౓͖ͭΩϡʔ (PriorityQueue) ߴ଎ʹ Top-K ΛٻΊΒΕΔσʔλߏ଄ • Java Ͱ͸ඪ४ϥΠϒϥϦʹೖ͍ͬͯΔ • όΠφϦώʔϓʢ̎෼ώʔϓʣͰ࣮૷͢Δ͜ͱ͕ଟ͍ • Elasticsearch Ͱ͸ɺ Lucene ͷ util ύοέʔδʹ͋Δ  org.apache.lucene.util.PriorityQueue ΫϥεΛ࢖͍ͬͯΔ https://github.com/apache/lucene-solr/blob/trunk/lucene/core/src/java/org/apache/lucene/util/PriorityQueue.java

Slide 22

Slide 22 text

Slide 23

Slide 23 text

More Like This Query API ͷΞϧΰϦζϜ  - ·ͱΊ - 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ (retrieveTerms()) IndexReader ͷϝιου getTermVectors(docNum) Ͱ term vectors ʢλʔϜͷ౷ܭ৘ใʣΛಡΈग़͠ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͰείΞϦϯάʣ 1. term vector ͔Β TF ΛΧ΢ϯτ 2. TFIDFSimilarity Ϋϥεʹ DF ͱ numDocs(શυΩϡϝϯτ਺) Λ౉ͯ͠ IDF Λܭࢉ 3. TF * IDF Λܭࢉʢ͜Ε͕λʔϜͷείΞʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ PriorityQueue ΫϥεͰ Top-K ͷλʔϜʢॏཁޠʣΛબ୒ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ XMoreLikeThis Ϋϥε  ʹ࣮૷͞Ε͍ͯΔ

Slide 24

Slide 24 text

Term Vectors API Λ࢖ͬͨ  ྨࣅจॻݕࡧ

Slide 25

Slide 25 text

ྨࣅจॻݕࡧͷΞϧΰϦζϜʢ࠶ܝʣ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢

Slide 26

Slide 26 text

Term Vectors API Λ࢖ͬͨྨࣅจॻݕࡧ λʔϜͷ֤छ౷ܭ৘ใ  ΛऔಘͰ͖Δ ͜ΕΒͷ౷ܭ৘ใ͔Β  λʔϜͷείΞΛܭࢉ͠  ಛ௃ޠΛબ୒͢Δ

Slide 27

Slide 27 text

Term Vectors API Λ࢖ͬͨ  ྨࣅจॻݕࡧͷΞϧΰϦζϜ 1. จॻʹؚ·ΕΔλʔϜͷ౷ܭ৘ใʢTF, DFͳͲʣΛऔಘ • Term Vectors API 2. ౷ܭ৘ใ͔Β֤λʔϜͷείΞΛܭࢉʢTF-IDFͳͲͰείΞϦϯάʣ • ౷ܭ৘ใ͔ΒಠࣗͷείΞؔ਺Ͱܭࢉ 3. λʔϜΛείΞॱʹฒͼସ্͑ͯҐͷλʔϜʢ=ॏཁޠʣΛநग़ • ॻ͖͍ͨݴޠͰϥΠϒϥϦ͕ͳ͔ͬͨΒࣗલͰ࣮૷ 4. ॏཁޠΛ࢖ͬͯ OR ΫΤϦΛ࡞੒ • Boolean Query ͷ should અʹλʔϜʢॏཁޠʣΛฒ΂Δ 5. ࡞੒ͨ͠ΫΤϦͰݕࡧͯ͠ɺ্ҐͷจॻΛྨࣅจॻͱͯ͠ฦ͢ • Boolean Query Λ Elasticsearch ʹ౤͛Δ

Slide 28

Slide 28 text

·ͱΊ 2 छྨͷྨࣅจॻݕࡧͷ΍ΓํΛ঺հ 1. More Like This Query API ʹΑΔྨࣅจॻݕࡧ ࠷ॳ͔Β૊Έࠐ·Ε͓ͯΓ API ͚ͩͰ؆୯ʹ࢖͑Δ ΫΤϦ֦ுͷܭࢉ͕ Elasticsearch ಺Ͱ׬݁ʢແବͳ IO ͕ൃੜ͠ͳ͍ʣ νϡʔχϯά͕͍͔ͭ͘ͷύϥϝʔλͷௐ੔ʹݶΒΕΔ 2. Term Vectors API ʹΑΔྨࣅจॻݕࡧ ΑΓॊೈͳΫΤϦ֦ு͕ߦ͑Δʢࣗવݴޠॲཧ΍ػցֶशͱͷ૊Έ߹Θ͕ͤ༰қʣ Term Vector API Ͱฦͬͯ͘ΔϨεϙϯεαΠζ͕େ͖͍ʢωοτϫʔΫෛՙ͕ߴ͍ʣ ΞϓϦέʔγϣϯαʔόͰେྔͷλʔϜΛॲཧ͢Δඞཁ͕༗ΔʢCPUෛՙ͕ߴ͍ʣ

Slide 29

Slide 29 text

We are hiring ! ৘ใݕࡧɾࣗવݴޠॲཧɾػցֶशʹڵຯͷ͋Δ͔ͨ  ϩάղੳج൫ߏஙɾϩά෼ੳʹڵຯͷ͋Δ͔ͨ ͝Ԡื͓଴͍ͪͯ͠·͢ʂ