Elasticsearchで多言語検索対応してみた話.pdf

1. ElasticsearchͰଟݴޠݕࡧର Ԡͯ͠Έͨ࿩
2. ࣗݾ঺հ ɹɾ໊લ ɹɹࠤ౻ ݩل ɹɾϝυϐΞྺ ɹɹ 2017/9 ʙ Webͷιϑτ΢ΤΞΤϯδχΞɹ ɹ
3. ͜Ε͔Β͓࿩͢͠Δ͜ͱ ɾӳޠυΩϡϝϯτΛ೔ຊޠͰݕࡧ͢ΔΑ͏ͳɺҟͳΔݴޠ ؒͷݕࡧ ɾ຋༁Λߦ͏ࣄ͕Ͱ͖ͳ͍৔߹ͷରԠ ɾElasticsearchͰߦͬͨࡍͷํ๏ͷ1ͭͱɺ
 ɹϝϦοτ/σϝϦοτ
4. ҩࢣઐ༻αΠτʮMedPeerʯ
5. 㲔 ৽αʔϏεͷ։ൃ
6. ʮPubmed࿦จΛ೔ຊޠͰޮ཰Α͘ݕࡧʯ
7. PubMed࿦จʁ
 ɹւ֎ҩֶจݙ৘ใͷσʔλϕʔε ɹɾӳޠ ʢ͘͝·Εʹผͷݴޠ΋ʣ ɾAPIɺFTPͰͷϑΝΠϧऔಘʹରԠ͍ͯ͠Δ
8. ɾӳจυΩϡϝϯτΛ೔ຊޠͰݕࡧ ͍ͨ͠ ɾݕࡧର৅͸λΠτϧɺຊจ ࣮ݱ͢Δ͜ͱ
9. ೔ຊޠʹ຋༁͓͚ͯ͠͹ ໰୊ͳ͍͸ͣ
10. ຋༁Λߦ͏ͨΊͷAPI ɾGoogle Translation API 100 ສจࣈ - 20υϧ ɹˠ Pubmed຋༁ʹෆࣗવͳ఺͕গͳ͍ ɾMicrosoft Translator API 100 ສจࣈ - 10υϧʢ1120ԁ) ɹˠ ྉۚ͸͍͕҆ɺPubmed຋༁͢Δͱෆࣗવͳ఺͕ΘΓͱ͋Δ ɾAmazon Translate ɹ೔ຊޠະରԠ(2017/ळࠒ)

11. ຋༁඼࣭Λ༏ઌͯ͠ɺPubmed຋༁ʹෆ ࣗવͳ఺͕গͳ͔ͬͨͷͰ Google Translation API ʹܾఆɻ
12. ຋༁ྉۚ
13. ɾPubMed࿦จ ɹ໿1700ສ݅ʢMedPeerͰͷऔΓࠐΈର৅ɻ೔ʑ૿Ճʣ ɾฏۉจࣈ਺ ɹ1300จࣈ (λΠτϧ100จࣈɺຊจ1200จࣈ) ɹɹɹɹɹˣɹɹɹɹ ɾ߹ܭจࣈ਺ ɹ໿221ԯจࣈ (100 + 1200) * 1700ສ݅
14. Google Translation API͸100 ສจࣈ - 20υϧ ɾֹۚ (221ԯ / 100ສจࣈ) * 20υϧ = 442000υϧ ɹ ೔ຊԁ = ໿4889ສԁ ɹ (2017/07/07࣌఺)
15. 4889ສԁ ߴ͍ʢฐࣾج४ʣ
16. 4889ສԁ͸ߴ͍ ɾશͯ຋༁͢Δͱߴ͗͢Δ ɾͱ͸͍͑ɺ݅਺͸ݮΒͨ͘͠ͳ͍ → ʮӳจυΩϡϝϯτΛ೔ຊޠͰݕࡧʯΛͲ ͏͢Δ͔
17. 1.ݩυΩϡϝϯτΛ຋༁͍ͯ͠ͳͯ͘΋ ݕࡧՄೳʹ → ೔ຊޠݕࡧʹࣙॻΛ࢖͏ɻ ɹElasticsearchͷʮSynonym Token Filterʯ
18. Elasticsearch Synonym Token Filter https://www.elastic.co/guide/en/elasticsearch/reference/current/ analysis-synonym-tokenfilter.html
19. ಉҙޠ΍ྨٛޠΛઃఆͰ͖Δػೳɻ ྫʣ ͱ͍͏ఆ͕ٛ͋Ε͹ɺ ʮi-podʯͰݕࡧ →ʮi podʯʯʮipodʯʹώοτ ʮi podʯͰݕࡧ →ʮi-podʯʯʮipodʯʹώοτ Synonym Token Filter i-pod, i pod => ipod
20. ߴ݂ѹ => hypertension ΠϯϑϧΤϯβ => influenza ͜ΕΛ࢖͍ɺ ೔ຊޠ/ӳޠΛؔ࿈͚ͮΔ
21. pubmed: { properties: { title_en: { type: "text", analyzer: “english_analyzer" }, title_ja: { type: "text", analyzer: "ja_analyzer" }, body_en: { type: “text”, analyzer: "english_analyzer" }, body_ja: { type: "text", analyzer: "ja_analyzer" }, }, }, Indexͷproperties(Ϛοϐϯά) ɹɾ೔ຊޠϑΟʔϧυ(title_ja/body_ja)ɺ
 ɹɹӳޠϑΟʔϧυ(title_en/body_en)ΛλΠτϧ/ຊจ
 ɹɹͦΕͧΕͰ༻ҙ ɾӳޠϑΟʔϧυɺ೔ຊޠϑΟʔϧυͰanalyzerΛ෼͚Δ Elasticsearch༻ͷઃఆ
22. Indexͷanalysisઃఆ ɹɾfilterʹtype:”synonym”Ͱઃఆ௥Ճ ɹɾӳޠϑΟʔϧυ༻ͷʮenglish_analyzerʯͷfillterʹɺ ɹɹsynonym filterΛ࢖͏Α͏ઃఆ(ଞެࣜͷEnglishઃఆΛϕʔεʹ) Elasticsearch༻ͷઃఆ { “index” : { “analysis“: { “filter“ : { “synonym“ : { “type“ : "synonym", “synonyms“ : [‘ߴ݂ѹ => hypertension’, …]}, “analyzer“: { “english_analyzer“: { “tokenizer”: "standard", “filter”: [“synoncym”,”english_possessive_stemmer”,”lowercase “,…] }, …}, }, }, }
23. Analyze݁Ռ (kibana) ʮߴ݂ѹʯˠʮhypertensionʯͷtokenͱͳΔ
 (࣮ࡍʹ͸ɺઌఔͷαϯϓϧͷଞͷfilterʹΑΓՃ޻͞Εͨtokenʣ
24. Ωʔϫʔυʮߴ݂ѹʯͰ೔ຊޠ/ӳޠݕࡧ͕Մೳʹ ɾ೔ຊޠͷʮߴ݂ѹʯ ɾӳจͷʮhypertensionʯ
25. ࣙॻ͸Ͳ͜ͰखʹೖΕΔ͔ 
 ༗໊ͳࣙॻ ɾJMdict (Japanese-Multilingual Dictionary)
 http://www.edrdg.org/jmdict/j_jmdict.html ɹӳޠҎ֎΋ೖ͍ͬͯΔΑ͏ͳͷͰɺ
 ɹෳ਺ݴޠͷઃఆ΋Ͱ͖Δ͔΋͠Ε·ͤΜɻ
26. ͲΜͳࣙॻͰ΋ྑ͍ͷ͔ ɾઐ໳༻ޠ͸ݫ͍͠
 ɹઐ໳༻ޠ(ҩྍ౳)͸ઐ໳ͷࣙॻͰͳ͍ͱ୯ޠ͕໢ཏͰ͖ͳ͍ࣄ͕ ଟ͍ɻ ɹˠ ઐ໳ྖҬͷࣙॻΛ୳͢ɺݕࡧϩά͔ΒࣙॻΛΞοϓσʔτ͢Δ ͳͲ ϝυϐΞ͸גࣜձࣾϩθολ༷ͱܖ໿͠ɺҩྍʹಛԽͨࣙ͠ॻΛఏ ڙͯ͠௖͖·ͨ͠ɻ
27. ࣙॻʹΑΔଟݴޠͷݕࡧ ϝϦοτ ɹɾશ຋༁͠ͳͯ͘΋ݕࡧͰ͖ΔΑ͏ʹͳΔ σϝϦοτ ɹɾࣙॻͷ༻ҙ͕ඞཁʹͳΔ ɹɾݕࡧਫ਼౓͕ࣙॻͷ඼࣭ʹࠨӈ͞Ε΍͍͢
 ɹɹˠ จষ಺ͷ୯ޠΛ໢ཏͰ͖ͳ͍Մೳੑ͸େ͖͍ ɾ୯ޠϕʔεͷݕࡧʹ͸ڧ͍͕ɺจͷݕࡧʹ͸ऑ͍ ɾݕࡧ͸Ͱ͖Δ͕ɺ݁Ռදࣔ͸ӳޠ
28. Ϣʔβʔ ΠϯλʔϑΣʔε͍ͩ͠Ͱ͕͢ɺ
 ɾλΠτϧ͸ࣄલ ɹ຋༁APIͷ଴͕ͪ࣌ؒͭΒ͍ʢҰཡͰͬ͞ͱݟΔఔ౓ͳͷͰʣ ɹຊจΑΓจࣈ਺͸গͳ͍ͷͰɺશ຋༁ͯ͠΋ֹۚ΋௿Ί 2. Ұ෦͸ࣄલ຋༁ɺ࢒Γ͸දࣔ࣌ʹ
29. ɾຊจ͸ৄࡉදࣔ࣌ ɹදࣔϖʔδ಺ͰඇಉظͰॲཧ͞Εͨ຋༁݁ՌΛදࣔɹ 2. Ұ෦͸ࣄલ຋༁ɺ࢒Γ͸දࣔ࣌ʹ
30. ྉۚ
 ɾ຋༁ྉ(ݻఆ) ɹ ࣄલʹλΠτϧ͚ͩ຋༁͢Δࣄʹͨ͠෼
 ɹ 1700ສ݅ * λΠτϧͷΈ(100จࣈ) = 376ສԁɹ(Google Translate API)
 ɾ຋༁ྉ(มಈ) ɹදࣔ࣌ʹ຋༁͢Δࣄʹͨ͠ = αສԁʢPV౳ඇެද਺஋ͷؔ܎Ͱग़ ͤͳ͍Ͱ͕͢ɺݕࡧ͞Εදࣔ͞ΕΔϖʔδ͸Ұ෦ͳͷͰ͔ͳΓ҆͘ʣ 
 ɾࣙॻ୅
 ɹ גࣜձࣾϩθολ༷ (ҩྍܥಛԽͷ࿨ӳࣙॻ)ɹ= bສԁʢ͜Ε΋ެ ද☓ͳͷͰ͕͢ɺۃΊ͓ͯ҆͘ఏڙ௖͍͍ͯ·͢ʣ
31. ࠷ऴతͳֹۚ 4889ສԁ → 376+α+b ສԁ େৎ෉ʢฐࣾج४ʣ
32. ิ଍ 1700ສ݅Ҏ߱ʢPubMedͷίϯςϯπ͸೔ʑ૿Ճʣ
 ͷ຋༁͸ɺΑΓҩྍಛԽͷ຋༁Λߦ͏ͨΊGoogle຋༁͚ͩͰ ͳ͘ɺԼهͷύʔτφʔ༷ͱܖ໿Λߦ͍APIΛར༻͍ͯ͠· ͢ɻ ɾגࣜձࣾγΣΞϝσΟΧϧ༷ ɹҩྍܥಛԽͷ຋༁API
 ɹhttps://www.ikotoba.jp/
 ɾגࣜձࣾϩθολ༷ PubMedಛԽͷ຋༁API
 ɹhttps://www.rozetta.jp/
33. ·ͱΊ - ݴޠ͕ҟͳΔυΩϡϝϯτݕࡧʹɺ ɹ ElasticsearchͷSynonym Token FilterΛ࢖ͬͨ 
 - ຋༁ྉ͸ݮΒͤΔ͕ɺϝϦοτ/σϝϦοτ͋Γ
 - ୯ޠͰͷݕࡧΛओͱ͢Δ৔߹ʹ͸ద༻͠΍͍͢
34. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠