Tweet
Tweet

Slide 1

Slide 1 text

ElasticsearchʹΑΔ શจݕࡧͷ࣮૷ in Rails #railsdm by FUJI Goro (gfx)

Slide 2

Slide 2 text

ࠓ೔࿩͢͜ͱ • ΢ΣϒαʔϏεͰશจݕࡧΛఏڙ͢Δͱ͖ʹ ͱ͖ʹߟ͑Δ͜ͱ • ElasticsearchʹΑΔશจݕࡧʹ͓͚Δ඼࣭ͷ ੍ޚʹ͍ͭͯ

Slide 3

Slide 3 text

ࠓ೔࿩͞ͳ͍͜ͱ • Elasticsearch vs ଞͷશจݕࡧΤϯδϯ • ϚωʔδυαʔϏε vs ࣗલ • ಛఆͷ elasticsearch client library ͷ࿩ • e.g. elasticsearch-rails, searchkick, elastomer-client

Slide 4

Slide 4 text

ൃදऀʹ͍ͭͯ • ໊લ: gfx / ౻ޗ࿠ • ৘ใڞ༗αʔϏεKibelaΛ։ൃ͍ͯ͠Δ • Kibela͸ Ruby on Rails੡αʔϏε • લճͷrailsdmͰ͸ʮϚϧνςφϯτɾ΢ΣϒΞϓϦέʔ γϣϯͷ࣮ફʯͱ͍͏ൃදΛͨ͠ • KibelaͰ͸ Amazon Elasticsearch Service Λར༻

Slide 5

Slide 5 text

΢ΣϒαʔϏεͰશจݕࡧΛ ఏڙ͢Δͱ͖ʹߟ͑Δ͜ͱ

Slide 6

Slide 6 text

શจݕࡧ in ΢ΣϒαʔϏε • ඼࣭Λ্͛Δʹ͸ఔ౓ίετֻ͕͔Δ • ≒ίετΛֻ͚Δ΄Ͳʹ඼࣭Λ্͛ΒΕΔ • Ͳͷ͘Β͍ίετΛֻ͚ΒΕΔ͔Ͱ΍Δ͜ͱ͕ ͔ͳΓมΘΔ • ίετΛֻ͚ΒΕͳ͍ͳΒͦ΋ͦ΋શจݕࡧ͸ ෆཁ͔΋͠Εͳ͍

Slide 7

Slide 7 text

ݕࡧͷ඼࣭ • ݕࡧͷ඼࣭͸Ұൠʹʮద߹཰ʯͱʮ࠶ݱ཰ʯ ͰݟΔ • ద߹཰͸ʮਖ਼֬ੑʯͱ΋͍͏ • ࠶ݱ཰͸ʮ໢ཏੑʯͱ΋͍͏

Slide 8

Slide 8 text

ద߹཰ vs ࠶ݱ཰ • ద߹཰:ʮݕࡧ͞Εͨద߹จॻͷ਺ ÷ ݕࡧ݁Ռͷจॻͷ ਺ʯ • ద߹཰͕ߴ͍ͱʮݕࡧ݁Ռͷଟ͕͘ద߹ੑ͕ߴ͍ʯ • ࠶ݱ཰:ʮݕࡧ͞Εͨద߹จॻͷ਺ ÷ શର৅จॻதͷਖ਼ ղจॻͷ਺ʯ • ࠶ݱ཰͕ߴ͍ͱʮద߹ੑ͸ͱ΋͔͘औΓ͜΅͠͸ͳ͍ʯ

Slide 9

Slide 9 text

ద߹཰ͱ࠶ݱ཰ͷࢉग़ • ԿΛ࣋ͬͯʮద߹ʯͱ͢Δ͔͸Ϣʔβʔͷ಄ͷதʹ ͷΈ͋Δ • ≒ ಉ͡ݕࡧΩʔϫʔυͰ΋Ϣʔβʔ͝ͱʹٻΊΔ ΋ͷ͸ҟͳΔՄೳੑ͕͋Δ • ࣮ࡍͷαʔϏεͰ͸௚઀ଌఆ͢Δ͜ͱ͸ෆՄೳ • ⇢ϢʔβʔͷߦಈΛ෼ੳͯ͠ධՁ͢Δඞཁ͕͋Δ

Slide 10

Slide 10 text

ద߹཰ͱ࠶ݱ཰Ͱॏࢹ͢Δͷ͸ʁ • Ұൠʹద߹཰ͱ࠶ݱ཰͸τϨʔυΦϑͷؔ܎͕͋ Δͱ͍ΘΕΔ • ͲͪΒ͕ॏཁ͔͸αʔϏεͷੑ࣭ʹΑΔ • KibelaͷΑ͏ͳاۀ಺ͷจॻݕࡧͰ͸ʮऔΓ͜΅ ͠ ≒ όάʯͱೝࣝ͞ΕΔ • ͭ·Γ࠶ݱ཰͸ʮ࠷௿ݶकΔ΂͖඼࣭ʯͰ͋Δ

Slide 11

Slide 11 text

ElasticsearchʹΑΔશจݕࡧ ʹ͓͚ΔݕࡧείΞͷߟ͑ํ

Slide 12

Slide 12 text

Elasticsearchͱ͸ • શจݕࡧΤϯδϯ • SQLͷLIKEݕࡧͱͷҧ͍: • ܗଶૉղੳޙͷ୯ޠͰݕࡧͰ͖Δ • ਖ਼نԽ͞Εͨ୯ޠͰݕࡧͰ͖Δ • ෳࡶͳείΞʹجͮ͘ιʔτ͕Ͱ͖Δ

Slide 13

Slide 13 text

Elasicsearch༻ޠ RDBMS Table Record Column Elasticsearch Index Document Field ※ Elasticsearch v5·Ͱ͸ table = type ͱઆ໌͞Ε͖͕ͯͨ v6 Ͱtype͸ඇ঑ྭɺ v7 Ͱ࡟আ͞ΕΔͷͰ஫ҙ

Slide 14

Slide 14 text

ܗଶૉղੳ

Slide 15

Slide 15 text

ܗଶૉղੳͯ͠ݕࡧ • શจݕࡧͷߴ଎ԽͷͨΊʹసஔΠϯσΫεͳͲͷ ΞϧΰϦζϜΛ࢖͍͍ͨ • సஔΠϯσΫεͷͨΊʹೖྗ͞ΕΔจষΛద౰ͳ ୯ҐͰ෼ׂ͢Δඞཁ͕͋Δ • ೔ຊޠͷΑ͏ʹ୯ޠͷ۠੾Γ͕͸͖ͬΓ͠ͳ͍ݴ ޠͷ৔߹N Gram and/or ܗଶૉղੳͰ෼ׂ͢Δ

Slide 16

Slide 16 text

N Gram • N͸௨ৗ2͔3 • 2 gram ͷ৔߹ɺʮ͜Μʹͪ͸ʯΛ • [“͜Μ”, “Μʹ”, “ʹͪ”, “ͪ͸”] ʹ෼͚Δ

Slide 17

Slide 17 text

ܗଶૉղੳ • ʮ౦ژ౎ʹҾͬӽͨ͠ʯΛ • [“౦ژ౎”, “ʹ”, “Ҿͬӽ͠”, “ͨ”] ͳͲʹ෼ׂ ͢Δ • ඼ࢺʢ໊ࢺ, ܗ༰ࢺʣ৘ใ͕ར༻Ͱ͖ΔͷͰɺ ޙͷਖ਼نԽͰͦͷ৘ใΛར༻

Slide 18

Slide 18 text

LIKEݕࡧͱͷࠩ: ܗଶૉղੳฤ • “౦ژ౎ʹҾͬӽͨ͠” LIKE “%ژ౎%” • Ϛον͢Δ • [“౦ژ౎”, “ʹ”, “Ҿͬӽ͠”, “ͨ”].include?(“ژ ౎”) • Ϛον͠ͳ͍

Slide 19

Slide 19 text

ܗଶૉղੳͱN Gramͷࠩ • ͨͱ͑͹ʮژ౎ʯͰʮ౦ژ౎ʯΛؚΉจॻΛ… • ܗଶૉղੳϕʔεͩͱݕࡧͰ͖ͳ͍ • ͭ·Γద߹཰ͷ໘Ͱ༏Ε͍ͯΔ • N GramϕʔεͩͱݕࡧͰ͖Δ • ͭ·Γ࠶ݱ཰ͷ໘Ͱ༏Ε͍ͯΔ

Slide 20

Slide 20 text

ਖ਼نԽ

Slide 21

Slide 21 text

ͳͥਖ਼نԽ͕ඞཁ͔ • ܗଶૉղੳʴసஔΠϯσΫεʹΑΔݕࡧͷ໛ࣜ Խ: [“Ҿͬӽ͠”, “ͨ”].include?(“Ҿͬӽ͠”) • “Ҿӽ”, “Ҿӽ͠” ΋ಉ༷ʹݕࡧ͍ͨ͠ ⇢ ਖ਼نԽ

Slide 22

Slide 22 text

ܗଶૉղੳͱਖ਼نԽ • ܗଶૉղੳث͕ࣙॻͱ඼ࢺ৘ใΛ࢖ͬͯਖ਼نԽ͢Δ • Elasticsearchͷ৔߹kuromoji plugin͕೔ຊޠͷܗଶૉղੳ ͱਖ਼نԽΛαϙʔτ • kuromoji_baseform: “ҿΈ” ⇢ “ҿΉ” • kuromoji_stemmer: “αʔόʔ” ⇢ “αʔό” • kuromoji_number: “ඦԁ” ⇢ “100ԁ”

Slide 23

Slide 23 text

είΞϦϯά

Slide 24

Slide 24 text

ݕࡧॱͷॏཁੑ • ͨͱ͑͹ɿʰژ౎ʱͰʰ౦ژ౎ʱͱ͍͏จॻ ͸ώοτͯ͠΄͍͕͠ɺʰژ౎ʱͰώοτ͢ Δจॻ͕͕͋ΔͳΒͦͪΒΛ্Ґʹ͍࣋ͬͯ ͖͍ͨ • ͭ·Γߴ͍࠶ݱ཰Λҡ࣋ͭͭ͠ݕࡧ݁Ռͷ্ Ґ͸ʮΑΓద߹͢ΔͱࢥΘΕΔʯจॻʹͨ͠ ͍

Slide 25

Slide 25 text

୯ޠ෼ׂ๏ʹΑΔॏΈ෇͚ • ܗଶૉղੳͰ෼ׂ͢Δͱద߹཰͸ߴ͘ͳΓ͕ ͕ͪͩ࠶ݱ཰͸௿͘ͳΓ͕ͪ • N GramͰ෼ׂ͢Δͱ࠶ݱ཰͸ߴ͘ͳΓ͕ͪͩ ͕ద߹཰͸௿͘ͳΓ͕ͪ • ⇢είΞϦϯάͷࡍʹॏΈ෇͚Λ͢Ε͹Α͍

Slide 26

Slide 26 text

ElasticsearchͰͷ࣮૷ํ๏ • elasticsearch-model Ͱͷσʔλιʔεఆٛ UJUMF͸LVSPNPKJͰ ෼ׂ͢Δ UJUMF@OHSBN͸/ (SBNͰ෼ׂ͢Δ

Slide 27

Slide 27 text

ϑΟʔϧυ͝ͱͷॏΈ෇͚ • ElasticsearchͰ͸ϑΟʔϧυ͝ͱʹॏΈ෇͚ ΛઃఆͰ͖Δ

Slide 28

Slide 28 text

ͦͷଞͷείΞϦϯάख๏ • Phrase Match / ϑϨʔζϚον • Field Value Factor / ϑΟʔϧυ܎਺ • Decay / είΞͷݮਰ

Slide 29

Slide 29 text

Phrase Match • Twitterݕࡧ΍GoogleݕࡧͰμϒϧΫΥʔτͰғΜͩͱ͖ʹ ʮϑϨʔζͦͷ΋ͷʯ͕ݕࡧର৅ʹͳΔͷ͕Phrase Match • Kibelaͷ৔߹ɺϑϨʔζʹϚονͨ͠ΒείΞΛϒʔετ ͍ͯ͠Δ • ྫ:ʮঁਆసੜʯͰݕࡧ͢ΔͱʮਅɾঁਆసੜIII NOCTURNEʯͷ΄͏͕ʮҟੈքʹঁਆͱͯ͠సੜ͠ ͨʯΑΓ΋είΞ͕ߴ͘ͳΔ

Slide 30

Slide 30 text

Field Value Factor • ಛఆͷϑΟʔϧυͷ஋ͰείΞΛϒʔετͤ͞ Δػೳ • Kibelaͷ৔߹ɺϥΠΫ਺ͱτϥοΫόοΫ਺Λ Field Value Factorͱͯ͠൓ө͍ͤͯ͞Δ • ͭ·ΓʮΑ͘ϥΠΫ͞ΕΔจॻʯʮΑ͘Ϧ ϯΫ͞ΕΔจॻʯͷείΞ͕ߴ͍

Slide 31

Slide 31 text

Decay • Field Value FactorͷٯͰɺಛఆϑΟʔϧυ஋ ͰείΞΛݮਰͤ͞Δػೳ • KibelaͰ͸ɺจॻͷλΠϜελϯϓΛར༻͠ ͕ͯ࣌ؒܦͭ΄ͲʹείΞ͕ݮਰ͢ΔΑ͏ʹ ͍ͯ͠Δ

Slide 32

Slide 32 text

DecayͷΠϝʔδ • origin = now / timestamp͸աڈͷҰ఺ͱ͢Δ

Slide 33

Slide 33 text

·ͱΊ • શจݕࡧͷ඼࣭͸ʮద߹཰ʯʮ࠶ݱ཰ʯʮε ίΞϦϯάʯͰߟ͑Δ • ʮద߹཰ʯͱʮ࠶ݱ཰ʯ͸୯ޠ෼ׂ๏ʹΑͬ ͯվળͰ͖Δ • ʮείΞϦϯάʯ͸༷ʑͳཁૉ͕͋ΔͷͰશ จݕࡧΤϯδϯͷυΩϡϝϯτΛख़ಡ͢Δ