Tweet
Tweet

Slide 1

Slide 1 text

2020/11/11 @kotaroooo0 ԶͷશจݕࡧΤϯδϯ(Go੡) Λ࡞Γ࢝Ίͨ

Slide 2

Slide 2 text

͜ͷLTΛฉ͘ͱ… 1. ͳΜͱͳ͘શจݕࡧΤϯδϯͷ࢓૊Έ͕෼͔ Δ 2. GoͰશจݕࡧΤϯδϯΛ࡞Γ࢝ΊΒΕΔ

Slide 3

Slide 3 text

-શจݕࡧΤϯδϯͷ࢓૊ΈΛ஌Δ -ElasticsearchͰ΋࢖ΘΕ͍ͯΔApache LuceneΆ͍࢓༷ͷ΋ͷΛ࡞Δ -GoΛֶΜͰ͍ΔͷͰԿ͔࡞Γ͍ͨ -Twitter BotΛ࡞͍ͬͯΔ͕ɺͪΐ͏Ͳ͍͍શจݕࡧΤϯδϯ͕ͳ͍ -ܰྔɺ͔ͭॏΈ෇͖ϨʔϕϯγϡλΠϯڑ཭ΛܭࢉͰ͖Δ΍ͭ શจݕࡧΤϯδϯΛ࡞Δཧ༝

Slide 4

Slide 4 text

3෼Ͱ෼͔ΔશจݕࡧΤϯδϯ

Slide 5

Slide 5 text

શจݕࡧͷ࢓૊Έ INDEXING ୯ޠ จॻ have 1,2 pen 1 we 2 Desk 2 จॻ1 “I have a pen.” จॻ2 “We have desk.” CHAR FILTER TOKENIZER TOKEN FILTER Analyzer

Slide 6

Slide 6 text

શจݕࡧͷ࢓૊Έ SEARCH ୯ޠ จॻ have 1,2 pen 1 we 2 Desk 2 ݕࡧϫʔυ: “pen” จॻ1͕ώοτ CHAR FILTER TOKENIZER TOKEN FILTER Analyzer

Slide 7

Slide 7 text

ANALYZER͸ͳͥඞཁ? - τʔΫϯ෼ׂͯ͘͠ΕΔͨΊ - “I have a pen.” ͜ͷ··Ͱ͸సஔΠϯσοΫε Λ࡞੒Ͱ͖ͳ͍ͷͰɺI, have, a, penͱτʔΫ ϯ෼ׂ͍ͨ͠ - ΫΤϦͷදه༳ΕΛٵऩͨ͠Γ͢ΔͨΊ - “GOD”ͱ͍͏୯ޠΛؚΉυΩϡϝϯτ ͸ɺ”god”Ͱ΋ώοτ͢ΔΑ͏ʹখจࣈʹ౷Ұ ͍ͤͨ͞ - ແବͳτʔΫϯͷϑΟϧλϦϯά - theͳͲΠϯσΩγϯάͯ͠΋ແବ Analyzeલ Analyzeޙ “I have a BIG pen!” have, big, pen

Slide 8

Slide 8 text

۩ମతͳANALYZERྫ - Char Filter(Tokenizerͷલʹॲཧ͢Δ) 0ݸҎ্ - Mapping: إจࣈΛ୯ޠʹม׵ͳͲ - HTMLstrip: HTMLΛύʔε - Tokenizer(τʔΫϯ෼ׂ͢Δ) 1ݸ - Standard: εϖʔεͳͲϧʔϧʹैͬͯ෼ׂ - Kuromoji: ܗଶૉղੳͰ෼ׂ - Ngram: Nจࣈ͝ͱʹ෼ׂ - Token Filter(Tokenizerͷޙʹॲཧ͢Δ) 0ݸҎ্ - Lowercase: খจࣈ - Stopword: ετοϓϫʔυআڈ - Stemming: දه༳Ε CHAR FILTER TOKENIZER TOKEN FILTER Analyzer

Slide 9

Slide 9 text

࣮૷

Slide 10

Slide 10 text

ANALYZERͷ࣮ߦΠϝʔδ Analyzeલ MappingCharFilter StandardTokenizer LowercaseFilter StopWordFilter StemmerFilter I have a lot of TASKs. I am very sad :( I have a lot of TASKs. I am very sad _sad_ I, have, a, lot, of, TASKs, I, am, very, sad, sad I, have, a, lot, of, tasks, I, am, very, sad, sad lot, tasks, am, very, sad, sad lot, task, am, very, sad, sad ॲཧͷྲྀΕ

Slide 11

Slide 11 text

ANALYZERͷ࣮૷

Slide 12

Slide 12 text

CHAR FILTERͷ࣮૷

Slide 13

Slide 13 text

TOKENIZERͷ࣮૷

Slide 14

Slide 14 text

TOKEN FILTERͷ࣮૷

Slide 15

Slide 15 text

INDEX - సஔΠϯσοΫε͸ map[string][]int - సஔΠϯσοΫεΛϑΟʔϧυ ໊͝ͱʹ࣋ͭ - υΩϡϝϯτ͸ɺIDͱϑΟʔϧ υΛ࣋ͭ - Indexing͢Δͱ͖΋AnalyzerΛ ௨͢

Slide 16

Slide 16 text

SEARCH -ANDݕࡧͱORݕࡧ -ΫΤϦ: “pink orange blue” -ANDݕࡧ: 3 -OR: ݕࡧ1,2,3,4,5,6 pink Orange blue 6 3 4 5 2 1

Slide 17

Slide 17 text

ಈ͔ͯ͠ΈΔ

Slide 18

Slide 18 text

ݕࡧྫ - ※Analyzer͸͖ͬ͞ͱಉ༷ - Index΋Search΋AnalyzerΛ௨͢ - ”foxes”Ͱ”fox”ΛؚΉυΩϡϝϯτ͕Ϛον - ”happy”Ͱ”:)”ΛؚΉυΩϡϝϯτ͕Ϛον - ANDݕࡧ - fine,FaX,foxes,happy͕શؚͯ·Ε͍ͯΔυΩϡϝ ϯτ1,2͕ώοτ͍ͯ͠Δ

Slide 19

Slide 19 text

ࠓޙ -͍͋·͍ݕࡧΛ࣮૷͢Δ -Fuzzy QueryɺSuggesters -είΞܭࢉΛ࣮૷͢Δ -IFIDF, BM25

Slide 20

Slide 20 text

ࢀߟ -https://github.com/kotaroooo0/stalefish -https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/