Upgrade to Pro — share decks privately, control downloads, hide ads and more …

俺の全文検索エンジン(Go製)を作り始めた

kotaroooo0
November 11, 2020

 俺の全文検索エンジン(Go製)を作り始めた

kotaroooo0

November 11, 2020
Tweet

More Decks by kotaroooo0

Other Decks in Programming

Transcript

  1. શจݕࡧͷ࢓૊Έ INDEXING ୯ޠ จॻ have 1,2 pen 1 we 2

    Desk 2 จॻ1 “I have a pen.” จॻ2 “We have desk.” CHAR FILTER TOKENIZER TOKEN FILTER Analyzer
  2. શจݕࡧͷ࢓૊Έ SEARCH ୯ޠ จॻ have 1,2 pen 1 we 2

    Desk 2 ݕࡧϫʔυ: “pen” จॻ1͕ώοτ CHAR FILTER TOKENIZER TOKEN FILTER Analyzer
  3. ANALYZER͸ͳͥඞཁ? - τʔΫϯ෼ׂͯ͘͠ΕΔͨΊ - “I have a pen.” ͜ͷ··Ͱ͸సஔΠϯσοΫε Λ࡞੒Ͱ͖ͳ͍ͷͰɺI,

    have, a, penͱτʔΫ ϯ෼ׂ͍ͨ͠ - ΫΤϦͷදه༳ΕΛٵऩͨ͠Γ͢ΔͨΊ - “GOD”ͱ͍͏୯ޠΛؚΉυΩϡϝϯτ ͸ɺ”god”Ͱ΋ώοτ͢ΔΑ͏ʹখจࣈʹ౷Ұ ͍ͤͨ͞ - ແବͳτʔΫϯͷϑΟϧλϦϯά - theͳͲΠϯσΩγϯάͯ͠΋ແବ Analyzeલ Analyzeޙ “I have a BIG pen!” have, big, pen
  4. ۩ମతͳANALYZERྫ - Char Filter(Tokenizerͷલʹॲཧ͢Δ) 0ݸҎ্ - Mapping: إจࣈΛ୯ޠʹม׵ͳͲ - HTMLstrip:

    HTMLΛύʔε - Tokenizer(τʔΫϯ෼ׂ͢Δ) 1ݸ - Standard: εϖʔεͳͲϧʔϧʹैͬͯ෼ׂ - Kuromoji: ܗଶૉղੳͰ෼ׂ - Ngram: Nจࣈ͝ͱʹ෼ׂ - Token Filter(Tokenizerͷޙʹॲཧ͢Δ) 0ݸҎ্ - Lowercase: খจࣈ - Stopword: ετοϓϫʔυআڈ - Stemming: දه༳Ε CHAR FILTER TOKENIZER TOKEN FILTER Analyzer
  5. ANALYZERͷ࣮ߦΠϝʔδ Analyzeલ MappingCharFilter StandardTokenizer LowercaseFilter StopWordFilter StemmerFilter I have a

    lot of TASKs. I am very sad :( I have a lot of TASKs. I am very sad _sad_ I, have, a, lot, of, TASKs, I, am, very, sad, sad I, have, a, lot, of, tasks, I, am, very, sad, sad lot, tasks, am, very, sad, sad lot, task, am, very, sad, sad ॲཧͷྲྀΕ