検索結果の品質向上

2017年4～5月開催「ブートキャンプ特別講座」の資料になります。

Transcript

1. Elasticsearch AP
2. ※
3. • • • • Term • index • • • •
4. • • • • Term • index • • • •
5. The Art of Computer Programming, Volume 3: Sorting and Searching • • •
6. The Art of Computer Programming, Volume 3: Sorting and Searching • • •
7. Introduction to Information Retrieval Modern Information Retrieval
8. Introduction to Information Retrieval Modern Information Retrieval : CS 276 Information Retrievaland Web Search http://web.stanford.edu/class/cs 276/ …
9. Java $ grep index index
10. Index • grep • Boyer-Moore • • N-gram •
11. Java $ grep index index Index
15. Index
16. Index Indexing
17. Index Search (AND )
18. Index Search (OR )
19. Index
20. Index index
21. • • • • Term • index • • • •
22. Inverted Index
23. Inverted Index
24. Inverted Index
25. Inverted Index
26. Term At A Time
32. Term At A Time
33. Inverted Index
34. Inverted Index
35. Document At A Time
40. Document At A Time
41. Inverted Index
42. – – – –
44. No
45. Java
46. Lucene • – • – • –
47. Document At A Time
48. Elasticsearch & Solr • • •
50. Elasticsearch
51. • • • • Term • index • • • •
52. • •
53. 映画
54. • • – • –
55. • • – • –
56. A A C C B B • • – • –
57. A A C C B B • • – • –
58. A A C C B B • • – • –
59. A A C C B B • • – • –
61. DAAT
62. DAAT
70. Index
74. AND
75. AND
76. • • • • Term • index • • • •
79. • • • • Term • index • • • •
80. Term At A Time
81. Term • • • •
82. Lucene Term Token lucene/core/src/java/org/apache/lucene/index/Term.java lucene/core/src/java/org/apache/lucene/analysis/Token.java
83. Lucene Tokenizer • ClassicTokenizerFactory • EdgeNGramTokenizerFactory • HMMChineseTokenizerFactory • ICUTokenizerFactory • JapaneseTokenizerFactory • KeywordTokenizerFactory • LetterTokenizerFactory • LowerCaseTokenizerFactory • NGramTokenizerFactory • PathHierarchyTokenizerFactory • PatternTokenizerFactory • StandardTokenizerFactory • ThaiTokenizerFactory • UAX29URLEmailTokenizerFactory • UIMAAnnotationsTokenizerFactory • UIMATypeAwareAnnotationsTokenize rFactory • WhitespaceTokenizerFactory • WikipediaTokenizerFactory
84. Lucene Tokenizer • ClassicTokenizerFactory • EdgeNGramTokenizerFactory • HMMChineseTokenizerFactory • ICUTokenizerFactory • JapaneseTokenizerFactory • KeywordTokenizerFactory • LetterTokenizerFactory • LowerCaseTokenizerFactory • NGramTokenizerFactory • PathHierarchyTokenizerFactory • PatternTokenizerFactory • StandardTokenizerFactory • ThaiTokenizerFactory • UAX29URLEmailTokenizerFactory • UIMAAnnotationsTokenizerFactory • UIMATypeAwareAnnotationsTokenize rFactory • WhitespaceTokenizerFactory • WikipediaTokenizerFactory
85. Lucene Tokenizer • ClassicTokenizerFactory • EdgeNGramTokenizerFactory • HMMChineseTokenizerFactory • ICUTokenizerFactory • JapaneseTokenizerFactory • KeywordTokenizerFactory • LetterTokenizerFactory • LowerCaseTokenizerFactory • NGramTokenizerFactory • PathHierarchyTokenizerFactory • PatternTokenizerFactory • StandardTokenizerFactory • ThaiTokenizerFactory • UAX29URLEmailTokenizerFactory • UIMAAnnotationsTokenizerFactory • UIMATypeAwareAnnotationsTokenize rFactory • WhitespaceTokenizerFactory • WikipediaTokenizerFactory
86. Tokenize WhitespaceTokenizer
87. Tokenize …
88. Lucene Tokenizer • ClassicTokenizerFactory • EdgeNGramTokenizerFactory • HMMChineseTokenizerFactory • ICUTokenizerFactory • JapaneseTokenizerFactory • KeywordTokenizerFactory • LetterTokenizerFactory • LowerCaseTokenizerFactory • NGramTokenizerFactory • PathHierarchyTokenizerFactory • PatternTokenizerFactory • StandardTokenizerFactory • ThaiTokenizerFactory • UAX29URLEmailTokenizerFactory • UIMAAnnotationsTokenizerFactory • UIMATypeAwareAnnotationsTokenize rFactory • WhitespaceTokenizerFactory • WikipediaTokenizerFactory
89. NgramTokenizer • • • Q: Bigram A: No… ⇒ N-gram 2-1gram
90. Japanese Tokenizer Kuromoji Surface form Part-of-Speech Base form Reading Pronunciation , ,*,* , , ,* , ,*,* , , ,* , ,*,* ,*,*,*
91. Ngram vs Ngram Good! hit! Bad… No hit. Bad… hit! Good! No hit. Index Bad… Good! index
92. Analyzer
93. Analyzer in Elasticseach
94. Lucene Tokenizer • ClassicTokenizerFactory • EdgeNGramTokenizerFactory • HMMChineseTokenizerFactory • ICUTokenizerFactory • JapaneseTokenizerFactory • KeywordTokenizerFactory • LetterTokenizerFactory • LowerCaseTokenizerFactory • NGramTokenizerFactory • PathHierarchyTokenizerFactory • PatternTokenizerFactory • StandardTokenizerFactory • ThaiTokenizerFactory • UAX29URLEmailTokenizerFactory • UIMAAnnotationsTokenizerFactory • UIMATypeAwareAnnotationsTokenize rFactory • WhitespaceTokenizerFactory • WikipediaTokenizerFactory
95. • • • • Term • index • • • •
96. index
97. Ngram vs Ngram Good! hit! Bad… No hit. Bad… hit! Good! No hit. Index Bad… Good! index
99. • • • • Term • index • • • •
101. C B A B A C A
102. C B A B A B A
103. C B A C A C A
104. C B A C A B A • •
105. C B A C A B A ↑ ↑
106. C B A C A B A ↓ ↓
107. C B A C A B A ↓ →
108. C B A C A B A ↑ →
109. C B A C A B A
110. C B A C A B A • •
111. C B A C A B A • • • •
112. C B A C A B A • • • •
113. C B A C A B A
114. C A B A F-measure F-measure
116. • – • – • –
118. 0 5000000 10000000 15000000 20000000 25000000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
122. Ngram vs Ngram Good! hit! Bad… No hit. Bad… hit! Good! No hit. Index Bad… Good! index
123. index • – • – • – •
125. index ngram index • • • •
128. index ngram index
131. • • • • Term • index • • • •
133. • – • – • – • –
134. • – • – • – • –
136. Linux Linux Linux Unix OS OS : Operating System OS OS … OS Linux FreeBSD… … 定量
138. TF-IDF • TF Term Frequency Term Term • IDF IDF Inverse Document Frequency Term • TF-IDF = TF IDF D Term T TF: D T IDF: T
139. TF-IDF TF Term Frequency Term Linux Linux Unix OS
140. TF-IDF IDF Inverse Document Frequency • Linux • IDF log Term T • • • • • Linux • • • IDF(Linux) log 2 5 IDF( ) log 3 5
141. TF-IDF IDF Inverse Document Frequency • – • log – TF IDF log Term T
142. TF-IDF
144. BM25: TF-IDF Linux Linux Unix OS Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux … TF(Linux) 14,352 TF(Linux) 2
145. BM25: TF-IDF Linux Linux Unix OS Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux Linux … TF(Linux) 14,352 TF(Linux) 2
146. BM25 Linux Linux Unix OS BM25(Linux) TF IDF TF+k1 (1-b+b ) k1 + 1 avgDL DL
147. BM25 BM25(Linux) TF IDF TF+k1 (1-b+b ) k1 + 1 avgDL DL 単に単語が沢山現れ る 場合は減点 単語 単語
148. BM25 and TF-IDF Linux OS … FreeBSD Unix … • Linux: 23 • OS: 11 • : 17 • : 0.331 • : 3.65 • : 0.003 • : 0.0001 • : 0.000053 • : 0.023 • FreeBSD: 65 • OS: 9 • : 5 • : 42 • : 58 • : 2 • :0.003 • : 0.00428 • : 0.00084 • : 90 • : 3 • : 1.8 • : 0.2 • : 0.00189
150. index • – • – • – •
156. • • • • Term • index • • • •
157. nDCG • • • • •
158. nDCG
159. nDCG
160. nDCG
161. nDCG
162. nDCG
163. nDCG
165. nDCG S1 S2 S3 S5 S4
166. nDCG DCG S1 S2 S3 S5 S4 DCG5 S1 + + + … log2 S2 log3 S3
167. nDCG S1 S2 S3 S5 S4 DCG5 S1 + + + … log2 S2 log3 S3
168. nDCG DCG=iDCG (ideal DCG) S1 S2 S4 S5 S3
170. DCG S1 S2 S3 S5 S4 DCG5 S1 + + + … = 439.23 log2 S2 log3 S3
171. iDCG S1 S2 S4 S5 S3 DCG5 S4 + + + … = 518.78 log2 S1 log3 S2
172. 518.78 439.23
173. DCG S1 S2 S3 S5 S4 DCG5 S4 + + + … = 489.23 log2 S2 log3 S3
174. 518.78 489.23
175. nDCG S1 S2 S3 S5 S4 S1 S2 S3 S5 S4
177. • • • • Term • index • • • •
179. MLR = Machine Learning Ranking S1 S2 S3 S5 S4 S2 S3 S5 S4 S1