Analyzer101 -Medcl #ESCC#4

什么情况下用什么分词器,搜索的时候又该选用哪种查询,搜索性能和它又有啥关系?配置为什么老是不成功?

Transcript

1. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Analyzer 101 how to work with analyzer in elas3csearch Medcl
2. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited About me •  Follow Elas3csearch Since v0.5,2010 •  Joined Elas3c Since September,2015 •  @medcl •  hDp://github.com/medcl
3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited We love Elas3csearch • Elas3csearch is build on top of Lucene! • Json IN Json OUT! • Fancy Distributed and Scalability! • …
4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited But What if … • “为什么查不到,明明有的” • “怎么出来这个鬼,数据怎么出来 的” • “这个字段可以用模糊匹配么” • “这个字段可以用Aggrega3on么” • “为什么索引这么大呀” 什么情况
5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited TELL ME WHY? If you don’t know why, you won’t know how to fix
6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Agenda • What’s the Analyzer • Why Analyzer MaDers • How Analyzer Works • Analyzer for Chinese • Analyzer In Elas3csearch • How to Choose Analyzer
7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 什么是ANALYZER What’s the Analyzer
8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Let’s go back to basis • Lucene & Invert Index • How index works? • How search works?
9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Inverted Index •  Doc1: – The quick brown fox jumped over the lazy dogs. •  Doc2: – The yellow dog is mine. •  Doc3: – I don’t have brown bag!
10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Term Name Document ID The Doc1,Doc2 quick Doc1, Brown Doc1,Doc3 Fox Doc1 … … … … Inverted Index
11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Search Index •  “The” – Doc1,Doc2 •  “The Fox” => “The” AND “Fox” – Doc1 Term Name Document ID The Doc1,Doc2 quick Doc1, Brown Doc1,Doc3 Fox Doc1 … … … …
12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Index workflow Brief version, we ignore details Prepare Document Analysis > Term[s] Build Inverted Index Save Index To Store
13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Search workflow Prepare Query String Analysis > Term[s] Match Inverted Index Return Search Result Brief version, we ignore details
14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Highlighted workflow Analysis > Term[s] Brief version, we ignore details
15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Text->Terms? Analysis Analyzer
16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited What is the Analysis •  Lucene is an indexing and search library, accepts only plain text input. •  Text analysis. – Lucene use Analyzer to Analysis,convert text into indexable/ searchable tokens.
17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited What is the Analyzer •  Analyzers create tokens from the character stream. •  An analyzer is an encapsula3on of the analysis process. An analyzer tokenizes text by performing any number of opera3ons on it, which could include extrac3ng words, discarding punctua3on, removing accents from characters, lowercasing (also called normalizing), removing common words, reducing words to a root form (stemming), or changing words into the basic form (lemma3za3on). 文本搅拌机
18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited WHY ANALYZER MATTERS
19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The Key of Hit Term[s] • Match - > HIT • Not Match -> MISS Of Inverted Index Of Analyzed Query String
20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The key of Condi3on • Parameter:default_operator – AND: must match all terms – OR: match or not is OK Example: _search?q=quick fox
21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The key of Condi3on •  BoolQuery – Must: must match the term – Must Not: must not match the term – Should: don’t really care match or not Example: _search?q=(“quick” AND “fox”) OR “dog”
22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The key of Term Query • Term Query: – “QueryString” will be direct used as term to match the index
23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The key of Text Query • Match_all / TextQuery / QueryStringQuery etc: – “QueryString” will be Analyzed to generate terms to match the index
24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited The key of Range Query • Range Query: – “QueryString” will also generated sequence terms to match the index
25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited It MaDers • Analyzer影响索引 • Analyzer影响查询
26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ANALYZER如何工作 How Analyzer Works
27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited How Analyzer works? •  Analyzer build with: –  One or More Char Filters [Chained] –  One Tokenizer –  One or More Token Filters [Chained] CharFilter Tokenizer TokenFilter TokenFilter TokenFilter … CharFilter CharFilter … Text Tokens
28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited How Analyzer works? •  Elas>c search IS REALLY Amazing! •  Elas>csearch IS REALLY Amazing! •  [Elas3csearch] [IS] [REALLY] [Amazing] [!] •  [elas3csearch] [is] [really] [amazing] [!] •  [elas3csearch] [really] [amazing] •  [elas3csearch] [really] [amaze] •  [elas3csearch] [really] [indeed] [amaze]
29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Built-In Analyzers Analzying "The quick brown fox jumped over the lazy dogs” •  org.apache.lucene.analysis.WhitespaceAnalyzer: [The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs] •  org.apache.lucene.analysis.SimpleAnalyzer: [the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dogs] •  org.apache.lucene.analysis.StopAnalyzer: [quick] [brown] [fox] [jumped] [over] [lazy] [dogs] •  org.apache.lucene.analysis.standard.StandardAnalyzer: [quick] [brown] [fox] [jumped] [over] [lazy] [dogs] •  org.apache.lucene.analysis.snowball.SnowballAnalyzer: [quick] [brown] [fox] [jump] [over] [lazi] [dog] Lucene has many built-in analyzers
30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 中文分析处理 Analyzer for Chinese
31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 中文博大精深
32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 一则新闻
33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 再看看英文
34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 换个姿势
35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 中文很复杂 • 同音、多音、多义、兼类词 和同形异构… … 1. 还欠款壹万元 2. 放弃美丽的女人让人心碎 3. 开刀的是他父亲
36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited “推” “敲”
37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Community Analyzers •  ICTCLAS •  ANSJ •  CC-CEDICT •  SCWS •  FudanNLP •  IK •  MMSEG •  JIEBA •  …
38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited More • 繁体 • 拼音
39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited ELASTICSEARCH中的ANALYZER Analyzer In Elas3csearch
40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 内置丰富
41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 调试Analyzer •  Tokenizer\Filter
42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Mapping
43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 自定义Analyzer •  elas3csearch.yml •  配置全局可见 •  修改需要重启集群
44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 动态组合Analyzer •  1.关闭Index •  2.修改IndexSerngs,创建Analyzer •  3.打开Index
45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 配置 Char-Filter
46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 如何选择ANALYZER How to Choose the Analyzer
47. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 关于停用词 • “To be or not to be”
48. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited 关于排序 • 字段能分词么?
49. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Prefix/Wildcard Query& Analyzers • 字段能分词么? • 和Ngram的比较
50. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Aggrega3on & Analyzers • 字段能分词么?
51. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Suggest • 一个Analyzer搞定所有场景? • 试试Mul3-Field • 试试多字段查询
52. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Final words Search well, choose right analyzer!
53. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Recommenda3on •  Apache Lucene Documenta>on hDp://lucene.apache.org/core/5_3_1/index.html •  Lucene In Ac3on 2
54. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written permission is strictly prohibited Thank You!