Share
Embed
Elasticsearch 日本語スキーマレス環境構築と、ついでに多言語対応
第6回elasticsearch勉強会「Elasticsearch 日本語スキーマレス環境構築と、ついでに多言語対応」資料
Kunihiko Kido
September 16, 2014
Transcript
48 slides
-
ࣗݾհ ށࠃ ,6/*)*,0,*%0 ! φϨοδϫʔΫεגࣜձࣾ φϨοδϫʔΫεגࣜձࣾͰݕࡧˍΫϥυࣄۀؔ࿈ͷϚωʔδϝϯτͯ͠·͢ɻ &MBTUJDTFBSDIؔ࿈Ͱɻɻ ϒϩάʮ)FMMP&MBTUJDTFBSDIʯͱ͔ &MBTUJDTFBSDIBTBTFSWJDFʮ4JCB γʔό
ʯͷ اըɾӡӦͯ͠·͢ɻ “Hello! Elasticsearch” “ݕࡧBaaS” ͖ͭͮ8FCͰˠ -
Ϛοϐϯάఆٛͱత ̍ ̎ ̏ ̐ ̑ ̒ ద߹Λ ্͍ͨ͠ ࠶ݱΛ
্͍ͨ͠ ਖ਼֬ʹ ߜΓࠐΈ͍ͨ ूܭ͍ͨ͠ ද͍ࣔͨ͠ BOENPSF ͔͠ϑΟʔϧυຖʹɻɻɻ ߟ͑Δ͜ͱ͕͍ͬͺ͍ʂ -
Ϛοϐϯάఆٛͱత ̍ ̎ ̏ ̐ ̑ ̒ શҰகͰ ݕࡧɾूܭ͍ͨ͠ ʮʯʮʯʮݝʯ
Λলུͯ͠ݕࡧ͍ͨ͠ ϤϛͰݕࡧ͍ͨ͠ ಓݝίʔυ ॱͰιʔτ͍ͨ͠ ʮؔ౦ʯͳͲ ผ໊Ͱݕࡧɾूܭ͍ͨ͠ BOENPSFʜ ྫ͑ɺಓݝ໊ϑΟʔϧυ̍ͭͱͬͯ -
%FNP DELETE /myindex! ! PUT /myindex/mytype/1! ! {! "title": "Elasticsearch
ಛ·ͱΊ",! "description": "Elasticsearch Features — ओʹγεςϜΛத৺ͱͨ͠ಛ·ͱΊ",! "author": "Kunihiko Kido",! "link": "https://medium.com/hello-elasticsearch/elasticsearch-500996e47c70",! "tags": ["Elasticsearch"],! "pubDate": "2014-03-12T11:09"! } ͍ͭ௨Γɺ͜Μͳײ͡Ͱσʔλొ͢Δ͚ͩͰɺ ϑΟʔϧυຖʹඞཁͳϚοϐϯάఆ͕ٛग़དྷ্͕Δ -
%FNP {! ...! "title" : {! "type" : "string",! "fields"
: {! "keyword" : {! "type" : "string",! "analyzer" : "keyword_analyzer"! },! "raw" : {! "type" : "string",! "index" : "not_analyzed"! },! "substring" : {! "type" : "string",! "analyzer" : "cjk"! },! "yomi" : {! "type" : "string",! "analyzer" : "katakana"! }! }! }! ...! }! UJUMFϑΟʔϧυͷϚοϐϯάఆٛ ˞ಈతʹ࡞͞ΕͨϚοϐϯάఆٛ -
%FNP {! ...! "description" : {! "type" : "string",! "fields"
: {! "substring" : {! "type" : "string",! "analyzer" : "cjk"! }! }! },! ...! } ˞ಈతʹ࡞͞ΕͨϚοϐϯάఆٛ EFTDSJQUJPOϑΟʔϧυͷϚοϐϯάఆٛ -
%FNP {! ...! "author" : {! "type" : "string",! "fields"
: {! "keyword" : {! "type" : "string",! "analyzer" : "keyword_analyzer"! },! "raw" : {! "type" : "string",! "index" : "not_analyzed"! },! "substring" : {! "type" : "string",! "analyzer" : "cjk"! },! "yomi" : {! "type" : "string",! "analyzer" : "katakana"! }! }! }! ...! }! BVUIPSϑΟʔϧυͷϚοϐϯάఆٛ ˞ಈతʹ࡞͞ΕͨϚοϐϯάఆٛ -
%FNP {! ...! "link" : {! "type" : "string",! "index"
: "not_analyzed",! "fields" : {! "domain" : {! "type" : "string",! "analyzer" : "domain_analyzer"! },! "keyword" : {! "type" : "string",! "analyzer" : "keyword_analyzer"! },! "substring" : {! "type" : "string",! "analyzer" : "standard"! }! }! },! ...! } ˞ಈతʹ࡞͞ΕͨϚοϐϯάఆٛ MJOLϑΟʔϧυͷϚοϐϯάఆٛ -
%FNP {! ...! "tags" : {! "type" : "string",! "fields"
: {! "keyword" : {! "type" : "string",! "analyzer" : "keyword_analyzer"! },! "raw" : {! "type" : "string",! "index" : "not_analyzed"! },! "substring" : {! "type" : "string",! "analyzer" : "cjk"! }! }! },! ...! }! ˞ಈతʹ࡞͞ΕͨϚοϐϯάఆٛ UBHTϑΟʔϧυͷϚοϐϯάఆٛ -
PUT /myindex/mytype/1! ! {! "title": "Elasticsearch Overview",! "description": "Elasticsearch Features —
System Overview",! "author": "Kunihiko Kido",! "link": "https://medium.com/hello-elasticsearch/elasticsearch-500996e47c70",! "tags": ["Elasticsearch"],! "pubDate": “2014-03-12T11:09”,! "language": “en",! } %FNP ʮӳޠʯͷจষͱͯ͠ΠϯσοΫε͍ͨ͠߹ ͜Ε͚ͩ -
PUT /myindex/mytype/1! ! {! "title": "Elasticsearch ѐਃ",! "description": "Elasticsearch Features —
दझమ ѐਃ",! "author": "Kunihiko Kido",! "link": "https://medium.com/hello-elasticsearch/elasticsearch-500996e47c70",! "tags": ["Elasticsearch"],! "pubDate": “2014-03-12T11:09”,! "language": “ko",! } %FNP ʮؖࠃޠʯͷจষͱͯ͠ΠϯσοΫε͍ͨ͠߹ ͍͕͍ͱ؆୯ʂ -
PUT /myindex/mytype/1! ! {! "title": "Elasticsearch Überblick",! "description": "Elasticsearch Features —
Systemübersicht",! "author": "Kunihiko Kido",! "link": "https://medium.com/hello-elasticsearch/elasticsearch-500996e47c70",! "tags": ["Elasticsearch"],! "pubDate": “2014-03-12T11:09”,! "language": “de",! } %FNP ʮυΠπޠʯͷจষͱͯ͠ΠϯσοΫε͍ͨ͠߹ ศར͔ʂ -
EZOBNJDUFNQMBUFT {! ...! "dynamic_templates": [! {! "my_field": {! "match": "*",
/* ᶃ ϑΟʔϧυ໊ͷύλʔϯ */! "match_pattern": "regex", /* ᶄ matchͷϚονϯάํࣜ ʢলུՄʣ*/! "match_mapping_type": "string", /* ᶅ JSONϑΥʔϚοτͷλΠϓ */! "mapping": {...} /* ᶆ Ϛοϐϯάఆٛ */! }! },! ...(ෳఆٛͰ͖Δ)! ],! ...! }! ! جຊతͳઃఆ -
EZOBNJDUFNQMBUFT {! "url_fields": {! "match": ".*url|.*link",! "match_pattern": "regex",! "match_mapping_type": "string",!
"mapping": {! "type": "string",! "index": "not_analyzed",! "fields": {! "keyword": {! "type": "string",! "analyzer": "keyword_analyzer"! },! "substring": {! "type": "string",! "analyzer": "standard"! },! "domain": {! "type": "string",! "analyzer": "domain_analyzer"! }! }! }! }! }! ྫʣ63-ϑΟʔϧυ͚ -
EZOBNJDUFNQMBUFT {! "special_string_fields": {! "match": ".*title.*|.*name.*|.*author.*",! "match_pattern": "regex",! "match_mapping_type": "string",!
"mapping": {! "type": "string",! "fields": {! "raw": {! "type": "string",! "index": "not_analyzed"! },! "keyword": {! "type": "string",! "analyzer": "keyword_analyzer"! },! "substring": {! "type": "string",! "analyzer": "ja-substring"! },! "yomi": {! "type": "string",! "analyzer": "katakana"! }! }! }! }! } ྫʣ໊শॏཁͳϑΟʔϧυ -
EZOBNJDUFNQMBUFT {! "long_string_fields": {! "match": ".*message.*|.*content.*|.*description.*| .*text.*|.*body.*|.*note.*|.*memo.*",! "match_pattern": "regex",! "match_mapping_type":
"string",! "mapping": {! "type": "string",! "fields": {! "substring": {! "type": "string",! "analyzer": "ja-substring"! }! }! }! }! } ྫʣจϑΟʔϧυ -
EZOBNJDUFNQMBUFT {! "short_string_fields": {! "match": "*",! "match_mapping_type": "string",! "mapping": {!
"type": "string",! "fields": {! "raw": {! "type": "string",! "index": "not_analyzed"! },! "keyword": {! "type": "string",! "analyzer": "keyword_analyzer"! },! "substring": {! "type": "string",! "analyzer": "ja-substring"! }! }! }! }! } ྫʣͦͷଞϑΟʔϧυ -
EZOBNJDUFNQMBUFT {! ...! "dynamic_templates": [! {! “url_field": {…}! },! {!
"special_string_fields": {…}! },! {! "long_string_fields": {…}! },! {! "short_string_fields": {…}! },! ],! ...! } ࠷ॳʹύλʔϯʹϚονͨ͠Ϛο ϐϯά͕ఆٛ͞ΕΔ -
JOEFYUFNQMBUF DPOpHUFNQMBUFTԼʹอଘ͢Δ͚ͩͰ४උ0, ˝FMBTUJDTFBSDI ˝DPOpH ˝UFNQMBUFT CBTFKTPO DVTUPN@BOBMZ[FSTKTPO KBQBOFTF@BOBMZ[FSTKTPO MBOHVBHF@BOBMZ[FSTKTPO TUSJOH@pFMETKTPO
%FNPͰ͍ͬͯΔJOEFY UFNQMBUFϑΝΠϧͨͪ ˡ ɾϊʔυͷ࠶ىಈඞཁͳ͍ ɾ৽نͰ࡞ͨ͠ΠϯσοΫεͷΈʹద༻͞ΕΔ ɾςϯϓϨʔτෳ࡞ΕΔ ɾ"1*ͰઃఆͰ͖Δ ɾEZOBNJDUFNQMBUFTͷఆؚٛΊΒΕΔ -
JOEFYUFNQMBUF {! "my_template":{ /* ςϯϓϨʔτͷ໊લ */! "template": "*", /* ςϯϓϨʔτΛద༻͢ΔΠϯσΫε໊ͷύλʔϯ
*/! "order": 0, /* ςϯϓϨʔτΛద༻͢Δ༏ઌॱҐ */! "settings": {...}, /* γϟʔυͷͱ͔ɺAnalysis ͷઃఆΠϯσοΫεఆٛʹؔΘΔઃఆ */! "mappings": {...} /* Ϛοϐϯάఆٛͷઃఆɻdynamic templates͜͜Ͱઃఆ */! . . .! }! }! ! ! ! ! ! ϑΝΠϧ໊ͱςϯϓϨʔτ໊ͱΓ͋͑ͣಉ͡ʹ͓͚ͯ͠ྑ͍͔ͳʁ جຊతͳઃఆ -
JOEFYUFNQMBUF {! "base": {! "template": "*",! "order": 1,! "mappings": {!
"_default_": {! "_timestamp" : {! "enabled" : true,! "store" : true! },! "_analyzer": {! "path": "language"! },! "_id": {! "path": "id"! },! "_source": {! "excludes" : ["attachment_file"]! }! }! }! }! }! UFNQMBUFTCBTFKTPO σϑΥϧτͷΞφϥΠβʔมߋ͢ΔͨΊͷMBOHVBHFϑΟʔϧυͷఆٛͳͲ -
JOEFYUFNQMBUF {! "language_analyzers": {! "template": "*",! "order": 2,! "settings": {!
"analysis": {! "filter": {! "arabic_stop": {! "type": "stop",! "stopwords": "_arabic_"! },! "arabic_stemmer": {! "type": "stemmer",! "language": "arabic"! },! "armenian_stop": {! "type": "stop",! "stopwords": "_armenian_"! },! "armenian_stemmer": {! "type": "stemmer",! "language": "armenian"! },! "basque_stop": {! "type": "stop",! "stopwords": "_basque_"! },! "basque_stemmer": {! "type": "stemmer",! "language": "basque"! },! "brazilian_stop": {! "type": "stop",! "stopwords": "_brazilian_"! },! "brazilian_stemmer": {! "type": "stemmer",! "language": "brazilian"! },! "bulgarian_stop": {! "type": "stop",! "stopwords": "_bulgarian_"! },! "bulgarian_stemmer": {! "type": "stemmer",! "language": "bulgarian"! },! "catalan_elision": {! UFNQMBUFTMBOHVBHF@BOBMZ[FSTKTPO ֤ࠃݴޠຖͷϑΟϧλʔΞφϥΠβʔͷఆٛͳͲ -
JOEFYUFNQMBUF {! "japanese_analyzers": {! "template": "*",! "order": 2,! "settings": {!
"analysis": {! "char_filter": {! "japanese_normalization": {! "type": "kuromoji_iteration_mark",! "normalize_kanji": true,! "normalize_kana": true! }! },! "filter": {! "romaji_readingform": {! "type": "kuromoji_readingform",! "use_romaji": true! },! "katakana_readingform": {! "type": "kuromoji_readingform",! "use_romaji": false! },! "katakana_stemmer": {! "type": "kuromoji_stemmer",! "minimum_length": 4! },! "japanese_stop": {! "type": "kuromoji_part_of_speech"! }! },! "tokenizer": {! "japanese_tokenizer": {! "type": "kuromoji_tokenizer",! "mode": "search"! }! },! "analyzer": {! "default": {! "alias": ["japanese", "ja"],! "type": "custom",! "char_filter": [! "html_strip",! "japanese_normalization"! ],! "tokenizer": "japanese_tokenizer",! "filter": [! "cjk_width",! "lowercase",! UFNQMBUFTKBQBOFTF@BOBMZ[FSTKTPO ຊޠؔ࿈ͷϑΟϧλʔΞφϥΠβʔͷఆٛͳͲ -
JOEFYUFNQMBUF {! "custom_analyzers": {! "template": "*",! "order": 2,! "settings": {!
"analysis": {! "filter": {! "domain_extractor" : {! "type" : "pattern_replace",! "preserve_original": false,! "pattern" : "https?://([^/]+).*",! "replacement": "$1"! }! },! "tokenizer": {! "comma_tokenizer": {! "type": "pattern",! "pattern":"[,ɺ]+"! }! },! "analyzer": {! "domain_analyzer" : {! "alias": ["domain"],! "tokenizer" : "uax_url_email",! "filter" : [ "domain_extractor", "lowercase", "unique" ]! },! "comma_analyzer":{! "alias": ["comma"],! "type": "custom",! "tokenizer": "comma_tokenizer",! "filter": ["cjk_width", "lowercase", "trim", "unique"]! },! "keyword_analyzer": {! "type": "custom",! "tokenizer": "keyword",! "filter": ["cjk_width", "lowercase", "trim"]! }! }! }! }! }! } UFNQMBUFTDVTUPN@BOBMZ[FSTKTPO 63-͔ΒυϝΠϯ໊Λநग़ͨ͠Γݴޠʹґଘ͠ͳ͍ΞφϥΠβʔͷఆٛ -
JOEFYUFNQMBUF {! "string_fields": {! "template": "*",! "order": 10,! "mappings": {!
"_default_": {! "dynamic_templates": [! {! "not_analyzed_fields": {! "match": "method|charSet|mimeType| content_type|language",! "match_pattern": "regex",! "match_mapping_type": "string",! "mapping": {! "type": "string",! "index": "not_analyzed"! }! }! },! {! "url_fields": {! "match": ".*url|.*link",! "match_pattern": "regex",! "match_mapping_type": "string",! "mapping": {! "type": "string",! "index": "not_analyzed",! "fields": {! "keyword": {! "type": "string",! "analyzer": "keyword_analyzer"! },! "substring": {! "type": "string",! "analyzer": "standard"! },! "domain": {! "type": "string",! "analyzer": "domain_analyzer"! }! }! }! }! },! {! "comma_separated_fields": {! "match": "meta_keywords",! "match_pattern": "regex",! "match_mapping_type": "string",! "mapping": {! UFNQMBUFTTUSJOH@pFMETKTPO TUSJOHܕϑΟʔϧυͷμΠφϛοΫςϯϓϨʔτఆٛ -
JOEFYUFNQMBUF ˝ΠϯσοΫε࡞ TFUUJOHTཁૉͷద༻ ˝λΠϓͷ࡞ NBQQJOHTཁૉͷద༻ ˝ϑΟʔϧυσʔλͷՃ EZOBNJDUFNQMBUFTͷద༻ γϟʔυͷઃఆͱ͔"OBMZTJTͷఆٛ ͳͲ͜ͷλΠϛϯάͰద༻ɻ ݻఆͷϑΟʔϧυͷఆٛͳͲ͜ͷλ
ΠϛϯάͰద༻ɻ EZOBNJDUFNQMBUFTͷఆٛ͜ͷλΠ ϛϯάͰద༻ɻ ˡ ˡ ˡ ద༻ͷྲྀΕ ✔️ ✔️ ✔️ -
JOEFYUFNQMBUF ˝ΠϯσοΫε࡞ TFUUJOHTཁૉͷద༻ NBQQJOHTཁૉͷద༻ ˝ϑΟʔϧυσʔλͷՃ EZOBNJDUFNQMBUFTͷద༻ ΠϨΪϡϥʔͳରԠ͍ͨ͠߹ ✔️ ✔️ ✔️
ରͷϑΟʔϧυఆ͕ٛଘࡏ͠ͳ͍ ㅟ ㅟ ㅟ ㅟ ㅟ ㅟ ߹ɺEZOBNJDUFNQMBUFTͷఆٛ͜ͷ ㅟ λΠϛϯάͰద༻ɻ ˡ ΠϨΪϡϥʔͳϚοϐϯάఆٛΛ͍ͨ͠ ߹͜ͷλΠϛϯάͰఆٛ͢Ε0, ˝λΠϓͷ࡞