Elasticsearch at Wantedly

2014/04/21 第 4 回 Elasticsearch 勉強会での発表資料。
当日の質問タイム及び懇親会で聞かれたことを追加しています。

Transcript

1. Elasticsearch at
2. SEIGO UCHIDA
3. !TQFTOPWB w 
   ೥
   :BIPP
   +"1"/
   ৽ଔೖࣾ
   w 
   ೥
   
   ݄
   8BOUFEMZ
   ʹδϣΠϯ
   w ޿ٛͷΠϯϑϥΛ୲౰
   w $IFG
   ͱ
   )VCPU
   ͱ͔͕޷͖
4. !TQFTOPWB
6. γΰτͰίίϩΦυϧਓΛ;΍͢
9. 8BOUFEMZ
   *OD w 
   ೥
   
   ݄ઃཱ
   w ΞϧόΠτɺΠϯλʔϯؚΊ
   
   ໊
   w 
   ໊ͷΤϯδχΞ͕׆༂
   w γΰτॆ࣮ԽΞϓϦ΍πʔϧͷఏڙ
   w 
   ࣾɺ
   ਓ͕࢖͏
10. ੵۃ࠾༻த https://www.wantedly.com/projects/5106
11. Wantedly
    Ͱ
    ES
    Λ
    ͲͷΑ͏ʹ࢖͍ͬͯΔͷ͔
12. ݕࡧର৅ w γΰτ
    w ձࣾ
    w Ϣʔβ
13. γΰτ w ձ໊ࣾ
    w ۈ຿஍
    w ࣾһ਺
15. ืू w ืू໊
    w ืूৄࡉ
    w ืू৬छ
17. Ϣʔβ w Ϣʔβ໊
    w ࣗݾ঺հจ
    w ͦͷଞ͍Ζ͍Ζʢޙड़ʣ
19. ΠϯσοΫε w ௨ৗͷݕࡧ༻ΠϯσοΫε
    w "VUPDPNQMFUF
    ༻ΠϯσοΫε
21. λΠϓ w γΰτλΠϓ
    w ձࣾλΠϓ
    w ϢʔβʔλΠϓ
22. ௨ৗͷݕࡧ༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿
    
    ສ
    w ϓϥΠϚϦαΠζɿ໿
    (#
23. "VUPDPNQMFUF༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿
    
    ສ
    w ϓϥΠϚϦαΠζɿ໿
    (#
24. ߏ੒ w 3BJMT
    PO
    )FSPLV
    w &MBTUJDTFBSDI
    PO
    &$
    w 1PTUHSFTRM
    PO
    3%4
25. FMBTUJDTFBSDISVCZ w UJSF
    ͸ϝϯς͕ऴྃ
    w TFBSDILJDL
    ͸
    UJSF
    ʹґଘ
    w ެࣜ
    HFN
    Λ࢖͏ͷ͕༨ܭͳ੍ݶ͕ͳ ͯ͘ྑ͍
26. Ϋϥελ w &$
    NTNBMM
    º
    
    ୆
    w $IFG
    Ͱ؅ཧ
    w +7.
    ׂ౰.&.ɿ౥ࡌ.&. 
27. ར༻ϓϥάΠϯ w )FBE
    w )2ʢ/PEF
    %JBHOPTUJDT
    ΍
    4UBUT
    ֬ೝʹར༻ʣ
    w .BSWFM
    TFOTF
    ͰΫΤϦ࡞੒ʹར༻ʣ ։ൃ؀ڥ ͷΈʣ
    w ,VSPNPKJ
    w *ORVJTJUPSʢ"OBMZ[FS
    ͷڍಈ֬ೝʣ
28. Ͳ͏΍ֶͬͯΜ͔ͩ
30. ެࣜϦϑΝϨϯε w ඞཁͳ͜ͱ͸ॻ͍ͯ͋Δɺؾ͕͢Δ
    w ΫΤϦͷྫ͕෦෼తʹॻ͔Ε͍ͯͯࠔΔ
    w ݕࡧॳ৺ऀʹ͸શମ૾͕͔ͭΈͮΒ͍
    w Ͱ΋ಡΜͩ
32. &MBTUJDTFBSDI
    8PSLTIPQ w શମ૾Λ௫Έͭͭɺ࣮ࡍʹ৮ΔͷͰΠ ϝʔδ͕௫Ίͨ
    w IUUQTTQFBLFSEFDLDPNEBEPPOFUFMBTUJDTFBSDIXPSLTIPQ
34. &MBTUJDTFBSDI
    νϡʔτϦΞϧ w ༻ޠɺશମ૾͕௫Έ΍͍͢
    w ίϯςΩετ͕͋Γ࣮ફత
    w IUUQDPEFIBUFOBCMPHDPNFOUSZ
35. ͜ͷຊ͕΋ͬͱૣ͘ग़ͯΕ͹ʜ
36. ۤ࿑ͨ͠ͱ͜Ζ
37. Ϣʔβʔݕࡧ w ϓϩϑΟʔϧ߲໨͕ଟ͍
    w ֤߲໨ຖʹϓϥΠόγʔϨϕϧ ͕͋Δ
    w NBQQJOH
    ΋
    RVFSZ
    ΋ෳࡶʹ
38. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ྫ͑͹ɺϓϩϑΟʔϧͷʮ໊લʯʮॴଐʯʮॴ ࡏ஍ʯΛجຊ৘ใͷηοτͱͯ͠ϓϥΠόγʔ ϨϕϧʮҰൠެ։ʯͱ͍ͯͨ͠ͱ͢Δ
    w ࠷ॳ͸
    /FTUFE
    EPDVNFOU
    Λ࢖ͬͯදݱ͠ ͍ͯͨ
39. "properties": {! "basic_privacy": {! "type": "nested"! },! "properties": {! "name": {! "type": "string"! },! "working_for": {! "type": "string"! },! "location": {! "type": "string"! },! "privacy_level": {! "type": "string"! }! }! }! mapping
40. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ͔͠͠ɺ/FTUFE
    ͩͱͰ͖ͳ͍͜ͱʢϋΠϥΠτͳͲʣ ͕ग़ͯ͘ΔͷͰϑϥοτͳߏ଄ͰɺϓϥΠόγʔϨϕ ϧΛදݱ
    w ܕ@σʔληοτ@@ϓϩύςΟ
    ͷΑ͏ʹϚοϐ ϯά
    w ࠷ॳͷܕ͕ඞཁͳͷ͸ɺEBUF
    ΍
    JOUFHFS
    ͳͲ
    TUSJOH
    Ҏ֎ʹ
    TJNQMF@RVFSZ
    Λ࢖͏ͱ
    FSSPS
    ͱͳΔͨ Ί
41. mapping query {! "query": {! "simple_query_string": {! "query": <QUERY>,! "fields": ["s_basic__*"],! "default_operator": "and"! }! }! },! "filter": {! "term": {! "basic_privacy": <PRIVACY_LEVEL>! }! }! }! "properties": {! "basic_privacy": {! "type": "string"! },! "s_basic__name": {! "type": "string"! },! "s_basic__working_for": {! "type": "string"! },! "s_basic__location": {! "type": "string"! }! }!
42. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ݁ՌతʹɺOFTUFE
    Λ࢖Θ͟ΔΛಘͳ͍৔໘ ͕ग़ͯདྷͯϑϥοτͳϓϩύςΟͱ
    OFTUFE
    Λซ༻͍ͯ͠Δ͕ɺࠓճ͸ͦ͜͸লུ
43. ࣭໰λΠϜ
44. 4PMS
    ΍
    $MPVETFBSDI
    ΋ධՁ্ͨ͠Ͱ
    
    &4
    ʹͨ͠ͷ͔ w ʮ༑ୡͱͷܨ͕ΓʯΛՃຯͯ͠ݕࡧͰ͖Δ͔ͱ͍͏͜ͱͰɺάϥϑ
    %#
    ͷ
    /FPK
    ͱҰൠతͳݕࡧͱͯ͠
    &4
    Λൺֱͯ͠ɺ
    &4
    ͷํ͕அ વύϑΥʔϚϯε͕ྑ͔ͬͨ
    w 4PMS
    ʹ͍ͭͯ͸
    4PMS
    Ϣʔβ͔Βࠓ͔Β࢖͏ͳΒ
    &4
    ͱ͍͏࿩Λฉ͍ ͍ͯͨ
    w ར༻Ϣʔβʢ(JU)VC
    GPVSTRVBSF
    4PVOE$MPVE
    ͳͲʣ΋ࢀߟʹ͠ ͨ
    w ݁Ռతʹɺ4PMS
    ʹͳ͍
    OFTUFE
    QBSFOUDIJME
    EPDVNFOU
    Λ࢖ͬͯ ͍ΔͷͰ
    &4
    Ͱྑ͔ͬͨ
45. ։ൃ؀ڥͰ
    &4
    Λ࢖͏ͱ
    
    NBQQJOH
    ͳͲόϥόϥͰਏ͘ͳΒͳ͍͔ w #PPUTUSBQ
    εΫϦϓτͱ͍͏΋ͷΛ༻ҙͯ͠
    HJU
    DMPOF
    ޙ ͷηοτΞοϓʢMPDBM
    ʹ
    &4
    ΛೖΕΔͳͲʣΛࣗಈԽ͠ ͍ͯΔ
    w .BQQJOH
    ΍υΩϡϝϯτͷ௥Ճ΋ηοτΞοϓεΫϦϓ τΛ༻ҙͯ͠ɺϝϯόʔ͝ͱͷ؀ڥͷࠩ෼͕ग़ͳ͍Α͏ʹ ͍ͯ͠Δ
46. ! #! # Check for Elasticsearch! #! if test ! $(which elasticsearch)! then! echo " x You need to install ElasticSearch:"! echo " + Installing Elasticsearch..."! brew install elasticsearch! else! echo " + ElasticSearch found."! fi! ! #! # Check for Elasticsearch version! #! ES_VERSION=$(elasticsearch -v | cut -d' ' -f2 | cut -d'.' -f1)! if test ! $ES_VERSION -eq 1! then! echo " x You need to upgrade elasticsearch."! brew upgrade elasticsearch! else! echo " + Your Elasticsearch version is correct."! fi! ! if test -d /usr/local/var/lib/elasticsearch/plugins/marvel! then! echo " + Elasticsearch Plugin Marvel found."! else! echo " + Installing Elasticsearch Plugins Marvel"! plugin --install elasticsearch/marvel/latest! fi! bootstrap script Ұ෦ൈਮ $ script/bootstrap! ! Wantedly! ! γΰτͰίίϩΦυϧਓΛ;΍͢! ! + Your Elasticsearch version is correct.! + ElasticSearch Plugin HQ found.! + ElasticSearch Plugin head found.! + ElasticSearch Plugin Kuromoji found.! + ElasticSearch Plugin Inquisitor found.! + ElasticSearch Plugin Marvel found.! ! Good work. We're ready.! ! Run 'elasticsearch' to start elasticsearch.! And you can see elasticsearch webui plugins 'locahost:9200/_plugin/head/'! ࣮ߦ݁Ռ γϯϓϧͳγΣϧεΫϦϓτɻelasticsearch ΍ϓϥάΠϯ͕Πϯετʔϧ͞ΕͯΔ ͔ΛνΣοΫͯ͠͞Εͯͳ͍ͳΒΠϯετʔϧ͢Δɻ ʢ࣮ࡍͷεΫϦϓτͰ͸ elasticsearch Ҏ֎ͷηοτΞοϓ΋΍ͬͯΔʣ
47. elasticsearch-ruby Λ࢖ͬͨ setup script ྫ class ES! def self.client! @client ||= Elasticsearch::Client.new({! url: ES.url,! log: true! })! end! end! ! ES.client.indices.create({! index: "index_name",! body: {! settings: {! analysis: {! filter: {! ...! },! tokenizer: {! ...! },! analyzer: {! ...! }! }! }! }! })! ! ES.client.indices.put_mapping({! index: “index_name”,! type: 'user',! body: {! company: {! _id: { path: 'id' },! _timestamp: { enabled: true, path: 'updated_at', store: true },! _source: { enabled: true },! _all: { enabled: true, analyzer: 'kuromoji_analyzer' },! properties: {! name: { type: 'string', store: true, index: 'analyzed', analyzer: 'kuromoji_analyzer' },! . . .! }! }! }! })! ! ! User.find_in_batches do |users|! body = users.map do |user|! {! index: {! _index: "index_name",! _type: 'user',! _id: user.id,! data: user.elasticsearch_data,! }! }! end! ES.client.bulk body: body! end!
48. εΩʔϚมߋ࣌ʹαʔϏεʹӨڹΛ༩͑ͳ͍Α͏ɺ Ͳ͏ΫϥελΛߋ৽ͯ͠Δ͔ w #MVF(SFFO
    %FQMPZNFOU
    Έ͍ͨͳ͜ͱ ʢ࣮ࡍ͸
    
    ৭ʣΛ΍͍ͬͯΔ
49. blue-es.wantedly.com blue-es-elb Nginx + ES Nginx + ES blue-es-cluster green-es.wantedly.com green-es-elb Nginx + ES Nginx + ES green-es-cluster چεΩʔϚ ৽εΩʔϚ
50. &4
    ͷ
    #MVF(SFFO
    %FQMPZNFOU w CMVFFTFMC
    HSFFOFTFMC
    ͷΑ͏ʹ
    &-#
    αϒυϝΠϯΛ
    
    ͭ༻ҙ͓ͯ͘͠
    w $IFG
    Ͱ
    lFMBTUJDTFBSDI
    SPMF
    lHSFFOz
    FOWJSPONFOU
    ͱͯ͠
    &4
    αʔόΛߏ ங͢ΔͱɺΫϥελϦϯάɺ&-# HSFFOFTFMC
    ΁ͷొ࿥·ͰࣗಈͰηοτ Ξοϓ͞ΕΔΑ͏ʹͯ͠Δɻ
    ෼͘Β͍ɻ
    w ࣾ಺ϝϯόʔͷΈΞΫηεՄೳͳ؀ڥʹ࠷৽ͷ
    3BJMT
    ΞϓϦʢ&4
    ͷ
    .BQQJOH
    ΛؚΉʣΛσϓϩΠޙɺ͔ͦ͜Β
    HSFFOFTDMVTUFS
    ʹ৽εΩʔϚ ͷΠϯσοΫε࡞੒ɺυΩϡϝϯτ௥ՃΛߦ͏
    w Ϋϥελʹσʔλ͕ೖͬͨΒɺຊ൪؀ڥͷ
    3BJMT
    ΋࠷৽ʹͨ͠ޙɺ3BJMT
    ͔Β ར༻͢Δ
    &4
    ͷΤϯυϙΠϯτʢ؀ڥม਺Ͱઃఆ͍ͯ͠ΔʣΛ੾Γସ͑ͯσϓ ϩΠ׬ྃ
51. ࠓޙͷ՝୊
52. ࠓޙͷ՝୊ w ύϑΥʔϚϯεͷ޲্
    w ϢʔβʔݕࡧͷϋΠϥΠτ
    w ࣙॻ࢖͏
    w ݕࡧਫ਼౓
53. ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ