Elasticsearch at Wantedly

Transcript

1. Elasticsearch at

2. SEIGO UCHIDA

3. !TQFTOPWB w 
   ೥
   :BIPP
   +"1"/
   ৽ଔೖࣾ
   w 
   ೥
   
   ݄
   8BOUFEMZ
   ʹδϣΠϯ
   w ޿ٛͷΠϯϑϥΛ୲౰
   w $IFG
   ͱ
   )VCPU
   ͱ͔͕޷͖

4. !TQFTOPWB

5. None

6. γΰτͰίίϩΦυϧਓΛ;΍͢

7. None
8. None

9. 8BOUFEMZ
   *OD w 
   ೥
   
   ݄ઃཱ
   w ΞϧόΠτɺΠϯλʔϯؚΊ
   
   ໊
   w 
   ໊ͷΤϯδχΞ͕׆༂
   w γΰτॆ࣮ԽΞϓϦ΍πʔϧͷఏڙ
   

   w 
   ࣾɺ
   ਓ͕࢖͏

10. ੵۃ࠾༻த https://www.wantedly.com/projects/5106

11. Wantedly
    Ͱ
    ES
    Λ
    ͲͷΑ͏ʹ࢖͍ͬͯΔͷ͔

12. ݕࡧର৅ w γΰτ
    w ձࣾ
    w Ϣʔβ

13. γΰτ w ձ໊ࣾ
    w ۈ຿஍
    w ࣾһ਺

14. None

15. ืू w ืू໊
    w ืूৄࡉ
    w ืू৬छ

16. None

17. Ϣʔβ w Ϣʔβ໊
    w ࣗݾ঺հจ
    w ͦͷଞ͍Ζ͍Ζʢޙड़ʣ

18. None

19. ΠϯσοΫε w ௨ৗͷݕࡧ༻ΠϯσοΫε
    w "VUPDPNQMFUF
    ༻ΠϯσοΫε

20. None

21. λΠϓ w γΰτλΠϓ
    w ձࣾλΠϓ
    w ϢʔβʔλΠϓ

22. ௨ৗͷݕࡧ༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿
    
    ສ
    w ϓϥΠϚϦαΠζɿ໿
    (#

23. "VUPDPNQMFUF༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿
    
    ສ
    w ϓϥΠϚϦαΠζɿ໿
    (#

24. ߏ੒ w 3BJMT
    PO
    )FSPLV
    w &MBTUJDTFBSDI
    PO
    &$
    w 1PTUHSFTRM
    PO
    3%4

25. FMBTUJDTFBSDISVCZ w UJSF
    ͸ϝϯς͕ऴྃ
    w TFBSDILJDL
    ͸
    UJSF
    ʹґଘ
    w ެࣜ
    HFN
    Λ࢖͏ͷ͕༨ܭͳ੍ݶ͕ͳ ͯ͘ྑ͍

26. Ϋϥελ w &$
    NTNBMM
    º
    
    ୆
    w $IFG
    Ͱ؅ཧ
    w +7.
    ׂ౰.&.ɿ౥ࡌ.&. 

27. ར༻ϓϥάΠϯ w )FBE
    w )2ʢ/PEF
    %JBHOPTUJDT
    ΍
    4UBUT
    ֬ೝʹར༻ʣ
    w .BSWFM
    TFOTF
    ͰΫΤϦ࡞੒ʹར༻ʣ ։ൃ؀ڥ ͷΈʣ
    

    w ,VSPNPKJ
    w *ORVJTJUPSʢ"OBMZ[FS
    ͷڍಈ֬ೝʣ

28. Ͳ͏΍ֶͬͯΜ͔ͩ

29. None

30. ެࣜϦϑΝϨϯε w ඞཁͳ͜ͱ͸ॻ͍ͯ͋Δɺؾ͕͢Δ
    w ΫΤϦͷྫ͕෦෼తʹॻ͔Ε͍ͯͯࠔΔ
    w ݕࡧॳ৺ऀʹ͸શମ૾͕͔ͭΈͮΒ͍
    w Ͱ΋ಡΜͩ

31. None

32. &MBTUJDTFBSDI
    8PSLTIPQ w શମ૾Λ௫Έͭͭɺ࣮ࡍʹ৮ΔͷͰΠ ϝʔδ͕௫Ίͨ
    w IUUQTTQFBLFSEFDLDPNEBEPPOFUFMBTUJDTFBSDIXPSLTIPQ

33. None

34. &MBTUJDTFBSDI
    νϡʔτϦΞϧ w ༻ޠɺશମ૾͕௫Έ΍͍͢
    w ίϯςΩετ͕͋Γ࣮ફత
    w IUUQDPEFIBUFOBCMPHDPNFOUSZ

35. ͜ͷຊ͕΋ͬͱૣ͘ग़ͯΕ͹ʜ

36. ۤ࿑ͨ͠ͱ͜Ζ

37. Ϣʔβʔݕࡧ w ϓϩϑΟʔϧ߲໨͕ଟ͍
    w ֤߲໨ຖʹϓϥΠόγʔϨϕϧ ͕͋Δ
    w NBQQJOH
    ΋
    RVFSZ
    ΋ෳࡶʹ

38. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ྫ͑͹ɺϓϩϑΟʔϧͷʮ໊લʯʮॴଐʯʮॴ ࡏ஍ʯΛجຊ৘ใͷηοτͱͯ͠ϓϥΠόγʔ ϨϕϧʮҰൠެ։ʯͱ͍ͯͨ͠ͱ͢Δ
    w

    ࠷ॳ͸
    /FTUFE
    EPDVNFOU
    Λ࢖ͬͯදݱ͠ ͍ͯͨ

39. "properties": {! "basic_privacy": {! "type": "nested"! },! "properties": {! "name":

    {! "type": "string"! },! "working_for": {! "type": "string"! },! "location": {! "type": "string"! },! "privacy_level": {! "type": "string"! }! }! }! mapping

40. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ͔͠͠ɺ/FTUFE
    ͩͱͰ͖ͳ͍͜ͱʢϋΠϥΠτͳͲʣ ͕ग़ͯ͘ΔͷͰϑϥοτͳߏ଄ͰɺϓϥΠόγʔϨϕ ϧΛදݱ
    w

    ܕ@σʔληοτ@@ϓϩύςΟ
    ͷΑ͏ʹϚοϐ ϯά
    w ࠷ॳͷܕ͕ඞཁͳͷ͸ɺEBUF
    ΍
    JOUFHFS
    ͳͲ
    TUSJOH
    Ҏ֎ʹ
    TJNQMF@RVFSZ
    Λ࢖͏ͱ
    FSSPS
    ͱͳΔͨ Ί

41. mapping query {! "query": {! "simple_query_string": {! "query": <QUERY>,! "fields":

    ["s_basic__*"],! "default_operator": "and"! }! }! },! "filter": {! "term": {! "basic_privacy": <PRIVACY_LEVEL>! }! }! }! "properties": {! "basic_privacy": {! "type": "string"! },! "s_basic__name": {! "type": "string"! },! "s_basic__working_for": {! "type": "string"! },! "s_basic__location": {! "type": "string"! }! }!

42. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ݁ՌతʹɺOFTUFE
    Λ࢖Θ͟ΔΛಘͳ͍৔໘ ͕ग़ͯདྷͯϑϥοτͳϓϩύςΟͱ
    OFTUFE
    Λซ༻͍ͯ͠Δ͕ɺࠓճ͸ͦ͜͸লུ

43. ࣭໰λΠϜ

44. 4PMS
    ΍
    $MPVETFBSDI
    ΋ධՁ্ͨ͠Ͱ
    
    &4
    ʹͨ͠ͷ͔ w ʮ༑ୡͱͷܨ͕ΓʯΛՃຯͯ͠ݕࡧͰ͖Δ͔ͱ͍͏͜ͱͰɺάϥϑ
    %#
    ͷ
    /FPK
    ͱҰൠతͳݕࡧͱͯ͠
    &4
    Λൺֱͯ͠ɺ
    &4
    ͷํ͕அ વύϑΥʔϚϯε͕ྑ͔ͬͨ
    w 4PMS
    ʹ͍ͭͯ͸
    4PMS
    Ϣʔβ͔Βࠓ͔Β࢖͏ͳΒ
    &4
    ͱ͍͏࿩Λฉ͍ ͍ͯͨ
    w

    ར༻Ϣʔβʢ(JU)VC
    GPVSTRVBSF
    4PVOE$MPVE
    ͳͲʣ΋ࢀߟʹ͠ ͨ
    w ݁Ռతʹɺ4PMS
    ʹͳ͍
    OFTUFE
    QBSFOUDIJME
    EPDVNFOU
    Λ࢖ͬͯ ͍ΔͷͰ
    &4
    Ͱྑ͔ͬͨ

45. ։ൃ؀ڥͰ
    &4
    Λ࢖͏ͱ
    
    NBQQJOH
    ͳͲόϥόϥͰਏ͘ͳΒͳ͍͔ w #PPUTUSBQ
    εΫϦϓτͱ͍͏΋ͷΛ༻ҙͯ͠
    HJU
    DMPOF
    ޙ ͷηοτΞοϓʢMPDBM
    ʹ
    &4
    ΛೖΕΔͳͲʣΛࣗಈԽ͠ ͍ͯΔ
    w .BQQJOH
    ΍υΩϡϝϯτͷ௥Ճ΋ηοτΞοϓεΫϦϓ τΛ༻ҙͯ͠ɺϝϯόʔ͝ͱͷ؀ڥͷࠩ෼͕ग़ͳ͍Α͏ʹ ͍ͯ͠Δ

46. ! #! # Check for Elasticsearch! #! if test !

    $(which elasticsearch)! then! echo " x You need to install ElasticSearch:"! echo " + Installing Elasticsearch..."! brew install elasticsearch! else! echo " + ElasticSearch found."! fi! ! #! # Check for Elasticsearch version! #! ES_VERSION=$(elasticsearch -v | cut -d' ' -f2 | cut -d'.' -f1)! if test ! $ES_VERSION -eq 1! then! echo " x You need to upgrade elasticsearch."! brew upgrade elasticsearch! else! echo " + Your Elasticsearch version is correct."! fi! ! if test -d /usr/local/var/lib/elasticsearch/plugins/marvel! then! echo " + Elasticsearch Plugin Marvel found."! else! echo " + Installing Elasticsearch Plugins Marvel"! plugin --install elasticsearch/marvel/latest! fi! bootstrap script Ұ෦ൈਮ $ script/bootstrap! ! Wantedly! ! γΰτͰίίϩΦυϧਓΛ;΍͢! ! + Your Elasticsearch version is correct.! + ElasticSearch Plugin HQ found.! + ElasticSearch Plugin head found.! + ElasticSearch Plugin Kuromoji found.! + ElasticSearch Plugin Inquisitor found.! + ElasticSearch Plugin Marvel found.! ! Good work. We're ready.! ! Run 'elasticsearch' to start elasticsearch.! And you can see elasticsearch webui plugins 'locahost:9200/_plugin/head/'! ࣮ߦ݁Ռ γϯϓϧͳγΣϧεΫϦϓτɻelasticsearch ΍ϓϥάΠϯ͕Πϯετʔϧ͞ΕͯΔ ͔ΛνΣοΫͯ͠͞Εͯͳ͍ͳΒΠϯετʔϧ͢Δɻ ʢ࣮ࡍͷεΫϦϓτͰ͸ elasticsearch Ҏ֎ͷηοτΞοϓ΋΍ͬͯΔʣ

47. elasticsearch-ruby Λ࢖ͬͨ setup script ྫ class ES! def self.client! @client

    ||= Elasticsearch::Client.new({! url: ES.url,! log: true! })! end! end! ! ES.client.indices.create({! index: "index_name",! body: {! settings: {! analysis: {! filter: {! ...! },! tokenizer: {! ...! },! analyzer: {! ...! }! }! }! }! })! ! ES.client.indices.put_mapping({! index: “index_name”,! type: 'user',! body: {! company: {! _id: { path: 'id' },! _timestamp: { enabled: true, path: 'updated_at', store: true },! _source: { enabled: true },! _all: { enabled: true, analyzer: 'kuromoji_analyzer' },! properties: {! name: { type: 'string', store: true, index: 'analyzed', analyzer: 'kuromoji_analyzer' },! . . .! }! }! }! })! ! ! User.find_in_batches do |users|! body = users.map do |user|! {! index: {! _index: "index_name",! _type: 'user',! _id: user.id,! data: user.elasticsearch_data,! }! }! end! ES.client.bulk body: body! end!

48. εΩʔϚมߋ࣌ʹαʔϏεʹӨڹΛ༩͑ͳ͍Α͏ɺ Ͳ͏ΫϥελΛߋ৽ͯ͠Δ͔ w #MVF(SFFO
    %FQMPZNFOU
    Έ͍ͨͳ͜ͱ ʢ࣮ࡍ͸
    
    ৭ʣΛ΍͍ͬͯΔ

49. blue-es.wantedly.com blue-es-elb Nginx + ES Nginx + ES blue-es-cluster green-es.wantedly.com

    green-es-elb Nginx + ES Nginx + ES green-es-cluster چεΩʔϚ ৽εΩʔϚ

50. &4
    ͷ
    #MVF(SFFO
    %FQMPZNFOU w CMVFFTFMC
    HSFFOFTFMC
    ͷΑ͏ʹ
    &-#
    αϒυϝΠϯΛ
    
    ͭ༻ҙ͓ͯ͘͠
    w $IFG
    Ͱ
    lFMBTUJDTFBSDI
    SPMF
    lHSFFOz
    FOWJSPONFOU
    ͱͯ͠
    &4
    αʔόΛߏ ங͢ΔͱɺΫϥελϦϯάɺ&-# HSFFOFTFMC

    
    ΁ͷొ࿥·ͰࣗಈͰηοτ Ξοϓ͞ΕΔΑ͏ʹͯ͠Δɻ
    ෼͘Β͍ɻ
    w ࣾ಺ϝϯόʔͷΈΞΫηεՄೳͳ؀ڥʹ࠷৽ͷ
    3BJMT
    ΞϓϦʢ&4
    ͷ
    .BQQJOH
    ΛؚΉʣΛσϓϩΠޙɺ͔ͦ͜Β
    HSFFOFTDMVTUFS
    ʹ৽εΩʔϚ ͷΠϯσοΫε࡞੒ɺυΩϡϝϯτ௥ՃΛߦ͏
    w Ϋϥελʹσʔλ͕ೖͬͨΒɺຊ൪؀ڥͷ
    3BJMT
    ΋࠷৽ʹͨ͠ޙɺ3BJMT
    ͔Β ར༻͢Δ
    &4
    ͷΤϯυϙΠϯτʢ؀ڥม਺Ͱઃఆ͍ͯ͠ΔʣΛ੾Γସ͑ͯσϓ ϩΠ׬ྃ

51. ࠓޙͷ՝୊

52. ࠓޙͷ՝୊ w ύϑΥʔϚϯεͷ޲্
    w ϢʔβʔݕࡧͷϋΠϥΠτ
    w ࣙॻ࢖͏
    w ݕࡧਫ਼౓

53. ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ