Elasticsearch at Wantedly

Elasticsearch at Wantedly

2014/04/21 第 4 回 Elasticsearch 勉強会での発表資料。
当日の質問タイム及び懇親会で聞かれたことを追加しています。

32f2e5ddb187baa2abac66d7e8b283fe?s=128

Seigo Uchida

April 22, 2014
Tweet

Transcript

  1. Elasticsearch at

  2. SEIGO UCHIDA

  3. !TQFTOPWB w ೥:BIPP+"1"/৽ଔೖࣾ w ೥݄8BOUFEMZʹδϣΠϯ w ޿ٛͷΠϯϑϥΛ୲౰ w $IFGͱ)VCPUͱ͔͕޷͖

  4. !TQFTOPWB

  5. None
  6. γΰτͰίίϩΦυϧਓΛ;΍͢

  7. None
  8. None
  9. 8BOUFEMZ *OD w ೥݄ઃཱ w ΞϧόΠτɺΠϯλʔϯؚΊ໊ w ໊ͷΤϯδχΞ͕׆༂ w γΰτॆ࣮ԽΞϓϦ΍πʔϧͷఏڙ

    w ࣾɺਓ͕࢖͏
  10. ੵۃ࠾༻த https://www.wantedly.com/projects/5106

  11. WantedlyͰESΛ ͲͷΑ͏ʹ࢖͍ͬͯΔͷ͔

  12. ݕࡧର৅ w γΰτ w ձࣾ w Ϣʔβ

  13. γΰτ w ձ໊ࣾ w ۈ຿஍ w ࣾһ਺

  14. None
  15. ืू w ืू໊ w ืूৄࡉ w ืू৬छ

  16. None
  17. Ϣʔβ w Ϣʔβ໊ w ࣗݾ঺հจ w ͦͷଞ͍Ζ͍Ζʢޙड़ʣ

  18. None
  19. ΠϯσοΫε w ௨ৗͷݕࡧ༻ΠϯσοΫε w "VUPDPNQMFUF༻ΠϯσοΫε

  20. None
  21. λΠϓ w γΰτλΠϓ w ձࣾλΠϓ w ϢʔβʔλΠϓ

  22. ௨ৗͷݕࡧ༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿ສ w ϓϥΠϚϦαΠζɿ໿(#

  23. "VUPDPNQMFUF༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿ສ w ϓϥΠϚϦαΠζɿ໿(#

  24. ߏ੒ w 3BJMTPO)FSPLV w &MBTUJDTFBSDIPO&$ w 1PTUHSFTRMPO3%4

  25. FMBTUJDTFBSDISVCZ w UJSF͸ϝϯς͕ऴྃ w TFBSDILJDL͸UJSFʹґଘ w ެࣜHFNΛ࢖͏ͷ͕༨ܭͳ੍ݶ͕ͳ ͯ͘ྑ͍

  26. Ϋϥελ w &$NTNBMMº୆ w $IFGͰ؅ཧ w +7.ׂ౰.&.ɿ౥ࡌ.&. 

  27. ར༻ϓϥάΠϯ w )FBE w )2ʢ/PEF%JBHOPTUJDT΍4UBUT֬ೝʹར༻ʣ w .BSWFM TFOTFͰΫΤϦ࡞੒ʹར༻ʣ ։ൃ؀ڥ ͷΈʣ

    w ,VSPNPKJ w *ORVJTJUPSʢ"OBMZ[FSͷڍಈ֬ೝʣ
  28. Ͳ͏΍ֶͬͯΜ͔ͩ

  29. None
  30. ެࣜϦϑΝϨϯε w ඞཁͳ͜ͱ͸ॻ͍ͯ͋Δɺؾ͕͢Δ w ΫΤϦͷྫ͕෦෼తʹॻ͔Ε͍ͯͯࠔΔ w ݕࡧॳ৺ऀʹ͸શମ૾͕͔ͭΈͮΒ͍ w Ͱ΋ಡΜͩ

  31. None
  32. &MBTUJDTFBSDI8PSLTIPQ w શମ૾Λ௫Έͭͭɺ࣮ࡍʹ৮ΔͷͰΠ ϝʔδ͕௫Ίͨ w IUUQTTQFBLFSEFDLDPNEBEPPOFUFMBTUJDTFBSDIXPSLTIPQ

  33. None
  34. &MBTUJDTFBSDIνϡʔτϦΞϧ w ༻ޠɺશମ૾͕௫Έ΍͍͢ w ίϯςΩετ͕͋Γ࣮ફత w IUUQDPEFIBUFOBCMPHDPNFOUSZ

  35. ͜ͷຊ͕΋ͬͱૣ͘ग़ͯΕ͹ʜ

  36. ۤ࿑ͨ͠ͱ͜Ζ

  37. Ϣʔβʔݕࡧ w ϓϩϑΟʔϧ߲໨͕ଟ͍ w ֤߲໨ຖʹϓϥΠόγʔϨϕϧ ͕͋Δ w NBQQJOH΋RVFSZ΋ෳࡶʹ

  38. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ྫ͑͹ɺϓϩϑΟʔϧͷʮ໊લʯʮॴଐʯʮॴ ࡏ஍ʯΛجຊ৘ใͷηοτͱͯ͠ϓϥΠόγʔ ϨϕϧʮҰൠެ։ʯͱ͍ͯͨ͠ͱ͢Δ w

    ࠷ॳ͸/FTUFEEPDVNFOUΛ࢖ͬͯදݱ͠ ͍ͯͨ
  39. "properties": {! "basic_privacy": {! "type": "nested"! },! "properties": {! "name":

    {! "type": "string"! },! "working_for": {! "type": "string"! },! "location": {! "type": "string"! },! "privacy_level": {! "type": "string"! }! }! }! mapping
  40. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ͔͠͠ɺ/FTUFEͩͱͰ͖ͳ͍͜ͱʢϋΠϥΠτͳͲʣ ͕ग़ͯ͘ΔͷͰϑϥοτͳߏ଄ͰɺϓϥΠόγʔϨϕ ϧΛදݱ w

    ܕ@σʔληοτ@@ϓϩύςΟͷΑ͏ʹϚοϐ ϯά w ࠷ॳͷܕ͕ඞཁͳͷ͸ɺEBUF΍JOUFHFSͳͲ TUSJOHҎ֎ʹTJNQMF@RVFSZΛ࢖͏ͱFSSPSͱͳΔͨ Ί
  41. mapping query {! "query": {! "simple_query_string": {! "query": <QUERY>,! "fields":

    ["s_basic__*"],! "default_operator": "and"! }! }! },! "filter": {! "term": {! "basic_privacy": <PRIVACY_LEVEL>! }! }! }! "properties": {! "basic_privacy": {! "type": "string"! },! "s_basic__name": {! "type": "string"! },! "s_basic__working_for": {! "type": "string"! },! "s_basic__location": {! "type": "string"! }! }!
  42. Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ݁ՌతʹɺOFTUFEΛ࢖Θ͟ΔΛಘͳ͍৔໘ ͕ग़ͯདྷͯϑϥοτͳϓϩύςΟͱOFTUFE Λซ༻͍ͯ͠Δ͕ɺࠓճ͸ͦ͜͸লུ

  43. ࣭໰λΠϜ

  44. 4PMS΍$MPVETFBSDI΋ධՁ্ͨ͠Ͱ &4ʹͨ͠ͷ͔ w ʮ༑ୡͱͷܨ͕ΓʯΛՃຯͯ͠ݕࡧͰ͖Δ͔ͱ͍͏͜ͱͰɺάϥϑ %#ͷ/FPKͱҰൠతͳݕࡧͱͯ͠&4Λൺֱͯ͠ɺ&4ͷํ͕அ વύϑΥʔϚϯε͕ྑ͔ͬͨ w 4PMSʹ͍ͭͯ͸4PMSϢʔβ͔Βࠓ͔Β࢖͏ͳΒ&4ͱ͍͏࿩Λฉ͍ ͍ͯͨ w

    ར༻Ϣʔβʢ(JU)VC GPVSTRVBSF 4PVOE$MPVEͳͲʣ΋ࢀߟʹ͠ ͨ w ݁Ռతʹɺ4PMSʹͳ͍OFTUFE QBSFOUDIJMEEPDVNFOUΛ࢖ͬͯ ͍ΔͷͰ&4Ͱྑ͔ͬͨ
  45. ։ൃ؀ڥͰ&4Λ࢖͏ͱ NBQQJOHͳͲόϥόϥͰਏ͘ͳΒͳ͍͔ w #PPUTUSBQεΫϦϓτͱ͍͏΋ͷΛ༻ҙͯ͠HJUDMPOFޙ ͷηοτΞοϓʢMPDBMʹ&4ΛೖΕΔͳͲʣΛࣗಈԽ͠ ͍ͯΔ w .BQQJOH΍υΩϡϝϯτͷ௥Ճ΋ηοτΞοϓεΫϦϓ τΛ༻ҙͯ͠ɺϝϯόʔ͝ͱͷ؀ڥͷࠩ෼͕ग़ͳ͍Α͏ʹ ͍ͯ͠Δ

  46. ! #! # Check for Elasticsearch! #! if test !

    $(which elasticsearch)! then! echo " x You need to install ElasticSearch:"! echo " + Installing Elasticsearch..."! brew install elasticsearch! else! echo " + ElasticSearch found."! fi! ! #! # Check for Elasticsearch version! #! ES_VERSION=$(elasticsearch -v | cut -d' ' -f2 | cut -d'.' -f1)! if test ! $ES_VERSION -eq 1! then! echo " x You need to upgrade elasticsearch."! brew upgrade elasticsearch! else! echo " + Your Elasticsearch version is correct."! fi! ! if test -d /usr/local/var/lib/elasticsearch/plugins/marvel! then! echo " + Elasticsearch Plugin Marvel found."! else! echo " + Installing Elasticsearch Plugins Marvel"! plugin --install elasticsearch/marvel/latest! fi! bootstrap script Ұ෦ൈਮ $ script/bootstrap! ! Wantedly! ! γΰτͰίίϩΦυϧਓΛ;΍͢! ! + Your Elasticsearch version is correct.! + ElasticSearch Plugin HQ found.! + ElasticSearch Plugin head found.! + ElasticSearch Plugin Kuromoji found.! + ElasticSearch Plugin Inquisitor found.! + ElasticSearch Plugin Marvel found.! ! Good work. We're ready.! ! Run 'elasticsearch' to start elasticsearch.! And you can see elasticsearch webui plugins 'locahost:9200/_plugin/head/'! ࣮ߦ݁Ռ γϯϓϧͳγΣϧεΫϦϓτɻelasticsearch ΍ϓϥάΠϯ͕Πϯετʔϧ͞ΕͯΔ ͔ΛνΣοΫͯ͠͞Εͯͳ͍ͳΒΠϯετʔϧ͢Δɻ ʢ࣮ࡍͷεΫϦϓτͰ͸ elasticsearch Ҏ֎ͷηοτΞοϓ΋΍ͬͯΔʣ
  47. elasticsearch-ruby Λ࢖ͬͨ setup script ྫ class ES! def self.client! @client

    ||= Elasticsearch::Client.new({! url: ES.url,! log: true! })! end! end! ! ES.client.indices.create({! index: "index_name",! body: {! settings: {! analysis: {! filter: {! ...! },! tokenizer: {! ...! },! analyzer: {! ...! }! }! }! }! })! ! ES.client.indices.put_mapping({! index: “index_name”,! type: 'user',! body: {! company: {! _id: { path: 'id' },! _timestamp: { enabled: true, path: 'updated_at', store: true },! _source: { enabled: true },! _all: { enabled: true, analyzer: 'kuromoji_analyzer' },! properties: {! name: { type: 'string', store: true, index: 'analyzed', analyzer: 'kuromoji_analyzer' },! . . .! }! }! }! })! ! ! User.find_in_batches do |users|! body = users.map do |user|! {! index: {! _index: "index_name",! _type: 'user',! _id: user.id,! data: user.elasticsearch_data,! }! }! end! ES.client.bulk body: body! end!
  48. εΩʔϚมߋ࣌ʹαʔϏεʹӨڹΛ༩͑ͳ͍Α͏ɺ Ͳ͏ΫϥελΛߋ৽ͯ͠Δ͔ w #MVF(SFFO%FQMPZNFOUΈ͍ͨͳ͜ͱ ʢ࣮ࡍ͸৭ʣΛ΍͍ͬͯΔ

  49. blue-es.wantedly.com blue-es-elb Nginx + ES Nginx + ES blue-es-cluster green-es.wantedly.com

    green-es-elb Nginx + ES Nginx + ES green-es-cluster چεΩʔϚ ৽εΩʔϚ
  50. &4ͷ#MVF(SFFO%FQMPZNFOU w CMVFFTFMC HSFFOFTFMCͷΑ͏ʹ&-# αϒυϝΠϯΛͭ༻ҙ͓ͯ͘͠ w $IFGͰlFMBTUJDTFBSDISPMF lHSFFOzFOWJSPONFOUͱͯ͠&4αʔόΛߏ ங͢ΔͱɺΫϥελϦϯάɺ&-# HSFFOFTFMC

    ΁ͷొ࿥·ͰࣗಈͰηοτ Ξοϓ͞ΕΔΑ͏ʹͯ͠Δɻ෼͘Β͍ɻ w ࣾ಺ϝϯόʔͷΈΞΫηεՄೳͳ؀ڥʹ࠷৽ͷ3BJMTΞϓϦʢ&4ͷ .BQQJOHΛؚΉʣΛσϓϩΠޙɺ͔ͦ͜ΒHSFFOFTDMVTUFSʹ৽εΩʔϚ ͷΠϯσοΫε࡞੒ɺυΩϡϝϯτ௥ՃΛߦ͏ w Ϋϥελʹσʔλ͕ೖͬͨΒɺຊ൪؀ڥͷ3BJMT΋࠷৽ʹͨ͠ޙɺ3BJMT͔Β ར༻͢Δ&4ͷΤϯυϙΠϯτʢ؀ڥม਺Ͱઃఆ͍ͯ͠ΔʣΛ੾Γସ͑ͯσϓ ϩΠ׬ྃ
  51. ࠓޙͷ՝୊

  52. ࠓޙͷ՝୊ w ύϑΥʔϚϯεͷ޲্ w ϢʔβʔݕࡧͷϋΠϥΠτ w ࣙॻ࢖͏ w ݕࡧਫ਼౓

  53. ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ