niconicoの検索を支えるElasticsearch / niconico Elasticsearch

6e4c7cb36e31e3b871f3ae0c83ac8110?s=47 shoito
November 18, 2014

niconicoの検索を支えるElasticsearch / niconico Elasticsearch

niconicoでのElasticsearch運用事例的なお話。
第7回elasticsearch勉強会 #elasticsearch #elasticsearchjp elasticsearch.doorkeeper.jp/events/16837

報告ブログ記事
http://ch.nicovideo.jp/dwango-engineer/blomaga/ar674946

運用事例や構成/設定などの情報を参考に伺いたいので @shoito / fb:shoito に是非お声かけを。

6e4c7cb36e31e3b871f3ae0c83ac8110?s=128

shoito

November 18, 2014
Tweet

Transcript

  1. OJDPOJDPͷݕࡧΛࢧ͑Δ
 &MBTUJDTFBSDI ҏ౻঵ ג υϫϯΰ 2014/11/18

  2. ҏ౻঵!TIPJUP 4PGUXBSF&OHJOFFS %XBOHP

  3. "HFOEB ❖౷߹ݕࡧج൫Λ࡞ͬͨཧ༝ ❖ͳͥ&MBTUJDTFBSDIͳͷ͔ʁ ❖Ͳ͏&MBTUJDTFBSDIΛ࢖͍ͬͯΔͷ͔ʁ ❖Ͳ͏&MBTUJDTFBSDIΛӡ༻͍ͯ͠Δͷ͔ʁ ❖'"2

  4. ג υϫϯΰ೥݄ظ௨ظܾࢉࢿྉΑΓ

  5. ౷߹ݕࡧج൫Λ࡞ͬͨཧ༝ ⿣ ݕࡧΛ௨ͨ͡OJDPOJDPશαʔϏεͷ17ɺ଺ࡏ࣌ؒͷ޲্ ⿣ ϦΞϧλΠϜݕࡧ ߴ͍Ԡ౴ੑೳɾߋ৽൓өස౓ Λ࣮ݱ͠ɺ ͞Βʹকདྷతʹ৽͍͠ݕࡧϨίϝϯυٕज़ͳͲ΁ͷରԠ ⿣ αʔϏεຖʹߏங͞Ε͍ͯΔݕࡧΠϯϑϥΛҰຊԽ͠ɺ


    ޮ཰Խ ˞ݱ࣌఺Ͱ͸
 ɹ͍͔ͭ͘ͷαʔϏε͕͜ͷج൫Λར༻͍ͯ͠Δͱ͍͏
 ɹεςʔλεͰ͋ΓɺશαʔϏε͕࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍
  6. ͳͥ&MBTUJDTFBSDIͳͷ͔ʁ

  7. ౷߹ݕࡧج൫ͷੑೳཁ݅ ⿣ χίಈͷϐʔΫλΠϜͷ214ͰϦΫΤετड͚ͭͭɺ
 ݕࡧ݁ՌΛඵҎ಺ʹฦͤΔ͜ͱ ⿣ ίϯςϯπ৘ใͷߋ৽͔Β෼Ҏ಺ʹݕࡧ݁Ռʹ
 ൓өͰ͖Δ͜ͱ ⿣ OJDPOJDPશαʔϏεͷݕࡧج൫ͱͳΓಘΔ͜ͱ 4DBMF0VU

    ;FSP EPXOUJNF NJHSBUJPO 1MVHHBCMF
  8. ౷߹ݕࡧج൫ͷੑೳཁ݅ ⿣ χίಈͷϐʔΫλΠϜͷ214ͰϦΫΤετड͚ͭͭɺ
 ݕࡧ݁ՌΛඵҎ಺ʹฦͤΔ͜ͱ ⿣ ίϯςϯπ৘ใͷߋ৽͔Β෼Ҏ಺ʹݕࡧ݁Ռʹ
 ൓өͰ͖Δ͜ͱ ⿣ OJDPOJDPશαʔϏεͷݕࡧج൫ͱͳΓಘΔ͜ͱ 13+։࢝౰࣌ͷ4PMSͰ͸


    ཁ݅Λຬͨͤͦ͏ʹͳ͔ͬͨ
  9. Ͳ͏&MBTUJDTFBSDIΛ࢖͍ͬͯΔͷ͔ʁ

  10. Ͳ͏&MBTUJDTFBSDIΛ࢖͍ͬͯΔͷ͔ʁ ⿣෼ࢄܕશจݕࡧΤϯδϯͱͯ͠ ⿣ϩάղੳͱՄࢹԽπʔϧͱͯ͠,JCBOB ⿣αδΣετݕࡧͷࣙॻੜ੒༻όοΫΤϯυͱͯ͠

  11. ݕࡧର৅͸Կ͔ʁ ⿣OJDPOJDPͷίϯςϯπ ˞ݱࡏ͸ݕࡧ݁ՌʹදࣔͰ͖Δίϯςϯπ͚ͩ  ⿣χίχίಈը໿   ⿣χίχίੜ์ૹ໿  ⿣χίχί੩ը໿

      ⿣χίχίνϟϯωϧ χίχίχϡʔε  ⿣ ·ͩ·ͩର৅͸૿͑Δ ˞ΠϕϯτͷൃੜຖʹΠϯσοΫεߋ৽
  12. ΞʔΩςΫνϟ ͍ͩͿϥϑʹ RabbitMQ Cluster MySQL Master/Slave Elasticsearch Cluster Indexer &


    Converter API front MessageReceiver niconico
 services niconico
 services ࢀরܥ ߋ৽ܥ Op module ids query contents on-premises
  13. ϊʔυΠϯσοΫεઃܭ ⿣ ϊʔυͷϩʔϧ෼͚ͳ͠ OPEFNBTUFS OPEFEBUBUSVF 
 
 
 
 ⿣

    αʔϏεΠϯσοΫελΠϓ ϨϓϦΧ ˞Ϛοϐϯάมߋɺ3FJOEFYͳͲͷϝϯςΛαʔϏεຖʹͰ͖ΔΑ͏ʹ Ϋϥελ ϊʔυ ΠϯσοΫε λΠϓ
  14. NBQQJOHT\ JUFN\  ུ  QSPQFSUJFT\ UJUMF\ UZQFTUSJOH  JOEFY@BOBMZ[FSJOEFYJOH"OBMZ[FS

      ུ  ^  EFTDSJQUJPO\^ WJFX@DPVOU\ UZQFJOUFHFS ^  NZMJTU@DPVOU\^  VQMPBE@UJNF\^  DPNNVOJUZ@JE\^ ϊʔυΠϯσοΫεઃܭ ⿣ Ϛοϐϯά͸ϑϥοτ ✤ χίχίಈը WJEFP  ✤ UJUMF ✤ EFTDSJQUJPO ✤ WJFX@DPVOU ✤ NZMJTU@DPVOU ✤ VQMPBE@UJNF ✤ DPNNVOJUZ@JE ✤  ✤ χίχίੜ์ૹ MJWF  ✤ UJUMF ✤ EFTDSJQUJPO ✤ WJFX@DPVOU ✤ DPNNFOU@DPVOU ✤ DPNNVOJUZ@MFWFM ✤ DPNNVOJUZ@JE ✤ 
  15. ϊʔυΠϯσοΫεઃܭ ⿣ ΤΠϦΞεΛશΠϯσοΫεͰར༻ WJEFP@ WJEFP WJEFP@ WJEFP WJEFP@ ϝϯςલ ϝϯςޙ

    WJEFP@ WJEFP WJEFP@ ϝϯςத reindex
 (number_of_replicas͸0ʹɺ
 optimizeޙɺ0͔Β1ʹ͢Δ) Close
  16. $6%ϝοηʔδ $SFBUF 3FBE 6QEBUF %FMFUF

  17. Create { "timestamp": 1415850040, "cudtype": "create", "issuer": "publisher", "service_name": "video",

    "cmsid": "sm9", "values": { "user_id": 1234567, "title": "৽ɾ߽݂ࣉҰ଒ -൥೰ղ์ - Ϩοπΰʔʂӄཅࢣ", "description": “Ϩοπΰʔʂӄཅࢣʢϑϧίʔϥεόʔδϣϯʣ”, "comment_count": 0, “…”, // লུ - શϑΟʔϧυσʔλ͕ૹΒΕΔ "tags": “ӄཅࢣ ެࣜ …” } }
  18. Update { "timestamp": 1415850040, "cudtype": "update", "issuer": "publisher", "service_name": "video",

    "cmsid": "sm9", "values": { “view_counter": 9876 } }
  19. Delete { "timestamp": 1415850040, "cudtype": "delete", "issuer": "publisher", "service_name": "video",

    "cmsid": "sm9", }
  20. CUD CUD CUD CUD CUD CUD CUD CUD CUD χίಈ

    χίੜ etc. MessageReceiver RabbitMQ CUD CUD CUD CUD CUD CUD CUD Elasticsearch MySQL Indexer & Converter αʔϏεڥք MessageReceiver MessageReceiver
  21. ϓϥάΠϯ ⿣ ಺੡ϓϥάΠϯ ⿣ OJDPOJDP༻ͷಠࣗτʔΫφΠβˍϑΟϧλ ⿣ GVODUJPOTDPSFͷܭࢉ༻ϩδοΫ ɹ˞Πϯετʔϧ๨Ε๷ࢭFMBTUJDTFBSDIZNM
 ɹɹQMVHJONBOEBUPSZOJDPBOBMZ[FS OJDPTDSJQU

    ⿣ ར༻ϓϥάΠϯ ⿣ IFBE LPQG CJHEFTL )2 ⿣ .BSWFM ݕূ࣌ʹɻຊ൪؀ڥʹ΋ཉ͍͚͠Ͳ
  22. Ͳ͏&MBTUJDTFBSDIΛӡ༻͍ͯ͠Δͷ͔ʁ

  23. &MBTUJDTFBSDIΫϥελͷ੒௕ 2012 2013 2014 WΫϥελߏங WҠߦ 2015 WҠߦ ϊʔυ਺✖️ WҠߦ

    ϊʔυ਺✖️ WYҠߦ ϊʔυ਺✖️ʁ ˞ΫϥελSFTUBSUʹΑΔ
 ɹఀࢭϝϯς ˞3PMMJOHSFTUBSUʹΑΔ
 ɹແఀࢭϝϯς ˞ΫϥελίϐʔεΠον
 ɹʹΑΔແఀࢭϝϯς ˞3PMMJOHSFTUBSUʹΑΔϝϯς༧ఆ
  24. σϓϩΠ؅ཧπʔϧ ⿣ ಺੡$BQJTUSBOP8FCXSBQQFS ⿣ &MBTUJDTFBSDIEFQMPZTUBSUTUPQSFTUBSU ⿣ ֤छϓϥάΠϯΠϯετʔϧ ⿣ 1VUNBQQJOH ⿣

    ΤΠϦΞε੾Γସ͑ ⿣ ($)FBQ$MBTTIJTUPHSBNEVNQग़ྗ ⿣ ֤छϩά֬ೝ ⿣ 
  25. None
  26. ϞχλϦϯά ⿣ $IFDL$MVTUFS)FBMUICZ;BCCJY
 HSFFOZFMMPXSFE ⿣ 4MPX-PH *OEFYJOH4FBSDIJOH  ⿣ ($MPH

    ZPVOHPME  ⿣ $16.FNPSZ)FBQ ⿣ FUD
  27. ͜Ε·Ͱૺ۰ͨ͠τϥϒϧ ⿣ 4QSJUCSBJO໰୊ ˞NJOJNVN@NBTUFS@OPEFTઃఆࡁΈ  ⿣ ϚελϊʔυͰͷ'VMM($ʹΑΔQJOHUJNFPVU ⿣ ϚελϊʔυͷSFTUBSUͱγϟʔυͷSFSPVUFͰճ෮ 01

    02 03 04 05 06 01 03 04 05 06 02 03 04 05 06 Ϋϥελ ϊʔυ Ϛελʔϊʔυ cluster_v1 cluster_v1 cluster_v1 ϊʔυ01͔ΒݟͨΫϥελ ϊʔυ02͔ΒݟͨΫϥελ ਖ਼ৗঢ়ଶͷΫϥελ
  28. ͜Ε·Ͱૺ۰ͨ͠τϥϒϧ ⿣ ૝ఆ֎ͷΠϯσοΫεEZOBNJDDSFBUJPOPGNBQQJOHT ⿣ ஌Β͵ؒʹҙਤ͍ͯ͠ͳ͍ΠϯσοΫε͕࡞ΒΕͯ͠·͍ͬͯͨ ⿣ BDUJPOBVUP@DSFBUF@JOEFYGBMTF ⿣ JOEFYNBQQFSEZOBNJDGBMTF ⿣

    ૝ఆ֎ͷϑΟʔϧυEZOBNJDDSFBUJPOPGpFMET ⿣ ϑΟʔϧυ௥Ճ͸Ͱ͖Δ͚Ͳফͤͳ͍ ⿣ ҙਤ͍ͯ͠ͳ͍ϑΟʔϧυΛؚΉ+40/Λ์ΓࠐΜͩ͜ͱ͕ݪҼ ⿣ EZOBNJDTUSJDU ⿣ JOEFYRVFSZQBSTFBMMPX@VONBQQFE@pFMETUSVFGBMTF  ʹ͠ͱ͘ͱ ͞Βʹྑ͍͔΋
  29. ͜Ε·Ͱૺ۰ͨ͠τϥϒϧ ⿣ 0VU0G.FNPSZFYDFQUJPO ⿣ େ͖ΊͷΠϯσοΫεͷϨϓϦΧγϟʔυͷ͕ͭ*OJUJBMJ[JOHͷ··ఀࢭ ⿣ ࢭΉແ͘OVNCFS@PG@SFQMJDBΛˠˠʹͨ͠ޙɺൃੜ ⿣ ϊʔυͷSFTUBSUͰ෮ؼ ⿣

    ң͕௧͘ͳΔτϥϒϧ͸ଞʹ΋
  30. ෛՙܭଌνϡʔχϯά ⿣ ෛՙܭଌπʔϧ(BUMJOH +.FUFS
 IUUQTHJUIVCDPNFMBTUJDTFBSDIFMBTUJDTFBSDIUSFFNBTUFSTSDUFTUSFTPVSDFTKNFUFS  ⿣ νϡʔχϯά ୅දతͳ΋ͷ 

    ⿣ Ϋϥελ ⿣ EJTDPWFSZ[FONJOJNVN@NBTUFS@OPEFT ⿣ EJTDPWFSZ[FOGEQJOH@JOUFSWBM UJNFPVU SFUSJFT ⿣ ΠϯσΫγϯά ⿣ SFGSFTI@JOUFSWBMTT ⿣ @BMM\FOBCMFEGBMTF^ ⿣ ݕࡧ ⿣ @SPVUJOH\QBUIHSPVQ@JE^ ⿣ "HHSFHBUJPO༻ʹCVDLFUT୯Ґʹγϟʔυ͕෼͔ΕΔΑ͏ϧʔςΟϯά
  31. 5IBOLT2VFUJPOT

  32. ྑ͘ݟΔ৘ใݯ ⿣ &MBTUJDTFBSDI#MPH
 IUUQXXXFMBTUJDTFBSDIPSHCMPH  ⿣ &MBTUJDTFBSDIUIFEFpOJUJWFHVJEF
 IUUQXXXFMBTUJDTFBSDIPSHHVJEFFOFMBTUJDTFBSDIHVJEFDVSSFOUJOEFYIUNM  ⿣

    'PVOE
 IUUQTXXXGPVOEOPEPDVNFOUBUJPO IUUQTXXXGPVOEOPGPVOEBUJPO  ⿣ !KPIUBOJͷ೔هOE
 IUUQCMPHKPIUBOJJOGP  ⿣ )FMMP&MBTUJDTFBSDI
 IUUQTNFEJVNDPNIFMMPFMBTUJDTFBSDI  ⿣ -VDFOFOJHIUMZCFODINBSLT
 IUUQQFPQMFBQBDIFPSHdNJLFNDDBOEMVDFOFCFODI
  33. ఏڙ גࣜձࣾυϫϯΰ υϫϯΰ͸ຊ෺ͷ ݕࡧcσʔλϚΠχϯάc8FCΞϓϦcεϚϗΞϓϦ 
 ΤϯδχΞΛืू͍ͯ͠·͢ɻ