日経テレコンを支えるElasticsearch

 日経テレコンを支えるElasticsearch

第38回Elasticsearch勉強会 2020.10.29 #elasticsearchjp
https://www.meetup.com/ja-JP/Tokyo-Elastic-Fantastics/events/273562532/

27a17f6dcd75a913de398d1ed4d75a9e?s=128

Taisuke Hinata

October 29, 2020
Tweet

Transcript

  1. ೔ܦςϨίϯΛࢧ͑Δ &MBTUJDTFBSDI ೔ຊܦࡁ৽ฉࣾɹ೔ᙛହี ୈ38ճ Elasticsearchษڧձ #elasticsearchjp 2020/10/29

  2.  w ໊લ w ೔ᙛହี 5BJTVLF)JOBUB  w !IJOBUBEFT w

    ॴଐ w ೔ຊܦࡁ৽ฉࣾΤϯδχΞ৽ଔ೥໨ w ࢓ࣄ w αʔόαΠυɾΠϯϑϥத৺ w 8FC"1*ݕࡧ w (PMBOH1ZUIPO&MBTUJDTFBSDI ࣗݾ঺հ
  3. ೔ܦͷσʔλϕʔεࣄۀ w ࣾҎ্ͷϝσΟΞاۀ༷ͱίϯςϯπ࿈ܞ w ৽ฉ શࠃࢴ஍ํࢴۀքࢴεϙʔπࢴ  w اۀ৘ใ w

    ۀքϨϙʔτ w ਓ෺ɾਓࣄ৘ใ w 104 ߪങ σʔλ 
  4. ࠓ೔࿩͢͜ͱ w ೔ܦςϨίϯͱ͸ w &MBTUJDTFBSDIΛಋೖͨ͠എܠ w Ͳ͏࡞ͬͨͷ͔ w ΞʔΩςΫνϟ w

    Πϯϑϥઃܭ w ؂ࢹ w Ͳ͏ͳͬͨͷ͔ 
  5. ೔ܦςϨίϯ w ೥ ʂ αʔϏε։࢝ w ࠃ಺࠷େڃͷϏδωεσʔλϕʔεαʔϏε w ౦ূҰ෦্৔اۀͷ໿ׂ͕ಋೖ w

    Ҏ্ͷهࣄഔମ w  ສࣾҎ্ͷࠃ಺֎اۀ৘ใ w ສਓҎ্ͷΩʔύʔιϯ৘ใ w ͦͷଞଟ਺ ϦϯΫࢀর  w ίϯςϯπΛશจݕࡧͰϫϯετοϓऩू  IUUQTUFMFDPNOJLLFJDPKQHVJEF
  6. ࠓ೥݄͔Β&MBTUJDTFBSDI΁ w ೔ܦςϨίϯݕࡧΤϯδϯͷྺ࢙  هࣄ 1999 2000 2001 2002 2003

    2004 2005 2006 2007 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2008 Pana Search Pana Search اۀ ਓࣄ + ೥શจݕࡧαʔϏε։࢝
  7. ೥݄શίϯςϯπΛ&MBTUJDTFBSDIʹҠߦ ࠓ೥݄͔Β&MBTUJDTFBSDI΁ w ೔ܦςϨίϯݕࡧΤϯδϯͷྺ࢙  هࣄ 1999 2000 2001 2002

    2003 2004 2005 2006 2007 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2008 Pana Search Pana Search اۀ ਓࣄ +
  8. &MBTUJDTFBSDIΛಋೖͨ͠എܠ 

  9. ᶃ୤ɾΦϯϓϨ  w ݕࡧΤϯδϯΛ୆Ҏ্ͷ෺ཧαʔόͰӡ༻ w ػثߋ৽ͷͨͼʹ೥Ҏ্͔͔Γίετେ w Ϋϥ΢υ্Ͱӡ༻ՄೳͳݕࡧΤϯδϯ͕ඞཁʹ

  10. ᶄίϯςϯπϓϥοτϑΥʔϜߏ૝  w Ϋϥ΢υҠߦ͠ɺ͞ΒʹͦΕΛશαʔϏεڞ௨ͷ౷߹ݕࡧج൫ʹ w ΑΓεέʔϥϏϦςΟͷ͋ΔݕࡧΤϯδϯ͕ඞཁʹ 2016೥ 2020೥

  11. ͳͥ&MBTUJDTFBSDIͳͷ͔ w ᶃΫϥ΢υԽ w %PDLFS*NBHFͷ഑෍ w Ϋϥ΢υ1MVHJO͕๛෋ ˞ޙड़  w

    ᶄϓϥοτϑΥʔϜԽ w ෼ࢄܕεέʔϥϏϦςΟ w ϊʔυ௥Ճ͕؆୯ w σʔλ෼ࢄͱϨϓϦΧࣗಈ഑ஔ 
  12. Ͳ͏࡞ͬͨͷ͔ 

  13. ΞʔΩςΫνϟ֎؍  ɾ ɾ ɾ ݕࡧ༻υϝΠϯ "1*(BUFXBZ API $%/

  14. ΞʔΩςΫνϟ֎؍  ɾ ɾ ɾ "1*(BUFXBZ API ೝূ ྲྀྔ੍ݶ

  15. ΞʔΩςΫνϟ֎؍  ɾ ɾ ɾ "1*(BUFXBZ API ΫΤϦόϦσʔγϣϯ ΫΤϦͷࣙॻల։ &4ΫΤϦ΁ͷॻ͖׵͑

  16. ΞʔΩςΫνϟ֎؍  ɾ ɾ ɾ "1*(BUFXBZ API &MBTUJDTFBSDI༻υϝΠϯ ϦΫΤετઌͷΫϥελΛεϫοϓ

  17. &4ΫϥελͷΠϯϑϥઃܭ֓ཁ w &MBTUJDTFBSDIPO%PDLFS w "NB[PO&$4PO&$ w ϩʔϦϯάΞοϓσʔτ w ΦʔτεέʔϦϯά w

    4IBSE"MMPDBUJPO"XBSFOFTT  D D D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ
  18. &MBTUJDTFBSDIPO%PDLFS  D D D D D D AZ-a AZ-c

    AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ w Մൖੑɾ؀ڥ࠶ݱੑͷ޲্ w *NBHF͸%PDLFSϨδετϦͰอଘ w όʔδϣϯ؅ཧ Dockerfile CI Amazon ECR
  19. "NB[PO&$4PO&$  w "NB[PO&$4 w ίϯςφΦʔέετϨʔγϣϯ w "NB[PO&$ w αʔόʔ

    w "NB[PO&$"VUP4DBMJOH w ϧʔϧϕʔεͰεέʔϦϯά D D D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ
  20. "NB[PO&$4PO&$  w "NB[PO&$4 w ίϯςφΦʔέετϨʔγϣϯ w "NB[PO&$ w αʔόʔ

    w "NB[PO&$"VUP4DBMJOH w ϧʔϧϕʔεͰεέʔϦϯά D D D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ
  21. "NB[PO&$4PO&$  w "NB[PO&$4 w ίϯςφΦʔέετϨʔγϣϯ w "NB[PO&$ w αʔόʔ

    w "NB[PO&$"VUP4DBMJOH w ϧʔϧϕʔεͰεέʔϦϯά D D D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ
  22. ᶃϩʔϦϯάΞοϓσʔτ  w &$4͕ίϯςφ܊ΛΑΖ͘͠؅ཧ w ίϯςφͷҰׅߋ৽ w ίϯςφͷࣗಈ෮چ w ϚΠφʔόʔδϣϯΞοϓ

    ୆ ͕਺෼Ͱ w ˞ॠஅ͕ൃੜ͢ΔͨΊຊ൪؀ڥͰ͸ D D D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ
  23. ᶄΦʔτεέʔϦϯά w ";ো֐౳Ͱγϟοτμ΢ϯ࣌ʹࣗಈ෮چ w ෮چޙɺ&4Ϋϥελ΁ͷొ࿥͕ඞཁ w &$%JTDPWFSZ1MVHJO w 5BHʹ&4Ϋϥελ໊ͷొ࿥ 

    D D D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ
  24. εςʔτϑϧΞϓϦ͸ σʔλӬଓԽͷ؍఺͔Β ͪ͜ΒΛબͿ͸ͣ ετϨʔδ͸ԿΛ࢖͏͔  w &MBTUJDTFBSDI͸εςʔτϑϧͰ͋ΓετϨʔδ*0͕ॏཁ w "NB[PO&$ w

    &#4 w σʔλӬଓԽՄೳ w ΠϯελϯεετΞ w شൃੑ͕ͩߴੑೳ ετϨʔδI/O Մ༻ੑ ΠϯελϯεετΞ &#4
  25. ετϨʔδ͸ԿΛ࢖͏͔  w &MBTUJDTFBSDI͸εςʔτϑϧͰ͋ΓετϨʔδ*0͕ॏཁ w "NB[PO&$ w &#4 w σʔλӬଓԽՄೳ

    w ΠϯελϯεετΞ w شൃੑ͕ͩߴੑೳ ετϨʔδI/O Մ༻ੑ ΠϯελϯεετΞ &#4 &4͸γϟʔυ͔Βσʔλ͕ ࣗಈ෮ݩɻࣗಈ෮چΛલఏ ͱ͢Ε͹બ୒Մೳʹ
  26. 4IBSE"MMPDBUJPO"XBSFOFTT  AZ-a AZ-c AZ-d ϓϥΠϚϦγϟʔυ ϨϓϦΧγϟʔυ ϓϥΠϚϦʹର͢Δ ϨϓϦΧ͕ผͷ";ʹ ෼ࢄ͞ΕߴՄ༻

    w ෺ཧαʔόͷ;POFΛߟྀͨ͠ϓϥΠϚϦɾϨϓϦΧγϟʔυͷ෼ࢄ഑ஔ w ྫ"; ϊʔυ ϓϥΠϚϦγϟʔυ ϨϓϦΧ
  27. 4IBSE"MMPDBUJPO"XBSFOFTT w ෺ཧαʔόͷ;POFΛߟྀͨ͠ϓϥΠϚϦɾϨϓϦΧγϟʔυͷ෼ࢄ഑ஔ w ྫ"; ϊʔυ ϓϥΠϚϦγϟʔυ ϨϓϦΧ  AZ-a

    AZ-c AZ-d ϓϥΠϚϦγϟʔυ ϨϓϦΧγϟʔυ ಉ";಺ʹ*OEFY͕׬੒͠ɺ ݕࡧ࣌ʹ";ؒ௨৴͕ෆཁʹ
  28. ؂ࢹʹ͍ͭͯ 

  29. ઐ༻ϞχλϦϯάΫϥελ w αʔϏεΫϥελͷෛՙ௿ݮ w (PMEϥΠηϯεҎ্Ͱར༻Մೳ  AZ-a 4JOHMF D D

    D D D D AZ-a AZ-c AZ-d .BTUFS %BUB $PPSEJOBUJOH Indexing Search Kibana ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ ɾ xpack.monitoring.exporters.my_remote.type: http xpack.monitoring.exporters.my_remote.host: https://XXX:9200
  30. ,JCBOB%BTICPBSEͷ׆༻ w &YQPSUͰ͖ΔͷͰҰ౓࡞ͬͯ͠·͑͹࠶ར༻Մೳ 

  31. ,JCBOB%BTICPBSEͷ׆༻ w ,JCBOBϞχλϦϯάʹࡌͬͯͳ͍ϝτϦΫεΛ࡞੒ˍ؂ࢹ  *OEFYผͷ214؂ࢹ

  32. ӡ༻਍அαʔϏε ΞΫϩΫΤετςΫϊϩδʔࣾ w िճͷΫϥελΠϯσοΫε਍அ w ݄ճͷϨϙʔτใࠂ w ਍அʹඞཁͳ,JCBOB%BTICPBSEͷ࡞੒  IUUQTXXXFOEPTOJQFDPNFMBTUJD@DMVTUFS@EJBHOPTJT

  33. Ͳ͏ͳͬͨͷ͔ 

  34. ᶃόʔδϣϯΞοϓ͕Մೳʹ w ΦϯϓϨ࣌୅͸όʔδϣϯΞοϓͷϋʔυϧ͕ߴ͘ɺಉόʔδϣϯΛػثߋ৽·Ͱอक w όʔδϣϯΞοϓ͕਺෼Ͱ࣮ߦՄೳʹ  ߋ৽ස౓͕ߴ͍͕ ͔ͬ͠Γ௥ਵͯ͠ ͍͖͍ͨ IUUQTXXXFMBTUJDDPKQTVQQPSUFPM

  35. ᶄ෮چ࣌ؒͷ୹ॖ w ΦϯϓϨ࣌୅͸ো֐ϊʔυͷ੾Γ཭͠ɺ෮ؼ·Ͱ͕͢΂ͯख࡞ۀ w &MBTUJDTFBSDIͷϨϓϦέʔγϣϯ΍1MVHJOʹΑΓશࣗಈ෮چՄೳʹ w ઌि  ͷ"84ো֐࣌΋ແࣄࣗಈ෮چ☺ 

    αʔόγϟʔυ࠶࡞੒Ͱݩͷঢ়ଶʹ
  36. ᶅϥϯχϯάίετ࡟ݮ w ෺ཧαʔό ୆͔ΒΫϥ΢υαʔό୆ʹ  ݕࡧΤϯδϯ ೥ؒϥϯχϯάίετ ໿22.5%ݮ ݕࡧΤϯδϯͷ ੑೳΞοϓͷ

    د༩͕େ͖͍ͱߟ࡯
  37. ·ͱΊ w ೔ܦςϨίϯ͸&MBTUJDTFBSDIʹେ͍ʹࢧ͑ΒΕ͍ͯΔ w ͋Γ͕ͱ͏ɺ&MBTUJDTFBSDI 

  38. ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ