日経テレコンを支えるElasticsearch

第38回Elasticsearch勉強会 2020.10.29 #elasticsearchjp
https://www.meetup.com/ja-JP/Tokyo-Elastic-Fantastics/events/273562532/

Transcript

1. ೔ܦςϨίϯΛࢧ͑Δ
   &MBTUJDTFBSDI
   ೔ຊܦࡁ৽ฉࣾɹ೔ᙛହี
   ୈ38ճ Elasticsearchษڧձ #elasticsearchjp
   2020/10/29
   

   View Slide

2. 
   w ໊લ
   w ೔ᙛହี 5BJTVLF)JOBUB
   
   w !IJOBUBEFT
   w ॴଐ
   w ೔ຊܦࡁ৽ฉࣾΤϯδχΞ৽ଔ೥໨
   w ࢓ࣄ
   w αʔόαΠυɾΠϯϑϥத৺
   w 8FC"1*ݕࡧ
   w (PMBOH1ZUIPO&MBTUJDTFBSDI
   ࣗݾ঺հ
   

   View Slide

3. ೔ܦͷσʔλϕʔεࣄۀ
   w ࣾҎ্ͷϝσΟΞاۀ༷ͱίϯςϯπ࿈ܞ
   w ৽ฉ શࠃࢴ஍ํࢴۀքࢴεϙʔπࢴ
   
   w اۀ৘ใ
   w ۀքϨϙʔτ
   w ਓ෺ɾਓࣄ৘ใ
   w 104 ߪങ
   σʔλ
   
   

   View Slide

4. ࠓ೔࿩͢͜ͱ
   w ೔ܦςϨίϯͱ͸
   w &MBTUJDTFBSDIΛಋೖͨ͠എܠ
   w Ͳ͏࡞ͬͨͷ͔
   w ΞʔΩςΫνϟ
   w Πϯϑϥઃܭ
   w ؂ࢹ
   w Ͳ͏ͳͬͨͷ͔
   
   

   View Slide

5. ೔ܦςϨίϯ
   w ೥ ʂ
   αʔϏε։࢝
   w ࠃ಺࠷େڃͷϏδωεσʔλϕʔεαʔϏε
   w ౦ূҰ෦্৔اۀͷ໿ׂ͕ಋೖ
   w Ҏ্ͷهࣄഔମ
   w ສࣾҎ্ͷࠃ಺֎اۀ৘ใ
   w ສਓҎ্ͷΩʔύʔιϯ৘ใ
   w ͦͷଞଟ਺ ϦϯΫࢀর
   
   w ίϯςϯπΛશจݕࡧͰϫϯετοϓऩू
   IUUQTUFMFDPNOJLLFJDPKQHVJEF
   

   View Slide

6. ࠓ೥݄͔Β&MBTUJDTFBSDI΁
   w ೔ܦςϨίϯݕࡧΤϯδϯͷྺ࢙
   
   هࣄ
   1999 2000 2001 2002 2003 2004 2005 2006 2007 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
   2008
   Pana Search
   Pana Search
   اۀ
   ਓࣄ
   +
   ೥શจݕࡧαʔϏε։࢝
   

   View Slide

7. ೥݄શίϯςϯπΛ&MBTUJDTFBSDIʹҠߦ
   ࠓ೥݄͔Β&MBTUJDTFBSDI΁
   w ೔ܦςϨίϯݕࡧΤϯδϯͷྺ࢙
   
   هࣄ
   1999 2000 2001 2002 2003 2004 2005 2006 2007 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022
   2008
   Pana Search
   Pana Search
   اۀ
   ਓࣄ
   +
   

   View Slide

8. &MBTUJDTFBSDIΛಋೖͨ͠എܠ
   
   

   View Slide

9. ᶃ୤ɾΦϯϓϨ
   
   w ݕࡧΤϯδϯΛ୆Ҏ্ͷ෺ཧαʔόͰӡ༻
   w ػثߋ৽ͷͨͼʹ೥Ҏ্͔͔Γίετେ
   w Ϋϥ΢υ্Ͱӡ༻ՄೳͳݕࡧΤϯδϯ͕ඞཁʹ
   

   View Slide

10. ᶄίϯςϯπϓϥοτϑΥʔϜߏ૝
    
    w Ϋϥ΢υҠߦ͠ɺ͞ΒʹͦΕΛશαʔϏεڞ௨ͷ౷߹ݕࡧج൫ʹ
    w ΑΓεέʔϥϏϦςΟͷ͋ΔݕࡧΤϯδϯ͕ඞཁʹ
    2016೥ 2020೥
    

    View Slide

11. ͳͥ&MBTUJDTFBSDIͳͷ͔
    w ᶃΫϥ΢υԽ
    w %PDLFS*NBHFͷ഑෍
    w Ϋϥ΢υ1MVHJO͕๛෋ ˞ޙड़
    
    w ᶄϓϥοτϑΥʔϜԽ
    w ෼ࢄܕεέʔϥϏϦςΟ
    w ϊʔυ௥Ճ͕؆୯
    w σʔλ෼ࢄͱϨϓϦΧࣗಈ഑ஔ
    
    

    View Slide

12. Ͳ͏࡞ͬͨͷ͔
    
    

    View Slide

13. ΞʔΩςΫνϟ֎؍
    
    ɾ
    ɾ
    ɾ
    ݕࡧ༻υϝΠϯ
    "1*(BUFXBZ
    API
    $%/
    

    View Slide

14. ΞʔΩςΫνϟ֎؍
    
    ɾ
    ɾ
    ɾ
    "1*(BUFXBZ
    API
    ೝূ
    ྲྀྔ੍ݶ
    

    View Slide

15. ΞʔΩςΫνϟ֎؍
    
    ɾ
    ɾ
    ɾ
    "1*(BUFXBZ
    API
    ΫΤϦόϦσʔγϣϯ
    ΫΤϦͷࣙॻల։
    &4ΫΤϦ΁ͷॻ͖׵͑
    

    View Slide

16. ΞʔΩςΫνϟ֎؍
    
    ɾ
    ɾ
    ɾ
    "1*(BUFXBZ
    API
    &MBTUJDTFBSDI༻υϝΠϯ
    ϦΫΤετઌͷΫϥελΛεϫοϓ
    

    View Slide

17. &4ΫϥελͷΠϯϑϥઃܭ֓ཁ
    w &MBTUJDTFBSDIPO%PDLFS
    w "NB[PO&$4PO&$
    w ϩʔϦϯάΞοϓσʔτ
    w ΦʔτεέʔϦϯά
    w 4IBSE"MMPDBUJPO"XBSFOFTT
    
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    

    View Slide

18. &MBTUJDTFBSDIPO%PDLFS
    
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    w Մൖੑɾ؀ڥ࠶ݱੑͷ޲্
    w *NBHF͸%PDLFSϨδετϦͰอଘ
    w όʔδϣϯ؅ཧ
    Dockerfile CI
    Amazon ECR
    

    View Slide

19. "NB[PO&$4PO&$
    
    w "NB[PO&$4
    w ίϯςφΦʔέετϨʔγϣϯ
    w "NB[PO&$
    w αʔόʔ
    w "NB[PO&$"VUP4DBMJOH
    w ϧʔϧϕʔεͰεέʔϦϯά
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    

    View Slide

20. "NB[PO&$4PO&$
    
    w "NB[PO&$4
    w ίϯςφΦʔέετϨʔγϣϯ
    w "NB[PO&$
    w αʔόʔ
    w "NB[PO&$"VUP4DBMJOH
    w ϧʔϧϕʔεͰεέʔϦϯά
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    

    View Slide

21. "NB[PO&$4PO&$
    
    w "NB[PO&$4
    w ίϯςφΦʔέετϨʔγϣϯ
    w "NB[PO&$
    w αʔόʔ
    w "NB[PO&$"VUP4DBMJOH
    w ϧʔϧϕʔεͰεέʔϦϯά
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    

    View Slide

22. ᶃϩʔϦϯάΞοϓσʔτ
    
    w &$4͕ίϯςφ܊ΛΑΖ͘͠؅ཧ
    w ίϯςφͷҰׅߋ৽
    w ίϯςφͷࣗಈ෮چ
    w ϚΠφʔόʔδϣϯΞοϓ ୆
    ͕਺෼Ͱ
    w ˞ॠஅ͕ൃੜ͢ΔͨΊຊ൪؀ڥͰ͸
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    

    View Slide

23. ᶄΦʔτεέʔϦϯά
    w ";ো֐౳Ͱγϟοτμ΢ϯ࣌ʹࣗಈ෮چ
    w ෮چޙɺ&4Ϋϥελ΁ͷొ࿥͕ඞཁ
    w &$%JTDPWFSZ1MVHJO
    w 5BHʹ&4Ϋϥελ໊ͷొ࿥
    
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    

    View Slide

24. εςʔτϑϧΞϓϦ͸
    σʔλӬଓԽͷ؍఺͔Β
    ͪ͜ΒΛબͿ͸ͣ
    ετϨʔδ͸ԿΛ࢖͏͔
    
    w &MBTUJDTFBSDI͸εςʔτϑϧͰ͋ΓετϨʔδ*0͕ॏཁ
    w "NB[PO&$
    w
    w σʔλӬଓԽՄೳ
    w ΠϯελϯεετΞ
    w شൃੑ͕ͩߴੑೳ
    ετϨʔδI/O
    Մ༻ੑ
    ΠϯελϯεετΞ
    
    

    View Slide

25. ετϨʔδ͸ԿΛ࢖͏͔
    
    w &MBTUJDTFBSDI͸εςʔτϑϧͰ͋ΓετϨʔδ*0͕ॏཁ
    w "NB[PO&$
    w
    w σʔλӬଓԽՄೳ
    w ΠϯελϯεετΞ
    w شൃੑ͕ͩߴੑೳ
    ετϨʔδI/O
    Մ༻ੑ
    ΠϯελϯεετΞ
    
    &4͸γϟʔυ͔Βσʔλ͕
    ࣗಈ෮ݩɻࣗಈ෮چΛલఏ
    ͱ͢Ε͹બ୒Մೳʹ
    

    View Slide

26. 4IBSE"MMPDBUJPO"XBSFOFTT
    
    AZ-a AZ-c AZ-d
    ϓϥΠϚϦγϟʔυ
    ϨϓϦΧγϟʔυ
    ϓϥΠϚϦʹର͢Δ
    ϨϓϦΧ͕ผͷ";ʹ
    ෼ࢄ͞ΕߴՄ༻
    w ෺ཧαʔόͷ;POFΛߟྀͨ͠ϓϥΠϚϦɾϨϓϦΧγϟʔυͷ෼ࢄ഑ஔ
    w ྫ"; ϊʔυ ϓϥΠϚϦγϟʔυ ϨϓϦΧ
    

    View Slide

27. 4IBSE"MMPDBUJPO"XBSFOFTT
    w ෺ཧαʔόͷ;POFΛߟྀͨ͠ϓϥΠϚϦɾϨϓϦΧγϟʔυͷ෼ࢄ഑ஔ
    w ྫ"; ϊʔυ ϓϥΠϚϦγϟʔυ ϨϓϦΧ
    
    AZ-a AZ-c AZ-d
    ϓϥΠϚϦγϟʔυ
    ϨϓϦΧγϟʔυ
    ಉ";಺ʹ*OEFY͕׬੒͠ɺ
    ݕࡧ࣌ʹ";ؒ௨৴͕ෆཁʹ
    

    View Slide

28. ؂ࢹʹ͍ͭͯ
    
    

    View Slide

29. ઐ༻ϞχλϦϯάΫϥελ
    w αʔϏεΫϥελͷෛՙ௿ݮ
    w (PMEϥΠηϯεҎ্Ͱར༻Մೳ
    
    AZ-a
    4JOHMF
    D D D
    D
    D
    D
    AZ-a AZ-c AZ-d
    .BTUFS %BUB $PPSEJOBUJOH
    Indexing
    Search
    Kibana
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    xpack.monitoring.exporters.my_remote.type: http
    xpack.monitoring.exporters.my_remote.host: https://XXX:9200
    

    View Slide

30. ,JCBOB%BTICPBSEͷ׆༻
    w &YQPSUͰ͖ΔͷͰҰ౓࡞ͬͯ͠·͑͹࠶ར༻Մೳ
    
    

    View Slide

31. ,JCBOB%BTICPBSEͷ׆༻
    w ,JCBOBϞχλϦϯάʹࡌͬͯͳ͍ϝτϦΫεΛ࡞੒ˍ؂ࢹ
    
    *OEFYผͷ214؂ࢹ
    

    View Slide

32. ӡ༻਍அαʔϏε ΞΫϩΫΤετςΫϊϩδʔࣾ
    
    w िճͷΫϥελΠϯσοΫε਍அ
    w ݄ճͷϨϙʔτใࠂ
    w ਍அʹඞཁͳ,JCBOB%BTICPBSEͷ࡞੒
    IUUQTXXXFOEPTOJQFDPNFMBTUJD@DMVTUFS@EJBHOPTJT
    

    View Slide

33. Ͳ͏ͳͬͨͷ͔
    
    

    View Slide

34. ᶃόʔδϣϯΞοϓ͕Մೳʹ
    w ΦϯϓϨ࣌୅͸όʔδϣϯΞοϓͷϋʔυϧ͕ߴ͘ɺಉόʔδϣϯΛػثߋ৽·Ͱอक
    w όʔδϣϯΞοϓ͕਺෼Ͱ࣮ߦՄೳʹ
    
    ߋ৽ස౓͕ߴ͍͕
    ͔ͬ͠Γ௥ਵͯ͠
    ͍͖͍ͨ
    IUUQTXXXFMBTUJDDPKQTVQQPSUFPM
    

    View Slide

35. ᶄ෮چ࣌ؒͷ୹ॖ
    w ΦϯϓϨ࣌୅͸ো֐ϊʔυͷ੾Γ཭͠ɺ෮ؼ·Ͱ͕͢΂ͯख࡞ۀ
    w &MBTUJDTFBSDIͷϨϓϦέʔγϣϯ΍1MVHJOʹΑΓશࣗಈ෮چՄೳʹ
    w ઌि
    ͷ"84ো֐࣌΋ແࣄࣗಈ෮چ☺
    
    αʔόγϟʔυ࠶࡞੒Ͱݩͷঢ়ଶʹ
    

    View Slide

36. ᶅϥϯχϯάίετ࡟ݮ
    w ෺ཧαʔό୆͔ΒΫϥ΢υαʔό୆ʹ
    
    ݕࡧΤϯδϯ
    ೥ؒϥϯχϯάίετ
    ໿22.5%ݮ
    ݕࡧΤϯδϯͷ
    ੑೳΞοϓͷ
    د༩͕େ͖͍ͱߟ࡯
    

    View Slide

37. ·ͱΊ
    w ೔ܦςϨίϯ͸&MBTUJDTFBSDIʹେ͍ʹࢧ͑ΒΕ͍ͯΔ
    w ͋Γ͕ͱ͏ɺ&MBTUJDTFBSDI
    
    

    View Slide

38. ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    
    

    View Slide