Elastic Stack 6.5で実現する快適Kibana生活 / Comfortable Kibana Life by Using Elastic Stack 6.5

62cc09bdd17296303d77a0d1f104dcb5?s=47 j-yama
January 16, 2019

Elastic Stack 6.5で実現する快適Kibana生活 / Comfortable Kibana Life by Using Elastic Stack 6.5

第28回 Elasticsearch勉強会の発表資料です。

勉強会の詳細はこちら
https://www.meetup.com/ja-JP/Tokyo-Elastic-Fantastics/events/257301177/

62cc09bdd17296303d77a0d1f104dcb5?s=128

j-yama

January 16, 2019
Tweet

Transcript

  1. &MBTUJD4UBDLͰ ࣮ݱ͢Δ շద,JCBOBੜ׆ @j__yama 2019/01/16 ୈ28ճElasticsearchษڧձ

  2. whoami • ࢁԼ৓࢘@NTTςΫϊΫϩε • Twitter: @j__yama • Favorite: Listening HR/HM

    , Watching Speedrun • Ҿ༻ݩɿIUUQTRJJUBDPNTFBSDI RKZBNB 35"J+ "(%2
  3. IUUQTFMBTUJDMPWFSIBUFOBCMPHKQ

  4. ࠓ೔ͷςʔϚ͸

  5. &MBTUJD4UBDL ৽ػೳ঺հ

  6. ͜ͷൃද͸,JCBOBͷ ৽ػೳΛ঺հ͠ɺ ͦͷັྗΛօ͞Μʹ఻͑Δ΋ͷͰ͢ʂ

  7. agenda ɾKibana 6.5ͷ৽ػೳ ɾಛʹ঺հ͍ͨͭ͠ͷػೳ  ɾ'JMF%BUB7JTVBMJ[FS  ɾ4QBDFT ɾ͓ΘΓʹ

  8. ,JCBOBͷ৽ػೳ

  9. Kibana 6.5ͷओͳ৽ػೳ(1/2) • ੝Γͩ͘͞Μ 4QBDFT $BOWBTͷϏϧτΠϯ 'JMF%BUB7JTVBMJ[FS *OGSB6*-PH6* #FUB #FUB

    ৽͍͠αϯϓϧσ ʔλ΋ͭ௥Ճ #FUB
  10. Kibana 6.5ͷओͳ৽ػೳ(2/2) • ੝Γͩ͘͞Μ 3PMMVQͷ؅ཧ6* 3PMMVQ݁ՌͷՄࢹԽ %FW5PPMTͷػೳڧԽ 4QZύωϧ͔Β*OTQFDUPS΁ͷҠߦ ͱͯ΋શ෦͸࿩ͤͳ͍ #FUB

  11. ಛʹ঺հ͍ͨͭ͠ͷػೳ

  12. ಛʹ঺հ͍ͨ͠2ͭͷػೳ • File Data Visualizer • ͜Ε͔ΒElastic StackΛ࢖͍࢝ΊΔਓ޲͚ • ௒؆୯ʹElasticsearchʹσʔλ౤ೖͰ͖Δػೳ

    • Spaces • طʹ։ൃɾӡ༻ʹ࢖͍ͬͯΔਓ޲͚ • Visualize΍Dashboard͕ཚཱ͍ͯͯ͠؅ཧʹࠔ͍ͬͯΔਓ ΁ͷղܾࡦ
  13. 'JMF%BUB7JTVBMJ[FS ʲ͜Ε͔Β&MBTUJD4UBDLΛ࢖͍࢝ΊΔਓ޲͚ʳ

  14. File Data Visualizer • ςΩετϑΝΠϧΛKibanaը໘͔Β௚઀ Elasticsearchʹ౤ೖͰ͖Δػೳ • ରԠσʔλܗࣜ: csv, ndjson,

    ஶ໊ͳϩάܗࣜ • ࣗಈͰύʔεɺϚοϐϯάੜ੒΋࣮ߦ • σϑΥϧτͰ͸࠷ॳͷ1000ߦΛ෼ੳͯࣝ͠ผ
  15. Before • ElasticsearchʹσʔλΛ౤ೖ͢ΔͨΊʹ͸
 ԿΒ͔ͷETLπʔϧ΍ϥΠϒϥϦΛ࢖͏ඞཁ͕ ͋ͬͨ ίϯϑΟάͷॻ͖ํͳͲ शख़͕ඞཁͰίετେ

  16. After • KibanaΛද͍ࣔͯ͠Δϒϥ΢β͔Β௚઀ ElasticsearchʹσʔλΛ౤ೖͰ͖Δ ࢼ͠ʹ࢖ͬͯΈ͍ͨ࣌ʹ ͱͯ΋ศརʂ

  17. Kibanaੜ׆ͷBefore / After • Before • After KibanaͰϩάͱ͔ՄࢹԽ͠ ·͠ΐ͏ʂ Ͱ͖ͳ͍ͷͰLogstash΍

    Filebeatͷ࢖͍ํΛ֮͑· ͠ΐ͏ csvͷσʔλͱ͔؆୯ʹը໘ ͔Β౤ೖͰ͖·͔͢ʁ ͑ͬ...େมͦ͏...࢖͏ͷ΍Ί Α͏͔ͳ… Ͱ͖·͢ΑʂKibanaͷը໘ ͔Β͜͏΍ͬͯબ୒͍ͯ͠ ͘ͱ... ͍͍͡ΌΜʂ࢖ͬͯΈΔΑ
  18. File Data Visualizerͷ࢖͍ํ .BDIJOF -FBSOJOHΛΫϦοΫ %BUB7JTVBMJ[FSΛΫϦοΫ ը໘தԝʹ౤ೖ͍ͨ͠ϑΝΠϧΛ υϥοάΞϯυυϩοϓ͢Δ

  19. File Data Visualizerͷ࢖͍ํ σʔλ܏޲͕දࣔ͞ΕΔ ͷͰɺ֬ೝͨ͠Β JNQPSUΛΫϦοΫ͢Δ

  20. File Data Visualizerͷ࢖͍ํ JOEFYOBNFʹઃఆ͍ͨ͠ ΠϯσοΫε໊Λೖྗ͢Δ JNQPSUΛΫϦοΫ͢Δ

  21. File Data Visualizerͷ࢖͍ํ ଴͍ͬͯΔͱϓϩάϨεόʔ͕ਐΈɺ σʔλ͕౤ೖ͞ΕΔ

  22. File Data Visualizerͷ࢖͍ํ ౤ೖ࣌ʹJOEFYQBUUFSO͕࡞੒͞ΕΔͨΊɺ ͦͷ··%JTDPWFS΍7JTVBMJ[FͰදࣔͰ͖Δ

  23. File Data Visualizerͷ࢖͍ํ JNQPSUEBUBͰ"EWBODFEʹ͢Δͱɺ ز͔ͭͷઃఆΛมߋͰ͖Δ ࢀߟ ΠϯσοΫε ઃఆ Ϛοϐϯάͷ ΧελϚΠζ

    σʔλՃ޻ઃఆ JOHFTUQJQFMJOF
  24. σϞ

  25. File Data VisualizerͰࢥ͏͜ͱ • Q. σʔλͷՃ޻Λ͢ΔͷʹIngest PipelineΛ࢖Θ ͳ͍ͱ͍͚ͳ͍ͷ͸େมͳͷͰ͸ʁ • A.

    ͦ΋ͦ΋౤ೖલʹExcelͳͲͰcsvΛՃ޻͓ͯ͠ ͚͹໰୊͋Γ·ͤΜ &MBTUJD͞Μͷ#MPHͰ΋ॳ৺ऀ޲͚ͷهࣄͰ͸ࣄલʹ &YDFMͳͲͰσʔλՃ޻͢ΔखॱͰ঺հ͞Ε͍ͯ·͢ɻ IUUQTXXXFMBTUJDDPCMPHBGUFSTIPDLUIFSBQZ XJUIFMBTUJDTFBSDIBOEDTWEBUBJNQPSU
  26. 4QBDFT ʲطʹ։ൃɾӡ༻ʹ࢖͍ͬͯΔਓ޲͚ʳ

  27. Spaces • Visualize/DashboardΛ੔ཧ੔಴Ͱ͖Δػೳ • εϖʔεͱݺ͹ΕΔྖҬΛ࡞੒͠ɺͦͷதͰVisualize΍ DashboardΛ࡞੒͢ΔͱɺͦͷεϖʔεʹډΔͱ͖͚ͩ ӾཡͰ͖ΔΑ͏ʹͳΔ ηΩϡϦςΟ ϚʔέςΟϯά ʓ

    º
  28. Spaces • Ϣʔβʔ͸Ӿཡ࣌ʹͲͷεϖʔεΛར༻ͨ͠ ͍͔બ୒͢Δ ؂ࠪϩάΛݟ͍ͨ࣌͸ ηΩϡϦςΟ༻ͷεϖʔε ച্৘ใΛݟ͍ͨ࣌͸ ϚʔέςΟϯά༻ͷεϖʔε

  29. Before • Visualize΍DashboardͷҰཡΛදࣔ͢Δͱଘ ࡏ͢Δશ͕ͯදࣔ͞Εͯ͠·͏ • ϫʔΫΞϥ΢ϯυͱͯ͠༻్ผʹVisualize ໊΁prefixΛ෇͚ͨΓ͍͕ͯͨ͠… • ී௨ʹԚ͍͠ɺҰཡը໘ͷݕࡧػೳ͸บ ͕͋ͬͯɺҾ͔͔ͬΒͳ͍͜ͱ΋͋Δ

  30. Beforeͷྫ #FBUTͷαϯϓϧμογϡϘʔυ Λ౤ೖ͢Δͱಋೖ࣌఺͔Βʜ

  31. Beforeͷྫ ϚΠφεݕࡧͰ.FUSJDCFBUͷ 7JTVBMJ[FΛඇදࣔʹ͠Α͏ͱͨ͠΋ͷͷɺҰ෦ 7JTVBMJ[Fͷ5JUMFʹ.FUSJDCFBUΛද͢Ωʔϫʔυ͕ ೖ͍ͬͯͳ͍ͨΊදࣔ͞Εͯ͠·͏ʜ

  32. Beforeͷྫ ͜Ε·Ͱ͸۠ผΛ͚ͭΔͨΊʹ<1SPKFDU">ͳͲ ͷ1SFpYΛ෇͚ͯରԠ͢Δͷ͕ਫ਼Ұഋ ݕࡧ͢Ε͹Ҿ͔͚ͬΒΕΔ͕ɺ 5JUMFΛ෇͚Δͱ͖ʹҙࣝ͢Δඞཁ͕͋Γɺ ίετ͕ߴ͍ ݕࡧ

  33. After • ࣄલʹద੾ʹ෼ྨ͓͚ͯ͠͹ɺͦͷ࣌ʑͰڵຯ ͋ΔVisualize/DashboardͷΈΛӾཡͰ͖Δ • SpaceΛ࡞੒͓͚ͯ͠͹ɺଞͷਓͷҰཡΛԚ͢ ͜ͱͳ͘ɺςετDashboardΛ࡞੒Ͱ͖Δ • SecurityΛ૊Έ߹ΘͤΕ͹ಛఆͷਓ͚͕ͩӾཡՄ ೳͳDashboard΋࡞੒Մೳ

  34. Kibanaੜ׆ͷBefore / After • Before • After ,JCBOB ,JCBOB ։ൃɾӡ༻

    ηΩϡϦςΟ ϚʔέςΟϯά ؂ࠪϩά ؂ࠪϩά ΞϓϦϩά Ϣʔβʔ࢖༻܏޲ ϝτϦοΫ ϝτϦοΫ ΞϓϦϩά ύέοτ৘ใ ύέοτ৘ใ Ϣʔβʔ࢖༻܏޲ ച্ ച্
  35. σϞ

  36. Spacesͷ׆༻Ҋͦͷ1 νʔϜ΍ۀ຿͝ͱʹ෼͚Δ • Elastic͞Μͷઆ໌Ͱ΋Α͘࢖ΘΕΔ • ଞνʔϜͷڵຯͳ͍Visualize/DashboardΛඇදࣔʹͰ͖Δ ։ൃɾӡ༻ ηΩϡϦςΟ ϚʔέςΟϯά ӡ༻

    ϝτϦοΫ ΞϓϦϩά ؂ࠪϩά ύέοτ৘ใ Ϣʔβʔ࢖༻܏޲ ച্
  37. Spacesͷ׆༻Ҋͦͷ2 ςφϯτ͝ͱʹ෼͚Δ • ͓٬༷͝ͱʹҟͳΔDashboardΛఏڙͰ͖Δ • ࣮͸͜Ε·ͰετϨʔτͳํ๏Ͱ͸࣮ݱͰ͖ͳ͔ͬͨ • ςφϯτ͝ͱʹKibanaΠϯελϯεΛݐͯͯݸผ ͷ".kibana"Λઃఆͯ͠…ͳͲͷϫʔΫΞϥ΢ϯυ͕ඞཁͩͬͨ ͓٬༷"

    ͓٬༷# ͓٬༷$ ӡ༻ #༷ࣾݶఆ "༷ࣾݶఆ $༷ࣾݶఆ
  38. Spacesͷ׆༻Ҋͦͷ3 Dashboardผʹ෼͚Δ • ։ൃத͸ಛʹVisualizeҰཡը໘͸ཚࡶʹͳΓ͕ͪͳͷΛ๷͛Δ • Space͕ཚཱ͢Δͷ͕एׯ໰୊ʁ • Dashboardͷ݅਺͕গͳ͚Ε͹͋Γ ΞϓϦϩά ϛυϧϩά

    ϝτϦοΫ ։ൃ "QBDIF 5PNDBU ΞϓϦ" ΞϓϦ# $16 ϝϞϦ
  39. Spacesͷ׆༻Ҋͦͷ4 ։ൃऀผʹ෼͚Δ • ։ൃத͸ಛʹVisualizeҰཡը໘͸ཚࡶʹͳΓ͕ͪͳͷΛ๷͛Δ • ଞͷਓ΁ͷӨڹΛߟ͑ͣؾָʹVisualize/DashboardΛ࡞੒Ͱ͖Δ • ਓ਺͕গͳ͚Ε͹Space΋ཚཱ͠ͳ͍ "͞Μ #͞Μ

    $͞Μ ։ൃ ؂ࠪ ੑೳ ͓ࢼ͠ ύέοτ ςετ
  40. Spacesͷ׆༻Ҋͦͷ5 DashboardͷϨϏϡΛKibanaͰߦ͏ • ςετతʹࢼ͍ͨ͠৔߹ʹ͸αϯυϘοΫεͰ࡞੒͢Δ • ϫʔΫεϖʔεͰ࡞੒͠ɺϨϏϡر๬ͷDashboard͸ϨϏϡ΁Ҡಈ • ϨϏϡ׬ྃͨ͠ΒϓϩμΫγϣϯ΁Ҡಈ αϯυϘοΫε ϫʔΫεϖʔε

    ϨϏϡ ։ൃ ϓϩμΫγϣϯ ࡞੒த ϨϏϡ଴ͪ ϨϏϡࡁΈ ϨϏϡࡁΈ ͓ࢼ͠
  41. SpacesͰࢥ͏͜ͱ • ͋·Γʹஅઈ͠ա͗Δͱྑ͘ͳ͍͔΋ • ӡ༻ɺ։ൃɺϚʔέςΟϯάͷਓͨͪڞ௨ Ͱݟ͍ͨDashboard΋͋Δ • ·ͨɺผͷSpaceͷVisualize͸ࢀরͰ͖ͳ ͘ͳͬͯ͠·͏ͷͰ࠶ར༻ੑ΋Լ͕Δ

  42. ͓ΘΓʹ

  43. ͜Ε͔ΒͷշదͳKibanaੜ׆ΛΠ ϝʔδͰ͖·͔ͨ͠ʁ • File Data Visualizer • ͜Ε͔ΒElastic StackΛ࢖͍࢝ΊΔਓ͸௒؆୯ʹσʔλ౤ೖ Ͱ͖ΔΑ͏ʹͳΔʂ

    • ETLπʔϧͷ࢖͍ํ͸ࣗಈԽͨ͘͠ͳ͔ͬͯΒ֮͑Ε͹OK • Spaces • طʹ։ൃ/ӡ༻Ͱ࢖͍ͬͯΔਓ͸Visualize/DashboardͷҰཡ Λ੔ཧ੔಴Ͱ͖ΔΑ͏ʹͳΔʂ
  44. 'JMF%BUB7JTVBMJ[FS͸ &YQFSJNFOUBMͳػೳͳͨΊɺࠓޙ ࢓༷มߋ͞ΕΔՄೳੑ͕͋Γ·͢  ·ͨɺόάΛଟؚ͘Ή͜ͱ͕༧૝͞ΕΔͨΊɺ ݱ࣌఺ͰϓϩμΫγϣϯͰ࢖༻͢Δ͜ͱ͸ΦεεϝͰ͖·ͤΜɻ όάΛൃݟͨ͠Β*TTVF13

  45. &MBTUJD4UBDLͰ ࣮ݱ͢Δ շద,JCBOBੜ׆ @j__yama 2019/01/16 ୈ28ճElasticsearchษڧձ

  46. ͓͠·͍

  47. "QQFOEJY

  48. File Data Visualizerͷ ରԠϑΝΠϧܗࣜ • ٙ໰ ճ౴ ΤϯίʔσΟϯά͸ԿʹରԠʁ UTF-8ɺBOM༗ແͷରԠ͸ʁ վߦίʔυ͸LFɺCRLFͷͲͪΒʹ΋ରԠʁ

    $47͸೚ҙͷ۠੾ΓจࣈʹରԠʁ ೔෇ͷରԠϑΥʔϚοτ͸ʁ ରԠ͍ͯ͠Δஶ໊ͳϩάܗࣜ͸ʁ όΠφϦϑΝΠϧʹ͸ରԠʁ
  49. File Data Visualizerͷίʔυ • ΫϥΠΞϯτ(Kibana)ଆͷίʔυ • https://github.com/elastic/kibana/blob/master/x- pack/plugins/ml/server/routes/file_data_visualizer.js • αʔόʔ(Elasticsearch)ଆͷίʔυ

    • https://github.com/elastic/elasticsearch/tree/ master/x-pack/plugin/ml/src/main/java/org/ elasticsearch/xpack/ml/filestructurefinder
  50. File Data VisualizerͷରԠ͍ͯ͠ ΔΤϯίʔσΟϯά • https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ml/src/main/ java/org/elasticsearch/xpack/ml/filestructurefinder/FileStructureFinderManager.java

  51. File Data VisualizerͷରԠ͍ͯ͠ Δ೔෇ϑΥʔϚοτ • https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ml/src/main/ java/org/elasticsearch/xpack/ml/filestructurefinder/TimestampFormatFinder.java

  52. File Data VisualizerͷରԠ͍ͯ͠ Δஶ໊ͳϩάܗࣜ • https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ml/src/main/ java/org/elasticsearch/xpack/ml/filestructurefinder/GrokPatternCreator.java IUUQTHJUIVCDPN MPHTUBTIQMVHJOTMPHTUBTIQBUUFSOT DPSFUSFFNBTUFSQBUUFSOT

    ʹఆٛ͞Ε͍ͯΔ(SPL1BUUFSO໊Ͱࢀর͞Εͯ ͍Δ
  53. File Data Visualizerͷ ରԠϑΝΠϧܗࣜ • ٙ໰ ճ౴ ΤϯίʔσΟϯά͸ԿʹରԠʁ 4+*4 &6$+1

    65'ͳͲ UTF-8ɺBOM༗ແͷରԠ͸ʁ ʓ վߦίʔυ͸LFɺCRLFͷͲͪΒʹ΋ରԠʁ ʓ $47͸೚ҙͷ۠੾ΓจࣈʹରԠʁ ʓ ೔෇ͷରԠϑΥʔϚοτ͸ʁ *40ͳͲछྨ ରԠ͍ͯ͠Δஶ໊ͳϩάܗࣜ͸ʁ "QBDIF 5PNDBUͳͲछྨ όΠφϦϑΝΠϧʹ͸ରԠʁ º
  54. SpaceؒͰVisualize/Dashboard ΛҠಈͤ͞Δํ๏ • ݱঢ়͸ManagementͷSaved Object͔Β Export/Import͢Δ͔͠ͳͦ͞͏ • ҠಈݩͷSaved Object͔ΒVisualize/ DashboardΛJSONʹExport

    • ҠಈઌͷSaved Object͔ΒJSONΛImport
  55. Dev Toolsͷػೳ֦ு • Grok DebuggerʹSyntax Highlight௥Ճ

  56. Dev Toolsͷػೳ֦ு • ConsoleͰCtrl/Cmd + /͢Δͱɺ࣮ߦ͠Α͏ͱ͍ͯ͠ΔAPIͷυΩ ϡϝϯτ͕৽͍͠΢Οϯυ΢Ͱ։͘Α͏ʹͳͬͨ