Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's new in Elastic Stack 6.3

What's new in Elastic Stack 6.3

第23回Elasticsarch勉強会で発表した6.3の新機能の紹介スライドです。

Webinarの録画ビデオではデモもありますので、参考にしていただければと。
https://www.elastic.co/jp/webinars/elastic-stack-6-3

Jun Ohtani

July 25, 2018
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. Jun Ohtani, Community Engineer

    Kosho Owa, Principal Solution Architect
    Elastic Stack 6.3 ঺հ

    View Slide

  2. !2
    ηογϣϯʹೖΔ

    ͦͷલʹɻ

    QRίʔυϦʔμʔͷ͝༻ҙΛʂ

    View Slide

  3. !3
    bit.ly/EnqEsMeetup

    View Slide

  4. !4

    View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. !8

    View Slide

  9. !9

    View Slide

  10. !10
    Elasticsearch SQL (6.3 - Experimental)
    ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ

    View Slide

  11. !11
    Elasticsearch SQL (6.3)
    ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ
    ● REST API & CLI Client (Basic License)
    ○ SQLγϯλοΫεͰElasticsearchͱ΍ΓͱΓ
    ○ දܗࣜͰ݁ՌΛදࣔ
    ● Translate API (Basic License)
    ○ SQLίϚϯυΛElasticsearchͷQuery DSLʹม׵
    ○ ElasticsearchͷQuery DSLγϯλοΫεʹ׳ΕΔͨΊͷศརͳํ๏
    ● JDBC Client (Platinum License)

    View Slide

  12. !12
    Data Rollups
    ● API for creating an Elasticsearch process
    to periodically store aggregate statistics
    ● Primary benefit is space savings
    ○ Faster queries
    ○ Potentially less nodes to manage
    ○ Smaller snapshots
    ○ Longer retention times
    ○ etc.
    ● Query rolled up data and “live” data
    together in a single query.
    Rollups API (6.3 - Experimental)
    ● ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥
    ● ओͳར఺͸༰ྔͷ࡟ݮ
    ● σʔλ͕গͳ͘ͳΔͨΊ
    ○ Query͕ΑΓߴ଎ʹ
    ○ গͳ͍ϊʔυͰσʔλΛ؅ཧ
    ○ Snapshot͕ΑΓখ͘͞
    ○ σʔλͷอ࣋ظ͕ؒΑΓ௕͘
    ● 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ
    ༰ྔ͕ɻɻɻ
    X-Pack feature (Basic, free)

    View Slide

  13. !13
    Raw Minute Hour Day
    Docs: 9,041,000 1,448,285 49,554 8,447
    Size: 2.23gb 1.25gb 48.40mb 9.10mb
    Docs % : -83.98% -99.45% -99.91%
    Size %: -43.68% -97.84% -99.59%
    (avg ~200 docs per minute, 32 days of data, single host)
    (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics)
    Rolling up Metricbeat data
    ༰ྔ࡟ݮͷҰྫ
    MetricbeatͷϩʔϧΞοϓ
    (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ)
    (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics)
    X-Pack feature (Basic, free)

    View Slide

  14. !14
    ͦͷଞʹ΋ …
    • Java 10 (6.3) for Elasticsearch
    • Painless execute API
    • ML jobs Ͱ Cross Cluster Search ͕Մೳʹ(X-Pack Platinum)
    • ML job ͷࣄલνΣοΫ (X-Pack Platinum)

    View Slide

  15. !15

    View Slide

  16. !16
    Kibana Query LanguageͰࣗಈิ׬

    View Slide

  17. !17
    Recently Viewed ΍ Recent Time Range
    ௚લʹԿΛ͔ͯͨ͠ͳ?...
    Recently viewed objects Recently used time range

    View Slide

  18. !18
    Kibana Lab Visualizations
    Input controls (supports chaining) Vega Plugin

    View Slide

  19. !19
    Index Management UI
    UIͰIndex؅ཧ
    X-Pack feature (Basic, free)

    View Slide

  20. !20
    License Management UI
    TrialΛ࢝ΊͨΓɺBasic΁໭ͨ͠ΓɺϥΠηϯεͷߋ৽͕UI͔Β

    View Slide

  21. !21
    ͦͷଞʹ΋…
    • DashboardͷύωϧͷαΠζมߋ͕ΑΓॊೈʹ
    • MonitoringͷػೳΛKibanaͷը໘͔Β༗ޮʹ(σϑΥϧτ͸ແޮ)
    • APMͱWatcherͷ࿈ܞ͕༰қʹ

    View Slide

  22. !22

    View Slide

  23. !23
    LogstashͷઃఆΛΑΓ؆ུԽ
    ྫ͑͹...
    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ

    View Slide

  24. !24
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Distributor

    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ

    View Slide

  25. !25
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Output Isolator

    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ

    View Slide

  26. !26
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Forked Path

    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ

    View Slide

  27. !27
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Collector
    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ

    View Slide

  28. !28
    SNMP Poller (Beta)
    Centralized polling of SNMP agents with a Logstash input plugin
    ϕʔλ൛ͷػೳ
    • TCP΋͘͠͸UDPͰSNMP v1 ͱ v2c ΛϙʔϦϯά
    • SNMP GETs ͱ WALKs Λαϙʔτ
    • ఆظతʹϙʔϦϯά
    • MIB ϑΝΠϧΠϯϙʔτ
    • ޓ׵ੑ: Logstash 2.4 Ҏ্ʹରԠ
    Blog: https://www.elastic.co/blog/logstash-lines-support-for-tcp-and-snmpv1-in-snmp-input-plugin
    Routers
    Switches
    Servers
    SNMP Agents
    v0.1.0.beta1

    View Slide

  29. !29
    ͦͷଞʹ΋...
    • Persistent Queue Breaking changes
    • S3ͷinput/outputͰcustom endpointͱregionͷࢦఆ͕Մೳʹ
    • PQͷσΟεΫͷ࢖༻ྔ΋MonitoringͰऔಘ (Basic)

    View Slide

  30. !30

    View Slide

  31. !31
    • at-least-once deliveryΛఏڙ

    • Metricbeat΍AuditbeatͷΑ͏ͳ

    λΠϓͰॏཁ
    • Ϧελʔτ΍ωοτϫʔΫো֐Ͱ΋
    σʔλ్͕੾Εͳ͍
    Disk΁Spool (Beta)
    ΤοδͰӬଓԽ
    # ____beat.yml
    queue:
    spool:
    file:
    path: "${path.data}/
    spool.dat"
    permissions: 0600 # One-
    time
    size: 100MiB # One-
    time
    #prealloc: true # One-
    time
    write:
    buffer_size: 1MiB
    flush.timeout: 1s
    flush.events: 16384

    View Slide

  32. !32
    ௥Ճ͞ΕͨModule
    Filebeat Metricbeat

    View Slide

  33. !33
    Filebeatͷೖྗ͕ଟ༷ʹ
    • TCP Input
    ‒ SSL/TLSΛαϙʔτ
    • UDP Input
    • Syslog Input
    ‒ BSD RFC3164Λαϙʔτ
    ‒ ϓϩτίϧ͸TCPͱUDP
    filebeat.inputs:

    - type: tcp

    max_message_size: 10MiB

    host: "localhost:9000"
    filebeat.inputs:

    - type: udp

    max_message_size: 10KiB

    host: "localhost:8080"
    filebeat.inputs:

    - type: syslog

    protocol.tcp:

    host: "localhost:9000"
    filebeat.yml

    View Slide

  34. !34
    ͦͷଞʹ΋...
    • Kubernetes/DockerͷAudodiscoverػೳͷվળ
    • add_host_metadataͷ௥Ճ

    View Slide

  35. !35
    6.3.1͕ϦϦʔε

    ͞Ε͍ͯ·͢ʂ

    View Slide

  36. !36
    Demo

    View Slide