What's new in Elastic Stack 6.3

第23回Elasticsarch勉強会で発表した6.3の新機能の紹介スライドです。

Webinarの録画ビデオではデモもありますので、参考にしていただければと。
https://www.elastic.co/jp/webinars/elastic-stack-6-3

Transcript

1. Jun Ohtani, Community Engineer

   Kosho Owa, Principal Solution Architect
   Elastic Stack 6.3 ঺հ
   

   View Slide

2. !2
   ηογϣϯʹೖΔ

   ͦͷલʹɻ

   QRίʔυϦʔμʔͷ͝༻ҙΛʂ
   

   View Slide

3. !3
   bit.ly/EnqEsMeetup
   

   View Slide

4. !4
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. !8
   

   View Slide

9. !9
   

   View Slide

10. !10
    Elasticsearch SQL (6.3 - Experimental)
    ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ
    

    View Slide

11. !11
    Elasticsearch SQL (6.3)
    ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ
    ● REST API & CLI Client (Basic License)
    ○ SQLγϯλοΫεͰElasticsearchͱ΍ΓͱΓ
    ○ දܗࣜͰ݁ՌΛදࣔ
    ● Translate API (Basic License)
    ○ SQLίϚϯυΛElasticsearchͷQuery DSLʹม׵
    ○ ElasticsearchͷQuery DSLγϯλοΫεʹ׳ΕΔͨΊͷศརͳํ๏
    ● JDBC Client (Platinum License)
    

    View Slide

12. !12
    Data Rollups
    ● API for creating an Elasticsearch process
    to periodically store aggregate statistics
    ● Primary benefit is space savings
    ○ Faster queries
    ○ Potentially less nodes to manage
    ○ Smaller snapshots
    ○ Longer retention times
    ○ etc.
    ● Query rolled up data and “live” data
    together in a single query.
    Rollups API (6.3 - Experimental)
    ● ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥
    ● ओͳར఺͸༰ྔͷ࡟ݮ
    ● σʔλ͕গͳ͘ͳΔͨΊ
    ○ Query͕ΑΓߴ଎ʹ
    ○ গͳ͍ϊʔυͰσʔλΛ؅ཧ
    ○ Snapshot͕ΑΓখ͘͞
    ○ σʔλͷอ࣋ظ͕ؒΑΓ௕͘
    ● 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ
    ༰ྔ͕ɻɻɻ
    X-Pack feature (Basic, free)
    

    View Slide

13. !13
    Raw Minute Hour Day
    Docs: 9,041,000 1,448,285 49,554 8,447
    Size: 2.23gb 1.25gb 48.40mb 9.10mb
    Docs % : -83.98% -99.45% -99.91%
    Size %: -43.68% -97.84% -99.59%
    (avg ~200 docs per minute, 32 days of data, single host)
    (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics)
    Rolling up Metricbeat data
    ༰ྔ࡟ݮͷҰྫ
    MetricbeatͷϩʔϧΞοϓ
    (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ)
    (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics)
    X-Pack feature (Basic, free)
    

    View Slide

14. !14
    ͦͷଞʹ΋ …
    • Java 10 (6.3) for Elasticsearch
    • Painless execute API
    • ML jobs Ͱ Cross Cluster Search ͕Մೳʹ(X-Pack Platinum)
    • ML job ͷࣄલνΣοΫ (X-Pack Platinum)
    

    View Slide

15. !15
    

    View Slide

16. !16
    Kibana Query LanguageͰࣗಈิ׬
    

    View Slide

17. !17
    Recently Viewed ΍ Recent Time Range
    ௚લʹԿΛ͔ͯͨ͠ͳ?...
    Recently viewed objects Recently used time range
    

    View Slide

18. !18
    Kibana Lab Visualizations
    Input controls (supports chaining) Vega Plugin
    

    View Slide

19. !19
    Index Management UI
    UIͰIndex؅ཧ
    X-Pack feature (Basic, free)
    

    View Slide

20. !20
    License Management UI
    TrialΛ࢝ΊͨΓɺBasic΁໭ͨ͠ΓɺϥΠηϯεͷߋ৽͕UI͔Β
    

    View Slide

21. !21
    ͦͷଞʹ΋…
    • DashboardͷύωϧͷαΠζมߋ͕ΑΓॊೈʹ
    • MonitoringͷػೳΛKibanaͷը໘͔Β༗ޮʹ(σϑΥϧτ͸ແޮ)
    • APMͱWatcherͷ࿈ܞ͕༰қʹ
    

    View Slide

22. !22
    

    View Slide

23. !23
    LogstashͷઃఆΛΑΓ؆ུԽ
    ྫ͑͹...
    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ
    

    View Slide

24. !24
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Distributor

    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ
    

    View Slide

25. !25
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Output Isolator

    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ
    

    View Slide

26. !26
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Forked Path

    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ
    

    View Slide

27. !27
    LogstashͷઃఆΛΑΓ؆ུԽ

    • Collector
    Pipelineؒ௨৴ (Beta)
    ઃఆΛΑΓγϯϓϧʹ
    

    View Slide

28. !28
    SNMP Poller (Beta)
    Centralized polling of SNMP agents with a Logstash input plugin
    ϕʔλ൛ͷػೳ
    • TCP΋͘͠͸UDPͰSNMP v1 ͱ v2c ΛϙʔϦϯά
    • SNMP GETs ͱ WALKs Λαϙʔτ
    • ఆظతʹϙʔϦϯά
    • MIB ϑΝΠϧΠϯϙʔτ
    • ޓ׵ੑ: Logstash 2.4 Ҏ্ʹରԠ
    Blog: https://www.elastic.co/blog/logstash-lines-support-for-tcp-and-snmpv1-in-snmp-input-plugin
    Routers
    Switches
    Servers
    SNMP Agents
    v0.1.0.beta1
    

    View Slide

29. !29
    ͦͷଞʹ΋...
    • Persistent Queue Breaking changes
    • S3ͷinput/outputͰcustom endpointͱregionͷࢦఆ͕Մೳʹ
    • PQͷσΟεΫͷ࢖༻ྔ΋MonitoringͰऔಘ (Basic)
    

    View Slide

30. !30
    

    View Slide

31. !31
    • at-least-once deliveryΛఏڙ

    • Metricbeat΍AuditbeatͷΑ͏ͳ

    λΠϓͰॏཁ
    • Ϧελʔτ΍ωοτϫʔΫো֐Ͱ΋
    σʔλ్͕੾Εͳ͍
    Disk΁Spool (Beta)
    ΤοδͰӬଓԽ
    # ____beat.yml
    queue:
    spool:
    file:
    path: "${path.data}/
    spool.dat"
    permissions: 0600 # One-
    time
    size: 100MiB # One-
    time
    #prealloc: true # One-
    time
    write:
    buffer_size: 1MiB
    flush.timeout: 1s
    flush.events: 16384
    

    View Slide

32. !32
    ௥Ճ͞ΕͨModule
    Filebeat Metricbeat
    

    View Slide

33. !33
    Filebeatͷೖྗ͕ଟ༷ʹ
    • TCP Input
    ‒ SSL/TLSΛαϙʔτ
    • UDP Input
    • Syslog Input
    ‒ BSD RFC3164Λαϙʔτ
    ‒ ϓϩτίϧ͸TCPͱUDP
    filebeat.inputs:

    - type: tcp

    max_message_size: 10MiB

    host: "localhost:9000"
    filebeat.inputs:

    - type: udp

    max_message_size: 10KiB

    host: "localhost:8080"
    filebeat.inputs:

    - type: syslog

    protocol.tcp:

    host: "localhost:9000"
    filebeat.yml
    

    View Slide

34. !34
    ͦͷଞʹ΋...
    • Kubernetes/DockerͷAudodiscoverػೳͷվળ
    • add_host_metadataͷ௥Ճ
    

    View Slide

35. !35
    6.3.1͕ϦϦʔε

    ͞Ε͍ͯ·͢ʂ
    

    View Slide

36. !36
    Demo
    

    View Slide