$30 off During Our Annual Pro Sale. View Details »

Intro Elastic Stack at Telemetry WG

Intro Elastic Stack at Telemetry WG

2018/06/29にTelemetryWGで使用したスライド。

Jun Ohtani

July 05, 2018
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. !1
    Jun Ohtani
    2018/06/29 at TelemetryWG
    @johtani
    Elastic stackͷ঺հ

    View Slide

  2. ‹#›

    View Slide

  3. about
    • Me, Jun Ohtani / Technical Advocate
    ‒ lucene-gosenίϛολʔ
    ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁
    ‒ http://blog.johtani.info

    • Elasticsearch, founded in 2012
    ‒ Products: Elasticsearch, Logstash, Kibana, Beats 

    X-Pack, Elastic Cloud

    Professional services: Support & development subscriptions
    ‒ Trainings, Consulting, SaaS
    !3

    View Slide

  4. 4
    ElasticελοΫ

    View Slide

  5. !5
    Elastic Stack
    Φʔϓϯιʔε
    ΛϦϦʔε

    View Slide

  6. Πϯετʔϧ͸؆୯
    &MBTUJD4UBDLΛ֦ு
    αϒεΫϦϓγϣϯʹؚΉ
    X-Pack
    !6
    Security
    Alerting
    Monitoring
    Reporting
    Graph
    Machine Learning

    View Slide

  7. ϝτϦΫε/ϩάղੳΛ
    ࢝ΊͯΈΑ͏

    View Slide

  8. ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ
    !8
    σʔλ Import Parse/

    Store/Search
    Visualize

    View Slide

  9. ܰྔσʔλγούʔ
    9
    Beats

    View Slide

  10. 10
    Beats
    ܰྔσʔλγούʔ
    ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ
    Logstashʹసૹ
    Elastic Cloudʹసૹ
    Libbeat: ΧελϜbeatsͷͨ
    ΊͷAPIϑϨʔϜϫʔΫ
    30Ҏ্ͷίϛϡχςΟbeats

    View Slide

  11. 11
    FILEBEAT
    ϩάϑΝΠϧ
    METRICBEAT
    ϝτϦοΫ৘ใ
    PACKETBEAT
    ωοτϫʔΫ
    WINGLOGBEAT
    WindowΠϕϯτ
    ͞Βʹ30Λ௒͑ΔίϛϡχςΟ
    Beats͕͋Γɺ૿Ճத
    Apachebeat, dockbeat, httpbeat,
    mysqlbeat, nginxbeat, redis beats,
    twitterbeat, and more

    View Slide

  12. Collect system
    and application
    metrics
    Metricbeat

    View Slide

  13. lots of modules
    Metricbeat

    View Slide

  14. tail log from
    file
    Filebeat

    View Slide

  15. many modules
    Filebeat

    View Slide

  16. Capture the
    Packet
    Packetbeat

    View Slide

  17. Capture the
    Packet
    Packetbeat

    View Slide

  18. Welcome
    to 1998
    winlogbeat

    View Slide

  19. Now
    winlogbeat

    View Slide

  20. 20
    Elasticsearch

    View Slide

  21. ݕࡧͱͯ͠ͷ

    Elasticsearch

    View Slide

  22. ؆୯ͳCRUD

    View Slide

  23. σʔλొ࿥
    23
    curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'

    View Slide

  24. σʔλߋ৽
    24
    curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'

    View Slide

  25. σʔλ࡟আ
    !25
    curl -X DELETE localhost:9200/books/book/1
    σʔλͷऔಘ
    curl —X GET localhost:9200/books/book/1
    curl —X GET localhost:9200/books/book/1/_source

    View Slide

  26. ݕࡧ
    !26
    curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]

    View Slide

  27. ݕࡧ - Query DSL
    !27
    curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0

    View Slide

  28. ෼ࢄߏ੒ɺ

    εέʔϧ

    View Slide

  29. Basic terms
    • ΠϯσοΫε
    ‒ σʔλͷ࿦ཧతͳू߹ɻ

    RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical
    • ϨϓϦέʔγϣϯ
    • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্
    • SPOFͷղফ
    • γϟʔσΟϯά
    • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ

    ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্

    σʔλϑϩʔ੍ޚ
    !29

    View Slide

  30. γϟʔυͱϨϓϦΧ
    !30
    node 1
    orders
    products
    1
    4
    1 2
    2
    3
    curl -X PUT localhost:9200/orders -d '{
    "settings.index.number_of_shards" : 4
    "settings.index.number_of_replicas" : 1
    }'
    curl -X PUT localhost:9200/products -d '{
    "settings.index.number_of_shards" : 2
    "settings.index.number_of_replicas" : 0
    }'

    View Slide

  31. γϟʔυͱϨϓϦΧ
    !31
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3

    View Slide

  32. ࣗಈతͳ෼ࢄ
    !32
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3

    View Slide

  33. ͦͷଞͷػೳ

    View Slide

  34. elasticsearch
    ͞·͟·ͳܗࣜͷσʔλͰ
    GeoݕࡧՄೳ


    Ң౓ܦ౓ɺGeoHashɺ
    GeoShape…
    GEO

    View Slide

  35. Ecosystem
    • Plugins
    ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ
    • ΫϥΠΞϯτϥΠϒϥϦ
    • Java, Ruby, python, php, perl, javascript, .NET
    • Scala, clojure, go
    !35

    View Slide

  36. Elasticsearch - The Definitive guide


    http://www.elastic.co/guide/en/
    elasticsearch/guide/current/index.html
    36
    ৄ͘͠஌Γ͍ͨํ͸

    View Slide

  37. 37
    KibanaͰՄࢹԽ

    View Slide

  38. Kibana 5
    • ElasticsearchͷσʔλΛՄࢹԽ
    • Node.js server & JavaScript
    • Apache License 2.0
    • Elastic Stackͷ૭ͷ໾ׂ
    • ༷ʑͳGUIΛPluginͱ͍ͯެ։
    • MarvelɺSenseɺTimelionͳͲ
    !38

    View Slide

  39. Kibana 5
    39

    View Slide

  40. Combining Search and Analytics
    !40

    View Slide

  41. σϞ for Kibana5
    Access Log
    41

    View Slide

  42. ຊ֨తʹղੳΛߦ͏ʹ͸ʁ

    View Slide

  43. View Slide

  44. 44
    Logstash

    View Slide

  45. Logstash in 10 seconds
    • ϩάɾσʔλͷऩूɾ؅ཧ
    • ऩूɺύʔεɾՃ޻ɺૹग़
    • ΦʔϓϯιʔεɿApache License 2.0
    • Ruby app (JRuby)
    !45

    View Slide

  46. Logstash architecture
    !46
    Input Output
    Filter
    ? ?
    collect and split alter and enrich store and visualize

    View Slide

  47. ઃఆ
    47
    input {

    }
    filter {

    }
    output {

    }

    View Slide

  48. ઃఆɿinput
    48
    input {
    file {
    path => “/Users/johtani/sample/*_log"
    start_position => "beginning"
    }
    }

    View Slide

  49. 1ߦ1σʔλ
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
    1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
    Firefox/5.0"
    49

    View Slide

  50. ઃఆɿfilter
    50
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  51. ύʔε
    !51
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",
    "clientip": "189.120.xx.xx",
    "ident": "-",
    "auth": "-",
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    "verb": "GET",
    "request": "/manager/html",

    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

    View Slide

  52. ઃఆɿfilter
    !52
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  53. ೔෇ͷύʔε
    53
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",

    "timestamp": "02/Dec/2014:12:18:29 +0900",

    }
    {…
    "@timestamp": "2014-12-02T03:18:29.000Z",

    "timestamp": "02/Dec/2014:12:18:29 +0900",

    }

    View Slide

  54. ઃఆɿfilter
    !54
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  55. IP͔ΒҢ౓ܦ౓ͳͲ෇༩
    55
    "clientip": "189.120.xx.xx",
    "clientip": "189.120.xx.xx",
    "geoip": {
    "ip": “189.120.xxx.xxx”,

    "country_name": "Brazil",
    "continent_code": "SA",
    "region_name": "27",
    "city_name": "São Paulo",
    "latitude":

    View Slide

  56. ઃఆɿfilter
    !56
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  57. ϢʔβΤʔδΣϯτͷύʔε
    57
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:
    5.0) Gecko/20100101 Firefox/5.0\""
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:
    5.0) Gecko/20100101 Firefox/5.0\""
    "useragent": {
    "name": "Firefox",
    "os": "Windows XP",
    "os_name": "Windows XP",
    "device": "Other",
    "major": "5",
    "minor": "0"

    View Slide

  58. ઃఆɿoutput
    58
    output {
    elasticsearch {
    hosts => ["localhost"]
    index => “demo_access_log-%{+YYYY.MM.dd}”
    }
    }

    View Slide

  59. ࢀߟจݙ
    • Elasticsearch - The Definitive guide
    ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
    • ॻ੶ʢ೔ຊޠʣ
    ‒ σʔλ෼ੳج൫ߏஙೖ໳
    ‒ Elasticsearch࣮ફΨΠυ
    !59

    View Slide

  60. ࢀߟαΠτ
    • Ϣʔεέʔε
    • https://www.elastic.co/use-cases
    • DiscussʢWebϑΥʔϥϜʣ
    • https://discuss.elastic.co
    • Elastic{ON}ͷϏσΦͱࢿྉ
    • https://www.elastic.co/elasticon/videos
    • αϙʔτϝχϡʔ
    • https://www.elastic.co/subscriptions
    !60

    View Slide

  61. Thanks for listening!
    Q & A
    We’re hiring!
    https://www.elastic.co/about/careers/
    We’re helping!
    https://www.elastic.co/subscriptions
    http://training.elastic.co

    View Slide

  62. QAͰ঺հͨ͠ػೳ

    View Slide

  63. !63
    Filebeatͷೖྗ͕ଟ༷ʹ
    • TCP Input
    ‒ SSL/TLSΛαϙʔτ
    • UDP Input
    • Syslog Input
    ‒ BSD RFC3164Λαϙʔτ
    ‒ ϓϩτίϧ͸TCPͱUDP
    filebeat.inputs:

    - type: tcp

    max_message_size: 10MiB

    host: "localhost:9000"
    filebeat.inputs:

    - type: udp

    max_message_size: 10KiB

    host: "localhost:8080"
    filebeat.inputs:

    - type: syslog

    protocol.tcp:

    host: "localhost:9000"
    filebeat.yml

    View Slide

  64. !64
    Data Rollups
    ● API for creating an Elasticsearch process
    to periodically store aggregate statistics
    ● Primary benefit is space savings
    ○ Faster queries
    ○ Potentially less nodes to manage
    ○ Smaller snapshots
    ○ Longer retention times
    ○ etc.
    ● Query rolled up data and “live” data
    together in a single query.
    Rollups API (6.3 - Experimental)
    ● ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥
    ● ओͳར఺͸༰ྔͷ࡟ݮ
    ● σʔλ͕গͳ͘ͳΔͨΊ
    ○ Query͕ΑΓߴ଎ʹ
    ○ গͳ͍ϊʔυͰσʔλΛ؅ཧ
    ○ Snapshot͕ΑΓখ͘͞
    ○ σʔλͷอ࣋ظ͕ؒΑΓ௕͘
    ● 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ
    ༰ྔ͕ɻɻɻ
    X-Pack feature (Basic, free)

    View Slide

  65. !65
    Raw Minute Hour Day
    Docs: 9,041,000 1,448,285 49,554 8,447
    Size: 2.23gb 1.25gb 48.40mb 9.10mb
    Docs % : -83.98% -99.45% -99.91%
    Size %: -43.68% -97.84% -99.59%
    (avg ~200 docs per minute, 32 days of data, single host)
    (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics)
    Rolling up Metricbeat data
    ༰ྔ࡟ݮͷҰྫ
    MetricbeatͷϩʔϧΞοϓ
    (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ)
    (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics)
    X-Pack feature (Basic, free)

    View Slide