Intro Elastic Stack at Telemetry WG

2018/06/29にTelemetryWGで使用したスライド。

Transcript

1. !1
   Jun Ohtani
   2018/06/29 at TelemetryWG
   @johtani
   Elastic stackͷ঺հ
   

   View Slide

2. ‹#›
   

   View Slide

3. about
   • Me, Jun Ohtani / Technical Advocate
   ‒ lucene-gosenίϛολʔ
   ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁
   ‒ http://blog.johtani.info

   • Elasticsearch, founded in 2012
   ‒ Products: Elasticsearch, Logstash, Kibana, Beats 

   X-Pack, Elastic Cloud

   Professional services: Support & development subscriptions
   ‒ Trainings, Consulting, SaaS
   !3
   

   View Slide

4. 4
   ElasticελοΫ
   

   View Slide

5. !5
   Elastic Stack
   Φʔϓϯιʔε
   ΛϦϦʔε
   

   View Slide

6. Πϯετʔϧ͸؆୯
   &MBTUJD4UBDLΛ֦ு
   αϒεΫϦϓγϣϯʹؚΉ
   X-Pack
   !6
   Security
   Alerting
   Monitoring
   Reporting
   Graph
   Machine Learning
   

   View Slide

7. ϝτϦΫε/ϩάղੳΛ
   ࢝ΊͯΈΑ͏
   

   View Slide

8. ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ
   !8
   σʔλ Import Parse/

   Store/Search
   Visualize
   

   View Slide

9. ܰྔσʔλγούʔ
   9
   Beats
   

   View Slide

10. 10
    Beats
    ܰྔσʔλγούʔ
    ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ
    Logstashʹసૹ
    Elastic Cloudʹసૹ
    Libbeat: ΧελϜbeatsͷͨ
    ΊͷAPIϑϨʔϜϫʔΫ
    30Ҏ্ͷίϛϡχςΟbeats
    

    View Slide

11. 11
    FILEBEAT
    ϩάϑΝΠϧ
    METRICBEAT
    ϝτϦοΫ৘ใ
    PACKETBEAT
    ωοτϫʔΫ
    WINGLOGBEAT
    WindowΠϕϯτ
    ͞Βʹ30Λ௒͑ΔίϛϡχςΟ
    Beats͕͋Γɺ૿Ճத
    Apachebeat, dockbeat, httpbeat,
    mysqlbeat, nginxbeat, redis beats,
    twitterbeat, and more
    

    View Slide

12. Collect system
    and application
    metrics
    Metricbeat
    

    View Slide

13. lots of modules
    Metricbeat
    

    View Slide

14. tail log from
    file
    Filebeat
    

    View Slide

15. many modules
    Filebeat
    

    View Slide

16. Capture the
    Packet
    Packetbeat
    

    View Slide

17. Capture the
    Packet
    Packetbeat
    

    View Slide

18. Welcome
    to 1998
    winlogbeat
    

    View Slide

19. Now
    winlogbeat
    

    View Slide

20. 20
    Elasticsearch
    

    View Slide

21. ݕࡧͱͯ͠ͷ

    Elasticsearch
    

    View Slide

22. ؆୯ͳCRUD
    

    View Slide

23. σʔλొ࿥
    23
    curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    

    View Slide

24. σʔλߋ৽
    24
    curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    

    View Slide

25. σʔλ࡟আ
    !25
    curl -X DELETE localhost:9200/books/book/1
    σʔλͷऔಘ
    curl —X GET localhost:9200/books/book/1
    curl —X GET localhost:9200/books/book/1/_source
    

    View Slide

26. ݕࡧ
    !26
    curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    

    View Slide

27. ݕࡧ - Query DSL
    !27
    curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    

    View Slide

28. ෼ࢄߏ੒ɺ

    εέʔϧ
    

    View Slide

29. Basic terms
    • ΠϯσοΫε
    ‒ σʔλͷ࿦ཧతͳू߹ɻ

    RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical
    • ϨϓϦέʔγϣϯ
    • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্
    • SPOFͷղফ
    • γϟʔσΟϯά
    • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ

    ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্

    σʔλϑϩʔ੍ޚ
    !29
    

    View Slide

30. γϟʔυͱϨϓϦΧ
    !30
    node 1
    orders
    products
    1
    4
    1 2
    2
    3
    curl -X PUT localhost:9200/orders -d '{
    "settings.index.number_of_shards" : 4
    "settings.index.number_of_replicas" : 1
    }'
    curl -X PUT localhost:9200/products -d '{
    "settings.index.number_of_shards" : 2
    "settings.index.number_of_replicas" : 0
    }'
    

    View Slide

31. γϟʔυͱϨϓϦΧ
    !31
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3
    

    View Slide

32. ࣗಈతͳ෼ࢄ
    !32
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3
    

    View Slide

33. ͦͷଞͷػೳ
    

    View Slide

34. elasticsearch
    ͞·͟·ͳܗࣜͷσʔλͰ
    GeoݕࡧՄೳ

    

    Ң౓ܦ౓ɺGeoHashɺ
    GeoShape…
    GEO
    

    View Slide

35. Ecosystem
    • Plugins
    ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ
    • ΫϥΠΞϯτϥΠϒϥϦ
    • Java, Ruby, python, php, perl, javascript, .NET
    • Scala, clojure, go
    !35
    

    View Slide

36. Elasticsearch - The Definitive guide

    

    http://www.elastic.co/guide/en/
    elasticsearch/guide/current/index.html
    36
    ৄ͘͠஌Γ͍ͨํ͸
    

    View Slide

37. 37
    KibanaͰՄࢹԽ
    

    View Slide

38. Kibana 5
    • ElasticsearchͷσʔλΛՄࢹԽ
    • Node.js server & JavaScript
    • Apache License 2.0
    • Elastic Stackͷ૭ͷ໾ׂ
    • ༷ʑͳGUIΛPluginͱ͍ͯެ։
    • MarvelɺSenseɺTimelionͳͲ
    !38
    

    View Slide

39. Kibana 5
    39
    

    View Slide

40. Combining Search and Analytics
    !40
    

    View Slide

41. σϞ for Kibana5
    Access Log
    41
    

    View Slide

42. ຊ֨తʹղੳΛߦ͏ʹ͸ʁ
    

    View Slide

43. View Slide

44. 44
    Logstash
    

    View Slide

45. Logstash in 10 seconds
    • ϩάɾσʔλͷऩूɾ؅ཧ
    • ऩूɺύʔεɾՃ޻ɺૹग़
    • ΦʔϓϯιʔεɿApache License 2.0
    • Ruby app (JRuby)
    !45
    

    View Slide

46. Logstash architecture
    !46
    Input Output
    Filter
    ? ?
    collect and split alter and enrich store and visualize
    

    View Slide

47. ઃఆ
    47
    input {
    …
    }
    filter {
    …
    }
    output {
    …
    }
    

    View Slide

48. ઃఆɿinput
    48
    input {
    file {
    path => “/Users/johtani/sample/*_log"
    start_position => "beginning"
    }
    }
    

    View Slide

49. 1ߦ1σʔλ
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
    1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
    Firefox/5.0"
    49
    

    View Slide

50. ઃఆɿfilter
    50
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

51. ύʔε
    !51
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"
    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",
    "clientip": "189.120.xx.xx",
    "ident": "-",
    "auth": "-",
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    "verb": "GET",
    "request": "/manager/html",
    …
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
    

    View Slide

52. ઃఆɿfilter
    !52
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

53. ೔෇ͷύʔε
    53
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",
    …
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    …
    }
    {…
    "@timestamp": "2014-12-02T03:18:29.000Z",
    …
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    …
    }
    

    View Slide

54. ઃఆɿfilter
    !54
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

55. IP͔ΒҢ౓ܦ౓ͳͲ෇༩
    55
    "clientip": "189.120.xx.xx",
    "clientip": "189.120.xx.xx",
    "geoip": {
    "ip": “189.120.xxx.xxx”,
    …
    "country_name": "Brazil",
    "continent_code": "SA",
    "region_name": "27",
    "city_name": "São Paulo",
    "latitude":
    

    View Slide

56. ઃఆɿfilter
    !56
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

57. ϢʔβΤʔδΣϯτͷύʔε
    57
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:
    5.0) Gecko/20100101 Firefox/5.0\""
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:
    5.0) Gecko/20100101 Firefox/5.0\""
    "useragent": {
    "name": "Firefox",
    "os": "Windows XP",
    "os_name": "Windows XP",
    "device": "Other",
    "major": "5",
    "minor": "0"
    

    View Slide

58. ઃఆɿoutput
    58
    output {
    elasticsearch {
    hosts => ["localhost"]
    index => “demo_access_log-%{+YYYY.MM.dd}”
    }
    }
    

    View Slide

59. ࢀߟจݙ
    • Elasticsearch - The Definitive guide
    ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
    • ॻ੶ʢ೔ຊޠʣ
    ‒ σʔλ෼ੳج൫ߏஙೖ໳
    ‒ Elasticsearch࣮ફΨΠυ
    !59
    

    View Slide

60. ࢀߟαΠτ
    • Ϣʔεέʔε
    • https://www.elastic.co/use-cases
    • DiscussʢWebϑΥʔϥϜʣ
    • https://discuss.elastic.co
    • Elastic{ON}ͷϏσΦͱࢿྉ
    • https://www.elastic.co/elasticon/videos
    • αϙʔτϝχϡʔ
    • https://www.elastic.co/subscriptions
    !60
    

    View Slide

61. Thanks for listening!
    Q & A
    We’re hiring!
    https://www.elastic.co/about/careers/
    We’re helping!
    https://www.elastic.co/subscriptions
    http://training.elastic.co
    

    View Slide

62. QAͰ঺հͨ͠ػೳ
    

    View Slide

63. !63
    Filebeatͷೖྗ͕ଟ༷ʹ
    • TCP Input
    ‒ SSL/TLSΛαϙʔτ
    • UDP Input
    • Syslog Input
    ‒ BSD RFC3164Λαϙʔτ
    ‒ ϓϩτίϧ͸TCPͱUDP
    filebeat.inputs:

    - type: tcp

    max_message_size: 10MiB

    host: "localhost:9000"
    filebeat.inputs:

    - type: udp

    max_message_size: 10KiB

    host: "localhost:8080"
    filebeat.inputs:

    - type: syslog

    protocol.tcp:

    host: "localhost:9000"
    filebeat.yml
    

    View Slide

64. !64
    Data Rollups
    ● API for creating an Elasticsearch process
    to periodically store aggregate statistics
    ● Primary benefit is space savings
    ○ Faster queries
    ○ Potentially less nodes to manage
    ○ Smaller snapshots
    ○ Longer retention times
    ○ etc.
    ● Query rolled up data and “live” data
    together in a single query.
    Rollups API (6.3 - Experimental)
    ● ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥
    ● ओͳར఺͸༰ྔͷ࡟ݮ
    ● σʔλ͕গͳ͘ͳΔͨΊ
    ○ Query͕ΑΓߴ଎ʹ
    ○ গͳ͍ϊʔυͰσʔλΛ؅ཧ
    ○ Snapshot͕ΑΓখ͘͞
    ○ σʔλͷอ࣋ظ͕ؒΑΓ௕͘
    ● 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ
    ༰ྔ͕ɻɻɻ
    X-Pack feature (Basic, free)
    

    View Slide

65. !65
    Raw Minute Hour Day
    Docs: 9,041,000 1,448,285 49,554 8,447
    Size: 2.23gb 1.25gb 48.40mb 9.10mb
    Docs % : -83.98% -99.45% -99.91%
    Size %: -43.68% -97.84% -99.59%
    (avg ~200 docs per minute, 32 days of data, single host)
    (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics)
    Rolling up Metricbeat data
    ༰ྔ࡟ݮͷҰྫ
    MetricbeatͷϩʔϧΞοϓ
    (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ)
    (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics)
    X-Pack feature (Basic, free)
    

    View Slide