Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticスタックではじめるログ解析入門 #osc16ep

Jun Ohtani
January 29, 2016

Elasticスタックではじめるログ解析入門 #osc16ep

オープンソースカンファレンス 2016 [email protected]での発表資料

Jun Ohtani

January 29, 2016
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. Elastic StackͰ͸͡ΊΔ
    ϩάղੳೖ໳
    Jun Ohtani
    @johtani
    2016/01/29

    View Slide

  2. View Slide

  3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    3
    • Elastic stack঺հ
    • Logstash - Logऩू
    • Elasticsearch - ݕࡧɾղੳ
    • Kibana - ՄࢹԽ
    • ঎༻ϓϥάΠϯ঺հ
    ΞδΣϯμ

    View Slide

  4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    4
    about
    • Me, Jun Ohtani / Technical Adovocate
    – lucene-gosenίϛολʔ
    – ElasticSearch Server೔ຊޠ൛ͷ຋༁
    – elasticsearch-extended-analysisͷ։ൃ
    – http://blog.johtani.info
    • Elasticsearch, founded in 2012
    – Products: Elasticsearch, Logstash, Kibana, Marvel, Sheild

    Professional services: Support & development subscriptions
    – Trainings

    View Slide

  5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    5
    ElasticελοΫ
    ElasticελοΫ

    View Slide

  6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    6
    ElasticελοΫ
    ML
    K
    B LE
    a
    B K
    B
    SWFB
    (
    M
    )
    W
    )

    View Slide

  7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    7
    ElasticελοΫɿOpen Source෦෼
    ML
    K
    B LE
    a
    B K
    B
    SWFB
    (
    M
    )
    W
    )

    View Slide

  8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    8
    Elastic stack
    Data Import/Parse/
    Export
    Store/Search
    Visualize

    View Slide

  9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    9
    Logstash
    Logstash

    View Slide

  10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    10
    Elastic stack
    Import/Parse/
    Export
    Store/Search
    Visualize
    Data

    View Slide

  11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    11
    Logstash in 10 seconds
    • ϩάɾσʔλͷऩूɾ؅ཧ
    • ऩूɺύʔεɾՃ޻ɺૹग़
    • ΦʔϓϯιʔεɿApache License 2.0
    • Ruby app (JRuby)
    • ࠷৽൛ɿ2.1.1

    View Slide

  12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    12
    Logstash architecture
    Logstash
    Input Output
    Filter
    ? ?
    collect and split alter and enrich store and visualize

    View Slide

  13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    13
    ઃఆ
    input {

    }
    filter {

    }
    output {

    }

    View Slide

  14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    14
    ઃఆɿinput
    input {
    file {
    path => “/Users/johtani/sample/*_log"
    start_position => "beginning"
    }
    }

    View Slide

  15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    15
    1ߦ1σʔλ
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
    1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
    Firefox/5.0"

    View Slide

  16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    16
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    17
    ύʔε
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
    1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
    Firefox/5.0"
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",
    "clientip": "189.120.xx.xx",
    "ident": "-",
    "auth": "-",
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    "verb": "GET",
    "request": "/manager/html",

    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\""
    }

    View Slide

  18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    18
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    19
    ೔෇ͷύʔε
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",

    "timestamp": "02/Dec/2014:12:18:29 +0900",

    }
    {…
    "@timestamp": "2014-12-02T03:18:29.000Z",

    "timestamp": "02/Dec/2014:12:18:29 +0900",

    }

    View Slide

  20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    20
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    21
    IP͔ΒҢ౓ܦ౓ͳͲ෇༩
    {…
    "clientip": "189.120.xx.xx",
    …}
    {…
    "clientip": "189.120.xx.xx",

    "geoip": {
    "ip": “189.120.xxx.xxx”,

    "country_name": "Brazil",
    "continent_code": "SA",
    "region_name": "27",
    "city_name": "São Paulo",
    "latitude": -23.473299999999995,
    "longitude": -46.66579999999999,

    View Slide

  22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    22
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }

    View Slide

  23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    23
    ϢʔβΤʔδΣϯτͷύʔε
    {…
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\""
    …}
    {…
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\""

    "useragent": {
    "name": "Firefox",
    "os": "Windows XP",
    "os_name": "Windows XP",
    "device": "Other",
    "major": "5",
    "minor": "0"
    }

    View Slide

  24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    24
    ઃఆɿoutput
    output {
    elasticsearch {
    hosts => ["localhost"]
    index => “demo_access_log-%{+YYYY.MM.dd}”
    }
    }

    View Slide

  25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    25
    Elasticsearch
    Elasticsearch

    View Slide

  26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    26
    ݕࡧͱͯ͠ͷ
    Elasticsearch
    ݕࡧͱͯ͠ͷElasticsearch

    View Slide

  27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    27
    Elasticsearchͱ͸ʁ
    Elasticsearchͱ͸ʁ

    View Slide

  28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    28
    ϑϦʔϫʔυݕࡧ

    View Slide

  29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    29
    ߜΓࠐΈ

    View Slide

  30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    30
    ϋΠϥΠτ

    View Slide

  31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    31
    ιʔτ

    View Slide

  32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    32
    ϖʔδϯά

    View Slide

  33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    33
    ूܭ

    View Slide

  34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    34
    αδΣετ

    View Slide

  35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    35
    Elasticsearch in 10 seconds
    • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON
    • Φʔϓϯιʔε: Apache License 2.0
    • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
    • JavaͰ࣮૷ɻ֦ு΋༰қ

    View Slide

  36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    36
    Powerful Search at Scale

    View Slide

  37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    37
    ؆୯ͳCRUD
    ؆୯ͳCRUD

    View Slide

  38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    38
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλొ࿥

    View Slide

  39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    39
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλߋ৽

    View Slide

  40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    40
    σʔλ࡟আ
    » curl -X DELETE localhost:9200/books/book/1
    σʔλͷऔಘ
    » curl —X GET localhost:9200/books/book/1
    » curl —X GET localhost:9200/books/book/1/_source

    View Slide

  41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    41
    ݕࡧ
    » curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    }
    }

    View Slide

  42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    42
    » curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    ...
    }
    }'
    ݕࡧ - Query DSL

    View Slide

  43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    43
    ෼ࢄߏ੒
    εέʔϧ
    ෼ࢄߏ੒ɺεέʔϧ

    View Slide

  44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    44
    Basic terms
    • ΠϯσοΫε
    – σʔλͷ࿦ཧతͳू߹ɻ

    RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical
    • ϨϓϦέʔγϣϯ
    • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্
    • SPOFͷղফ
    • γϟʔσΟϯά
    • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ

    ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্

    σʔλϑϩʔ੍ޚ

    View Slide

  45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    45
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1 2
    2
    2
    curl -X PUT localhost:9200/orders -d '{
    "settings.index.number_of_shards" : 4
    "settings.index.number_of_replicas" : 1
    }'
    curl -X PUT localhost:9200/products -d '{
    "settings.index.number_of_shards" : 2
    "settings.index.number_of_replicas" : 0
    }'

    View Slide

  46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    46
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3

    View Slide

  47. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    47
    ࣗಈతͳ෼ࢄ
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3

    View Slide

  48. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    48
    શจݕࡧͱ͸ʁ
    શจݕࡧͱ͸ʁ

    View Slide

  49. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    49
    શจݕࡧͱ͸ʁ
    • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจ
    ॻʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕ
    ࡧʯ΍ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·ͨ
    ͕ͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞
    ΕΔɻ

    ʢWikipediaΑΓʣ

    View Slide

  50. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    50
    ༻ޠ
    • ΠϯσοΫε
    ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ
    • υΩϡϝϯτʢจॻʣ
    – ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ
    • ϑΟʔϧυ
    – υΩϡϝϯτʹؚ·ΕΔଐੑ
    • ΫΤϦ
    – ݕࡧ৚݅ɺݕࡧࣜ

    View Slide

  51. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    51
    ༻ޠ
    • εΩʔϚ
    – υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ
    • λʔϜʢTermʣɺτʔΫϯʢTokenʣ
    – ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ
    – จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ
    – ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ

    View Slide

  52. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    52
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    υΩϡϝϯτͷొ࿥

    View Slide

  53. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    53
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ


    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ

    View Slide

  54. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    54
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ


    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    ୯ޠ͔Βidͷ഑ྻ͕
    Ҿ͚ΔΑ͏ʹ

    View Slide

  55. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    55
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ݕࡧ৚݅ೖྗ
    ΧπΦɹαβΤ

    View Slide

  56. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    56
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  57. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    57
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  58. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    58
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  59. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    59
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  60. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    60
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  61. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    61
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ

    ϫΧϝ 2
    1 2
    1 2
    1

    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ

    View Slide

  62. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    62
    ͦͷଞͷػೳ
    ͦͷଞͷػೳ

    View Slide

  63. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    63
    Geo
    • ͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ
    • σʔλɿ
    • Ң౓ܦ౓ɺGeoHashɺGeoShape
    • ݕࡧɿ
    • geo shape queryɺgeo shape filterɺgeo distance filterɺ
    • geo distance range filterɺgeo bounding box filterɺ
    • geo polygon filter

    View Slide

  64. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    64
    Percolator
    • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥
    • _percolate APIʹυΩϡϝϯτΛૹ৴
    • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ

    Ϛονͨ͠ΫΤϦΛฦ͢

    View Slide

  65. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    65
    Snapshot/Restore
    • 1.0͔Βಋೖ
    • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ
    • อଘઌʢRepositoryʣ
    – Shard FSɺS3ɺHDFSɺAzure…

    View Slide

  66. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    66
    Ecosystem
    • Plugins
    – ϓϥάΠϯʹΑΔػೳͷ௥Ճ
    • ΫϥΠΞϯτϥΠϒϥϦ
    • Ruby, python, php, perl, javascript, .NET
    • Scala, clojure, go
    • Hadoop integration

    View Slide

  67. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    67
    ৄ͘͠஌Γ͍ͨํ͸
    • Elasticsearch - The Definitive guide
    – http://www.elastic.co/guide/en/elasticsearch/guide/current/
    index.html

    View Slide

  68. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    68
    ղੳͱͯ͠ͷ
    Elasticsearch
    ղੳͱͯ͠ͷElasticsearch

    View Slide

  69. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    69
    aggregation
    aggregation

    View Slide

  70. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    70
    Aggregationͱ͸
    • 1.0͔Βಋೖ
    • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ
    • ֊૚తͳूܭɺάϧʔϓԽ

    ಈతͳूܭɺάϧʔϓԽ
    • େ͖͘2छྨ
    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά
    • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ

    View Slide

  71. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    71
    ྫɿݴޠ͝ͱͷϑΝηοτ
    » curl -XGET twitter-2014.08.22/_search -d '
    GET twitter-2014.08.22/_search
    {
    "facets": {
    "lang_facet": {
    "terms": {"field": "lang"}
    }
    }
    }

    View Slide

  72. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    72
    ྫɿݴޠ͝ͱͷϑΝηοτ
    "facets": {
    "lang_facet": {
    "_type": "terms",
    "missing": 0,
    "total": 4521075,
    "other": 320228,
    "terms": [
    {
    "term": "en",
    "count": 1471819
    },
    {
    "term": "ja",
    "count": 980145
    },

    View Slide

  73. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    73
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    » curl -XGET twitter-2014.08.22/_search -d '
    {
    "aggs": {
    "lang": {
    "terms": {"field": "lang" },
    "aggs": {
    "place": {
    "terms": {
    "field": “place.full_name", "size": 10
    }
    }
    }
    }
    }
    }

    View Slide

  74. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    74
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    "aggregations": {
    "lang": {
    "buckets": [{…}, {
    "key": "ja",
    "doc_count": 980145,
    "place": {
    "buckets": [
    { "key": "ژ౎ࢢ෬ݟ۠, ژ౎",
    "doc_count":252 },
    { "key": "ઍ୅ా۠, ౦ژ",
    "doc_count": 39 },…

    View Slide

  75. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    75
    bucket
    • filter
    • filters (>=1.4.0)
    • missing
    • nested
    • reverse nested (>=1.2.0)
    • children
    • terms
    • significant terms (>=1.1.0)
    • range
    • date range
    • ipv4 range
    • histogram
    • date histogram
    • geo distance
    • geohash grid

    View Slide

  76. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    76
    metric
    • min
    • max
    • sum
    • avg
    • stats
    • extended_stats
    • value count
    • percentiles (>=1.1.0)
    • percentile ranks (>=1.3)
    • cardinality (>=1.1.0)
    • geo bounds (>=1.3.0)
    • top_hits (>=1.3.0)
    • scripted metric 

    (>= 1.4.0)

    View Slide

  77. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    77
    KibanaͰՄࢹԽ
    KibanaͰՄࢹԽ

    View Slide

  78. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    78
    Kibana4
    • ElasticsearchͷσʔλΛՄࢹԽ
    • Node.js server & JavaScript
    • Apache License 2.0

    View Slide

  79. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    79
    Kibana 4

    View Slide

  80. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    80
    σϞ for Kibana4
    Access Log
    demo for Kibana4

    View Slide

  81. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    81
    Combining Search and Analytics

    View Slide

  82. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    82
    ͦͷଞͷ
    ElasticελοΫ
    ͦͷଞͷElasticελοΫ

    View Slide

  83. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    83
    Beats
    19
    #!"



    a ab


    U EC W e
    Pc i b c W e
    g


    b b ol st
    hU g


    c st r
    L

    View Slide

  84. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    84
    elasticsearch-hadoop
    20
    -
    •  D E H
    •  PD ecd
    ER
    •  g D
    • 
    CH
    •  Ca M DMS
    D FERC

    View Slide

  85. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    85
    ঎༻ϓϥάΠϯ
    ঎༻ϓϥάΠϯ

    View Slide

  86. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    86
    Shield
    Shield

    View Slide

  87. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    87
    Shieldͷಛ௃
    • User Authentication
    • LDAP/Active Directory
    • ϑΝΠϧϕʔε
    • Authorization
    • ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ
    • ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ
    • υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ
    • ηΩϡΞͳ௨৴
    • ElasticsearchϊʔυؒͷSSL/TLS
    • IPϑΟϧλϦϯά
    • ؂ࠪϩά

    View Slide

  88. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    88
    ΨΠυͳͲ
    • ϓϩμΫτϖʔδ
    – https://www.elastic.co/products/shield
    • ΨΠυ
    – https://www.elastic.co/guide/en/shield/current/quick-getting-
    started.html

    View Slide

  89. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    89
    Watcher
    Watcher

    View Slide

  90. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    90
    Watcherͷಛ௃
    • ΫΤϦʹΑΔWatch
    – ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ
    • ৚݅ͷઃఆ
    – ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ
    • εέδϡʔϧ
    – ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ
    • ΞΫγϣϯͷఆٛ
    – ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ
    • ཤྺͷอଘ
    – ElasticsearchͷΠϯσοΫεͱ࣮ͯ͠ߦ݁ՌΛอଘ

    View Slide

  91. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    91
    ΨΠυͳͲ
    • ϓϩμΫτϖʔδ
    – https://www.elastic.co/products/watcher


    • ΨΠυ
    – https://www.elastic.co/guide/en/watcher/current/index.html

    View Slide

  92. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    92
    found.no

    View Slide

  93. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    93
    found.no
    • Elasticsearch as a Service
    • Elasticsearchͷػೳ͕ར༻Մೳ
    • ϓϥάΠϯͷར༻΋Մೳ
    • ΞοϓάϨʔυ΋؆୯
    • ΧελϜࣙॻɺϓϥάΠϯ΋ར༻Մೳ
    • 14೔ؒͷ͓ࢼ͠ظؒ͋Γ

    View Slide

  94. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    94
    found premium
    • SLAϕʔεͷαϙʔτ
    – ElasticͷαϙʔτϝϯόʔʹΑΔαϙʔτ
    • Shield
    • Watcher
    • Marvel

    View Slide

  95. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    95
    ࢀߟจݙ
    • Elasticsearch - The Definitive guide
    – http://www.elastic.co/guide/en/elasticsearch/guide/current/
    index.html
    • ॻ੶ʢ೔ຊޠʣ
    – ElasticSearchServer೔ຊޠ൛

    αʔό/ΠϯϑϥΤϯδχΞ

    ɹཆ੒ಡຊɹϩάऩू

    View Slide

  96. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    96
    ࢀߟαΠτ
    • Ϣʔεέʔε
    • https://www.elastic.co/use-cases
    • DiscussʢWebϑΥʔϥϜʣ
    • https://discuss.elastic.co
    • Elastic{ON}ͷϏσΦͱࢿྉ
    • https://www.elastic.co/elasticon/videos
    • αϙʔτϝχϡʔ
    • https://www.elastic.co/subscriptions

    View Slide

  97. Thanks for listening!
    Q & A
    We’re hiring!
    https://www.elastic.co/about/careers/
    We’re helping!
    https://www.elastic.co/subscriptions
    http://purchases.elastic.co

    View Slide