Elasticスタックではじめるログ解析入門 #osc16ep

オープンソースカンファレンス 2016 Enterprise@Osakaでの発表資料

Transcript

1. Elastic StackͰ͸͡ΊΔ
   ϩάղੳೖ໳
   Jun Ohtani
   @johtani
   2016/01/29
   

   View Slide

2. View Slide

3. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   3
   • Elastic stack঺հ
   • Logstash - Logऩू
   • Elasticsearch - ݕࡧɾղੳ
   • Kibana - ՄࢹԽ
   • ঎༻ϓϥάΠϯ঺հ
   ΞδΣϯμ
   

   View Slide

4. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   4
   about
   • Me, Jun Ohtani / Technical Adovocate
   – lucene-gosenίϛολʔ
   – ElasticSearch Server೔ຊޠ൛ͷ຋༁
   – elasticsearch-extended-analysisͷ։ൃ
   – http://blog.johtani.info
   • Elasticsearch, founded in 2012
   – Products: Elasticsearch, Logstash, Kibana, Marvel, Sheild

   Professional services: Support & development subscriptions
   – Trainings
   

   View Slide

5. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   5
   ElasticελοΫ
   ElasticελοΫ
   

   View Slide

6. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   6
   ElasticελοΫ
   ML
   K
   B LE
   a
   B K
   B
   SWFB
   (
   M
   )
   W
   )
   

   View Slide

7. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   7
   ElasticελοΫɿOpen Source෦෼
   ML
   K
   B LE
   a
   B K
   B
   SWFB
   (
   M
   )
   W
   )
   

   View Slide

8. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   8
   Elastic stack
   Data Import/Parse/
   Export
   Store/Search
   Visualize
   

   View Slide

9. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
   distributing without written permission is strictly prohibited
   9
   Logstash
   Logstash
   

   View Slide

10. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    10
    Elastic stack
    Import/Parse/
    Export
    Store/Search
    Visualize
    Data
    

    View Slide

11. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    11
    Logstash in 10 seconds
    • ϩάɾσʔλͷऩूɾ؅ཧ
    • ऩूɺύʔεɾՃ޻ɺૹग़
    • ΦʔϓϯιʔεɿApache License 2.0
    • Ruby app (JRuby)
    • ࠷৽൛ɿ2.1.1
    

    View Slide

12. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    12
    Logstash architecture
    Logstash
    Input Output
    Filter
    ? ?
    collect and split alter and enrich store and visualize
    

    View Slide

13. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    13
    ઃఆ
    input {
    …
    }
    filter {
    …
    }
    output {
    …
    }
    

    View Slide

14. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    14
    ઃఆɿinput
    input {
    file {
    path => “/Users/johtani/sample/*_log"
    start_position => "beginning"
    }
    }
    

    View Slide

15. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    15
    1ߦ1σʔλ
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
    1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
    Firefox/5.0"
    

    View Slide

16. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    16
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

17. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    17
    ύʔε
    189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/
    1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101
    Firefox/5.0"
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",
    "clientip": "189.120.xx.xx",
    "ident": "-",
    "auth": "-",
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    "verb": "GET",
    "request": "/manager/html",
    …
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\""
    }
    

    View Slide

18. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    18
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

19. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    19
    ೔෇ͷύʔε
    {…
    "@timestamp": "2015-04-10T09:07:49.325Z",
    …
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    …
    }
    {…
    "@timestamp": "2014-12-02T03:18:29.000Z",
    …
    "timestamp": "02/Dec/2014:12:18:29 +0900",
    …
    }
    

    View Slide

20. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    20
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

21. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    21
    IP͔ΒҢ౓ܦ౓ͳͲ෇༩
    {…
    "clientip": "189.120.xx.xx",
    …}
    {…
    "clientip": "189.120.xx.xx",
    …
    "geoip": {
    "ip": “189.120.xxx.xxx”,
    …
    "country_name": "Brazil",
    "continent_code": "SA",
    "region_name": "27",
    "city_name": "São Paulo",
    "latitude": -23.473299999999995,
    "longitude": -46.66579999999999,
    …
    

    View Slide

22. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    22
    ઃఆɿfilter
    filter {
    grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
    break_on_match => false
    }
    date {
    match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]
    locale => en
    }
    geoip { source => ["clientip"] }
    useragent {
    source => "agent"
    target => "useragent"
    }
    }
    

    View Slide

23. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    23
    ϢʔβΤʔδΣϯτͷύʔε
    {…
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\""
    …}
    {…
    "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\""
    …
    "useragent": {
    "name": "Firefox",
    "os": "Windows XP",
    "os_name": "Windows XP",
    "device": "Other",
    "major": "5",
    "minor": "0"
    }
    …
    

    View Slide

24. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    24
    ઃఆɿoutput
    output {
    elasticsearch {
    hosts => ["localhost"]
    index => “demo_access_log-%{+YYYY.MM.dd}”
    }
    }
    

    View Slide

25. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    25
    Elasticsearch
    Elasticsearch
    

    View Slide

26. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    26
    ݕࡧͱͯ͠ͷ
    Elasticsearch
    ݕࡧͱͯ͠ͷElasticsearch
    

    View Slide

27. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    27
    Elasticsearchͱ͸ʁ
    Elasticsearchͱ͸ʁ
    

    View Slide

28. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    28
    ϑϦʔϫʔυݕࡧ
    

    View Slide

29. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    29
    ߜΓࠐΈ
    

    View Slide

30. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    30
    ϋΠϥΠτ
    

    View Slide

31. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    31
    ιʔτ
    

    View Slide

32. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    32
    ϖʔδϯά
    

    View Slide

33. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    33
    ूܭ
    

    View Slide

34. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    34
    αδΣετ
    

    View Slide

35. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    35
    Elasticsearch in 10 seconds
    • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON
    • Φʔϓϯιʔε: Apache License 2.0
    • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
    • JavaͰ࣮૷ɻ֦ு΋༰қ
    

    View Slide

36. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    36
    Powerful Search at Scale
    

    View Slide

37. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    37
    ؆୯ͳCRUD
    ؆୯ͳCRUD
    

    View Slide

38. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    38
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : "Clinton Gormley",
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλొ࿥
    

    View Slide

39. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    39
    » curl -XPUT localhost:9200/books/book/1 -d '
    {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : "2013-02-04",
    "pages" : 230
    }'
    σʔλߋ৽
    

    View Slide

40. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    40
    σʔλ࡟আ
    » curl -X DELETE localhost:9200/books/book/1
    σʔλͷऔಘ
    » curl —X GET localhost:9200/books/book/1
    » curl —X GET localhost:9200/books/book/1/_source
    

    View Slide

41. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    41
    ݕࡧ
    » curl -XGET localhost:9200/books/_search?q=elasticsearch
    {
    "took" : 2, "timed_out" : false,
    "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },
    "hits" : {
    "total" : 1, "max_score" : 0.076713204,
    "hits" : [ {
    "_index" : “books", "_type" : “book", "_id" : "1",
    "_score" : 0.076713204, "_source" : {
    "title" : "Elasticsearch - The definitive guide",
    "authors" : [ "Clinton Gormley", "Zachary Tong" ],
    "started" : “2013-02-04", "pages" : 230
    }
    } ]
    }
    }
    

    View Slide

42. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    42
    » curl -XGET ‘localhost:9200/books/book/_search' -d '{
    "query": {
    "filtered" : {
    "query" : {
    "match": {
    "text" : {
    "query" : “To Be Or Not To Be",
    "cutoff_frequency" : 0.01
    }
    }
    },
    "filter" : {
    "range": {
    "price": {
    "gte": 20.0
    "lte": 50.0
    ...
    }
    }'
    ݕࡧ - Query DSL
    

    View Slide

43. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    43
    ෼ࢄߏ੒
    εέʔϧ
    ෼ࢄߏ੒ɺεέʔϧ
    

    View Slide

44. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    44
    Basic terms
    • ΠϯσοΫε
    – σʔλͷ࿦ཧతͳू߹ɻ

    RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical
    • ϨϓϦέʔγϣϯ
    • ಡΈࠐΈͷεέʔϥϏϦςΟ޲্
    • SPOFͷղফ
    • γϟʔσΟϯά
    • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ

    ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্

    σʔλϑϩʔ੍ޚ
    

    View Slide

45. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    45
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1 2
    2
    2
    curl -X PUT localhost:9200/orders -d '{
    "settings.index.number_of_shards" : 4
    "settings.index.number_of_replicas" : 1
    }'
    curl -X PUT localhost:9200/products -d '{
    "settings.index.number_of_shards" : 2
    "settings.index.number_of_replicas" : 0
    }'
    

    View Slide

46. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    46
    γϟʔυͱϨϓϦΧ
    node 1
    orders
    products
    1
    4
    1
    node 2
    orders
    products
    2
    2
    3 4
    1 2
    3
    

    View Slide

47. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    47
    ࣗಈతͳ෼ࢄ
    node 1
    orders
    products
    2
    1
    4
    1
    node 2
    orders
    products
    2
    2
    node 3
    orders
    products
    3 4
    1
    3
    

    View Slide

48. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    48
    શจݕࡧͱ͸ʁ
    શจݕࡧͱ͸ʁ
    

    View Slide

49. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    49
    શจݕࡧͱ͸ʁ
    • શจݕࡧʢFull text searchʣͱ͸ɺίϯϐϡʔλʹ͓͍ͯɺෳ਺ͷจ
    ॻʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕ
    ࡧʯ΍ʮ୯ҰϑΝΠϧ಺ͷจࣈྻݕࡧʯͱҟͳΓɺʮෳ਺จॻʹ·ͨ
    ͕ͬͯɺจॻʹؚ·ΕΔશจΛର৅ͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ࢖༻͞
    ΕΔɻ

    ʢWikipediaΑΓʣ
    

    View Slide

50. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    50
    ༻ޠ
    • ΠϯσοΫε
    ݕࡧΤϯδϯ͕ݕࡧʹ࢖༻͢Δσʔλͷอଘઌ
    • υΩϡϝϯτʢจॻʣ
    – ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ
    • ϑΟʔϧυ
    – υΩϡϝϯτʹؚ·ΕΔଐੑ
    • ΫΤϦ
    – ݕࡧ৚݅ɺݕࡧࣜ
    

    View Slide

51. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    51
    ༻ޠ
    • εΩʔϚ
    – υΩϡϝϯτͷߏ଄Λఆٛ͢Δ΋ͷ
    • λʔϜʢTermʣɺτʔΫϯʢTokenʣ
    – ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ
    – จষΛҰఆͷ๏ଇͰ۠੾ͬͨ୯ޠ
    – ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲ΋ؚΉ
    

    View Slide

52. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    52
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    υΩϡϝϯτͷొ࿥
    

    View Slide

53. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    53
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ
    ఋ
    ࢞
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    

    View Slide

54. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    54
    υΩϡϝϯτͷొ࿥
    1
    2
    ΧπΦ͸αβΤͷఋ
    αβΤ͸ϫΧϝͷ࢞
    1
    2
    ΧπΦ
    αβΤ
    ͸
    ͸
    ͷ
    ͷ
    αβΤ
    ϫΧϝ
    ఋ
    ࢞
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    υΩϡϝϯτͷొ࿥
    ୯ޠʹ෼ׂ
    ୯ޠ͔Βidͷ഑ྻ͕
    Ҿ͚ΔΑ͏ʹ
    

    View Slide

55. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    55
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ݕࡧ৚݅ೖྗ
    ΧπΦɹαβΤ
    

    View Slide

56. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    56
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

57. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    57
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

58. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    58
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

59. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    59
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

60. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    60
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

61. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    61
    ݕࡧ
    ΧπΦ
    αβΤ
    1
    1 2 ͸
    ͷ
    ࢞
    ϫΧϝ 2
    1 2
    1 2
    1
    ఋ
    2
    ΧπΦ αβΤ
    AND
    ݕࡧ৚݅ೖྗ
    ݕࡧ৚݅ͷύʔε

    ݕࡧΫΤϦԽ
    ΧπΦɹαβΤ
    

    View Slide

62. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    62
    ͦͷଞͷػೳ
    ͦͷଞͷػೳ
    

    View Slide

63. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    63
    Geo
    • ͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ
    • σʔλɿ
    • Ң౓ܦ౓ɺGeoHashɺGeoShape
    • ݕࡧɿ
    • geo shape queryɺgeo shape filterɺgeo distance filterɺ
    • geo distance range filterɺgeo bounding box filterɺ
    • geo polygon filter
    

    View Slide

64. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    64
    Percolator
    • υΩϡϝϯτͰ͸ͳ͘ΫΤϦΛొ࿥
    • _percolate APIʹυΩϡϝϯτΛૹ৴
    • ొ࿥͞ΕͨΫΤϦʹϚονͨ͠৔߹ɺ

    Ϛονͨ͠ΫΤϦΛฦ͢
    

    View Slide

65. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    65
    Snapshot/Restore
    • 1.0͔Βಋೖ
    • ΠϯσοΫε୯ҐͰSnapshot/RestoreՄೳ
    • อଘઌʢRepositoryʣ
    – Shard FSɺS3ɺHDFSɺAzure…
    

    View Slide

66. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    66
    Ecosystem
    • Plugins
    – ϓϥάΠϯʹΑΔػೳͷ௥Ճ
    • ΫϥΠΞϯτϥΠϒϥϦ
    • Ruby, python, php, perl, javascript, .NET
    • Scala, clojure, go
    • Hadoop integration
    

    View Slide

67. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    67
    ৄ͘͠஌Γ͍ͨํ͸
    • Elasticsearch - The Definitive guide
    – http://www.elastic.co/guide/en/elasticsearch/guide/current/
    index.html
    

    View Slide

68. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    68
    ղੳͱͯ͠ͷ
    Elasticsearch
    ղੳͱͯ͠ͷElasticsearch
    

    View Slide

69. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    69
    aggregation
    aggregation
    

    View Slide

70. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    70
    Aggregationͱ͸
    • 1.0͔Βಋೖ
    • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ
    • ֊૚తͳूܭɺάϧʔϓԽ

    ಈతͳूܭɺάϧʔϓԽ
    • େ͖͘2छྨ
    • BucketɹυΩϡϝϯτΛ஋͝ͱʹ݁ՌΛάϧʔϐϯά
    • Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ
    

    View Slide

71. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    71
    ྫɿݴޠ͝ͱͷϑΝηοτ
    » curl -XGET twitter-2014.08.22/_search -d '
    GET twitter-2014.08.22/_search
    {
    "facets": {
    "lang_facet": {
    "terms": {"field": "lang"}
    }
    }
    }
    

    View Slide

72. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    72
    ྫɿݴޠ͝ͱͷϑΝηοτ
    "facets": {
    "lang_facet": {
    "_type": "terms",
    "missing": 0,
    "total": 4521075,
    "other": 320228,
    "terms": [
    {
    "term": "en",
    "count": 1471819
    },
    {
    "term": "ja",
    "count": 980145
    },
    

    View Slide

73. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    73
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    » curl -XGET twitter-2014.08.22/_search -d '
    {
    "aggs": {
    "lang": {
    "terms": {"field": "lang" },
    "aggs": {
    "place": {
    "terms": {
    "field": “place.full_name", "size": 10
    }
    }
    }
    }
    }
    }
    

    View Slide

74. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    74
    ྫɿݴޠ͓Αͼ஍Ҭͷूܭ
    "aggregations": {
    "lang": {
    "buckets": [{…}, {
    "key": "ja",
    "doc_count": 980145,
    "place": {
    "buckets": [
    { "key": "ژ౎ࢢ෬ݟ۠, ژ౎",
    "doc_count":252 },
    { "key": "ઍ୅ా۠, ౦ژ",
    "doc_count": 39 },…
    

    View Slide

75. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    75
    bucket
    • filter
    • filters (>=1.4.0)
    • missing
    • nested
    • reverse nested (>=1.2.0)
    • children
    • terms
    • significant terms (>=1.1.0)
    • range
    • date range
    • ipv4 range
    • histogram
    • date histogram
    • geo distance
    • geohash grid
    

    View Slide

76. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    76
    metric
    • min
    • max
    • sum
    • avg
    • stats
    • extended_stats
    • value count
    • percentiles (>=1.1.0)
    • percentile ranks (>=1.3)
    • cardinality (>=1.1.0)
    • geo bounds (>=1.3.0)
    • top_hits (>=1.3.0)
    • scripted metric 

    (>= 1.4.0)
    

    View Slide

77. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    77
    KibanaͰՄࢹԽ
    KibanaͰՄࢹԽ
    

    View Slide

78. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    78
    Kibana4
    • ElasticsearchͷσʔλΛՄࢹԽ
    • Node.js server & JavaScript
    • Apache License 2.0
    

    View Slide

79. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    79
    Kibana 4
    

    View Slide

80. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    80
    σϞ for Kibana4
    Access Log
    demo for Kibana4
    

    View Slide

81. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    81
    Combining Search and Analytics
    

    View Slide

82. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    82
    ͦͷଞͷ
    ElasticελοΫ
    ͦͷଞͷElasticελοΫ
    

    View Slide

83. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    83
    Beats
    19
    #!"
    
    
    
    a ab
    
    
    U EC W e
    Pc i b c W e
    g
    
    
    b b ol st
    hU g
    
    
    c st r
    L
    

    View Slide

84. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    84
    elasticsearch-hadoop
    20
    -
    •  D E H
    •  PD ecd
    ER
    •  g D
    • 
    CH
    •  Ca M DMS
    D FERC
    

    View Slide

85. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    85
    ঎༻ϓϥάΠϯ
    ঎༻ϓϥάΠϯ
    

    View Slide

86. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    86
    Shield
    Shield
    

    View Slide

87. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    87
    Shieldͷಛ௃
    • User Authentication
    • LDAP/Active Directory
    • ϑΝΠϧϕʔε
    • Authorization
    • ϩʔϧϕʔεͷΞΫηείϯτϩʔϧ
    • ΠϯσοΫε͝ͱɺΞΫγϣϯ͝ͱͷઃఆ͕Մೳ
    • υΩϡϝϯτɾϑΟʔϧυ͝ͱͷઃఆ΋Մೳʹ
    • ηΩϡΞͳ௨৴
    • ElasticsearchϊʔυؒͷSSL/TLS
    • IPϑΟϧλϦϯά
    • ؂ࠪϩά
    

    View Slide

88. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    88
    ΨΠυͳͲ
    • ϓϩμΫτϖʔδ
    – https://www.elastic.co/products/shield
    • ΨΠυ
    – https://www.elastic.co/guide/en/shield/current/quick-getting-
    started.html
    

    View Slide

89. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    89
    Watcher
    Watcher
    

    View Slide

90. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    90
    Watcherͷಛ௃
    • ΫΤϦʹΑΔWatch
    – ElasticsearchͷΫΤϦΛར༻ͯ͠σʔλͷ؂ࢹ
    • ৚݅ͷઃఆ
    – ΞΫγϣϯΛ࣮ߦ͢Δ͔Ͳ͏͔ͷઃఆ
    • εέδϡʔϧ
    – ΫΤϦΛ࣮ߦ͠ɺ৚݅ΛνΣοΫ͢Δස౓ͷࢦఆ
    • ΞΫγϣϯͷఆٛ
    – ϝʔϧͷૹ৴ɺଞγεςϜ΁ͷσʔλૹ৴ͳͲͷಈ࡞Λઃఆ
    • ཤྺͷอଘ
    – ElasticsearchͷΠϯσοΫεͱ࣮ͯ͠ߦ݁ՌΛอଘ
    

    View Slide

91. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    91
    ΨΠυͳͲ
    • ϓϩμΫτϖʔδ
    – https://www.elastic.co/products/watcher

    

    • ΨΠυ
    – https://www.elastic.co/guide/en/watcher/current/index.html
    

    View Slide

92. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    92
    found.no
    

    View Slide

93. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    93
    found.no
    • Elasticsearch as a Service
    • Elasticsearchͷػೳ͕ར༻Մೳ
    • ϓϥάΠϯͷར༻΋Մೳ
    • ΞοϓάϨʔυ΋؆୯
    • ΧελϜࣙॻɺϓϥάΠϯ΋ར༻Մೳ
    • 14೔ؒͷ͓ࢼ͠ظؒ͋Γ
    

    View Slide

94. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    94
    found premium
    • SLAϕʔεͷαϙʔτ
    – ElasticͷαϙʔτϝϯόʔʹΑΔαϙʔτ
    • Shield
    • Watcher
    • Marvel
    

    View Slide

95. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    95
    ࢀߟจݙ
    • Elasticsearch - The Definitive guide
    – http://www.elastic.co/guide/en/elasticsearch/guide/current/
    index.html
    • ॻ੶ʢ೔ຊޠʣ
    – ElasticSearchServer೔ຊޠ൛

    αʔό/ΠϯϑϥΤϯδχΞ

    ɹཆ੒ಡຊɹϩάऩू
    

    View Slide

96. www.elastic.co Copyright Elastic 2015 Copying, publishing and/or
    distributing without written permission is strictly prohibited
    96
    ࢀߟαΠτ
    • Ϣʔεέʔε
    • https://www.elastic.co/use-cases
    • DiscussʢWebϑΥʔϥϜʣ
    • https://discuss.elastic.co
    • Elastic{ON}ͷϏσΦͱࢿྉ
    • https://www.elastic.co/elasticon/videos
    • αϙʔτϝχϡʔ
    • https://www.elastic.co/subscriptions
    

    View Slide

97. Thanks for listening!
    Q & A
    We’re hiring!
    https://www.elastic.co/about/careers/
    We’re helping!
    https://www.elastic.co/subscriptions
    http://purchases.elastic.co
    

    View Slide