様々なメトリクスやログを集めてシステム解析 
- Elastic Stackの入門と活用 - / Intro Elastic Stack

Transcript

1. !1 2018/07/06 Developer Advocate at Elastic Jun Ohtani @johtani ༷ʑͳϝτϦΫε΍ϩάΛूΊͯγεςϜղੳ

   
 - Elastic Stackͷೖ໳ͱ׆༻ -

2. !2

3. !3 ΞδΣϯμ • ϝτϦΫεʗϩάͱ͸ʁ • γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹ΍ͬͯΈΑ͏ • Beats - Elasticsearch

   - KibanaͰղੳ • ຊ֨తʹղੳΛ΍Δʹ͸ʁ • LogstashͰϩά΍ϝτϦΫεΛதܧɾू໿ • ͞Βʹ৭ʑࢼͯ͠ΈΔʹ͸ʁ

4. !4 about • Me, Jun Ohtani / Developer Advocate ‒

   lucene-gosenίϛολʔ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ڞஶ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 Elastic APM, 
 X-Pack, Elastic Cloud, Swiftype 
 Professional services: Support & development subscriptions
 Trainings, Consulting, SaaS

5. !5 ͲΜͳϝτϦΫεɺ
 ϩάΛूΊ͍ͯ·͔͢ʁ

6. !6 ϝτϦΫε • CPUɺϝϞϦ࢖༻཰ɺσΟεΫ࢖༻཰ • ΞΫηε਺ɺωοτϫʔΫసૹྔ • Ԡ౴࣌ؒ • ίωΫγϣϯ਺

   • τϥϯβΫγϣϯ਺ɺച্ • ίϯςφͷ্ͷ֤छϝτϦΫε

7. !7 ϩά • ೝূϩά • γεςϜϩά • ΞϓϦέʔγϣϯϩά • Slow

   log • ΞΫηεϩά • ίϯςφͷதͷϩά

8. !8 Ͱ͖Ε͹ϩάͱϝτϦΫεΛ
 ·ͱΊͯ1ͭͷը໘Ͱ
 ݟ͍ͨͰ͢ΑͶʁ

9. !9 Elastic Stack

10. 10 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ

11. !11 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

    Graph Machine Learning

12. ఆܕͷϝτϦΫε/ϩάղੳΛ Elastic StackͰ

13. !13 ϝτϦΫεɾϩάͷ෼ੳʢ؆қ൛ʣ σʔλ Import Parse/
 Store/Search Visualize

14. !14

15. 15 Beats ܰྔσʔλγούʔ ιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू໿ ม׵ͱύʔεͷͨΊ Logstashʹసૹ Elastic Cloudʹసૹ Libbeat:

    ΧελϜbeatsͷͨ ΊͷAPIϑϨʔϜϫʔΫ 30Ҏ্ͷίϛϡχςΟbeats

16. The Beats family Heartbeat Uptime monitoring Filebeat Log files Winlogbeat

    Windows Event Logs Packetbeat Network data +40 community Beats Metricbeat Metrics Auditbeat Audit data

17. Collect system and application metrics Metricbeat

18. lots of modules Metricbeat

19. tail log from file Filebeat

20. many modules Filebeat

21. Capture the Packet Packetbeat

22. Capture the Packet Packetbeat

23. Welcome to 1998 winlogbeat

24. Now winlogbeat

25. !25 • Kubernetes module in Metricbeat ‒ CPU, memory, ωοτϫʔΫసૹྔͳͲ

    • add_docker_metadataϓϩηοα ‒ Container ID, name, image, labels • add_kubernetes_metadataϓϩηοα ‒ Pod name, pod namespace, container name, pod labels Beats <3 ίϯςφ Docker΍KubernetesͰͷσϓϩΠΛ؆୯ʹ

26. !26

27. 27 Elasticsearch Heart of the Elastic Stack ෼ࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ

    ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ

28. Elasticsearchͱ͸ʁ

29. !29 ϑϦʔϫʔυݕࡧ

30. !30 ߜΓࠐΈ

31. !31 ϋΠϥΠτ

32. !32 ιʔτ

33. !33 ϖʔδϯά

34. !34 ूܭ

35. !35 αδΣετ

36. !36 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON •

    Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

37. ؆୯ͳCRUD

38. σʔλొ࿥ 38 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : "Clinton Gormley", "started" : "2013-02-04", "pages" : 230 }'

39. σʔλߋ৽ 39 curl -XPUT localhost:9200/books/doc/1 -d ' { "title" :

    "Elasticsearch - The definitive guide", "authors" : [ "Clinton Gormley", "Zachary Tong" ], "started" : "2013-02-04", "pages" : 230 }'

40. σʔλ࡟আ !40 curl -X DELETE localhost:9200/books/doc/1 σʔλͷऔಘ curl —X GET

    localhost:9200/books/doc/1 curl —X GET localhost:9200/books/doc/1/_source

41. ݕࡧ - Query DSL !41 curl -XGET ‘localhost:9200/books/doc/_search' -d '{

    "query": { "bool": { "must": [ { "match": { "title": "Search" }}, { "match": { "content": "Elasticsearch" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ] } } }'

42. ෼ࢄߏ੒ɺ
 εέʔϧ

43. Basic terms • ΠϯσοΫε ‒ σʔλͷ࿦ཧతͳू߹ɻ
 RDBͷσʔλϕʔεͷΑ͏ͳ΋ͷLogical • ϨϓϦέʔγϣϯ •

    ಡΈࠐΈͷεέʔϥϏϦςΟ޲্ • SPOFͷղফ • γϟʔσΟϯά • ෳ਺Ϛγϯ΁σʔλΛ෼ׂ
 ॻ͖ࠐΈͷεέʔϥϏϦςΟ޲্
 σʔλϑϩʔ੍ޚ !43

44. γϟʔυͱϨϓϦΧ !44 node 1 orders products 1 4 1 2

    2 3 curl -X PUT localhost:9200/orders -d '{ "settings.index.number_of_shards" : 4 "settings.index.number_of_replicas" : 1 }' curl -X PUT localhost:9200/products -d '{ "settings.index.number_of_shards" : 2 "settings.index.number_of_replicas" : 0 }'

45. γϟʔυͱϨϓϦΧ !45 node 1 orders products 1 4 1 node

    2 orders products 2 2 3 4 1 2 3

46. ࣗಈతͳ෼ࢄ !46 node 1 orders products 2 1 4 1

    node 2 orders products 2 2 node 3 orders products 3 4 1 3

47. ͦͷଞͷػೳ

48. elasticsearch ͞·͟·ͳܗࣜͷσʔλͰ GeoݕࡧՄೳ
 
 Ң౓ܦ౓ɺGeoHashɺ GeoShape… GEO

49. Ecosystem • Plugins ‒ ϓϥάΠϯʹΑΔػೳͷ௥Ճ • ΫϥΠΞϯτϥΠϒϥϦ • Java, Ruby,

    python, php, perl, javascript, .NET • Scala, clojure, go !49

50. Elasticsearch - The Definitive guide
 
 http://www.elastic.co/guide/en/ elasticsearch/guide/current/index.html 50 ৄ͘͠஌Γ͍ͨํ͸

51. !51

52. 52 Kibana Window into the Elastic Stack ՄࢹԽͱ෼ੳ ஍ཧۭؒ ΧελϚΠζͱ

    Ϩϙʔτͷڞ༗ άϥϑ୳ࡧ Elastic Stack΁ͷ ηΩϡΞͳΞΫηεͱ؅ཧ ΧελϜAppsͷ࡞੒

53. !53 Kibana 6

54. !54 σϞ σʔλ౤ೖ͔ΒՄࢹԽ·Ͱ

55. ຊ֨తʹղੳΛߦ͏ʹ͸ʁ

56. !56

57. 57 Logstash σʔλՃ޻ύΠϓϥΠϯ શͯͷܗࣜɺαΠζͱσʔλιʔ εͷ౤ೖ ύʔεͱಈతͳ σʔλม׵ ͋ΒΏΔग़ྗʹ σʔλసૹ ҆શͰ҉߸Խ͞Εͨ


    σʔλೖྗ ಠࣗͷύΠϓϥΠϯॲཧ ͷ࡞੒ 200Ҏ্ͷϓϥάΠϯ

58. Logstash architecture !58 Input Output Filter ? ? collect and

    split alter and enrich store and visualize

59. ઃఆ 59 input { … } filter { … }

    output { … }

60. 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" 60

61. ઃఆɿfilter 61 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

62. ύʔε !62 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"

    404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/

63. ઃఆɿfilter !63 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

64. ೔෇ͷύʔε 64 {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", …

    } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }

65. ઃఆɿfilter !65 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

66. IP͔ΒҢ౓ܦ౓ͳͲ෇༩ 66 "clientip": "189.120.xx.xx", "clientip": "189.120.xx.xx", "geoip": { "ip": “189.120.xxx.xxx”,

    … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude":

67. ઃఆɿfilter !67 filter { grok { match => { "message"

    => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

68. ϢʔβΤʔδΣϯτͷύʔε 68 "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101

    Firefox/5.0\"" "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv: 5.0) Gecko/20100101 Firefox/5.0\"" "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0"

69. ͦͷ΄͔ʹ͸ʁ

70. !70 elasticsearch-hadoop - •  D E H •  PD ecd

    ER •  g D •  CH •  Ca M DMS D FERC

71. !71

72. !72 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

    Graph Machine Learning

73. !73

74. !74 ࢀߟจݙ • Elasticsearch - The Definitive guide ‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/

    index.html • ॻ੶ʢ೔ຊޠʣ ‒ σʔλ෼ੳج൫ߏஙೖ໳ ‒ Elasticsearch࣮ફΨΠυ

75. !75 ࢀߟαΠτ • Ϣʔεέʔε • https://www.elastic.co/use-cases • DiscussʢWebϑΥʔϥϜʣ • https://discuss.elastic.co

    • Elastic{ON}ͷϏσΦͱࢿྉ • https://www.elastic.co/elasticon/videos • αϙʔτϝχϡʔ • https://www.elastic.co/subscriptions

76. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co