Stack Elasticsearch - Matinale Zenika

Présentation de la stack elastic et du x-pack

Transcript

1. Elastic Stack @EmmanuelDemey @NicolaZnk #elastic
4. Elasticsearch • Open source project • Based on Apache Lucene • Add many features ◦ REST API ◦ High Availability ◦ Many Clients : Java, JavaScript, GO, Haskell, .net, ...
5. Elasticsearch • Search & Analytics: ◦ Full-text search ◦ Aggregations ◦ Geospatial ◦ Multilingual
6. Elasticsearch • Use cases: ◦ Single bar search ◦ Catalog indexation ◦ Localized searches ◦ Logs management
7. Elasticsearch • Success stories: ◦ Ebay: 800 millions items ◦ Github: repositories code search ◦ Deezer: catalog 40m entries ◦ NYT: 15m articles since 160 years
8. Elasticsearch • Infrastructure: ◦ Start from 3 nodes, up to ? ◦ Add / remove nodes on the fly ◦ Heterogeneous servers ◦ Easy monitoring & supervision ◦ Modest hardware configuration
9. PUT http://localhost:9200/index/type/1 { "title": "Olivier Twist" } >> {“acknowledged”: true} POST http://localhost:9200/index/type/ { "title": "Les Aventures de Monsieur Pickwick" } >> {“acknowledged”: true} GET http://localhost:9200/index/type/1 >> {“title”: “Olivier Twist”} PUT http://localhost:9200/index/type/2 { "title": "Oliver Twist" } >> {“acknowledged”: true} DELETE http://localhost:9200/index/type/2 >> {“acknowledged”: true} Elasticsearch
10. POST http://localhost:9200/index/type/_search { "query": "match" : { "title" : { "query" : "OLIVER" } } } } >> { "hits": { "hits": [ { "_index": "index", "_type": "type", "_id": "1", "_score": 0.5, "_source": {"title": "Oliver Twist"} } ], ... } ... } Elasticsearch
11. POST http://localhost:9200/index/type/_search { "query": { "bool": { "must": [ { "match": { "title": "Oliver" }}, { "match": { "author": "Dickens" }} ], "filter": [ { "term": { "status": "published" }}, { "range": { "publish_date": { "lte": "1950-01-01" }}} ] } } } Elasticsearch
13. Elasticsearch • Developped in java & jruby • Dynamic data pipeline: ◦ Multiple input / output ◦ Centralize logs ◦ Parse ◦ Store / forward • Plugin based
14. Logstash Logs REST API Broker Unix commands Files REST API Broker Elastic Search Filter 1 Filter 2 Filter 3
15. Logstash Logs REST API Broker Unix commands Files REST API Broker Elastic Search Filter 1 Filter 2 Filter 3
16. Logstash # logstash -f logtash.conf input { } filter { } output { } { } “message”: “127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"”, file { path => "/var/access*.log” } grok {match => { "message" => "%{COMMONAPACHELOG}" }} “clientip”: “127.0.0.1”, “verb”: “GET”, “request”: “/london”/, “response”: 200, ... if [request] == “/london” { mutate { add_field => { “inEnglish” => true } } “inEnglish”: true mutate { remove_field => [ "message" ] elasticsearch { hosts => [ “localhost:9200” ] }
17. Logstash
18. Logstash - Complex architecture Logs / App1 Files REST API Broker Elastic Search Logstash 2 Kafka Logstash Logs / App2 Logs / App3 Logstash 3 Logstash 1
20. Beats • Stack developped with GO • More efficient (I/O, …) than JRuby code • Easy to install / configure • TopBeat, PacketBeat, FileBeat, MetricBeat, WinlogBeat, ...
21. Logstash - Complex architecture Logs / App1 Files REST API Broker Elastic Search Filebeat 2 Kafka Logstash Logs / App2 Logs / App3 Filebeat 3 Filebeat 1
22. # logstash -f logtash.conf input { } filter { } output { } beats { port => 5044 } grok {match => { "message" => "%{COMMONAPACHELOG}" }} if [request] == “/london” { mutate { add_field => { “inEnglish” => true } } mutate { remove_field => [ "message" ] elasticsearch { hosts => [ “localhost:9200” ] } Beats # filebeat -c filebeat-conf.yml filebeat: prospectors: - paths: - "/var/access*.log” output: logstash: hosts: ["localhost:5044"]
23. Beats
25. Kibana
26. • Discover / Visualize / Dashboard • Many plugged-in widgets ◦ map, pie, metric, area chart, line chart, table… ◦ Many plugins available • Period selector and auto-refresh behaviors • Authentication Management System • PDF Report : automate & email Kibana
27. and many more products...
30. Security
31. • Define roles for you ES cluster • Many authentication levels : ◦ cluster ◦ indices ◦ documents ◦ properties • Authentication Providers : Basic Auth, LDAP, Active Directory, your own provider Security
32. Security # PUT /_xpack/security/role/clicks_admin { "cluster": [ "monitor" ], "indices": [ { "names": [ "events-*" ], "privileges": [ "read" ], "query": "{\"match\": {\"category\": \"click\"}}" } ] } # PUT /_xpack/security/user/manu { "password": "password", "roles": [ "clicks_admin"], "full_name": "Emmanuel Demey”, "email": "emmanuel.demey@zenika.com”, "metadata": { "event": "Matinale ES" } } # role_mapping.yml monitoring: - "cn=admins,dc=example,dc=com" user: - "cn=John Doe,cn=contractors,dc=example,dc=com" - "cn=users,dc=example,dc=com" - "cn=admins,dc=example,dc=com"
33. • Encrypted communication between nodes • Encrypted access • IP Filtering • Audit logging • Add authentication to Kibana and Monitoring Security
34. Alerting
35. • Detect changes in your data • Get notified • Learn from alert history Alerting
36. PUT _xpack/watcher/watch/log_errors { "trigger" : { schedule" : { "interval" : "5m" } }, "input" : { "search" : { "request" : { "indices" : "log-events", "body" : { "query" : { "match" : { "status" : "error" } } } } } }, ... ... "condition" : { "compare" : { "ctx.payload.hits.total" : { "gt" : 5 }} }, "actions" : { "email_administrator" : { "email" : { "to" : "sys.admino@host.domain", "subject" : " {{ctx.payload.hits.total}} errors", "body" : "Too many error" } } } } } Alerting
37. Monitoring
38. Monitoring
39. Emmanuel DEMEY Zenika LILLE @EmmanuelDemey Nicolas LASSALLE Zenika LILLE @NicolaZnk