Elastic{ON} Tour Toronto - Ingest

Elastic{ON} Tour Toronto - September 27, 2017

Learn how our ingest technologies, such as Elasticsearch, Beats, Logstash, and ES-Hadoop, help get you to a deployment of the Elastic Stack.

Suyog Rao |Logstash Team Lead | Elastic

Transcript

1. Suyog Rao, Logstash Team Lead Toronto | Sept 27, 2017 Ingest
2. 2 Data Ingestion The process of collecting and importing data for immediate use in a datastore
3. Simple things should be simple. 3 ? Shay Banon Elastic{ON}’17
4. 4 Ingest Technologies Lightweight Data Shippers Beats Centralized Data Collection Engine Logstash Hadoop Ecosystem Connector ES-Hadoop APIs Ingest Node Elasticsearch
5. 5 Elastic Ingestion Technologies API Elasticsearch Transform Store ingest node data node
6. 6 Elastic Ingestion Technologies CUSTOM CONNECTORS Elasticsearch Transform Store ingest node data node es-hadoop
7. 7 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats servers, containers Transform Store ingest node data node Metrics Logs
8. 8 Elastic Ingestion Technologies network devices DB data CENTRALIZED COLLECTION Logstash DISTRIBUTED COLLECTION Beats servers, containers Elasticsearch Transform Store ingest node data node Flows JDBC
9. 9 Ingestion Architecture Scalable and robust centralized ETL • Java event rewrite • Multiple pipelines
10. 10 Ingestion Architecture Scalable and robust centralized ETL • Persistent queues • Dead letter queues
11. Elastic Ingestion Technologies Elasticsearch Transform Store ingest node data node 11 DISTRIBUTED COLLECTION Beats servers, containers
12. Elastic Ingestion Technologies CENTRALIZED COLLECTION Logstash Elasticsearch Transform Store ingest node data node 12 network devices DISTRIBUTED COLLECTION Beats servers, containers
13. Elastic Ingestion Technologies CENTRALIZED COLLECTION Logstash Elasticsearch Transform Store ingest node data node 13 network devices DISTRIBUTED COLLECTION Beats servers, containers
14. 14 Easy migration between ingest technologies Ingest Node to Logstash conversion tool Elasticsearch ingest node Logstash ingest node
15. Data Sources
16. 16 Use Cases & Data Sources Common Log Formats System Web Servers Queues Turnkey Monitoring Infrastructure Containers Databases SecOps Dashboards Audit Firewalls, IDS/IPS SIEM Augmentation Logging Metrics Security
17. 17 Modules: Data sources made easy • Collect specific type of data • Parse and enrich it • Default dashboards, alerts, ML jobs ./filebeat -e -modules=system -setup
18. 18 Packetbeat (it all started with Beats 1.0)
19. 19 Metricbeat modules (introduced in 5.0) Aerospike Apache Ceph Couchbase Docker Dropwizard Elasticsearch Golang Graphite HAProxy HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vSphere Windows ZooKeeper
20. 20 Filebeat modules (introduced in 5.3) Apache2 Auditd Icinga Kafka MySQL Nginx PostgreSQL Redis System
21. 21 Logstash modules (introduced in 5.6) ArcSight Netflow
22. 22 DEMO
23. 23 Logging Data Sources System • Linux / MacOS • Windows Events Containers • Docker (6.0) • Kubernetes (6.0) Infrastructure Applications Databases • MySQL • PostgreSQL (6.1) Queues • Kafka (6.1) • Redis (6.0) Web servers • Apache • Nginx Other • HAProxy* • Zookeeper* WINLOGBEAT FILEBEAT * Near-term roadmap
24. 24 Metrics & Event Data System • Linux • MacOS • Windows • Perfmon (6.0) • WMI* Infrastructure Cloud • AWS • GCP • Azure* • DigitalOcean …. Containers • Docker • Kubernetes (6.0) Virtualization • vSphere (6.0) PACKETBEAT METRICBEAT Network • Netflow (5.6) • Packets Storage • Ceph LOGSTASH * Near-term roadmap
25. 25 Metrics & Event Data Applications Datastores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike (6.0) • Graphite (6.1) Web servers • Apache • Nginx Other • HAProxy • Zookeeper • Prometheus Queues • Kafka • Redis • RabbitMQ (6.0) Caches • Memcached (6.0) METRICBEAT Uptime • Heartbeat Custom apps • JMX/Jolokia • PHP-FPM • Golang (6.0) • Dropwizard (6.0) HEARTBEAT * Near-term roadmap LOGSTASH
26. 26 Security Data Sources Security Activity SIEM Augmentation • ArcSight (5.6) • more* Audit • Auditd • Auditbeat (6.0) Systems • Access • SSH Applications • Connections • Users Network • IPs / GeoIP • DNS Packets • Netflow (5.6) • Firewalls* • IDS/IPS* FILEBEAT PACKETBEAT METRICBEAT LOGSTASH * Near-term roadmap
27. 27 Business Analytics Structured Activity Databases • JDBC input • JDBC filter SaaS services • Salesforce • Heroku • Github • Azure* LOGSTASH * Near-term roadmap Social media • Twitter
28. 28 http://go.es.io/2gEBoLN Data Sources Survey
29. Administration
30. 30 Monitoring & Management Logstash • Centralized monitoring (5.3) • Centralized management (6.0) * Near-term roadmap
31. 31 DEMO
32. 32 Monitoring & Management Logstash • Centralized monitoring (5.3) • Centralized management (6.0) Beats (Roadmap) • Centralized monitoring • Centralized management
33. 33 Calls to action • Familiarize yourself with latest integrations (including in X-Pack) • Watch UI roadmap for additional add-data workflows • Take the Data Sources Survey: http://go.es.io/2gEBoLN • Come talk to us at the AMA booth
34. Thank You
35. Thank You Find me at AMA booth or email tanya@elastic.co
36. Appendix
37. 37 Tools: Grok Debugger
38. 38 DEMO
39. 39 Demo: Filebeat Nginx Backup - Won’t be shown
40. 40 Demo: Nginx Module ML job (5.6) Backup - Won’t be shown
41. 41 Demo: Nginx Module ML job (5.6) Backup - Won’t be shown
42. 42 Demo: Filebeat Docker (6.0) Backup - Won’t be shown
43. 43 Demo: Logstash Arcsight Module (5.6) Backup - Won’t be shown
44. 44 DEMO
45. 45 Demo: Central Management Backup - Won’t be shown
46. 46 Demo: Central Management Backup - Won’t be shown
47. 47 Demo: Monitoring Backup - Won’t be shown
48. 48 Demo: Monitoring Backup - Won’t be shown
49. Elastic Ingestion Technologies CUSTOM CONNECTORS CENTRALIZED COLLECTION Logstash API Clients CUSTOM CONNECTORS Elasticsearch Transform Store ingest node data node es-hadoop 49 network devices DBs DISTRIBUTED COLLECTION Beats servers, containers