Ingest and the Elastic Stack - Elastic{ON} Tour Seoul 2017

Walk through all things ingest for Logstash 5.x, from dead letter and persistent queues to the Grok Debugger and new monitoring APIs. Then get caught up on new lightweight data shipper additions like Heartbeat and Metricbeat, as well as new modules that simplify the getting started process.

Dead letter와 Persistent queues 부터 Grok Debugger 및 새로운 모니터링 API들에 이르는 Logstash 5.x의 데이터 수집에 대해서 배워보세요. 또한 경량 데이터 수집기 에디션인 Heartbeat와 Metricbeat 그리고 프로젝트를 더욱 쉽게 만들어 주는 새로운 모듈들에 대해서도 알아가시기 바랍니다.

Aaron Mildenstein | Consulting Architect | Elastic

Transcript

1. Aaron Mildenstein, Consulting Architect Seoul | Dec 12, 2017 Ingest
2. 2 Data Ingestion The process of collecting and importing data for immediate use
3. Simple things should be simple. 3 ? Shay Banon Elastic{ON}’17
4. 4 Ingest Technologies Lightweight Data Shippers Beats Centralized Data Collection Engine Logstash Hadoop Ecosystem Connector ES-Hadoop APIs Ingest Node Elaticsearch
5. 5 Elastic Ingestion Technologies API Elasticsearch Transform Store ingest node data node
6. 6 Elastic Ingestion Technologies CUSTOM CONNECTORS Elasticsearch Transform Store ingest node data node es-hadoop
7. 7 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats servers, containers Transform Store ingest node data node Metrics Logs
8. 8 Elastic Ingestion Technologies network devices DB data CENTRALIZED COLLECTION Logstash DISTRIBUTED COLLECTION Beats servers, containers Elasticsearch Transform Store ingest node data node Flows JDBC
9. 9 Ingestion Architecture Scalable and robust centralized ETL • Persistent queues • Dead letter queues
10. 10 Ingestion Architecture Scalable and robust centralized ETL • Java event rewrite • Multiple pipelines
11. 11 Ingestion Architecture Scalable and robust centralized ETL • Java event rewrite • Multiple pipelines Logstash 5.x
12. 12 Ingestion Architecture Scalable and robust centralized ETL • Java event rewrite • Multiple pipelines Logstash 6.0
13. 13 Elastic Ingestion Technologies network devices DB data CENTRALIZED COLLECTION Logstash DISTRIBUTED COLLECTION Beats servers, containers Elasticsearch Transform Store ingest node data node Flows JDBC
14. 14 Elastic Ingestion Technologies network devices DB data CENTRALIZED COLLECTION Logstash DISTRIBUTED COLLECTION Beats servers, containers Elasticsearch Transform Store ingest node data node Flows JDBC
15. 15 Easy migration between ingest technologies Ingest Node to Logstash conversion tool Elasticsearch ingest node Logstash ingest node
16. Data Sources
17. 17 Use Cases & Data Sources Common Log Formats System Web Servers Queues Turnkey Monitoring Infrastructure Containers Databases SecOps Dashboards Audit Firewalls, IDS/IPS SIEM Augmentation Logging Metrics Security
18. 18 Modules: The Data to Dashboard Experience • Collect specific type of data • Parse and enrich it • Default dashboards, alerts, ML jobs ./filebeat -e -modules=system -setup
19. 19 Packetbeat (It all started with Beats 1.0)
20. 20 Metricbeat Modules (Introduced in 5.0) Aerospike Apache Ceph Couchbase Docker Dropwizard Elasticsearch Golang Graphite HAProxy HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vSphere Windows ZooKeeper
21. 21 Filebeat Modules (Introduced in 5.3) Apache2 Auditd Icinga Kafka MySQL Nginx PostgreSQL Redis System
22. 22 Logstash Modules (Introduced in 5.6) ArcSight Netflow
23. 23 ArcSight Module (Introduced in 5.6)
24. 24 Modules Demo NGINX Netflow ArcSight
25. 25 Logging Data Sources System • Linux / MacOS • Windows Events Containers • Docker (6.0) • Kubernetes (6.0) Infrastructure Applications Databases • MySQL • PostgreSQL (6.1) Queues • Kafka (6.1) • Redis (6.0) Web servers • Apache • Nginx Other • HAProxy* • Zookeeper* WINLOGBEAT FILEBEAT * Near-term roadmap
26. 26 Metrics & Event Data System • Linux • MacOS • Windows • Perfmon (6.0) • WMI* Infrastructure Cloud • AWS • GCP • Azure* • DigitalOcean …. Containers • Docker • Kubernetes (6.0) Virtualization • vSphere (6.0) PACKETBEAT METRICBEAT Network • Netflow (5.6) • Packets Storage • Ceph LOGSTASH * Near-term roadmap
27. 27 Metrics & Event Data Applications Datastores • MySQL • PostgreSQL • MongoDB • Couchbase • Aerospike (6.0) • Graphite (6.1) Web servers • Apache • Nginx Other • HAProxy • Zookeeper • Prometheus Queues • Kafka • Redis • RabbitMQ (6.0) Caches • Memcached (6.0) METRICBEAT Uptime • Heartbeat Custom apps • JMX/Jolokia • PHP-FPM • Golang (6.0) • Dropwizard (6.0) HEARTBEAT * Near-term roadmap LOGSTASH
28. 28 Security Data Sources Security Activity SIEM Augmentation • ArcSight (5.6) • more* Audit • Auditd • Auditbeat (6.0) Systems • Access • SSH Applications • Connections • Users Network • IPs / GeoIP • DNS Packets • Netflow (5.6) • Firewalls* • IDS/IPS* FILEBEAT PACKETBEAT METRICBEAT LOGSTASH * Near-term roadmap
29. 29 Business Analytics Structured Activity Databases • JDBC input • JDBC filter SaaS services • Salesforce • Heroku • Github • Azure* LOGSTASH * Near-term roadmap Social media • Twitter
30. Administration
31. 31 Monitoring & Management Logstash • Centralized monitoring (5.3) • Centralized management (6.0)
32. 32 Demo Monitoring & Management
33. 33 Monitoring & Management Logstash • Centralized monitoring (5.3) • Centralized management (6.0) Beats (Roadmap) • Centralized monitoring • Centralized management
34. 34 Calls to Action • Familiarize yourself with latest integrations (including in X-Pack) • Watch UI roadmap for additional add-data workflows • Come talk to us at the AMA booth
35. Thank You Find me at AMA booth or email untergeek@elastic.co