Share
Embed
Elastic{ON} Tour 2017 New York : Ingest
Elastic{ON} Tour New York - November 9, 2017
Walk through all things ingest for Logstash 5.x, from dead letter and persistent queues to the Grok Debugger and new monitoring APIs. Then get caught up on new lightweight data shipper additions like Heartbeat and Metricbeat, as well as new modules that simplify the getting started process.
Kevin Kluge | Senior VP of Engineering | Elastic
Elastic Co
November 09, 2017
Transcript
51 slides
-
2 Data Ingestion The process of collecting and importing data
for immediate use in a datastore -
4 Ingest Technologies Lightweight Data Shippers Beats Centralized Data Collection
Engine Logstash Hadoop Ecosystem Connector ES-Hadoop APIs Ingest Node Elasticsearch -
6 Elastic Ingestion Technologies CUSTOM CONNECTORS Elasticsearch Transform Store ingest
node data node es-hadoop -
7 Elastic Ingestion Technologies DISTRIBUTED COLLECTION Elasticsearch Beats servers, containers
Transform Store ingest node data node Metrics Logs -
8 Elastic Ingestion Technologies network devices DB data CENTRALIZED COLLECTION
Logstash DISTRIBUTED COLLECTION Beats servers, containers Elasticsearch Transform Store ingest node data node Flows JDBC -
9 Ingestion Architecture Scalable and robust centralized ETL • Java
event rewrite • Multiple pipelines -
10 Ingestion Architecture Scalable and robust centralized ETL • Persistent
queues • Dead letter queues -
Cooperative Ingestion Elasticsearch Transform Store ingest node data node 11
DISTRIBUTED COLLECTION Beats servers, containers -
Elastic Ingestion Technologies CENTRALIZED COLLECTION Logstash Elasticsearch Transform Store ingest
node data node 12 network devices DISTRIBUTED COLLECTION Beats servers, containers -
Elastic Ingestion Technologies CENTRALIZED COLLECTION Logstash Elasticsearch Transform Store ingest
node data node 13 network devices DISTRIBUTED COLLECTION Beats servers, containers -
14 Easy migration between ingest technologies Ingest Node to Logstash
conversion tool Elasticsearch ingest node Logstash ingest node -
16 Use Cases & Data Sources Common Log Formats System
Web Servers Queues Turnkey Monitoring Infrastructure Containers Databases SecOps Dashboards Audit Firewalls, IDS/IPS SIEM Augmentation Logging Metrics Security -
17 Modules: Data sources made easy • Collect specific type
of data • Parse and enrich it • Default dashboards, alerts, ML jobs ./filebeat -e -modules=system -setup -
18 Metricbeat modules (introduced in 5.0) Aerospike Apache Ceph Couchbase
Docker Dropwizard Elasticsearch Golang Graphite HAProxy HTTP Jolokia Kafka Kibana Kubernetes Memcached MongoDB MySQL Nginx PHP_FPM PostgreSQL Prometheus RabbitMQ Redis System vSphere Windows ZooKeeper -
19 Filebeat modules (introduced in 5.3) Apache2 Auditd Icinga Kafka
MySQL Nginx PostgreSQL Redis System -
23 Logging Data Sources System • Linux / MacOS •
Windows Events Containers • Docker (6.0) • Kubernetes (6.0) Infrastructure Applications Databases • MySQL • PostgreSQL (6.1) Queues • Kafka (6.1) • Redis (6.0) Web servers • Apache • Nginx Other • HAProxy* • Zookeeper* WINLOGBEAT FILEBEAT * Near-term roadmap -
24 Metrics & Event Data System • Linux • MacOS
• Windows • Perfmon (6.0) • WMI* Infrastructure Cloud • AWS • GCP • Azure* • DigitalOcean …. Containers • Docker • Kubernetes (6.0) Virtualization • vSphere (6.0) PACKETBEAT METRICBEAT Network • Netflow (5.6) • Packets Storage • Ceph LOGSTASH * Near-term roadmap -
25 Metrics & Event Data Applications Datastores • MySQL •
PostgreSQL • MongoDB • Couchbase • Aerospike (6.0) • Graphite (6.1) Web servers • Apache • Nginx Other • HAProxy • Zookeeper • Prometheus Queues • Kafka • Redis • RabbitMQ (6.0) Caches • Memcached (6.0) METRICBEAT Uptime • Heartbeat Custom apps • JMX/Jolokia • PHP-FPM • Golang (6.0) • Dropwizard (6.0) HEARTBEAT * Near-term roadmap LOGSTASH -
26 Security Data Sources Security Activity SIEM Augmentation • ArcSight
(5.6) • more* Audit • Auditd • Auditbeat (6.0) Systems • Access • SSH Applications • Connections • Users Network • IPs / GeoIP • DNS Packets • Netflow (5.6) • Firewalls* • IDS/IPS* FILEBEAT PACKETBEAT METRICBEAT LOGSTASH * Near-term roadmap -
27 Business Analytics Structured Activity Databases • JDBC input •
JDBC filter SaaS services • Salesforce • Heroku • Github • Azure* LOGSTASH * Near-term roadmap Social media • Twitter -
30 Monitoring & Management Logstash • Centralized monitoring (5.3) •
Centralized management (6.0) * Near-term roadmap -
32 Monitoring & Management Logstash • Centralized monitoring (5.3) •
Centralized management (6.0) Beats (Roadmap) • Centralized monitoring • Centralized management -
33 Next steps • Familiarize yourself with latest integrations (including
in X-Pack) • Watch UI roadmap for additional add-data workflows • Take the Data Sources Survey: http://go.es.io/2gEBoLN • Come talk to us at the AMA booth -
Machine Learning1 Algorithms and methods for data driven prediction, decision
making, and modelling Supervised Learning Prediction based on examples of correct behavior 1Machine Learning Overview, Tommi Jaakkola, MIT Unsupervised Learning No explicit target, only data, goal to model/discover Semi-supervised Learning Supplement limited annotations with unsupervised learning Active Learning Learn to query the examples actually needed for learning Transfer Learning How to apply what you have learned from A to B Reinforcement Learning Learning to act, not just predict; goal to optimize the consequences of actions Other! … -
Machine Learning1 Algorithms and methods for data driven prediction, decision
making, and modelling Supervised Learning Prediction based on examples of correct behavior 1Machine Learning Overview, Tommi Jaakkola, MIT Unsupervised Learning No explicit target, only data, goal to model/discover Semi-supervised Learning Supplement limited annotations with unsupervised learning Active Learning Learn to query the examples actually needed for learning Transfer Learning How to apply what you have learned from A to B Reinforcement Learning Learning to act, not just predict; goal to optimize the consequences of actions Other! … Time Series Anomaly Detection and Forecasting -
Anomaly Detection in Time Series Data • Learn models from
past behaviour (training, modelling) • Use models to predict future behaviour (prediction) • Use predictions to make decisions Expected value @ 15:05 = 1859 Actual value @ 15:05 = 280 Probability = 0.0000174025 -
41 Rules Don’t Scale • Where do you set the
threshold? • Who maintains the rules? -
42 DNS Are there signs of data exfiltration? packetbeat Traffic
Is one of my users an insider threat? metricbeat Auth Logs Is a brute- force attack underway? filebeat Security Analytics -
43 Unusual spike in user latency Server woes or regional
outage Rare event from sensor Failing device Metrics -
44 It All Begins with Data Discovering information in NGINX
logs 68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/ 2017:12:16:27 +0200] "GET /sites/default/files/styles/ company_profile_cover_crop/public/1500x500_1_10.jpg? itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36” -
45 68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/
company_profile_cover_crop/public/1500x500_1_10.jpg?itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 221.247.242.171, 162.158.166.51, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/ company_profile_logo/public/company_logos/aaeaaqaaaaaaaawvaaaajdk3n2vkzme0lte0zjctngy3ms1inmm4lta4ntnhzwqymzvmoq.png? itok=H2B05xX0 HTTP/1.1" 200 9296 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 192.228.32.190, 108.162.246.21, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /jobs/24237/it-back-end HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover_crop/public/1500x500_1_10.jpg? itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 221.247.242.171, 162.158.166.51, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/ company_profile_logo/public/company_logos/aaeaaqaaaaaaaawvaaaajdk3n2vkzme0lte0zjctngy3ms1inmm4lta4ntnhzwqymzvmoq.png? itok=H2B05xX0 HTTP/1.1" 200 9296 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 192.228.32.190, 108.162.246.21, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /jobs/24237/it-back-end HTTP/1.1" 301 5 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 137.56.184.63, 162.158.165.50, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/ company_profile_cover/public/1500x500_1_10.jpg?itok=1cNqdGYK HTTP/1.1" 200 102268 "https://www.startus.cc/company/ finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/ 537.36" 92.222.165.172, 162.158.167.202, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "POST /jstats.php HTTP/1.0" 200 13 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 68.75.44.178, 172.68.146.54, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/company_profile_cover_crop/public/1500x500_1_10.jpg? itok=RUgim2UQ&sc=297009042628d7de3f0eb50e807d29e4 HTTP/1.1" 200 92763 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 221.247.242.171, 162.158.166.51, 127.0.0.1 - - [15/May/2017:12:16:27 +0200] "GET /sites/default/files/styles/ company_profile_logo/public/company_logos/aaeaaqaaaaaaaawvaaaajdk3n2vkzme0lte0zjctngy3ms1inmm4lta4ntnhzwqymzvmoq.png? itok=H2B05xX0 HTTP/1.1" 200 9296 "https://www.startus.cc/company/finleap" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" -
46 Ingest, Enrich, Visualize, Analyze, Alert Elasticsearch X-pack Master Nodes
(3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Beats Log Files Metrics Wire Data your(beat) Filebeat Module NGINX Kibana X-pack Instances (X) -
49 Population Analytics Unusual clients (nginx.access.remote_ip) high_distinct_count("nginx.access.url") over "nginx.access.remote_ip" high_count("nginx.access.url")
over "nginx.access.remote_ip" Scanners Heavy Hitters