Get real-time insights from your application with Packetbeat & Elasticsearch

This presentation explains how Packetbeat extracts data from packets and how you can use Elasticsearch to get insights from these data. It takes you from the raw bytes up to doing anomaly detection via moving averages.

Transcript

1. Get real-time insights from your application with Packetbeat & Elasticsearch
2. www.elastic.co 2 In today’s program • Lots of JSON objects • A cow, an elk, a violin, a guitar and a set of drums • Anomaly detection via moving averages
3. www.elastic.co 3 Tudor Golubenco twitter.com/tudor_g github.com/tsg I am
4. www.elastic.co 4 I work for
5. www.elastic.co 5 the company behind Elasticsearch Logstash Kibana
6. www.elastic.co 6 also known as the ELK stack Photo  credit:  https://www.flickr.com/photos/lsmith2010/8215026548
7. www.elastic.co 7 Open source culture Image  credit:  https://www.flickr.com/photos/tappnel/5798812875 • We live in GitHub • We talk Pull Requests • Conferences • Community • Blog posts
8. www.elastic.co 8 Started an open source project Packetbeat
9. www.elastic.co 9 We wanted to make Monitoring and troubleshooting tools for complex applications and infrastructures
10. www.elastic.co 10 Idea look at the communication between services
11. www.elastic.co 11 Capture network packets • Visibility into the infrastructure by • Passively listening to network packets • It doesn’t add latency • It cannot break your application Image  credit:  https://www.flickr.com/photos/bigdrumthump/3223280727
12. www.elastic.co 12 Packet capturing 1.  Using  port  mirroring 2.  As  an  “agent”
13. www.elastic.co 13 Sniffing from a technical PoV • libpcap (tcpdump), supports all Unix like systems • Winpcap, supports Windows • For Go, gopacket provides bindings and more • High speed API for packet capturing on Linux: af_packet Image  credit:  https://www.flickr.com/photos/57881779@N04/7930362242/
14. www.elastic.co 14 Decoding
15. www.elastic.co 15 Matching requests and responses • Pipelining complicates matching the requests with the responses.
16. www.elastic.co 16 Create a JSON object for each request-response pair HTTP  transaction GET  method Response  code Response  time
17. www.elastic.co 17 SQL example SQL  method Query Error  message Bandwidth   information
18. www.elastic.co 18 DNS example Query  type Domain  name Response  time
19. www.elastic.co 19 Packetbeat: Overview
20. www.elastic.co 20 There’s more to apps than packets Packetbeat   Listens  to  the  “beat”  of   the  network  packets. Topbeat   Listens  to  the  “beat”  of   the  operating  system   metrics. Image  credits:   https://www.flickr.com/photos/7147684@N03/921738874/   https://www.flickr.com/photos/bigdrumthump/3223280727   https://www.flickr.com/photos/jadeashleyphotography/6584949945/   https://www.flickr.com/photos/mitosettembremusica/2839965900/   Filebeat   Listens  to  the  “beat”  of   logs. Metricsbeat   Listens  to  the  internal   “beat”  of  systems  via   APIs.
21. www.elastic.co 21 Topbeat • Like the Unix top command but sending the data periodically to Elasticsearch • Works also on Windows
22. www.elastic.co 22 Topbeat system wide and per process stats CPU  “steal”  time Total  /  used  /  free   memory CPU  stats Per  process  stats CPU  time     consumed Process  pid,  name,   parent  pid,  etc. Memory  used
23. www.elastic.co 23 Topbeat output objects File  system  stats Mount  point Device  name Total,  used,  free   disk  space
24. www.elastic.co 24 Filebeat • A “Beat” based on the Logstash-Forwarder source code • Do one thing well: • Send log files to Logstash & Elasticsearch • Light on consumed resources • Easy to deploy on multiple platforms
25. www.elastic.co 25 Filebeat JSON output The  log  message The  timestamp The  log  level
26. www.elastic.co 26 Beats have libbeat in common • Go library • Provides common things for all Beats: • logging, service handling, configuration file handling, CLI flags • Outputs and filters Dev  guide  for  creating  a  new  Beat:  https://www.elastic.co/guide/en/beats/libbeat/current/index.html
27. www.elastic.co 27 Deployment: directly to ES • Option 1: Insert directly into Elasticsearch via the bulk API • Security can be provided via Shield and HTTPs
28. www.elastic.co 28 Deployment: Send to Logstash • Option 2: Insert via Logstash • Uses the Lumberjack protocol which offers security • Gives the opportunity of enriching or modifying the data
29. www.elastic.co 29 Getting insights from the data • Elasticsearch aggregations • Split the data into buckets • Apply a function over the data • Freely combine them by nesting • Work with multiple shards Image  credit:  https://www.flickr.com/photos/sheeprus/4551642374/
30. www.elastic.co 30 Date histogram •Splits data in buckets of time
31. www.elastic.co 31 Date histogram response
32. www.elastic.co 32 Percentiles aggregation 95th percentile value means that 95% of the values are smaller than it
33. www.elastic.co 33 Percentile aggregation response
34. www.elastic.co 34 Percentile aggregation •Approximate values •T-digests algorithm by Ted Dunning •Accurate for small sets of values •More accurate for extreme percentiles
35. www.elastic.co 35 Date histogram nested with percentiles
36. www.elastic.co 36 Date_time + percentiles response
37. www.elastic.co 37 Result
38. www.elastic.co 38 Histogram by response time • Splits data in buckets by response time • [0-10ms), [10ms-20ms), …
39. www.elastic.co 39 Latency histogram
40. www.elastic.co 40 Add a date histogram
41. www.elastic.co 41 Response times repartition
42. www.elastic.co 42 Kibana config
43. www.elastic.co 43 Slowest RPC methods •Combines terms and percentiles aggregations
44. www.elastic.co 44 Terms aggregation • Buckets are dynamically built: one per unique value • By default: top 10 by document count • Approximate because each shard can have a different
45. www.elastic.co 45 Order by 99th percentile
46. www.elastic.co 46 • New in Elasticsearch 2.0 (currently in beta) • Work on the results of other aggregations Pipeline aggregations
47. www.elastic.co 47 Derivative aggregation • Metric constantly growing • Take first order derivate to see the speed of growth
48. www.elastic.co 48 Simple moving average • [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] • with a window size of 5: • (1 + 2 + 3 + 4 + 5) / 5 = 3 • (2 + 3 + 4 + 5 + 6) / 5 = 4 • (3 + 4 + 5 + 6 + 7) / 5 = 5 • etc.
49. www.elastic.co 49 Exponentially weighted moving average • Older values become exponentially less important
50. www.elastic.co 50 Moving average - dynamic thresholds • yellow - measured values • purple - moving average (ewma) • green - threshold, mean + (3 * standard deviation)
51. www.elastic.co 51 Request Extended  stats  agg  for   mean  and  std   deviation Moving  averages   aggs  for  mean  and   std Bucket  script  agg Details:  https://www.elastic.co/blog/staying-­‐in-­‐control-­‐with-­‐moving-­‐averages-­‐part-­‐1
52. www.elastic.co 52 Cyclic trends - anomalies • EWMA lags behind too much • The values constantly hit the threshold
53. www.elastic.co 53 Cyclic trends - anomalies • Holt-Winters (triple exponential) model works better for seasonal data • Requires two periods to bootstrap the algorithm
54. www.elastic.co 54 Thanks • Live demo: http://demo.elastic.co/packetbeat/ • Twitter: @tudor_g • Come by the booth, we have stickers!