PLUG AT - Practical Intro to Stream Processsing

A practical introduction to stream processing with Mozilla's Heka

Transcript

1. The HEK Stack at Treehouse lessons learned running heka in production PLUG AT, Aug 2015
2. Nathan Williams Systems Developer/SRE teamtreehouse.com @nathwhal / github.com/nathwill
3. What is Stream Processing • Formally: Given a continuous data set, a series of operations is performed on each element in the stream • In English: chunk-wise real-time analysis/normalization/mutation of an event stream, analogous to signal processing in audio
4. Why Stream Processing • Real-time – Speed matters! • Scalability • Not exclusive to later batch processing • Multi-purpose – Data analysis/enrichment – Alerting (abuse/anomaly detection)
5. Stream Processing Systems • Apache Spark • Apache Storm • Apache Samza • AWS Kinesis • IBM InfoSphere Streams • Logstash • Heka
6. HEK Stack? • Like the ELK stack, but not – Heka – https://hekad.readthedocs.org/ – Elasticsearch – https://elastic.co/products/elasticsearch – Kibana – https://elastic.co/products/kibana
7. But first there was ELK ELK is a misnomer, there's typically more involved
8. Logstash Design
9. Logstash Configuration
10. Typical ELK Topology
11. This is gonna be awesome
12. Graphs!
13. Until… Reality Sets In
14. Pages… Pages Everywhere
15. Problems Encountered • Agent installation problems • High resource utilization in parsing • Buffer backups • Random hangs/slowdowns • Data discarded on error
16. Why not ELK stack? • Elasticsearch, Great! • Logstash, Meh. • Kibana, Great!
17. Well, then...
18. Heka Design
19. Heka Configuration
20. HEK Topology
21. Now you're talkin'!
22. Operational Benefits • Distributed, “smart” agent, all the power out at the collection point • Support for aggregation, downsampling, anomaly detection • Efficient indexing, Go is awesome • Resilient under error conditions, back- pressure • Multipurpose design (more than logs) • More...
23. Efficient, Really!
24. Insights
25. Insights
26. Facilitating Dev-Ops Cooperation
27. It's working!
28. App Log Formatting
29. Things to know • Complexity – Learning curve • Lua – Sandbox – PEGs • Go – Recompile to extend • Smaller ecosystem – Fewer integrations – Very helpful (IRC, mailing list)
30. Lua plugins are easy
31. Example Lua telemetry plugins • Nginx Stub Status Decoder • Load Avg Decoder • Disk Stats Decoder
32. What Next? • More Enrichment – GeoIP data – Session Data • More Metrics – Collectd – StatsD – ProcessInput • More Anomaly Detection – Aggregate • More Outputs – AWS S3 via HTTPOutput – Hipchat/Slack – Sensu → PagerDuty
33. Questions? Slide Deck - http://bit.ly/1NslHjc