June 2015, PDXDevops - HEK Stack

Data stream processing at Treehouse with Heka

Transcript

1. The HEK Stack at Treehouse lessons learned running heka in production PDXDevOps, June 2015
2. Nathan Williams Systems Developer teamtreehouse.com @nathwhal / github.com/nathwill
3. HEK Stack? • Like the ELK stack, but not – Heka – https://hekad.readthedocs.org/ – Elasticsearch – https://elastic.co/products/elasticsearch – Kibana – https://elastic.co/products/kibana
4. But first there was ELK ELK is a misnomer, there's typically more involved
5. Logstash Design
6. Logstash Configuration
7. ELK Topology
8. This is gonna be awesome
9. Graphs!
10. Until… Reality Sets In
11. Pages… Pages Everywhere
12. Problems Encountered • Agent installation problems • High resource utilization in parsing • Buffer backups • Random hangs/slowdowns • Data discarded on error
13. Why not ELK stack? • Elasticsearch, Great! • Logstash, Meh. • Kibana, Great!
14. Well, then...
15. Heka Design
16. Heka Configuration
17. HEK Topology
18. Now you're talkin'!
19. Operational Benefits • Distributed, “smart” agent, all the power out at the edge • Efficient indexing, Go is awesome • Resilient under error conditions, back- pressure • Multipurpose design (more than logs) • More...
20. Facilitating Dev-Ops Cooperation
21. It's working!
22. App Log Formatting
23. Things to know • Complexity – Learning curve • Lua – Sandbox – PEGs • Go – Recompile to extend • Smaller ecosystem – Fewer integrations – Very helpful (IRC, mailing list)
24. Lua Plugins
25. What Next? • Metrics – Collectd – StatsD • Anomaly Detection – Local – Aggregate • More Outputs – Hipchat/Slack – Sensu → PagerDuty
26. Questions?