Integrating your Go Service with the ELK Stack

This is a slide deck, which supplements a workshop I did for Golab Conf 2020 - https://golab.io

Workshop abstract follow...
In this workshop, we will start from a sample Go web application & integrate it, step by step, with the ELK stack. This will enable you to create shiny dashboards, tracking your service's logs & monitoring any issues your users might encounter. It will be VERY hands-on, so be prepared.

The workshop consists of a small lecture (up to 40 mins) & a lot of hands-on exercises, the participants go through. These cover integrating a Go web application from scratch \w the ELK stack - Elasticsearch, Logstash, Kibana. That is the most popular & widely used system for monitoring web application logs. Participants will learn all the basic configuration they'll need to bootstrap their own ELK instances, how to emit structured logs, optimised for analysing your service's domain-specific production issues & also how to actually use their new shiny dashboards to find some "sample bugs" in our web application.

Transcript

1. Monitoring Go Services - Logging Preslav Mihaylov, Software Engineer @ Uber, Technical Trainer & Consultant
2. Agenda 1. The classic approach to logging 2. Introduction to Structured Logging 3. Introduction to the ELK Stack 4. Analysing production traffic with Kibana
3. The classic approach to logging
5. Problems
6. What does it take to start debugging?
7. Debugging via Logs - Runbook 1. ssh to prod 2. download the log locally 3. filter out the lines you don’t need (grep to the rescue) 4. open the IDE 5. prepare a sheet of paper to trace what’s going on 6. That’s it!
8. Simple... right?
9. And what if an outage happens while you’re ...
10. ...
11. There is a better way…
12. Introduction to Structured Logging
14. Structured logs == logs with schema
15. How does that help?
16. Structured logs can be parsed by software Or at least make grepping easier...
17. Introduction to the ELK Stack
18. The ELK stack is a set of tools which gives you this...
20. How is it better you ask?
21. With a single glance you can... 1. Evaluate the success to error ratio 2. Which endpoints have the most errors 3. What kinds of errors is your application throwing 4. Which input parameters cause the highest % of errors 5. And much more...
22. DEMO
23. So what is ELK anyways?
24. E == Elasticsearch This stores your logs... L == Logstash This filters & enriches your logs... B == Beats (surprise!) Various applications which collect your logs... K == Kibana This visualizes your logs
25. Enough theory... Learn it yourself
26. It’s your turn 4 x EXERCISE
27. Conclusion
28. Classical logs can work well for analysing issues on your server, but are often clumsy to use...
29. Structured logging can give you “the best bang for your buck” from your logs
30. The ELK Stack enables you to capitalize on your structured logs and effectively monitor production traffic
31. Q&A Time