Lean log monitoring in Go

Why It is not a bad thing to build something small when you need it if nothing else fits.
In this talk I will share what went wrong, and what went well, and hopefully bring some inspiration for your team when they face a similar challenge.

Transcript

1. Hello! I am Shyamala Site Reliability Engineer @ Volkswagen Digital:Lab Berlin @shyamala_u shyamz-22 dev.to/shyamala_u
2. A Lean log monitoring system in Go
3. @shyamala_u There was an issue... “The Login is slow”
4. @shyamala_u There was an issue... “One does not Simply fix Its slow”
5. @shyamala_u The next step was ... To find a logging solution that was reliable and sophisticated
6. @shyamala_u In the process we figured out... Getting a hosted log management solution is complicated
7. @shyamala_u Because …. • Reluctance to let third party to process and store logs • Long processes to get through security and legalities • Not enough budget • Not enough capacity to host and manage an on premise solution
8. While waiting for a proper solution to be in place ⏳⌛⏳
9. @shyamala_u What if we build a simple solution that can... • Receive logs from multiple applications • Store pattern configuration per application and where to alert. • Process the incoming log of an application against the configuration • Send alert to configured channel when a match is found
10. @shyamala_u
11. Syslog to the rescue
12. @shyamala_u What is Syslog? In computing, syslog is a standard for message logging. It allows separation of : • the software that generates messages • the system that stores them • and the software that reports and analyzes them
13. @shyamala_u <165>1 2003-10-11T22:14:15.003Z mymachine.example.com evntslog - ID47 [exampleSDID@32473 iut="3" eventID="1011"] An application event log entry... | Value | Type | |--------------------------------------------|-----------------------| | <86> Facility=10, Severity=6 | Priority | | 1 | Version | | 2003-10-11T22:14:15.003Z | TimeStamp | | mymachine.example.com | Hostname | | evntslog | App Name | | - | ProcID | | [exampleSDID@32473 iut="3" eventID="1011"] | Structured Data | | An applicationevent log entry... | UTF-8 encoded message | What is Syslog?
14. Building the solution
15. @shyamala_u we wanted to build a solution that is also... • Robust • Low Resource footprint (CPU/RAM) • Low maintenance cost • Reliable
16. Ready, Steady Go ‍♀
17. Sample snippet
18. When we went live
19. @shyamala_u • Significantly long processing times under load Some hiccups..
20. @shyamala_u • Out of Memory Error Logging processed logs is a bad idea ‍♀ Replacing `json.Marshal` with `json.Encoder` with buffer from `sync.pool` made significant improvement under load name old time/op new time/op delta Cache/test_put-8 34.2µs ± 1% 33.8µs ± 1% -1.09% (p=0.000 n=10+9) name old alloc/op new alloc/op delta Cache/test_put-8 1.47kB ± 0% 1.33kB ± 0% -9.71% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Cache/test_put-8 42.0 ± 0% 40.0 ± 0% -4.76% (p=0.000 n=10+10) Some hiccups..
21. @shyamala_u A surprise... Negative LookAhead was not supported
22. Let numbers speak for Go
23. @shyamala_u 177k Average number of logs processed per minute
24. @shyamala_u 30 MB Average RAM Utilization for constant load 177k Logs Per Minute
25. @shyamala_u 99.5% Service availability in the last quarter
26. Is the login fast?
27. @shyamala_u Yes!! And some interesting usage patterns emerged
28. Thank you! @shyamala_u shyamz-22 dev.to/shyamala_u
29. @shyamala_u • Syslog Definition https://en.wikipedia.org/wiki/Syslog • go-syslog https://github.com/mcuadros/go-syslog • syslog-example https://github.com/shyamz-22/syslog-example Credits