Adopting observability at scale

Live recording: https://www.youtube.com/watch?v=HEAgbPJhIzg

In this retrospective talk Igor looks back on his experience in establishing and improving observability in microservice stacks. Whether it's 5, 50 or 500 services, whether the stack is built in Go, Node, Java, Python, PHP or all of them at once - operating a complex system at scale means both performance and stability have to have actionable metrics and ways to efficiently find and fix problems.

Key takeaways:
- Ways to meaningfully connect metrics, traces and logs
- Lessons learned when adopting observability across multi-technology stacks
- How observability helps to transition from a monolith or few microservices to large microservice stacks with minimal disruption
- Why standards matter

Igor is a tech lead with strong background in distributed systems. He started his career in a broadband provider helping to design and scale the ISP's core. Igor has been in the industry for 13 years, leading engineering function in a fintech until 2019. He's current challenge is with Skyscanner where Igor is working on a next generation booking platform as a Principal Engineer.

Transcript

1. Adopting observability at scale Igor Karpovich @ikarpovich
2. Who I Am @ikarpovich Until 2014 2014 - 2019 2019+ Engineer to CTO in digital consultancies and telecoms Head of Engineering at Currency Solutions Principal Engineer at Skyscanner Fun facts: - First public linux server at 13yo - Had 19” rack in a flat - Still listed as admin for IPv4 AS
3. The Monolith Crew complement
 up to 179,000
5. Traditional answer is LOGS!
6. We’re in 2019, we have 5 micro services!
7. Will you be fast enough to read through logs?
8. And what if
 we have 50?
9. Or 500?
10. Observability ❤
12. Framework
 Common envelope
 Context propagation Logging driver Generation 2 Generation 1 JSON Text Yet Another Logging Platform Where is ?
13. Skyscanner Data Platform
14. Logging • Structured logging (in JSON at least, or maybe in Proto?) • Defined data structure • Common tags to describe the context (request or session, order funnel, identity, application release, platform) • Logging decoupled from the application, or at least asynchronous • Reliable log delivery (transport and platform) • Considerations around PII { “message”: “Error ”, “error”: { “file”: “/usr/local/…”, “line”: 322, “stack”: “…”, “code”: “921-575” }, “context”: { “userId”: “uuid”, “requestId”: “uuid”, “traceId”: “uuid”, “sessionId”: “uuid”, “version”: “1.5.1123”, “pod”: “uuid” } }
15. Metrics • Consider push vs poll - InfluxDB/OpenTSDB/Graphite vs Prometheus/Thanos • Don’t think metrics as just CPU and memory • Define a standard around metrics for typical operations like http or database calls • Gather/calculate business metrics (orders, funnel steps) • Action!
16. Tracing
17. Why tracing? • Network-related issues • API calls latency • Internal language calls latency • Threads and goroutines • Exceptions, crashes and UX errors • Structured logging • Dependency analysis
19. • HTTP • gRPC • MQ (e.g. SQS) • Streams (e.g. Kafka/Kinesis) • Databases • Caches • Emails …across multiple transports
21. Personal record so far 20+ services 1000+ spans in a single trace
22. Lessons learned • Define practical standards • Sell the tech and the standards • Consider building an abstraction • Track the adoption • Use metrics Microservice 1 …….. ✅ Microservice 2 …….. ✅ Microservice 3 …….. Microservice 4 …….. Microservice 5 …….. ✅ Microservice 6 …….. ✅ Microservice 7 …….. ✅ Adoption plan
23. Iterate!
24. How to sell?
25. Thank you @ikarpovich I’m hiring! https://grnh.se/deb0961b1