Monitoring and Debugging Containers

@rakyll monitoring and debugging containerized systems Jaana B. Dogan, Google
[email protected]

@rakyll me overly frustrated engineer 15+ years in networking systems
making systems more reliable

@rakyll the new old monitoring? (maybe)

@rakyll systems are growing... and you are not in control

@rakyll bare metal kernel network stack cloud stack libraries frameworks
your code

@rakyll

@rakyll complexity is inevitable

@rakyll container

@rakyll container container

@rakyll container container message queue

@rakyll container container storage/database

@rakyll container container load balancer location=us-west location=europe-central

@rakyll host host container container load balancer

@rakyll container container container container container orchestrated hot mess

@rakyll areas of issues: - lack of locality - networking
- scheduling - dependencies

@rakyll bare metal kernel network stack cloud stack libraries frameworks
your code

@rakyll “my job is done here”

@rakyll after going to production... 1. monitor 2. alert 3.
troubleshoot 4. fix

@rakyll

@rakyll load balancer

@rakyll load balancer critical path

@rakyll discovering critical paths making them reliable then fast making
them debuggable

@rakyll

@rakyll Latency Numbers Every Programmer Should Know by Jeff Dean

@rakyll

@rakyll ping pong pongservice:6996 project: ping the pong server.

@rakyll opencensus.io

@rakyll not my team!

@rakyll where is the source code?

@rakyll who to page?

@rakyll give me the logs, runtime events, profiles...

@rakyll

@rakyll http://server:9999/tracez

@rakyll challenges...

@rakyll no wire standards

@rakyll

@rakyll traceparent: <version>-<traceid>-<spanid>-<opts> Example: traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

@rakyll no export standards

@rakyll areas of issues: - locality - networking - scheduling
- dependencies

@rakyll fin [email protected]

More Decks by JBD