Debugging & Troubleshooting Microservices in Kubernetes & Stackdriver

It’s easier than ever to be able to deploy and manage microservices architecture with container and contain orchestration tool like Kubernetes. However, debugging multiple microservices across network boundaries is hard. Trying to what’s going on in a production microservices deployment at scale is impossible without proper tools! Google has spent over a decade deploying containerized Java applications at unprecedented scale. During that time Google has built an ecosystem of infrastructure and tools to manage, troubleshoot, and debug, at scale. These tools were originally used to diagnose problems with software running on Borg. With Google Cloud Platform we’re bringing some of the best kept internal technology to developers all over the world. In this session, we’ll deploy and scale Java microservices in Kubernetes, walk through common troubleshooting techniques with logs, shell access, isolating troubled containers, and also connecting to specific containers via port forwarding.

Beyond the basic tools, we’ll see how easy production diagnostic we can enjoy with Stackdriver and Google Cloud Platform, including capturing traces & logs without managing your own infrastructure, and live debugging and injecting log messages without needing to rewrite nor redeploy your code!

Transcript

1. @saturnism @googlecloud Debugging & Troubleshooting Microservices w/ Kubernetes It runs in production, but it's not enough!
2. @saturnism @googlecloud Ray Tsang Developer Advocate Google Cloud Platform @saturnism | +RayTsang
3. @saturnism @googlecloud Ray Tsang Developer Architect Traveler Photographer flickr.com/saturnism
4. @saturnism @googlecloud
5. @saturnism @googlecloud #kubernetes #devoxx
6. @saturnism @googlecloud
7. @saturnism @googlecloud Hello World Service - Greet Guestbook Service - Create Guestbook Service - Retrieve
8. @saturnism @googlecloud Guestbook UI Hello World Service Redis session replication greeting MySQL Guestbook Service CRUD
9. @saturnism @googlecloud Visibility & Insight is Key
10. @saturnism @googlecloud Let's see it!
11. @saturnism @googlecloud Production vs Non-Production :(
12. @saturnism @googlecloud Distributed Systems are hard!
13. @saturnism @googlecloud We've been there, done that
14. @saturnism @googlecloud Information - Logs! kubectl logs -f pod_id
15. @saturnism @googlecloud Inspecting the process / container kubectl exec -ti pod_id /bin/bash
16. @saturnism @googlecloud Port forward to container kubectl port-forward pod_id local_port:remote_port
17. @saturnism @googlecloud Isolate problematic instance w/ Labels kubectl label pod pod_id --overewrite serving=false
18. @saturnism @googlecloud Pod frontend Pod serving = true version = 1.0 Pod serving = true version = 1.0 Service Label selectors: version = 1.0 serving = true Pod serving = true version = 1.0
19. @saturnism @googlecloud Pod frontend Pod serving = true version = 1.0 Pod serving = true version = 1.0 Service Label selectors: version = 1.0 serving = true Pod serving = false version = 1.0
20. @saturnism @googlecloud Log aggregation, at scale
21. @saturnism @googlecloud Stackdriver Logging https://cloud.google.com/logging/
22. @saturnism @googlecloud
23. @saturnism @googlecloud Metrics from Logs → Alerting Near-realtime insight
24. @saturnism @googlecloud Search and Query Your Logs
25. @saturnism @googlecloud Export to BigQuery, Storage, Pub/Sub Near-realtime insight
26. @saturnism @googlecloud
27. @saturnism @googlecloud
28. @saturnism @googlecloud Tracing at Google - Dapper https://research.google.com/pubs/pub36356.html
29. @saturnism @googlecloud Zipkin
30. @saturnism @googlecloud Stackdriver Trace https://cloud.google.com/trace/
31. @saturnism @googlecloud Zipkin → Stackdriver Trace https://cloud.google.com/trace/docs/zipkin
32. @saturnism @googlecloud
33. @saturnism @googlecloud
34. @saturnism @googlecloud
35. @saturnism @googlecloud
36. @saturnism @googlecloud Automatic reports Detect performance regression
37. @saturnism @googlecloud Oops… I forgot a log message. Darn!
38. @saturnism @googlecloud Don't you wish you can use a debugger?
39. @saturnism @googlecloud Stackdriver Debugger - Production Debugger https://cloud.google.com/debugger/
40. @saturnism @googlecloud Add debugger agent java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE
41. @saturnism @googlecloud Logpoint - dynamically add log messages! No redeployment, Magic. gcloud debug logpoints create HelloworldService.java:35 \ "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT
42. @saturnism @googlecloud Snapshots Inspect call stack and variables
43. @saturnism @googlecloud
44. @saturnism @googlecloud
45. @saturnism @googlecloud Trace Log Debug
46. @saturnism @googlecloud Trace → Stackdriver Trace Logs → Stackdriver Logging Debug → Stackdriver Debugger https://cloud.google.com/stackdriver/
47. @saturnism @googlecloud Thanks! Source: https://github.com/saturnism/spring-boot-docker Stackdriver: https://cloud.google.com/stackdriver/ Ray Tsang @saturnism Google Cloud Platform