Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Debugging & Troubleshooting Microservices in Kubernetes & Stackdriver

Debugging & Troubleshooting Microservices in Kubernetes & Stackdriver

It’s easier than ever to be able to deploy and manage microservices architecture with container and contain orchestration tool like Kubernetes. However, debugging multiple microservices across network boundaries is hard. Trying to what’s going on in a production microservices deployment at scale is impossible without proper tools! Google has spent over a decade deploying containerized Java applications at unprecedented scale. During that time Google has built an ecosystem of infrastructure and tools to manage, troubleshoot, and debug, at scale. These tools were originally used to diagnose problems with software running on Borg. With Google Cloud Platform we’re bringing some of the best kept internal technology to developers all over the world. In this session, we’ll deploy and scale Java microservices in Kubernetes, walk through common troubleshooting techniques with logs, shell access, isolating troubled containers, and also connecting to specific containers via port forwarding.

Beyond the basic tools, we’ll see how easy production diagnostic we can enjoy with Stackdriver and Google Cloud Platform, including capturing traces & logs without managing your own infrastructure, and live debugging and injecting log messages without needing to rewrite nor redeploy your code!

Ray Tsang

May 08, 2017

More Decks by Ray Tsang

Other Decks in Technology


  1. @saturnism @googlecloud Debugging & Troubleshooting Microservices w/ Kubernetes It runs

    in production, but it's not enough!
  2. @saturnism @googlecloud Ray Tsang Developer Advocate Google Cloud Platform @saturnism

    | +RayTsang
  3. @saturnism @googlecloud Ray Tsang Developer Architect Traveler Photographer flickr.com/saturnism

  4. @saturnism @googlecloud

  5. @saturnism @googlecloud #kubernetes #devoxx

  6. @saturnism @googlecloud

  7. @saturnism @googlecloud Hello World Service - Greet Guestbook Service -

    Create Guestbook Service - Retrieve
  8. @saturnism @googlecloud Guestbook UI Hello World Service Redis session replication

    greeting MySQL Guestbook Service CRUD
  9. @saturnism @googlecloud Visibility & Insight is Key

  10. @saturnism @googlecloud Let's see it!

  11. @saturnism @googlecloud Production vs Non-Production :(

  12. @saturnism @googlecloud Distributed Systems are hard!

  13. @saturnism @googlecloud We've been there, done that

  14. @saturnism @googlecloud Information - Logs! kubectl logs -f pod_id

  15. @saturnism @googlecloud Inspecting the process / container kubectl exec -ti

    pod_id /bin/bash
  16. @saturnism @googlecloud Port forward to container kubectl port-forward pod_id local_port:remote_port

  17. @saturnism @googlecloud Isolate problematic instance w/ Labels kubectl label pod

    pod_id --overewrite serving=false
  18. @saturnism @googlecloud Pod frontend Pod serving = true version =

    1.0 Pod serving = true version = 1.0 Service Label selectors: version = 1.0 serving = true Pod serving = true version = 1.0
  19. @saturnism @googlecloud Pod frontend Pod serving = true version =

    1.0 Pod serving = true version = 1.0 Service Label selectors: version = 1.0 serving = true Pod serving = false version = 1.0
  20. @saturnism @googlecloud Log aggregation, at scale

  21. @saturnism @googlecloud Stackdriver Logging https://cloud.google.com/logging/

  22. @saturnism @googlecloud

  23. @saturnism @googlecloud Metrics from Logs → Alerting Near-realtime insight

  24. @saturnism @googlecloud Search and Query Your Logs

  25. @saturnism @googlecloud Export to BigQuery, Storage, Pub/Sub Near-realtime insight

  26. @saturnism @googlecloud

  27. @saturnism @googlecloud

  28. @saturnism @googlecloud Tracing at Google - Dapper https://research.google.com/pubs/pub36356.html

  29. @saturnism @googlecloud Zipkin

  30. @saturnism @googlecloud Stackdriver Trace https://cloud.google.com/trace/

  31. @saturnism @googlecloud Zipkin → Stackdriver Trace https://cloud.google.com/trace/docs/zipkin

  32. @saturnism @googlecloud

  33. @saturnism @googlecloud

  34. @saturnism @googlecloud

  35. @saturnism @googlecloud

  36. @saturnism @googlecloud Automatic reports Detect performance regression

  37. @saturnism @googlecloud Oops… I forgot a log message. Darn!

  38. @saturnism @googlecloud Don't you wish you can use a debugger?

  39. @saturnism @googlecloud Stackdriver Debugger - Production Debugger https://cloud.google.com/debugger/

  40. @saturnism @googlecloud Add debugger agent java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE

  41. @saturnism @googlecloud Logpoint - dynamically add log messages! No redeployment,

    Magic. gcloud debug logpoints create HelloworldService.java:35 \ "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT
  42. @saturnism @googlecloud Snapshots Inspect call stack and variables

  43. @saturnism @googlecloud

  44. @saturnism @googlecloud

  45. @saturnism @googlecloud Trace Log Debug

  46. @saturnism @googlecloud Trace → Stackdriver Trace Logs → Stackdriver Logging

    Debug → Stackdriver Debugger https://cloud.google.com/stackdriver/
  47. @saturnism @googlecloud Thanks! Source: https://github.com/saturnism/spring-boot-docker Stackdriver: https://cloud.google.com/stackdriver/ Ray Tsang @saturnism

    Google Cloud Platform