Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Debugging & Troubleshooting Microservices in Kubernetes & Stackdriver

Debugging & Troubleshooting Microservices in Kubernetes & Stackdriver

It’s easier than ever to be able to deploy and manage microservices architecture with container and contain orchestration tool like Kubernetes. However, debugging multiple microservices across network boundaries is hard. Trying to what’s going on in a production microservices deployment at scale is impossible without proper tools! Google has spent over a decade deploying containerized Java applications at unprecedented scale. During that time Google has built an ecosystem of infrastructure and tools to manage, troubleshoot, and debug, at scale. These tools were originally used to diagnose problems with software running on Borg. With Google Cloud Platform we’re bringing some of the best kept internal technology to developers all over the world. In this session, we’ll deploy and scale Java microservices in Kubernetes, walk through common troubleshooting techniques with logs, shell access, isolating troubled containers, and also connecting to specific containers via port forwarding.

Beyond the basic tools, we’ll see how easy production diagnostic we can enjoy with Stackdriver and Google Cloud Platform, including capturing traces & logs without managing your own infrastructure, and live debugging and injecting log messages without needing to rewrite nor redeploy your code!

Ray Tsang

May 08, 2017
Tweet

More Decks by Ray Tsang

Other Decks in Technology

Transcript

  1. @saturnism @googlecloud
    Debugging & Troubleshooting Microservices
    w/ Kubernetes
    It runs in production, but it's not enough!

    View full-size slide

  2. @saturnism @googlecloud
    Ray Tsang
    Developer Advocate
    Google Cloud Platform
    @saturnism | +RayTsang

    View full-size slide

  3. @saturnism @googlecloud
    Ray Tsang
    Developer
    Architect
    Traveler
    Photographer
    flickr.com/saturnism

    View full-size slide

  4. @saturnism @googlecloud

    View full-size slide

  5. @saturnism @googlecloud #kubernetes #devoxx

    View full-size slide

  6. @saturnism @googlecloud

    View full-size slide

  7. @saturnism @googlecloud
    Hello World
    Service - Greet
    Guestbook
    Service - Create
    Guestbook Service -
    Retrieve

    View full-size slide

  8. @saturnism @googlecloud
    Guestbook
    UI
    Hello World
    Service
    Redis
    session replication
    greeting
    MySQL
    Guestbook
    Service
    CRUD

    View full-size slide

  9. @saturnism @googlecloud
    Visibility & Insight is Key

    View full-size slide

  10. @saturnism @googlecloud
    Let's see it!

    View full-size slide

  11. @saturnism @googlecloud
    Production vs Non-Production :(

    View full-size slide

  12. @saturnism @googlecloud
    Distributed Systems are hard!

    View full-size slide

  13. @saturnism @googlecloud
    We've been there, done that

    View full-size slide

  14. @saturnism @googlecloud
    Information - Logs!
    kubectl logs -f pod_id

    View full-size slide

  15. @saturnism @googlecloud
    Inspecting the process / container
    kubectl exec -ti pod_id /bin/bash

    View full-size slide

  16. @saturnism @googlecloud
    Port forward to container
    kubectl port-forward pod_id local_port:remote_port

    View full-size slide

  17. @saturnism @googlecloud
    Isolate problematic instance w/ Labels
    kubectl label pod pod_id --overewrite serving=false

    View full-size slide

  18. @saturnism @googlecloud
    Pod
    frontend
    Pod
    serving = true
    version = 1.0
    Pod
    serving = true
    version = 1.0
    Service
    Label selectors:
    version = 1.0
    serving = true
    Pod
    serving = true
    version = 1.0

    View full-size slide

  19. @saturnism @googlecloud
    Pod
    frontend
    Pod
    serving = true
    version = 1.0
    Pod
    serving = true
    version = 1.0
    Service
    Label selectors:
    version = 1.0
    serving = true
    Pod
    serving = false
    version = 1.0

    View full-size slide

  20. @saturnism @googlecloud
    Log aggregation, at scale

    View full-size slide

  21. @saturnism @googlecloud
    Stackdriver Logging
    https://cloud.google.com/logging/

    View full-size slide

  22. @saturnism @googlecloud

    View full-size slide

  23. @saturnism @googlecloud
    Metrics from Logs → Alerting
    Near-realtime insight

    View full-size slide

  24. @saturnism @googlecloud
    Search and Query Your Logs

    View full-size slide

  25. @saturnism @googlecloud
    Export to BigQuery, Storage, Pub/Sub
    Near-realtime insight

    View full-size slide

  26. @saturnism @googlecloud

    View full-size slide

  27. @saturnism @googlecloud

    View full-size slide

  28. @saturnism @googlecloud
    Tracing at Google - Dapper
    https://research.google.com/pubs/pub36356.html

    View full-size slide

  29. @saturnism @googlecloud
    Zipkin

    View full-size slide

  30. @saturnism @googlecloud
    Stackdriver Trace
    https://cloud.google.com/trace/

    View full-size slide

  31. @saturnism @googlecloud
    Zipkin → Stackdriver Trace
    https://cloud.google.com/trace/docs/zipkin

    View full-size slide

  32. @saturnism @googlecloud

    View full-size slide

  33. @saturnism @googlecloud

    View full-size slide

  34. @saturnism @googlecloud

    View full-size slide

  35. @saturnism @googlecloud

    View full-size slide

  36. @saturnism @googlecloud
    Automatic reports
    Detect performance regression

    View full-size slide

  37. @saturnism @googlecloud
    Oops… I forgot a log message. Darn!

    View full-size slide

  38. @saturnism @googlecloud
    Don't you wish you can use a debugger?

    View full-size slide

  39. @saturnism @googlecloud
    Stackdriver Debugger - Production Debugger
    https://cloud.google.com/debugger/

    View full-size slide

  40. @saturnism @googlecloud
    Add debugger agent
    java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE

    View full-size slide

  41. @saturnism @googlecloud
    Logpoint - dynamically add log messages!
    No redeployment, Magic.
    gcloud debug logpoints create HelloworldService.java:35 \
    "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT

    View full-size slide

  42. @saturnism @googlecloud
    Snapshots
    Inspect call stack and variables

    View full-size slide

  43. @saturnism @googlecloud

    View full-size slide

  44. @saturnism @googlecloud

    View full-size slide

  45. @saturnism @googlecloud
    Trace
    Log
    Debug

    View full-size slide

  46. @saturnism @googlecloud
    Trace → Stackdriver Trace
    Logs → Stackdriver Logging
    Debug → Stackdriver Debugger
    https://cloud.google.com/stackdriver/

    View full-size slide

  47. @saturnism @googlecloud
    Thanks!
    Source: https://github.com/saturnism/spring-boot-docker
    Stackdriver: https://cloud.google.com/stackdriver/
    Ray Tsang
    @saturnism
    Google Cloud Platform

    View full-size slide