Tweet
Tweet

Slide 1

Slide 1 text

@saturnism @googlecloud Debugging & Troubleshooting Microservices w/ Kubernetes It runs in production, but it's not enough!

Slide 2

Slide 2 text

@saturnism @googlecloud Ray Tsang Developer Advocate Google Cloud Platform @saturnism | +RayTsang

Slide 3

Slide 3 text

@saturnism @googlecloud Ray Tsang Developer Architect Traveler Photographer flickr.com/saturnism

Slide 4

Slide 4 text

@saturnism @googlecloud

Slide 5

Slide 5 text

@saturnism @googlecloud #kubernetes #devoxx

Slide 6

Slide 6 text

@saturnism @googlecloud

Slide 7

Slide 7 text

@saturnism @googlecloud Hello World Service - Greet Guestbook Service - Create Guestbook Service - Retrieve

Slide 8

Slide 8 text

@saturnism @googlecloud Guestbook UI Hello World Service Redis session replication greeting MySQL Guestbook Service CRUD

Slide 9

Slide 9 text

@saturnism @googlecloud Visibility & Insight is Key

Slide 10

Slide 10 text

@saturnism @googlecloud Let's see it!

Slide 11

Slide 11 text

@saturnism @googlecloud Production vs Non-Production :(

Slide 12

Slide 12 text

@saturnism @googlecloud Distributed Systems are hard!

Slide 13

Slide 13 text

@saturnism @googlecloud We've been there, done that

Slide 14

Slide 14 text

@saturnism @googlecloud Information - Logs! kubectl logs -f pod_id

Slide 15

Slide 15 text

@saturnism @googlecloud Inspecting the process / container kubectl exec -ti pod_id /bin/bash

Slide 16

Slide 16 text

@saturnism @googlecloud Port forward to container kubectl port-forward pod_id local_port:remote_port

Slide 17

Slide 17 text

@saturnism @googlecloud Isolate problematic instance w/ Labels kubectl label pod pod_id --overewrite serving=false

Slide 18

Slide 18 text

@saturnism @googlecloud Pod frontend Pod serving = true version = 1.0 Pod serving = true version = 1.0 Service Label selectors: version = 1.0 serving = true Pod serving = true version = 1.0

Slide 19

Slide 19 text

@saturnism @googlecloud Pod frontend Pod serving = true version = 1.0 Pod serving = true version = 1.0 Service Label selectors: version = 1.0 serving = true Pod serving = false version = 1.0

Slide 20

Slide 20 text

@saturnism @googlecloud Log aggregation, at scale

Slide 21

Slide 21 text

@saturnism @googlecloud Stackdriver Logging https://cloud.google.com/logging/

Slide 22

Slide 22 text

@saturnism @googlecloud

Slide 23

Slide 23 text

@saturnism @googlecloud Metrics from Logs → Alerting Near-realtime insight

Slide 24

Slide 24 text

@saturnism @googlecloud Search and Query Your Logs

Slide 25

Slide 25 text

@saturnism @googlecloud Export to BigQuery, Storage, Pub/Sub Near-realtime insight

Slide 26

Slide 26 text

@saturnism @googlecloud

Slide 27

Slide 27 text

@saturnism @googlecloud

Slide 28

Slide 28 text

@saturnism @googlecloud Tracing at Google - Dapper https://research.google.com/pubs/pub36356.html

Slide 29

Slide 29 text

@saturnism @googlecloud Zipkin

Slide 30

Slide 30 text

@saturnism @googlecloud Stackdriver Trace https://cloud.google.com/trace/

Slide 31

Slide 31 text

@saturnism @googlecloud Zipkin → Stackdriver Trace https://cloud.google.com/trace/docs/zipkin

Slide 32

Slide 32 text

@saturnism @googlecloud

Slide 33

Slide 33 text

@saturnism @googlecloud

Slide 34

Slide 34 text

@saturnism @googlecloud

Slide 35

Slide 35 text

@saturnism @googlecloud

Slide 36

Slide 36 text

@saturnism @googlecloud Automatic reports Detect performance regression

Slide 37

Slide 37 text

@saturnism @googlecloud Oops… I forgot a log message. Darn!

Slide 38

Slide 38 text

@saturnism @googlecloud Don't you wish you can use a debugger?

Slide 39

Slide 39 text

@saturnism @googlecloud Stackdriver Debugger - Production Debugger https://cloud.google.com/debugger/

Slide 40

Slide 40 text

@saturnism @googlecloud Add debugger agent java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE

Slide 41

Slide 41 text

@saturnism @googlecloud Logpoint - dynamically add log messages! No redeployment, Magic. gcloud debug logpoints create HelloworldService.java:35 \ "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT

Slide 42

Slide 42 text

@saturnism @googlecloud Snapshots Inspect call stack and variables

Slide 43

Slide 43 text

@saturnism @googlecloud

Slide 44

Slide 44 text

@saturnism @googlecloud

Slide 45

Slide 45 text

@saturnism @googlecloud Trace Log Debug

Slide 46

Slide 46 text

@saturnism @googlecloud Trace → Stackdriver Trace Logs → Stackdriver Logging Debug → Stackdriver Debugger https://cloud.google.com/stackdriver/

Slide 47

Slide 47 text

@saturnism @googlecloud Thanks! Source: https://github.com/saturnism/spring-boot-docker Stackdriver: https://cloud.google.com/stackdriver/ Ray Tsang @saturnism Google Cloud Platform