CPDD

4. systems? who does that?
5. jaana b. dogan 6+ years at Google, touched many projects
6. early days (of a company)
8. growing...
9. what growth looks like service A service B service C service D service E service auth email
10. one becomes many failure in isolation who to ping in failure?
11. and it goes larger...
12. good guy jeff
15. code search
16. go_library( name = "logs", srcs = ["logs.go"], visibility = ["//visibility:public"], deps = [ …. ], ) References (641 occurrences) - //source/ads/monitoring/BUILD - //source/ads/analysis/BUILD - //source/ads/mobile/BUILD ...
17. frontend server authentication users images memcache blobservice memcache memcache (metadata) (disks) load balancer
18. frontend server authentication users images memcache blobservice memcache memcache (metadata) (disks) load balancer critical path
19. cpdd (critical path driven development)
20. discover the critical paths make them reliable and fast make them debuggable
21. how do we get there? events or tracing
22. why? why? why? why? why?
23. GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get
24. bare metal kernel process scheduler network stack cloud stack user process frameworks your code
25. GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get not my fault
26. GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get cache.Get mysql.Query blob.Get where is the source code?
27. GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get cache.Get mysql.Query blob.Get who to call?
28. GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get cache.Get mysql.Query blob.Get give me the logs, runtime events, profiles...
29. challenges...
30. this is an organizational problem CPDD CHALLENGE #1:
31. github.com/w3c/distributed-tracing
32. engineers don’t know where to start CPDD CHALLENGE #2:
33. infra is still a blackbox CPDD CHALLENGE #3:
34. instrumentation is expensive CPDD CHALLENGE #4:
35. dynamic capabilities are underestimated CPDD CHALLENGE #5:
37. cpdd: a tool to close knowledge gaps (which we don’t talk about)
38. fin jbd@google.com