Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CPDD

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
October 02, 2018

 CPDD

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

October 02, 2018
Tweet

Transcript

  1. None
  2. None
  3. None
  4. systems? who does that?

  5. jaana b. dogan 6+ years at Google, touched many projects

  6. early days (of a company)

  7. None
  8. growing...

  9. what growth looks like service A service B service C

    service D service E service auth email
  10. one becomes many failure in isolation who to ping in

    failure?
  11. and it goes larger...

  12. good guy jeff

  13. None
  14. None
  15. code search

  16. go_library( name = "logs", srcs = ["logs.go"], visibility = ["//visibility:public"],

    deps = [ …. ], ) References (641 occurrences) - //source/ads/monitoring/BUILD - //source/ads/analysis/BUILD - //source/ads/mobile/BUILD ...
  17. frontend server authentication users images memcache blobservice memcache memcache (metadata)

    (disks) load balancer
  18. frontend server authentication users images memcache blobservice memcache memcache (metadata)

    (disks) load balancer critical path
  19. cpdd (critical path driven development)

  20. discover the critical paths make them reliable and fast make

    them debuggable
  21. how do we get there? events or tracing

  22. why? why? why? why? why?

  23. GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile cache.Get

    mysql.Query images.Filter blobstore.Get
  24. bare metal kernel process scheduler network stack cloud stack user

    process frameworks your code
  25. GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile cache.Get

    mysql.Query images.Filter blobstore.Get not my fault
  26. GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get

    cache.Get mysql.Query blob.Get where is the source code?
  27. GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get

    cache.Get mysql.Query blob.Get who to call?
  28. GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get

    cache.Get mysql.Query blob.Get give me the logs, runtime events, profiles...
  29. challenges...

  30. this is an organizational problem CPDD CHALLENGE #1:

  31. github.com/w3c/distributed-tracing

  32. engineers don’t know where to start CPDD CHALLENGE #2:

  33. infra is still a blackbox CPDD CHALLENGE #3:

  34. instrumentation is expensive CPDD CHALLENGE #4:

  35. dynamic capabilities are underestimated CPDD CHALLENGE #5:

  36. None
  37. cpdd: a tool to close knowledge gaps (which we don’t

    talk about)
  38. fin jbd@google.com