Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Critical Path Analysis

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
February 06, 2019
110

Critical Path Analysis

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

February 06, 2019
Tweet

Transcript

  1. @rakyll

  2. @rakyll

  3. @rakyll We use facilities that are not reserved for our

    flight but are essential in making it happen.
  4. @rakyll We use facilities that are not reserved for our

    flight but are essential in making it happen.
  5. @rakyll

  6. @rakyll Systems? who understands them?

  7. @rakyll Jaana B. Dogan jbd@google.com

  8. @rakyll

  9. @rakyll Growing...

  10. @rakyll Tech stack gets fragmented, monoliths are gone...

  11. @rakyll one becomes many failure in isolation escalation in failure?

  12. @rakyll and it may get larger...

  13. @rakyll good guy jeff

  14. @rakyll

  15. @rakyll

  16. @rakyll code search

  17. @rakyll go_library( name = "logs", srcs = ["logs.go"], visibility =

    ["//visibility:public"], deps = [ …. ], ) References (641 occurrences) - //source/ads/monitoring/BUILD - //source/ads/analysis/BUILD - //source/ads/mobile/BUILD ...
  18. @rakyll frontend server authentication users images memcache blobservice memcache memcache

    (metadata) (disks) load balancer
  19. @rakyll frontend server authentication users images memcache blobservice memcache memcache

    (metadata) (disks) load balancer critical path
  20. @rakyll frontend server authentication users images memcache blobservice memcache memcache

    (metadata) (disks) load balancer critical path
  21. @rakyll frontend server authentication users images memcache blobservice memcache memcache

    (metadata) (disks) load balancer critical path
  22. @rakyll CPA (critical path analysis)

  23. @rakyll discover the critical paths make them reliable and fast

    make them debuggable
  24. @rakyll How do we get there? (tracing, events, logs)

  25. @rakyll Why? Why? Why? Why? Why?

  26. @rakyll GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile

    cache.Get mysql.Query images.Filter blobstore.Get
  27. @rakyll bare metal kernel network stack cloud stack libraries frameworks

    your code
  28. @rakyll GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile

    cache.Get mysql.Query images.Filter blobstore.Get not my fault
  29. @rakyll GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter

    blobstore.Get cache.Get mysql.Query blob.Get where is the source code?
  30. @rakyll GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter

    blobstore.Get cache.Get mysql.Query blob.Get who to call?
  31. @rakyll GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter

    blobstore.Get cache.Get mysql.Query blob.Get give me the logs, runtime events, profiles...
  32. @rakyll Challenges...

  33. @rakyll This is an organizational problem. CHALLENGE #1:

  34. @rakyll Engineers don’t know where to start. CHALLENGE #2:

  35. @rakyll Infra is still a blackbox. CHALLENGE #3:

  36. @rakyll Instrumentation is expensive. CHALLENGE #4:

  37. @rakyll Dynamic capabilities are underestimated. CHALLENGE #5:

  38. @rakyll

  39. @rakyll A tool to close knowledge gaps (which we don’t

    talk about)
  40. @rakyll Fin jbd@google.com