Critical Path Analysis

Transcript

1. @rakyll
2. @rakyll
3. @rakyll We use facilities that are not reserved for our flight but are essential in making it happen.
4. @rakyll We use facilities that are not reserved for our flight but are essential in making it happen.
5. @rakyll
6. @rakyll Systems? who understands them?
7. @rakyll Jaana B. Dogan jbd@google.com
8. @rakyll
9. @rakyll Growing...
10. @rakyll Tech stack gets fragmented, monoliths are gone...
11. @rakyll one becomes many failure in isolation escalation in failure?
12. @rakyll and it may get larger...
13. @rakyll good guy jeff
14. @rakyll
15. @rakyll
16. @rakyll code search
17. @rakyll go_library( name = "logs", srcs = ["logs.go"], visibility = ["//visibility:public"], deps = [ …. ], ) References (641 occurrences) - //source/ads/monitoring/BUILD - //source/ads/analysis/BUILD - //source/ads/mobile/BUILD ...
18. @rakyll frontend server authentication users images memcache blobservice memcache memcache (metadata) (disks) load balancer
19. @rakyll frontend server authentication users images memcache blobservice memcache memcache (metadata) (disks) load balancer critical path
20. @rakyll frontend server authentication users images memcache blobservice memcache memcache (metadata) (disks) load balancer critical path
21. @rakyll frontend server authentication users images memcache blobservice memcache memcache (metadata) (disks) load balancer critical path
22. @rakyll CPA (critical path analysis)
23. @rakyll discover the critical paths make them reliable and fast make them debuggable
24. @rakyll How do we get there? (tracing, events, logs)
25. @rakyll Why? Why? Why? Why? Why?
26. @rakyll GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get
27. @rakyll bare metal kernel network stack cloud stack libraries frameworks your code
28. @rakyll GET /timeline edge-lb sched api-server auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get not my fault
29. @rakyll GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get cache.Get mysql.Query blob.Get where is the source code?
30. @rakyll GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get cache.Get mysql.Query blob.Get who to call?
31. @rakyll GET /timeline auth.Auth cache.Get mysql.Query user.Profile cache.Get mysql.Query images.Filter blobstore.Get cache.Get mysql.Query blob.Get give me the logs, runtime events, profiles...
32. @rakyll Challenges...
33. @rakyll This is an organizational problem. CHALLENGE #1:
34. @rakyll Engineers don’t know where to start. CHALLENGE #2:
35. @rakyll Infra is still a blackbox. CHALLENGE #3:
36. @rakyll Instrumentation is expensive. CHALLENGE #4:
37. @rakyll Dynamic capabilities are underestimated. CHALLENGE #5:
38. @rakyll
39. @rakyll A tool to close knowledge gaps (which we don’t talk about)
40. @rakyll Fin jbd@google.com