Critical Path Analysis

Transcript

1. @rakyll
   

   View Slide

2. @rakyll
   

   View Slide

3. @rakyll
   We use facilities that are not
   reserved for our flight but are
   essential in making it happen.
   

   View Slide

4. @rakyll
   We use facilities that are not
   reserved for our flight but are
   essential in making it happen.
   

   View Slide

5. @rakyll
   

   View Slide

6. @rakyll
   Systems?
   who understands them?
   

   View Slide

7. @rakyll
   Jaana B. Dogan
   jbd@google.com
   

   View Slide

8. @rakyll
   

   View Slide

9. @rakyll
   Growing...
   

   View Slide

10. @rakyll
    Tech stack gets fragmented,
    monoliths are gone...
    

    View Slide

11. @rakyll
    one becomes many
    failure in isolation
    escalation in failure?
    

    View Slide

12. @rakyll
    and it may get larger...
    

    View Slide

13. @rakyll
    good guy jeff
    

    View Slide

14. @rakyll
    

    View Slide

15. @rakyll
    

    View Slide

16. @rakyll
    code search
    

    View Slide

17. @rakyll
    go_library(
    name = "logs",
    srcs = ["logs.go"],
    visibility = ["//visibility:public"],
    deps = [
    ….
    ],
    )
    References (641 occurrences)
    - //source/ads/monitoring/BUILD
    - //source/ads/analysis/BUILD
    - //source/ads/mobile/BUILD
    ...
    

    View Slide

18. @rakyll
    frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    

    View Slide

19. @rakyll
    frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    critical path
    

    View Slide

20. @rakyll
    frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    critical path
    

    View Slide

21. @rakyll
    frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    critical path
    

    View Slide

22. @rakyll
    CPA
    (critical path analysis)
    

    View Slide

23. @rakyll
    discover the critical paths
    make them reliable and fast
    make them debuggable
    

    View Slide

24. @rakyll
    How do we get there?
    (tracing, events, logs)
    

    View Slide

25. @rakyll
    Why? Why?
    Why? Why? Why?
    

    View Slide

26. @rakyll
    GET /timeline
    edge-lb
    sched
    api-server
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    

    View Slide

27. @rakyll
    bare metal
    kernel
    network stack
    cloud stack
    libraries
    frameworks
    your code
    

    View Slide

28. @rakyll
    GET /timeline
    edge-lb
    sched
    api-server
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    not my fault
    

    View Slide

29. @rakyll
    GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    where is the source code?
    

    View Slide

30. @rakyll
    GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    who to call?
    

    View Slide

31. @rakyll
    GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    give me the logs, runtime
    events, profiles...
    

    View Slide

32. @rakyll
    Challenges...
    

    View Slide

33. @rakyll
    This is an organizational
    problem.
    CHALLENGE #1:
    

    View Slide

34. @rakyll
    Engineers don’t know
    where to start.
    CHALLENGE #2:
    

    View Slide

35. @rakyll
    Infra is still
    a blackbox.
    CHALLENGE #3:
    

    View Slide

36. @rakyll
    Instrumentation is
    expensive.
    CHALLENGE #4:
    

    View Slide

37. @rakyll
    Dynamic capabilities are
    underestimated.
    CHALLENGE #5:
    

    View Slide

38. @rakyll
    

    View Slide

39. @rakyll
    A tool to close
    knowledge gaps
    (which we don’t talk about)
    

    View Slide

40. @rakyll
    Fin
    jbd@google.com
    

    View Slide