CPDD

Transcript

1. View Slide

2. View Slide

3. View Slide

4. systems?
   who does that?
   

   View Slide

5. jaana b. dogan
   6+ years at Google, touched many projects
   

   View Slide

6. early days
   (of a company)
   

   View Slide

7. View Slide

8. growing...
   

   View Slide

9. what growth looks like
   service A
   service B
   service C
   service D
   service E
   service auth
   email
   

   View Slide

10. one becomes many
    failure in isolation
    who to ping in failure?
    

    View Slide

11. and it goes larger...
    

    View Slide

12. good guy jeff
    

    View Slide

13. View Slide

14. View Slide

15. code search
    

    View Slide

16. go_library(
    name = "logs",
    srcs = ["logs.go"],
    visibility = ["//visibility:public"],
    deps = [
    ….
    ],
    )
    References (641 occurrences)
    - //source/ads/monitoring/BUILD
    - //source/ads/analysis/BUILD
    - //source/ads/mobile/BUILD
    ...
    

    View Slide

17. frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    

    View Slide

18. frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    critical path
    

    View Slide

19. cpdd
    (critical path driven development)
    

    View Slide

20. discover the critical paths
    make them reliable and fast
    make them debuggable
    

    View Slide

21. how do we get there?
    events or tracing
    

    View Slide

22. why? why?
    why? why? why?
    

    View Slide

23. GET /timeline
    edge-lb
    sched
    api-server
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    

    View Slide

24. bare metal
    kernel
    process scheduler
    network stack
    cloud stack
    user process
    frameworks
    your code
    

    View Slide

25. GET /timeline
    edge-lb
    sched
    api-server
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    not my fault
    

    View Slide

26. GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    where is the source code?
    

    View Slide

27. GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    who to call?
    

    View Slide

28. GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    give me the logs, runtime
    events, profiles...
    

    View Slide

29. challenges...
    

    View Slide

30. this is an organizational
    problem
    CPDD CHALLENGE #1:
    

    View Slide

31. github.com/w3c/distributed-tracing
    

    View Slide

32. engineers don’t know
    where to start
    CPDD CHALLENGE #2:
    

    View Slide

33. infra is still
    a blackbox
    CPDD CHALLENGE #3:
    

    View Slide

34. instrumentation is
    expensive
    CPDD CHALLENGE #4:
    

    View Slide

35. dynamic capabilities are
    underestimated
    CPDD CHALLENGE #5:
    

    View Slide

36. View Slide

37. cpdd: a tool to close
    knowledge gaps
    (which we don’t talk about)
    

    View Slide

38. fin
    jbd@google.com
    

    View Slide