$30 off During Our Annual Pro Sale. View Details »

CPDD

JBD
October 02, 2018

 CPDD

JBD

October 02, 2018
Tweet

More Decks by JBD

Other Decks in Technology

Transcript

  1. View Slide

  2. View Slide

  3. View Slide

  4. systems?
    who does that?

    View Slide

  5. jaana b. dogan
    6+ years at Google, touched many projects

    View Slide

  6. early days
    (of a company)

    View Slide

  7. View Slide

  8. growing...

    View Slide

  9. what growth looks like
    service A
    service B
    service C
    service D
    service E
    service auth
    email

    View Slide

  10. one becomes many
    failure in isolation
    who to ping in failure?

    View Slide

  11. and it goes larger...

    View Slide

  12. good guy jeff

    View Slide

  13. View Slide

  14. View Slide

  15. code search

    View Slide

  16. go_library(
    name = "logs",
    srcs = ["logs.go"],
    visibility = ["//visibility:public"],
    deps = [
    ….
    ],
    )
    References (641 occurrences)
    - //source/ads/monitoring/BUILD
    - //source/ads/analysis/BUILD
    - //source/ads/mobile/BUILD
    ...

    View Slide

  17. frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer

    View Slide

  18. frontend server
    authentication
    users images
    memcache
    blobservice
    memcache
    memcache
    (metadata) (disks)
    load balancer
    critical path

    View Slide

  19. cpdd
    (critical path driven development)

    View Slide

  20. discover the critical paths
    make them reliable and fast
    make them debuggable

    View Slide

  21. how do we get there?
    events or tracing

    View Slide

  22. why? why?
    why? why? why?

    View Slide

  23. GET /timeline
    edge-lb
    sched
    api-server
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get

    View Slide

  24. bare metal
    kernel
    process scheduler
    network stack
    cloud stack
    user process
    frameworks
    your code

    View Slide

  25. GET /timeline
    edge-lb
    sched
    api-server
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    not my fault

    View Slide

  26. GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    where is the source code?

    View Slide

  27. GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    who to call?

    View Slide

  28. GET /timeline
    auth.Auth
    cache.Get
    mysql.Query
    user.Profile
    cache.Get
    mysql.Query
    images.Filter
    blobstore.Get
    cache.Get
    mysql.Query
    blob.Get
    give me the logs, runtime
    events, profiles...

    View Slide

  29. challenges...

    View Slide

  30. this is an organizational
    problem
    CPDD CHALLENGE #1:

    View Slide

  31. github.com/w3c/distributed-tracing

    View Slide

  32. engineers don’t know
    where to start
    CPDD CHALLENGE #2:

    View Slide

  33. infra is still
    a blackbox
    CPDD CHALLENGE #3:

    View Slide

  34. instrumentation is
    expensive
    CPDD CHALLENGE #4:

    View Slide

  35. dynamic capabilities are
    underestimated
    CPDD CHALLENGE #5:

    View Slide

  36. View Slide

  37. cpdd: a tool to close
    knowledge gaps
    (which we don’t talk about)

    View Slide

  38. View Slide