Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Instrumenting CI Pipelines

Clay Smith
February 20, 2018

Instrumenting CI Pipelines

Clay Smith

February 20, 2018
Tweet

More Decks by Clay Smith

Other Decks in Programming

Transcript

  1. Instrumenting
    CI Pipelines
    @smithclay
    Testing in Production Meetup, San Francisco
    2/20/2018

    View Slide

  2. A BRIEF HISTORY
    OF OBSERVABILITY THOUGHT-
    LEADERSHIP CENTERED ON THE URBAN
    COAST OF THE WESTERN UNITED STATES
    intro.
    FOR
    CURIOUS DEVELOPMENT AND
    OPERATIONAL PROFESSIONALS

    View Slide

  3. THAT TWITTER POST
    REGARDING OBSERVABILITY
    (credit: @gphat, https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html
    fig. 1

    View Slide

  4. THE THREE PILLARS
    OF OBSERVABILITY
    Metrics Logs Traces
    (credit: @copyconstruct, https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38)
    fig. 3

    View Slide

  5. THE bourgonIAN
    VENN DIAGRAM
    (credit: @peterbourgon, https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html)
    fig. 2

    View Slide

  6. ?
    HOW SHOULD WE DEFINE
    'OBSERVABILITY' FOR CI
    TOOLS?
    q. 1

    View Slide

  7. AN EXPIERMENT
    IN INSTRUMENTING A CI
    PIPELINE IN THE CLOUD

    View Slide

  8. CI PIPELINE:
    TYPICAL MONITORING BATTERIES
    INCLUDED
    https://github.com/capitalone/Hygieia
    (as seen at AWS re:Invent)
    (diagram: @peterbourgon)

    View Slide

  9. CI PIPELINE:
    WHAT DOES TRACING LOOK LIKE?
    Waterfall/Gantt Chart
    'CI run scoped (?)'
    (diagram: @peterbourgon)

    View Slide

  10. CASE STUDY: AWS CodeBuild
    (no servers required!)
    Converting steps to segments in a trace

    View Slide

  11. HOW TO INSTRUMENT THE PIPELINE?
    (I.E. NO OPERATING SYSTEM ACCESS)
    Log Events converted to a trace.
    Needs: timestamp and relationship to other events
    Optional: other useful attributes (i.e. container name)
    Traces are just related events! We can convert!
    (diagram: @peterbourgon)

    View Slide

  12. COLLECTING TRACE EVENTS
    THE REALLY LAZY WAY
    tail -f $temp_file | ./traceformer -app-name DevOpsTest &> output.log &
    Just read events in from STDIN and convert them to a tracing format (in this case
    New Relic APM) with some hacked together golang code.

    View Slide

  13. CI TRACES, VISUALIZED
    (APM STYLE)
    Latency breakdown of different build steps.
    Metrics created from traces.

    View Slide

  14. CONNECTING CI TRACES
    TO PROD PERFORMANCE
    Attributes add additional context to traces.
    ... this can then be correlated with production performance.

    View Slide

  15. LESSONs LEARNED
    INSTRUMENTING THE CI PIPELINE
    1) If you have a complex CI pipeline, consider traces.
    2) We should be connecting
    production performance to the processes (i.e.
    pipeline runs) that deployed and tested them.
    (diagram: @peterbourgon)

    View Slide

  16. THANK YOU.
    @smithclay—New Relic
    Testing in Production Meetup, San Francisco
    2/20/2018

    View Slide