Instrumenting CI Pipelines

Transcript

1. Instrumenting
   CI Pipelines
   @smithclay
   Testing in Production Meetup, San Francisco
   2/20/2018
   

   View Slide

2. A BRIEF HISTORY
   OF OBSERVABILITY THOUGHT-
   LEADERSHIP CENTERED ON THE URBAN
   COAST OF THE WESTERN UNITED STATES
   intro.
   FOR
   CURIOUS DEVELOPMENT AND
   OPERATIONAL PROFESSIONALS
   

   View Slide

3. THAT TWITTER POST
   REGARDING OBSERVABILITY
   (credit: @gphat, https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html
   fig. 1
   

   View Slide

4. THE THREE PILLARS
   OF OBSERVABILITY
   Metrics Logs Traces
   (credit: @copyconstruct, https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38)
   fig. 3
   

   View Slide

5. THE bourgonIAN
   VENN DIAGRAM
   (credit: @peterbourgon, https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html)
   fig. 2
   

   View Slide

6. ?
   HOW SHOULD WE DEFINE
   'OBSERVABILITY' FOR CI
   TOOLS?
   q. 1
   

   View Slide

7. AN EXPIERMENT
   IN INSTRUMENTING A CI
   PIPELINE IN THE CLOUD
   

   View Slide

8. CI PIPELINE:
   TYPICAL MONITORING BATTERIES
   INCLUDED
   https://github.com/capitalone/Hygieia
   (as seen at AWS re:Invent)
   (diagram: @peterbourgon)
   

   View Slide

9. CI PIPELINE:
   WHAT DOES TRACING LOOK LIKE?
   Waterfall/Gantt Chart
   'CI run scoped (?)'
   (diagram: @peterbourgon)
   

   View Slide

10. CASE STUDY: AWS CodeBuild
    (no servers required!)
    Converting steps to segments in a trace
    

    View Slide

11. HOW TO INSTRUMENT THE PIPELINE?
    (I.E. NO OPERATING SYSTEM ACCESS)
    Log Events converted to a trace.
    Needs: timestamp and relationship to other events
    Optional: other useful attributes (i.e. container name)
    Traces are just related events! We can convert!
    (diagram: @peterbourgon)
    

    View Slide

12. COLLECTING TRACE EVENTS
    THE REALLY LAZY WAY
    tail -f $temp_file | ./traceformer -app-name DevOpsTest &> output.log &
    Just read events in from STDIN and convert them to a tracing format (in this case
    New Relic APM) with some hacked together golang code.
    

    View Slide

13. CI TRACES, VISUALIZED
    (APM STYLE)
    Latency breakdown of different build steps.
    Metrics created from traces.
    

    View Slide

14. CONNECTING CI TRACES
    TO PROD PERFORMANCE
    Attributes add additional context to traces.
    ... this can then be correlated with production performance.
    

    View Slide

15. LESSONs LEARNED
    INSTRUMENTING THE CI PIPELINE
    1) If you have a complex CI pipeline, consider traces.
    2) We should be connecting
    production performance to the processes (i.e.
    pipeline runs) that deployed and tested them.
    (diagram: @peterbourgon)
    

    View Slide

16. THANK YOU.
    @smithclay—New Relic
    Testing in Production Meetup, San Francisco
    2/20/2018
    

    View Slide