Instrumenting CI Pipelines

2ad55a66b8325eb097612c069ab76231?s=47 Clay Smith
February 20, 2018

Instrumenting CI Pipelines

2ad55a66b8325eb097612c069ab76231?s=128

Clay Smith

February 20, 2018
Tweet

Transcript

  1. Instrumenting CI Pipelines @smithclay Testing in Production Meetup, San Francisco

    2/20/2018
  2. A BRIEF HISTORY OF OBSERVABILITY THOUGHT- LEADERSHIP CENTERED ON THE

    URBAN COAST OF THE WESTERN UNITED STATES intro. FOR CURIOUS DEVELOPMENT AND OPERATIONAL PROFESSIONALS
  3. THAT TWITTER POST REGARDING OBSERVABILITY (credit: @gphat, https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html fig. 1

  4. THE THREE PILLARS OF OBSERVABILITY Metrics Logs Traces (credit: @copyconstruct,

    https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38) fig. 3
  5. THE bourgonIAN VENN DIAGRAM (credit: @peterbourgon, https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html) fig. 2

  6. ? HOW SHOULD WE DEFINE 'OBSERVABILITY' FOR CI TOOLS? q.

    1
  7. AN EXPIERMENT IN INSTRUMENTING A CI PIPELINE IN THE CLOUD

  8. CI PIPELINE: TYPICAL MONITORING BATTERIES INCLUDED https://github.com/capitalone/Hygieia (as seen at

    AWS re:Invent) (diagram: @peterbourgon)
  9. CI PIPELINE: WHAT DOES TRACING LOOK LIKE? Waterfall/Gantt Chart 'CI

    run scoped (?)' (diagram: @peterbourgon)
  10. CASE STUDY: AWS CodeBuild (no servers required!) Converting steps to

    segments in a trace
  11. HOW TO INSTRUMENT THE PIPELINE? (I.E. NO OPERATING SYSTEM ACCESS)

    Log Events converted to a trace. Needs: timestamp and relationship to other events Optional: other useful attributes (i.e. container name) Traces are just related events! We can convert! (diagram: @peterbourgon)
  12. COLLECTING TRACE EVENTS THE REALLY LAZY WAY tail -f $temp_file

    | ./traceformer -app-name DevOpsTest &> output.log & Just read events in from STDIN and convert them to a tracing format (in this case New Relic APM) with some hacked together golang code.
  13. CI TRACES, VISUALIZED (APM STYLE) Latency breakdown of different build

    steps. Metrics created from traces.
  14. CONNECTING CI TRACES TO PROD PERFORMANCE Attributes add additional context

    to traces. ... this can then be correlated with production performance.
  15. LESSONs LEARNED INSTRUMENTING THE CI PIPELINE 1) If you have

    a complex CI pipeline, consider traces. 2) We should be connecting production performance to the processes (i.e. pipeline runs) that deployed and tested them. (diagram: @peterbourgon)
  16. THANK YOU. @smithclay—New Relic Testing in Production Meetup, San Francisco

    2/20/2018