Tweet
Tweet

Slide 1

Slide 1 text

Instrumenting CI Pipelines @smithclay Testing in Production Meetup, San Francisco 2/20/2018

Slide 2

Slide 2 text

A BRIEF HISTORY OF OBSERVABILITY THOUGHT- LEADERSHIP CENTERED ON THE URBAN COAST OF THE WESTERN UNITED STATES intro. FOR CURIOUS DEVELOPMENT AND OPERATIONAL PROFESSIONALS

Slide 3

Slide 3 text

THAT TWITTER POST REGARDING OBSERVABILITY (credit: @gphat, https://blog.twitter.com/engineering/en_us/a/2013/observability-at-twitter.html fig. 1

Slide 4

Slide 4 text

THE THREE PILLARS OF OBSERVABILITY Metrics Logs Traces (credit: @copyconstruct, https://medium.com/@copyconstruct/logs-and-metrics-6d34d3026e38) fig. 3

Slide 5

Slide 5 text

THE bourgonIAN VENN DIAGRAM (credit: @peterbourgon, https://peter.bourgon.org/blog/2017/02/21/metrics-tracing-and-logging.html) fig. 2

Slide 6

Slide 6 text

? HOW SHOULD WE DEFINE 'OBSERVABILITY' FOR CI TOOLS? q. 1

Slide 7

Slide 7 text

AN EXPIERMENT IN INSTRUMENTING A CI PIPELINE IN THE CLOUD

Slide 8

Slide 8 text

CI PIPELINE: TYPICAL MONITORING BATTERIES INCLUDED https://github.com/capitalone/Hygieia (as seen at AWS re:Invent) (diagram: @peterbourgon)

Slide 9

Slide 9 text

CI PIPELINE: WHAT DOES TRACING LOOK LIKE? Waterfall/Gantt Chart 'CI run scoped (?)' (diagram: @peterbourgon)

Slide 10

Slide 10 text

CASE STUDY: AWS CodeBuild (no servers required!) Converting steps to segments in a trace

Slide 11

Slide 11 text

HOW TO INSTRUMENT THE PIPELINE? (I.E. NO OPERATING SYSTEM ACCESS) Log Events converted to a trace. Needs: timestamp and relationship to other events Optional: other useful attributes (i.e. container name) Traces are just related events! We can convert! (diagram: @peterbourgon)

Slide 12

Slide 12 text

COLLECTING TRACE EVENTS THE REALLY LAZY WAY tail -f $temp_file | ./traceformer -app-name DevOpsTest &> output.log & Just read events in from STDIN and convert them to a tracing format (in this case New Relic APM) with some hacked together golang code.

Slide 13

Slide 13 text

CI TRACES, VISUALIZED (APM STYLE) Latency breakdown of different build steps. Metrics created from traces.

Slide 14

Slide 14 text

CONNECTING CI TRACES TO PROD PERFORMANCE Attributes add additional context to traces. ... this can then be correlated with production performance.

Slide 15

Slide 15 text

LESSONs LEARNED INSTRUMENTING THE CI PIPELINE 1) If you have a complex CI pipeline, consider traces. 2) We should be connecting production performance to the processes (i.e. pipeline runs) that deployed and tested them. (diagram: @peterbourgon)

Slide 16

Slide 16 text

THANK YOU. @smithclay—New Relic Testing in Production Meetup, San Francisco 2/20/2018