Why is my pipeline so slow?

Is my CI pipeline getting slower? It feels like more end-to-end tests are failing
lately. Since when is deployment taking so long?

Have questions like these bothered you before? You are probably not alone.
But what can we do about it? Well, we could add some facts and measurements
to our gut-feeling and prove that something is not right.

Luckily, there are APIs that we can leverage to get the timing and
success rates of our CI jobs.

In this talk we will show how we can use a simple Deno script to retrieve that data,
pipe into ElasticSearch and visualize it with Kibana, so that we can see what is
really going on.

Transcript

1. Analyzing CI/CD-Pipelines with Deno, ElasticSearch and Kibana Why is my pipeline so slow? nknapp nknapp-1979
2. git push origin main pipeline running … Done. Next ticket. Trunk based development
3. The pipeline is failing for over 4 hours now and nothing gets deployed. Trunk based anger
4. Early warning system
7. Some simple code
8. LIVE DEMO Watching main branch pipeline
9. Gut feelings Do you also think that our tests have become slower lately? Yeah, sure feels like it. But how can we be sure?
10. Statistics over all jobs How can you be sure? What caused it? When did it start?
12. Goal: Pinpoint time and change What happened here?
13. Iterating all jobs
16. Downsides of REST Too much data Missing data
17. GraphQL !!!
18. https://gitlab.cosee.biz/-/graphql-explorer
20. The query Pagination
21. Dump all Pagination
22. Where to put the data?
23. Set up Kibana and ElasticSearch
24. Set up Kibana and ElasticSearch
25. Create ElasticSearch index
26. Add job to index
27. Add job to index
28. SPIEL.digital - Frontend • Digital board-game fair • Developed June 2020 - Oktober 2020 • CMS for exhibitors, media creators etc. • Fancy UI for visitors LIVE DEMO Analysis of SPIEL.digital pipelines
29. A real life example • Monorepo for frontend and backend • Lots of tests • Merge requests • Review apps + staging environment • E2E-Tests • Continuous deployment
30. A real life example pipeline for main branch
31. Duration of frontend unit tests (jest 27) Node 18.x Node 20.x Node 18.x
32. Jest 27 vs vitest with parallel shards 20 18 18 vitest shards 20 22 vue 3 vitest 0.34 to vitest 2
33. Failing e2e-tests, last 24 hours Worth looking at, should not fail on main branch
34. E2E failure rate, main branch usual noise something is wrong something is wrong
35. Successful e2e test, average duration usual noise something is wrong something is wrong Warning! Graph not proportional
36. Conclusion • Patterns are easier to spot in graphs • Good decision helper for fixing e2e-tests • Addictive: Donʼt waste your time • Better to use Node instead of Deno? (More mainstream)
37. Thank you for listening
38. https://github.com/cosee/talk-pipeline-statistics
39. git push origin main pipeline running … Done. Next ticket. The pipeline is failing for over 4 hours now and nothing gets deployed.