Advanced Code Coverage with Jenkins, GitHub and API Mocking

In 2015-2018, I talked about how to use the Jenkins Pipeline and custom libraries to do advanced integration tests and analyze code coverage. Coverage plugins were rather weak, and one needed some scripts and hacks to make it work, and to DIY for distributed testing. In 2021 the situation has changed significantly thanks to the Coverage and Checks API plugins. Distributed integration testing also became easier thanks to better coverage collectors and integrations with API mocking tools. So, good time to be alive… and use Jenkins!

We will talk about how modern Jenkins allows you to improve and analyze code coverage. We will talk about unit and integration testing with WireMock, the new Coverage Plugin, support for standard formats (Cobertura, JaCoCo, gcov, JUnit, etc.), parallelization for heavy integration tests and API mocking, and integration with GitHub Checks API. How can you analyze code coverage in Jenkins and when do you need to create your own libraries? And what’s the fuzz about Testcontainers and WireMock for integration testing?

Transcript

1. / Advanced Code Coverage with Jenkins, GitHub and API Mocks Oleg Nenashev
2. / > Hello from Lugano, CH @oleg_nenashev oleg-nenashev
3. / bit.ly/open-roadmaps-for-your-open-communities
4. / > whoami @oleg_nenashev oleg-nenashev Dr. Nenashev / Mr. Jenkins Developer tools hacker Community builder & DevRel consultant #RussiansAgainstPutin #StandWithUkraine
5. / P.S: @jbaruch cosplay
6. / Agenda • Jenkins TL;DR • Code Coverage in Jenkins. 2017 Edition • Modern Code Coverage in Jenkins, done right • Boosting coverage with Integration tests • GitHub Checks API in Jenkins • WireMock and Testcontainers These slides: speakerdeck.com/onenashev/advanced-code- coverage-with-jenkins-github-and-api-mocking
7. / community.jenkins.io/t/coming-soon-advanced-code-coverage-with-jen kins-github-and-api-mocks/11874 Q&A:
8. / @oleg_nenashev oleg-nenashev > whoami --jenkins
9. / Disclaimer: Taking a break in Jenkins • On sabbatical in the community • Reasons: personal events, ongoing war • I participate in governance and GSoC • I will be back! CDF Statement We stand with the people of Ukraine. Please assist humanitarian efforts for the Ukrainian people and those affected by the military invasion of Ukraine by supporting international aid organizations, including UNICEF.
10. / Image: dronetrest.com/t/diy-self-build-drones-or-ready-to-fly-rtf-drones/1414
11. / Expectation
12. / Expectation Reality
13. / Jenkins is an AUTOMATION FRAMEWORK
14. / Система Docs Plugins Configs Infrastruc ture Backend Samples Demos CI/CD System from Jenkins Infrastruc ture Infrastruc ture
15. / Система Docs Plugins Configs Infra- structure Backend Samples Demos Infra- structure Infra- structure ? without proper maintenance Any Automation Framework
16. / Team A(utomation) Jenkins is a part of the CI system, Dev Productivity Team’s Best Friend (?proof)
17. / Code Coverage
18. /
19. / A FEW PANDEMICS AGO… Jenkins was different
20. / Back to 2017… Cobertura publishing In every job???
21. / 1. Take library components 2. Build your own bicycle Pipeline Add some glue! Back to 2017… Using Pipeline libraries
22. / Back to 2017… Global variable wrapper in a lib
23. / Back to 2017… Using the global variable as a step
24. / Back to 2017… Parallelizing tests
25. / Back to 2017… Parallelizing coverage
26. / Coverage Plugin
27. / Coverage Plugin • 2018 - Created as Code Coverage API Plugin during GSoC 2018 by Shenyu Zheng • jenkins.io/projects/gsoc/2018/code-coverage-api-plugin • 2020/2022 - Major UI/UX revamps • 2023 - replaced by the Coverage Plugin plugins.jenkins.io/coverage
28. / Code Coverage API Plugin (EoL). Supported Adapters • JaCoCo • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
29. / Coverage Plugin. Supported Adapters • JaCoCo • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
30. / Build Unit tests Publish Coverage
31. / Usage • Records coverage produced by your build tool • Compatible with Jenkins Pipeline and Freestyle jobs
32. / Coverage Plugin. Features • Coverage charts and trends • Fine grain reporting • Source code navigation • Parallel execution support • Reports merging • Flexible thresholds • REST API
33. / Features: Statistics
34. / Features: Heatmaps
35. / Features: Fine grain reporting
36. / … and Code Browsing* * Disabled by default due to space/CPU consumption. Enable for projects when you need them
37. / Examples Coverage Plugin • Reference implementation & Dogfooding • github.com/jenkinsci/coverage-plugin • ci.jenkins.io/job/Plugins/job/coverage-plugin • With sources ci.jenkins.io/job/Plugins/job/code-coverage-api-plugin/view/change-requests/job/PR-651/10 Jenkinsfile Core: • github.com/jenkinsci/jenkins • Cobertura • Standard publishing
38. / Examples Jenkinsfile Runner: • https://github.com/jenkinsci/jenkinsfile-runner/ • JaCoCo / JaCoCo Maven Plugin • https://github.com/jenkinsci/jenkinsfile-runner/blob/master/pom.xml • publishCoverage adapters: [jacocoAdapter('target/site/jacoco/jacoco.xml')]
39. / How do we do it in Jenkins?
40. / Jenkinsfile github.com/jenkinsci/code-coverage- api-plugin/blob/master/Jenkinsfile
41. / And the Pipeline library • buildPlugin() and buildPluginWithGradle in the shared Pipeline Library • 400 codelines but easy for users • Default settings to meet project quality standards github.com/jenkins-infra/pipeline-library/blob/ master/vars/buildPluginWithGradle.groovy
42. / Let’s Integrate with GitHub!
43. / • “GitHub Checks API for Jenkins project” by Kezhi Xiong • Google Summer of Code 2021 • Implementation: Checks API Plugin + GitHub Adapter • jenkins.io/projects/gsoc/2020/projects/github-checks/
44. / Checks API 101 • Status reporting to GitHub • Available in GitHub and GitHub Enterprise • Requires GitHub App authentication • Checks API documentation: https://docs.github.com/en/rest/reference/checks
45. / GitHub App Authentication • Supported in GitHub Branch Source Plugin 2.7.1+. Patch by Tim Jacomb in April 2020 • https://plugins.jenkins.io/github-branch-source • Announcement: https://www.jenkins.io/blog/2020/04/16/github-app-auth entication/
46. / Setting up GitHub App Authentication Guide: https://github.com/jenkinsci/github-branch-source-plugin/blob/m aster/docs/github-app.adoc 1. Register a new GitHub App in GitHub 2. Download access tokens 3. Install the App in your organization/repository 4. Configure Jenkins credentials 5. Use credentials in Branch Source / Org folders
47. / Checks Per Commit
48. / • Multiple checks • Status reporting • Summary and Details • GitHub Markdown support • Annotations https://github.com/jenkinsci/gradle- plugin/pull/382/checks?check_run_i d=19876936412
49. / https://github.com/jenkinsci/code-coverage-api-plugin/runs/2673064852
50. / https://github.com/jenkinsci/coverag e-plugin/runs/20604906277 Summary Warnings for the diff
51. /
52. / Integration testing • Multiple technology stacks and toolchains • Many teams involved • Sacred knowledge • [Almost] always too late
53. / Build Unit tests Publish Reports Integration tests Publish Reports WAS
54. /
55. / xkcd.com/303 www.jenkins.io/artwork
56. /
57. /
58. / xkcd.com/303 www.jenkins.io/artwork
59. / Source: https://en.wikipedia.org/wiki/Groundhog_Day_(film)
60. / Build Unit tests Publish Coverage Integration tests Publish Coverage • High infrastructure requirements • Needs parallelization • Reports merging • Virtualization
61. /
62. / Build Unit tests Publish Reports Integration tests Publish Reports NOW “Shift Left” Fast integrations tests are critical
63. / Ways to do API integration testing 1. Testing against Production/Staging servers 2. Testing against a simplified portable instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level Slow Fast Accu- rate Nope
64. / Staging & QA Tests Build Integration tests Build Tests Gaining confidence in you software
65. / Shifting ITs left with Testcontainers and WireMock
66. /
67. / SDKs C/C++ testcontainers.org
68. /
69. / Containers are not always slow! • On-demand image build • Caching Docker image builds • Suspending containers between tests • Graceful termination ◦ github.com/testcontainers/moby-ryuk testcontainers.org
70. /
71. / wiremock.org/docs/stubbing IF (request_url) THEN (response) * * it gets MUCH more complex WireMock Config JSON: Client library response request Client App Mock API Server HTTP/2
72. / WireMock in Java wiremock.org/docs/stubbing
73. / WireMock Ecosystem & Features wiremock.org/docs
74. / WireMock Ecosystem & Features wiremock.org/docs
75. / Comparison [1/2] 1. Testing against Production/Staging servers 2. Testing against a simplified/containerized instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level Slow Fast Nope Accu- rate
76. /
77. / WireMock has an official Testcontainers module! testcontainers.com/modules/wiremock
78. / Example - JUnit 5
79. /
80. / WireMock as a Proxy Tests Real API Provider • Fault injection • Recording • Protocol Verification Testcontainers
81. /
82. / Code Coverage: Parallel Pipeline Support
83. / Code Coverage API: Merging reports • Parameter in older versions • Default behavior in new versions • Keeps diffs and code browsing
84. /
85. / Coverage Plugin. Supported Adapters • JaCoCo • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
86. / wiremock.org/docs/solutions/testcontainers
87. / wiremock/wiremock-testcontainers-go
88. /
89. / Credits to • All Jenkins contributors • All WireMock contributors • Code Coverage API / Coverage Plugin ◦ Shenyu Zheng ◦ Ulli Hafner ◦ Florian Orendi - docs ❤ • GitHub Checks API ◦ Kezhi Xiong ◦ Ulli Hafner ◦ Tim Jacomb • Google Summer of Code!
90. / Takeaways - Coverage Reporting • Jenkins is one of the strongest platforms for coverage reporting and analysis • Code Coverage API supports parallelization and reports mergings • Your technology stack can be supported OOTB or via report format converters • Jenkins integrates with GitHub Checks
91. / Integration Testing • Great asset if done right • Shift left your integration testing • Mocking APIs and services, reducing external dependencies, containers • There are tools for that, including WireMock and Testcontainers
92. / What to do after the talk? 1. Check out our CI in Jenkins repositories ◦ github.com/jenkinsci 2. Try out coverage reporting in Jenkins 3. Shift left your integration tests with WireMock and Testcontainers ◦ wiremock.org ◦ testcontainers.com 4. Share feedback with us! ◦ www.jenkins.io/participate/connect ◦ slack.wiremock.org
93. / It’s [always] a great time to contribute ;) jenkins.io/participate wiremock.org/participate gradle.org/resources
94. / It is a great time to contribute! https://jenkins.io/participate
95. / Contacts: E-mail: o.v.nenashev@gmail.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? community.jenkins.io/t/coming-soon-advanced-code -coverage-with-jenkins-github-and-api-mocks/11874