Advanced Code Coverage with Jenkins, GitHub and API Mocking

In 2015-2018, I talked about how to use the Jenkins Pipeline and custom libraries to do advanced integration tests and analyze code coverage. Coverage plugins were rather weak, and one needed some scripts and hacks to make it work, and to DIY for distributed testing. In 2021 the situation has changed significantly thanks to the Coverage and Checks API plugins. Distributed integration testing also became easier thanks to better coverage collectors and integrations with API mocking tools. So, good time to be alive… and use Jenkins!

We will talk about how modern Jenkins allows you to improve and analyze code coverage. We will talk about unit and integration testing with WireMock, the new Coverage Plugin, support for standard formats (Cobertura, JaCoCo, gcov, JUnit, etc.), parallelization for heavy integration tests and API mocking, and integration with GitHub Checks API. How can you analyze code coverage in Jenkins and when do you need to create your own libraries? And what’s the fuzz about Testcontainers and WireMock for integration testing?

Copyright Notice: Gradle, Inc. Gradle®, Develocity®, Build Scan®, and the Gradlephant logo are registered trademarks of Gradle, Inc. "Gradle" means "Gradle Build Tool" and does not reference Gradle, Inc. and/or its subsidiaries

Transcript

1. Advanced Code Coverage with Jenkins, GitHub and API Mocks Oleg Nenashev
2. > Hello from Lugano, CH @oleg_nenashev oleg-nenashev
3. bit.ly/open-roadmaps-for-your-open-communities
4. > whoami @oleg_nenashev oleg-nenashev Dr. Nenashev / Mr. Jenkins Developer tools hacker Community builder & DevRel consultant #RussiansAgainstPutin #StandWithUkraine
5. Agenda • Jenkins TL;DR • Code Coverage in Jenkins. 2017 Edition • Modern Code Coverage in Jenkins, done right • Boosting coverage with Integration tests • GitHub Checks API in Jenkins • WireMock and Testcontainers These slides: speakerdeck.com/onenashev/advanced-code- coverage-with-jenkins-github-and-api-mocking
6. community.jenkins.io/t/coming-soon-advanced-code-coverage-with-jen kins-github-and-api-mocks/11874 Q&A:
7. @oleg_nenashev oleg-nenashev > whoami --jenkins
8. Disclaimer: Taking a break in Jenkins • On sabbatical in the community • Reasons: personal events, ongoing war • I participate in governance and GSoC • I will be back! CDF Statement We stand with the people of Ukraine. Please assist humanitarian efforts for the Ukrainian people and those affected by the military invasion of Ukraine by supporting international aid organizations, including UNICEF.
9. Image: dronetrest.com/t/diy-self-build-drones-or-ready-to-fly-rtf-drones/1414
10. Expectation
11. Expectation Reality
12. Jenkins is an AUTOMATION FRAMEWORK
13. Система Docs Plugins Configs Infrastruc ture Backend Samples Demos CI/CD System from Jenkins Infrastruc ture Infrastruc ture
14. Система Docs Plugins Configs Infra- structure Backend Samples Demos Infra- structure Infra- structure ? without proper maintenance Any Automation Framework
15. Team A(utomation) Jenkins is a part of the CI system, Dev Productivity Team’s Best Friend (?proof)
16. Code Coverage
18. A FEW PANDEMICS AGO… Jenkins was different
19. Back to 2017… Cobertura publishing In every job???
20. 1. Take library components 2. Build your own bicycle Pipeline Add some glue! Back to 2017… Using Pipeline libraries
21. Back to 2017… Global variable wrapper in a lib
22. Back to 2017… Using the global variable as a step
23. Back to 2017… Parallelizing tests
24. Back to 2017… Parallelizing coverage
25. Coverage Plugin
26. Coverage Plugin • 2018 - Created as Code Coverage API Plugin during GSoC 2018 by Shenyu Zheng • jenkins.io/projects/gsoc/2018/code-coverage-api-plugin • 2020/2022 - Major UI/UX revamps • 2023 - replaced by the Coverage Plugin plugins.jenkins.io/coverage
27. Code Coverage API Plugin (EoL). Supported Adapters • JaCoCo • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
28. Coverage Plugin. Supported Adapters • JaCoCo • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
29. Build Unit tests Publish Coverage
30. Usage • Records coverage produced by your build tool • Compatible with Jenkins Pipeline and Freestyle jobs
31. Coverage Plugin. Features • Coverage charts and trends • Fine grain reporting • Source code navigation • Parallel execution support • Reports merging • Flexible thresholds • REST API
32. Features: Statistics
33. Features: Heatmaps
34. Features: Fine grain reporting
35. … and Code Browsing* * Disabled by default due to space/CPU consumption. Enable for projects when you need them
36. Examples Coverage Plugin • Reference implementation & Dogfooding • github.com/jenkinsci/coverage-plugin • ci.jenkins.io/job/Plugins/job/coverage-plugin • With sources ci.jenkins.io/job/Plugins/job/code-coverage-api-plugin/view/change-requests/job/PR-651/10 Jenkinsfile Core: • github.com/jenkinsci/jenkins • Cobertura • Standard publishing
37. Examples Jenkinsfile Runner: • https://github.com/jenkinsci/jenkinsfile-runner/ • JaCoCo / JaCoCo Maven Plugin • https://github.com/jenkinsci/jenkinsfile-runner/blob/master/pom.xml • publishCoverage adapters: [jacocoAdapter('target/site/jacoco/jacoco.xml')]
38. How do we do it in Jenkins?
39. Jenkinsfile github.com/jenkinsci/code-coverage- api-plugin/blob/master/Jenkinsfile
40. And the Pipeline library • buildPlugin() and buildPluginWithGradle()* in the shared Pipeline Library • 400 codelines but easy for users • Default settings to meet project quality standards github.com/jenkins-infra/pipeline-library/blob/ master/vars/buildPluginWithGradle.groovy * NOTE: Gradle® and the Gradlephant logo are registered trademarks of Gradle, Inc. "Gradle" means "Gradle Build Tool" and does not reference Gradle, Inc. and/or its subsidiaries.
41. Let’s Integrate with GitHub!
42. GitHub Checks in Jenkins • “GitHub Checks API for Jenkins project” by Kezhi Xiong • Google Summer of Code 2021 • Implementation: Checks API Plugin + GitHub Adapter • jenkins.io/projects/gsoc/2020/projects/github-checks/
43. Checks API 101 • Status reporting to GitHub • Available in GitHub and GitHub Enterprise • Requires GitHub App authentication • Checks API documentation: https://docs.github.com/en/rest/reference/checks
44. GitHub App Authentication • Supported in GitHub Branch Source Plugin 2.7.1+. Patch by Tim Jacomb in April 2020 plugins.jenkins.io/github-branch-source • Announcement: jenkins.io/blog/2020/04/16/github-app-authentication
45. Setting up GitHub App Authentication Guide: https://github.com/jenkinsci/github-branch-source-plugin/blob/m aster/docs/github-app.adoc 1. Register a new GitHub App in GitHub 2. Download access tokens 3. Install the App in your organization/repository 4. Configure Jenkins credentials 5. Use credentials in Branch Source / Org folders
46. Checks Per Commit 46
47. • Multiple checks • Status reporting • Summary and Details • GitHub-Flavoured Markdown support • Annotations 47 https://github.com/jenkinsci/je nkins/pull/11148/checks?chec k_run_id=52443616008
48. https://github.com/jenkinsci/code-coverage-api-plugin/runs/2673064852
49. https://github.com/jenkinsci/coverag e-plugin/runs/20604906277 Summary Warnings for the diff
51. Integration testing • Multiple technology stacks and toolchains • Many teams involved • Sacred knowledge • [Almost] always too late
52. Build Unit tests Publish Reports Integration tests Publish Reports WAS
54. xkcd.com/303 www.jenkins.io/artwork
57. xkcd.com/303 www.jenkins.io/artwork
58. Source: https://en.wikipedia.org/wiki/Groundhog_Day_(film)
59. Build Unit tests Publish Coverage Integration tests Publish Coverage • High infrastructure requirements • Needs parallelization • Reports merging • Virtualization
61. Build Unit tests Publish Reports Integration tests Publish Reports NOW “Shift Left” Fast integrations tests are critical
62. Ways to do API integration testing 1. Testing against Production/Staging servers 2. Testing against a simplified portable instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level Slow Fast Accu- rate Nope
63. Staging & QA Tests Build Integration tests Build Tests Gaining confidence in you software
64. Shifting ITs left with Testcontainers and WireMock
66. SDKs C/C++ testcontainers.org
68. Containers are not always slow! • On-demand image build • Caching Docker image builds • Suspending containers between tests • Graceful termination ◦ github.com/testcontainers/moby-ryuk testcontainers.org
70. wiremock.org/docs/stubbing IF (request_url) THEN (response) * * it gets MUCH more complex WireMock Config JSON: Client library response request Client App Mock API Server HTTP/2
71. WireMock in Java wiremock.org/docs/stubbing
72. WireMock Ecosystem & Features wiremock.org/docs
73. WireMock Ecosystem & Features wiremock.org/docs
74. Comparison [1/2] 1. Testing against Production/Staging servers 2. Testing against a simplified/containerized instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level Slow Fast Nope Accu- rate
76. WireMock has an official Testcontainers module! testcontainers.com/modules/wiremock
77. Example - JUnit 5
79. WireMock as a Proxy Tests Real API Provider • Fault injection • Recording • Protocol Verification Testcontainers
81. Code Coverage: Parallel Pipeline Support
82. Code Coverage API: Merging reports • Parameter in older versions • Default behavior in new versions • Keeps diffs and code browsing
84. Coverage Plugin. Supported Adapters • JaCoCo • Cobertura, gcov (Cobertura Plugin) • llvm-cov (llvm-cov plugin) • OpenCover (OpenCover Plugin) • Istanbul - via Cobertura Reporter • PIT Mutation Testing • SonarQube Generic Coverage Adapter
85. wiremock.org/docs/solutions/testcontainers
86. wiremock/wiremock-testcontainers-go
88. Credits to • All Jenkins contributors • All WireMock contributors • Code Coverage API / Coverage Plugin ◦ Shenyu Zheng ◦ Ulli Hafner ◦ Florian Orendi - docs ❤ • GitHub Checks API ◦ Kezhi Xiong ◦ Ulli Hafner ◦ Tim Jacomb • Google Summer of Code!
89. Takeaways - Coverage Reporting • Jenkins is one of the strongest platforms for coverage reporting and analysis • Code Coverage API supports parallelization and reports mergings • Your technology stack can be supported OOTB or via report format converters • Jenkins integrates with GitHub Checks
90. Integration Testing • Great asset if done right • Shift left your integration testing • Mocking APIs and services, reducing external dependencies, containers • There are tools for that, including WireMock and Testcontainers
91. What to do after the talk? 1. Check out our CI in Jenkins repositories ◦ github.com/jenkinsci 2. Try out coverage reporting in Jenkins 3. Shift left your integration tests with WireMock and Testcontainers ◦ wiremock.org ◦ testcontainers.com 4. Share feedback with us! ◦ www.jenkins.io/participate/connect ◦ slack.wiremock.org
92. It’s [always] a great time to contribute ;) jenkins.io/participate wiremock.org/participate
93. It is a great time to contribute! https://jenkins.io/participate
94. Contacts: E-mail: o.v.nenashev@gmail.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? community.jenkins.io/t/coming-soon-advanced-code -coverage-with-jenkins-github-and-api-mocks/11874