Testing the build with TestKit

Build code contributes to the success of a company to a large extent. It enables an organization to deliver their software to the end user fast, frequently and reliably. With Continuous Delivery becoming a standard practice throughout the industry and build logic increasing in complexity, a need arises to verify the logic powering these processes. Gradle provides powerful support for testing build code out-of-the-box. In this demo-driven session, we will discuss the ins and outs of TestKit, a Gradle core library that helps with functional testing of build logic.

In the course of this talk, we’ll touch on aspects such as:

– Setting up TestKit for a build
– Executing a build with a specific Gradle distribution
– Asserting expected build outcome and output
– Verifying proper runtime behavior of custom plugins and extensions
– Cross-version testing of builds
– Debugging test execution in the IDE

Transcript

1. Gradle Summit Testing the build with TestKit
 Benjamin Muschko 2016
2. About to ship code to production…
3. …but the build logic contained bugs
4. Why not test build logic?
5. Testing build logic Build logic needs to be testable on multiple levels.
6. Writing unit tests in Gradle Class under test does not use Gradle API Test logic in isolation Testing a single class
7. Example: Writing a unit test with Spock 
 package com.bmuschko.gradle.docker.tasks.image
 
 import spock.lang.Specification
 
 import static com.bmuschko.gradle.docker.tasks.image.Dockerfile.*
 
 class DockerfileTest extends Specification {
 def "Instruction String representation is built correctly"() {
 expect:
 instructionInstance.keyword == keyword
 instructionInstance.build() == builtInstruction 
 where:
 instructionInstance | keyword | builtInstruction
 new FromInstruction('ubuntu:14.04') | 'FROM' | 'FROM ubuntu:14.04'
 new FromInstruction({ 'ubuntu:14.04' }) | 'FROM' | 'FROM ubuntu:14.04'
 ...
 }
 }
8. Writing integration tests in Gradle Class(es) under test use Gradle API Usually interacts with Project instance Does not execute a full build script
9. Example: Writing an integration test using ProjectBuilder 
 import org.gradle.api.Project
 import org.gradle.testfixtures.ProjectBuilder class DockerJavaApplicationPluginIntegrationTest extends Specification {
 @Rule TemporaryFolder temporaryFolder = new TemporaryFolder()
 Project project
 
 def setup() {
 project = ProjectBuilder.builder().withProjectDir(temporaryFolder.root).build()
 }
 
 def "Creates tasks out-of-the-box when application plugin is applied"() {
 when:
 project.apply(plugin: DockerJavaApplicationPlugin)
 project.apply(plugin: 'application')
 
 then: project.tasks.findByName(DockerJavaApplicationPlugin .COPY_DIST_RESOURCES_TASK_NAME)
 project.tasks.findByName(DockerJavaApplicationPlugin.DOCKERFILE_TASK_NAME)
 project.tasks.findByName(DockerJavaApplicationPlugin.BUILD_IMAGE_TASK_NAME)
 project.tasks.findByName(DockerJavaApplicationPlugin.PUSH_IMAGE_TASK_NAME)
 }
 }
10. Writing functional tests in Gradle Executes build script similar to end user Examines build outcome and output Isolated test environment
11. Example: Writing a functional test with TestKit 
 def "can successfully create Dockerfile"() {
 given:
 buildFile << """
 import com.bmuschko.gradle.docker.tasks.image.Dockerfile
 
 task dockerfile(type: Dockerfile) {
 from 'ubuntu:14.04'
 maintainer 'Benjamin Muschko "benjamin.muschko@gmail.com"'
 } """
 
 when:
 def result = GradleRunner.create()
 .withProjectDir(testProjectDir.root)
 .withArguments('dockerfile')
 .build()
 
 then:
 result.task(":dockerfile").outcome == SUCCESS testProjectDir.file('Dockerfile').exists()
 }
12. What’s the Gradle TestKit? - Uses Tooling API as test execution harness - Agnostic of test framework - Assertions made based on build output, build logging or executed tasks + their result Functional testing support in Gradle core. Gradle 2.6
13. TestKit usage Declaring TestKit dependency 
 dependencies {
 testCompile gradleTestKit()
 }
 Declaring dependency on test framework 
 dependencies {
 testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
 }
14. DEMO Functional testing of a build script
15. Cross-version compatibility tests Forward and backward compatibility independent of Gradle version used to build logic. 2.4 2.5 2.6 2.3 2.2 Version used for writing build logic
16. Retrieving Gradle distributions Enterprise Server Local or Shared Disk Build Machine Public Server
17. Specifying a Gradle version Using a Gradle version available on server Using a Gradle installation on disk Using a Gradle distribution via URL GradleRunner.withGradleVersion(String) GradleRunner.withGradleInstallation(File) GradleRunner.withGradleDistribution(URI)
18. DEMO Cross-version compatibility testing
19. Testing a plugin - Tooling API runs in separate process - Does not share classpath and classloaders as test process - Requires injection of code under test Requires additional work!
20. DEMO Manually injecting the plugin classpath
21. That’s some complex machinery!
22. Simplifying setup with plugin development plugin - Declaration of gradleApi() and gradleTestKit() - Generates plugin classpath manifest file - Plugin classpath injection needs to be declared for GradleRunner instance explicitly Apply sensible default and conventions…
23. Automatic injection of plugin code Applying the Java Gradle plugin dev. plugin 
 apply plugin: 'java-gradle-plugin' Injecting classpath into GradleRunner 
 GradleRunner.create()
 .withPluginClasspath()
 .build() Gradle 2.13
24. DEMO Testing a plugin with TestKit
25. Configurable conventions Source set containing code under test Source set used for injecting the plugin classpath Reconfigurable with the help of the class GradlePluginDevelopmentExtension sourceSets.main sourceSets.test
26. DEMO Configuring a dedicated test source set
27. Debugging test execution Setting system property for ad-hoc testing Enabling debugging programmatically -Dorg.gradle.testkit.debug=true GradleRunner.withDebug(true)
28. DEMO Debugging tests from the IDE
29. Feature limitations Support for features determined by version of Tooling API used to execute test. More info: https:/ /docs.gradle.org/current/userguide/test_kit.html
30. Future enhancements for TestKit - Convenience test fixtures - Hooking into IDE plugins - Integration with JaCoCo plugin More info: https://github.com/gradle/gradle/blob/master/ design-docs/testing-toolkit.md
31. Thank You! Please ask questions… https:/ /www.github.com/bmuschko @bmuschko
32. Introducing Gradle Cloud Services - Insights into your build - View and share via URL - Debug, optimize and refine - Analyze all of your builds Stop by the Gradle Lounge and create a Build Scan! The first service, Gradle Build Scans, is now available.
33. Who Are We Motto: Build Happiness Mission: To revolutionize the way software is built and shipped. We exist to end once-and-for-all the worst things about big software and restore the reason you got into coding in the first place. We’re Hiring: Gradle is hiring front-end, back-end, and core software engineers. Visit gradle.org/jobs to apply
34. Learn more at www.gradle.org