Testing the build with TestKit

Build code contributes to the success of a company to a large extent. It enables an organization to deliver their software to the end user fast, frequently and reliably. With Continuous Delivery becoming a standard practice throughout the industry and build logic increasing in complexity, a need arises to verify the logic powering these processes. Gradle provides powerful support for testing build code out-of-the-box. In this demo-driven session, we will discuss the ins and outs of TestKit, a Gradle core library that helps with functional testing of build logic.

In the course of this talk, we’ll touch on aspects such as:

– Setting up TestKit for a build
– Executing a build with a specific Gradle distribution
– Asserting expected build outcome and output
– Verifying proper runtime behavior of custom plugins and extensions
– Cross-version testing of builds
– Debugging test execution in the IDE

Transcript

1. Gradle Summit
   Testing the build with TestKit

   Benjamin Muschko
   2016
   

   View Slide

2. About to ship code to production…
   

   View Slide

3. …but the build logic contained bugs
   

   View Slide

4. Why not test build logic?
   

   View Slide

5. Testing build logic
   Build logic needs to be testable on multiple
   levels.
   

   View Slide

6. Writing unit tests in Gradle
   Class under test does
   not use Gradle API
   Test logic in isolation
   Testing a single class
   

   View Slide

7. Example: Writing a unit test with Spock
   

   package com.bmuschko.gradle.docker.tasks.image

   

   import spock.lang.Specification

   

   import static com.bmuschko.gradle.docker.tasks.image.Dockerfile.*

   

   class DockerfileTest extends Specification {

   def "Instruction String representation is built correctly"() {

   expect:

   instructionInstance.keyword == keyword

   instructionInstance.build() == builtInstruction
   

   where:

   instructionInstance | keyword | builtInstruction

   new FromInstruction('ubuntu:14.04') | 'FROM' | 'FROM ubuntu:14.04'

   new FromInstruction({ 'ubuntu:14.04' }) | 'FROM' | 'FROM ubuntu:14.04'

   ...

   }

   }
   

   View Slide

8. Writing integration tests in Gradle
   Class(es) under test
   use Gradle API
   Usually interacts with
   Project instance
   Does not execute
   a full build script
   

   View Slide

9. Example: Writing an integration test using ProjectBuilder
   

   import org.gradle.api.Project

   import org.gradle.testfixtures.ProjectBuilder
   class DockerJavaApplicationPluginIntegrationTest extends Specification {

   @Rule TemporaryFolder temporaryFolder = new TemporaryFolder()

   Project project

   

   def setup() {

   project = ProjectBuilder.builder().withProjectDir(temporaryFolder.root).build()

   }

   

   def "Creates tasks out-of-the-box when application plugin is applied"() {

   when:

   project.apply(plugin: DockerJavaApplicationPlugin)

   project.apply(plugin: 'application')

   

   then:
   project.tasks.findByName(DockerJavaApplicationPlugin
   .COPY_DIST_RESOURCES_TASK_NAME)

   project.tasks.findByName(DockerJavaApplicationPlugin.DOCKERFILE_TASK_NAME)

   project.tasks.findByName(DockerJavaApplicationPlugin.BUILD_IMAGE_TASK_NAME)

   project.tasks.findByName(DockerJavaApplicationPlugin.PUSH_IMAGE_TASK_NAME)

   }

   }
   

   View Slide

10. Writing functional tests in Gradle
    Executes build script
    similar to end user
    Examines build
    outcome and output
    Isolated test
    environment
    

    View Slide

11. Example: Writing a functional test with TestKit
    

    def "can successfully create Dockerfile"() {

    given:

    buildFile << """

    import com.bmuschko.gradle.docker.tasks.image.Dockerfile

    

    task dockerfile(type: Dockerfile) {

    from 'ubuntu:14.04'

    maintainer 'Benjamin Muschko "benjamin.muschko@gmail.com"'

    }
    """

    

    when:

    def result = GradleRunner.create()

    .withProjectDir(testProjectDir.root)

    .withArguments('dockerfile')

    .build()

    

    then:

    result.task(":dockerfile").outcome == SUCCESS
    testProjectDir.file('Dockerfile').exists()

    }
    

    View Slide

12. What’s the Gradle TestKit?
    - Uses Tooling API as test execution harness
    - Agnostic of test framework
    - Assertions made based on build output, build
    logging or executed tasks + their result
    Functional testing support in Gradle core.
    Gradle 2.6
    

    View Slide

13. TestKit usage
    Declaring TestKit dependency
    

    dependencies {

    testCompile gradleTestKit()

    }

    Declaring dependency on test framework
    

    dependencies {

    testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'

    }
    

    View Slide

14. DEMO
    Functional testing of a build script
    

    View Slide

15. Cross-version compatibility tests
    Forward and backward compatibility independent
    of Gradle version used to build logic.
    2.4 2.5 2.6
    2.3
    2.2
    Version used for
    writing build logic
    

    View Slide

16. Retrieving Gradle distributions
    Enterprise
    Server
    Local or
    Shared Disk
    Build Machine
    Public Server
    

    View Slide

17. Specifying a Gradle version
    Using a Gradle version available on server
    Using a Gradle installation on disk
    Using a Gradle distribution via URL
    GradleRunner.withGradleVersion(String)
    GradleRunner.withGradleInstallation(File)
    GradleRunner.withGradleDistribution(URI)
    

    View Slide

18. DEMO
    Cross-version compatibility testing
    

    View Slide

19. Testing a plugin
    - Tooling API runs in separate process
    - Does not share classpath and classloaders as
    test process
    - Requires injection of code under test
    Requires additional work!
    

    View Slide

20. DEMO
    Manually injecting the plugin classpath
    

    View Slide

21. That’s some complex machinery!
    

    View Slide

22. Simplifying setup with plugin development plugin
    - Declaration of gradleApi() and
    gradleTestKit()
    - Generates plugin classpath manifest file
    - Plugin classpath injection needs to be declared
    for GradleRunner instance explicitly
    Apply sensible default and conventions…
    

    View Slide

23. Automatic injection of plugin code
    Applying the Java Gradle plugin dev. plugin
    

    apply plugin: 'java-gradle-plugin'
    Injecting classpath into GradleRunner
    

    GradleRunner.create()

    .withPluginClasspath()

    .build()
    Gradle 2.13
    

    View Slide

24. DEMO
    Testing a plugin with TestKit
    

    View Slide

25. Configurable conventions
    Source set containing code under test
    Source set used for injecting the plugin classpath
    Reconfigurable with the help of the class
    GradlePluginDevelopmentExtension
    sourceSets.main
    sourceSets.test
    

    View Slide

26. DEMO
    Configuring a dedicated test source set
    

    View Slide

27. Debugging test execution
    Setting system property for ad-hoc testing
    Enabling debugging programmatically
    -Dorg.gradle.testkit.debug=true
    GradleRunner.withDebug(true)
    

    View Slide

28. DEMO
    Debugging tests from the IDE
    

    View Slide

29. Feature limitations
    Support for features determined by version of
    Tooling API used to execute test.
    More info:
    https:/
    /docs.gradle.org/current/userguide/test_kit.html
    

    View Slide

30. Future enhancements for TestKit
    - Convenience test fixtures
    - Hooking into IDE plugins
    - Integration with JaCoCo plugin
    More info:
    https://github.com/gradle/gradle/blob/master/
    design-docs/testing-toolkit.md
    

    View Slide

31. Thank You!
    Please ask questions…
    https:/
    /www.github.com/bmuschko
    @bmuschko
    

    View Slide

32. Introducing Gradle Cloud Services
    - Insights into your build
    - View and share via URL
    - Debug, optimize and refine
    - Analyze all of your builds
    Stop by the Gradle Lounge and create a Build Scan!
    The first service, Gradle Build Scans, is now available.
    

    View Slide

33. Who Are We
    Motto: Build Happiness
    Mission: To revolutionize the way software is built and
    shipped. We exist to end once-and-for-all the worst things
    about big software and restore the reason you got into coding
    in the first place.
    We’re Hiring: Gradle is hiring front-end, back-end, and core
    software engineers. Visit gradle.org/jobs to apply
    

    View Slide

34. Learn more at www.gradle.org
    

    View Slide