Continuous Integration for Android Using Jenkins

This talk explains how Jenkins can be used to improve product quality, stability and delivery speed by embracing continuous integration. We’ll walk through how Android developers and testers can benefit from using Jenkins, and then give some pro tips for more advanced capabilities, including how to speed up your tests by running them in parallel across multiple Jenkins slaves.

It was presented at the Android Summit 2016: http://androidsummit.org/

VIDEO: https://www.youtube.com/watch?v=wNXJi75WHcA

Dan Jarvis: http://careers.stackoverflow.com/dj
Sam Edwards: https://twitter.com/handstandsam

Transcript

1. CONTINUOUS
   INTEGRATION
   FOR ANDROID
   USING JENKINS
   

   View Slide

2. 05
   About Us – Dan Jarvis & Sam Edwards
   • Tech Leads for Capital One Wallet on Android
   • Top rated finance app by a US bank
   • First US bank with native Android contactless payments
   • Each has 12+ years professional experience
   • Passionate about automation and knowledge sharing
   @HandstandSam
   

   View Slide

3. • Continuous Integration (CI) fundamentals
   • Productivity tips
   • Capital One Wallet case study
   • Jenkins as a platform
   • Pro tips - how to run tests in parallel
   05
   Agenda
   

   View Slide

4. CONTINUOUS
   INTEGRATION
   FUNDAMENTALS
   

   View Slide

5. What is CI?
   

   View Slide

6. Continuous Integration (CI) is a development practice
   that requires developers to integrate code into a shared
   repository several times a day. Each check-in is then
   verified by an automated build, allowing teams to
   detect problems early.
   ”
   

   View Slide

7. Why do CI?
   

   View Slide

8. View Slide

9. Why?
   • Deliver faster and higher quality
   • Your product is always ready to deploy
   • Capture institutional knowledge in tests
   • Automation hugely improves productivity
   • Multiply your impact!
   

   View Slide

10. What is Jenkins?
    

    View Slide

11. A server that executes jobs
    based on triggers.
    ”
    

    View Slide

12. Jenkins Terminology
    •Jobs
    •Plugins
    •Master, slaves and executors
    

    View Slide

13. Jenkins Jobs – “A Job is essentially a script”
    

    View Slide

14. Jenkins Jobs - Configuration
    

    View Slide

15. Jenkins Jobs – Execution & History
    

    View Slide

16. View Slide

17. View Slide

18. Jenkins Plugins
    • Plugins can be written to extend Jenkins
    capabilities
    Default Install Plugins Added
    

    View Slide

19. 05
    Android Emulator Plugin
    

    View Slide

20. Jenkins Master, Slaves and Executors
    • Jobs are configured on the master node
    • The master dispatches jobs to run on
    executors on slave nodes (which could
    be on the same machine as the master)
    

    View Slide

21. What Makes Jenkins so Great?
    • Easy, installation, configuration & a web based interface
    • Open source https://github.com/jenkinsci/jenkins
    • >1000 plugins
    • Distributed and scalable
    • Long history and widespread adoption
    • Enterprise integrations
    

    View Slide

22. PRODUCTIVITY
    TIPS
    

    View Slide

23. Real-time Notifications
    

    View Slide

24. View Slide

25. View Slide

26. Spoon - https://github.com/square/spoon
    

    View Slide

27. View Slide

28. View Slide

29. View Slide

30. Screenshots on Assertion Failure
    Spoon.screenshot(activity,"test_failure");
    

    View Slide

31. Spoon & Screenshots
    

    View Slide

32. More on Spoon, Espresso and Screenshots
    http://droidcon.nyc/
    http://www.devfestdc.org/
    

    View Slide

33. How to Run Spoon
    

    View Slide

34. Custom Links to Spoon Results
    

    View Slide

35. Running Specific Instrumentation Tests (docs)
    Run By Instrumentation Argument(s) Code
    Method -e class com.FooTest#test1
    Class -e class com.FooTest
    Standard
    Annotations
    -e size small
    -e size medium
    -e size large
    @SmallTest
    @MediumTest
    @LargeTest
    Custom
    Annotations
    -e annotation
    -e notAnnotation
    @MyCustomAnnotation
    Package –e package
    

    View Slide

36. Android Instrumentation Runner (docs)
    

    View Slide

37. Spoon Arguments
    

    View Slide

38. CASE STUDY
    WALLET
    

    View Slide

39. View Slide

40. The Testing Pyramid - Wallet Examples
    • Android Espresso Tests, Real Servers
    • Android Tests, Real Servers
    • JVM Tests, Mock Data
    • Android Component Tests, Mock Data
    Black
    Box Tests
    Integration Tests
    Component Tests
    Unit Tests
    

    View Slide

41. Auto-run Tests for Every PR
    

    View Slide

42. GitHub Pull Request Builder Plugin
    

    View Slide

43. Capital One Wallet for Android on Jenkins Today
    Git Commit
    Pull Request (PR)
    Opened or Updated
    Unit Tests on PR Branch ✓
    PR Merged
    Assemble APKs
    and Test APKs
    Integration Tests
    Unit Tests
    Component Tests
    Blackbox Tests
    ✓ ✓
    ✓
    ✓
    ✓
    ✓
    

    View Slide

44. JENKINS
    AS A PLATFORM
    

    View Slide

45. Linked Jobs & Triggers
    

    View Slide

46. Newbie – One Job
    Compile/Build
    Integration Tests
    Unit Tests
    Component Tests
    Blackbox Tests
    

    View Slide

47. Better – Four Jobs
    Run Component Tests
    Run Unit Tests
    Compile/Build
    Compile/Build Compile/Build
    Run Blackbox Tests
    Run Integration Tests
    Compile/Build
    

    View Slide

48. Better – Four Jobs
    Run Component Tests
    Run Unit Tests
    Compile/Build
    Compile/Build Compile/Build
    Run Blackbox Tests
    Run Integration Tests
    Compile/Build
    Limitations:
    • Not great if you want to repeatedly re-run the tests
    • Wastes time/money if you have less than four slaves/executors
    

    View Slide

49. Better – Five Jobs
    Run Component Tests
    Run Unit Tests
    Compile/Build
    Run Blackbox Tests
    Run Integration Tests
    

    View Slide

50. Configuration - Triggering
    

    View Slide

51. Better – Five Jobs
    Run Component Tests
    Run Unit Tests
    Compile/Build
    Run Blackbox Tests
    Run Integration Tests
    

    View Slide

52. Configuration – Copy Build Artifacts From Other Job
    

    View Slide

53. Continuous Delivery
    

    View Slide

54. Building a Release Candidate
    Compile/Build
    ✓
    Manual Testing
    and Upload
    Alpha
    

    View Slide

55. What if I told you every build
    that passes your automated tests
    is a release candidate?
    ”
    

    View Slide

56. Continuous Delivery for Clients
    Compile/Build
    Integration Tests
    Unit Tests
    Component Tests
    Blackbox Tests
    ✓
    ✓
    ✓
    ✓
    ✓
    Alpha
    

    View Slide

57. Continuous Delivery for All
    

    View Slide

58. Server
    Unit Tests
    Continuous Delivery for Servers
    ✓
    ✓
    Server
    Integration Tests
    ✓
    

    View Slide

59. Your automated client tests are
    now an API.
    ”
    

    View Slide

60. iOS Stable
    Integration Tests
    Server
    Unit Tests
    Continuous Delivery for Servers
    ✓
    Server
    Integration Tests
    Android Stable
    Integration Tests
    ✓
    ✓
    ✓
    ✓
    

    View Slide

61. Continuous Delivery for Servers
    1. Server developer runs unit tests locally
    

    View Slide

62. Continuous Delivery for Servers
    1. Server developer runs unit tests locally
    2. On pull request, unit tests and all server and client integration tests run
    on a transient server instance
    

    View Slide

63. Continuous Delivery for Servers
    1. Server developer runs unit tests locally
    2. On pull request, unit tests and all server and client integration tests run
    on a transient server instance
    3. On merge, server code gets automatically deployed to shared QA
    region
    

    View Slide

64. Pipeline Views
    

    View Slide

65. Pipeline
    Jenkins Pipelines: https://jenkins.io/doc/pipeline/
    

    View Slide

66. View Slide

67. Manage Scripts in Separate Repository
    Benefits:
    • Versioned with great tooling to view history
    

    View Slide

68. Manage Scripts in Separate Repository
    Benefits:
    • Versioned with great tooling to view history
    • Great when you have to update multiple or similar scripts like a library
    version number
    

    View Slide

69. Manage Scripts in Separate Repository
    Benefits:
    • Versioned with great tooling to view history
    • Great when you have to update multiple or similar scripts like a library
    version number
    • Great when re-building Jenkins
    

    View Slide

70. Manage Scripts in Separate Repository
    Benefits:
    • Versioned with great tooling to view history
    • Great when you have to update multiple or similar scripts like a library
    version number
    • Great when re-building Jenkins
    

    View Slide

71. PRO TIPS
    RUNNING TESTS
    IN PARALLEL
    

    View Slide

72. Recap
    Run Component Tests
    Run Unit Tests
    Compile/Build
    Run Blackbox Tests
    Run Integration Tests
    

    View Slide

73. • Split them across devices
    and/or executors
    …or both!
    05
    Parallel Tests
    

    View Slide

74. Sharding Your Tests
    Run Component Tests
    Run Unit Tests
    Compile/Build
    Run Blackbox Tests
    Run Integration Tests
    Blackbox Tests [0]
    Blackbox Tests [1]
    Blackbox Tests [2]
    Blackbox Tests [3]
    

    View Slide

75. Parallel Tests Attempt #1
    

    View Slide

76. View Slide

77. View Slide

78. • The main job uses up an executor while waiting for Phases to
    complete (which each use their own executor)
    • Phases need to do their own partitioning of the test suite
    • Test results are not aggregated back into a single report
    05
    Problems with MultiJob Plugin
    

    View Slide

79. 05
    Android Instrumentation Sharding (docs)
    

    View Slide

80. Matrix Project Plugin
    

    View Slide

81. View Slide

82. 05
    Configuring The Matrix
    

    View Slide

83. 05
    Tell The Job Which Test Shard To Run
    

    View Slide

84. • Easier to pass sharding index to build executors
    • Starts parallel jobs for all combinations of configurations
    05
    Benefits of Matrix Plugin
    

    View Slide

85. Supercharging Your Matrix
    

    View Slide

86. 05
    Android Emulator Plugin
    • Built in support for
    Multi-configuration (matrix)
    jobs!
    

    View Slide

87. 05
    Android Emulator Plugin
    

    View Slide

88. THE
    END
    

    View Slide

89. ENCORE
    

    View Slide

90. 05
    Android Emulator
    PROS
    • Multiple screen sizes are easy
    • FREE!
    • Fast (10x faster install speed then normal devices)
    CONS
    • “Snapshot” feature is unreliable
    • Workaround: Re-create the AVD every time
    

    View Slide

91. 05
    Genymotion Emulator
    PROS
    • Multiple screen sizes are easy
    • Fast
    • Snapshot feature works pretty well
    CONS
    • Requires Licenses
    • Can be difficult to manage with sharding
    

    View Slide

92. 05
    Real Devices
    PROS
    • They are REAL Devices
    • There are things like Google Test Cloud, Test Droid and Amazon
    Device Farm
    CONS
    • You have to buy all the devices & hook them up
    • OR you have to pay a lot for cloud based use of devices which
    can be slow
    

    View Slide

93. 05
    Jenkins Architecture
    

    View Slide

94. THAT’S ALL FOLKS
    

    View Slide