Continuous Integration for Android Using Jenkins

Continuous Integration for Android Using Jenkins

This talk explains how Jenkins can be used to improve product quality, stability and delivery speed by embracing continuous integration. We’ll walk through how Android developers and testers can benefit from using Jenkins, and then give some pro tips for more advanced capabilities, including how to speed up your tests by running them in parallel across multiple Jenkins slaves.

It was presented at the Android Summit 2016: http://androidsummit.org/

VIDEO: https://www.youtube.com/watch?v=wNXJi75WHcA

Dan Jarvis: http://careers.stackoverflow.com/dj
Sam Edwards: https://twitter.com/handstandsam

5701f31a8433a22ae736282de8d08cd6?s=128

Sam Edwards

August 26, 2016
Tweet

Transcript

  1. CONTINUOUS INTEGRATION FOR ANDROID USING JENKINS

  2. 05 About Us – Dan Jarvis & Sam Edwards •

    Tech Leads for Capital One Wallet on Android • Top rated finance app by a US bank • First US bank with native Android contactless payments • Each has 12+ years professional experience • Passionate about automation and knowledge sharing @HandstandSam
  3. • Continuous Integration (CI) fundamentals • Productivity tips • Capital

    One Wallet case study • Jenkins as a platform • Pro tips - how to run tests in parallel 05 Agenda
  4. CONTINUOUS INTEGRATION FUNDAMENTALS

  5. What is CI?

  6. Continuous Integration (CI) is a development practice that requires developers

    to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. ”
  7. Why do CI?

  8. None
  9. Why? • Deliver faster and higher quality • Your product

    is always ready to deploy • Capture institutional knowledge in tests • Automation hugely improves productivity • Multiply your impact!
  10. What is Jenkins?

  11. A server that executes jobs based on triggers. ”

  12. Jenkins Terminology •Jobs •Plugins •Master, slaves and executors

  13. Jenkins Jobs – “A Job is essentially a script”

  14. Jenkins Jobs - Configuration

  15. Jenkins Jobs – Execution & History

  16. None
  17. None
  18. Jenkins Plugins • Plugins can be written to extend Jenkins

    capabilities Default Install Plugins Added
  19. 05 Android Emulator Plugin

  20. Jenkins Master, Slaves and Executors • Jobs are configured on

    the master node • The master dispatches jobs to run on executors on slave nodes (which could be on the same machine as the master)
  21. What Makes Jenkins so Great? • Easy, installation, configuration &

    a web based interface • Open source https://github.com/jenkinsci/jenkins • >1000 plugins • Distributed and scalable • Long history and widespread adoption • Enterprise integrations
  22. PRODUCTIVITY TIPS

  23. Real-time Notifications

  24. None
  25. None
  26. Spoon - https://github.com/square/spoon

  27. None
  28. None
  29. None
  30. Screenshots on Assertion Failure Spoon.screenshot(activity,"test_failure");

  31. Spoon & Screenshots

  32. More on Spoon, Espresso and Screenshots http://droidcon.nyc/ http://www.devfestdc.org/

  33. How to Run Spoon

  34. Custom Links to Spoon Results

  35. Running Specific Instrumentation Tests (docs) Run By Instrumentation Argument(s) Code

    Method -e class com.FooTest#test1 Class -e class com.FooTest Standard Annotations -e size small -e size medium -e size large @SmallTest @MediumTest @LargeTest Custom Annotations -e annotation -e notAnnotation @MyCustomAnnotation Package –e package
  36. Android Instrumentation Runner (docs)

  37. Spoon Arguments

  38. CASE STUDY WALLET

  39. None
  40. The Testing Pyramid - Wallet Examples • Android Espresso Tests,

    Real Servers • Android Tests, Real Servers • JVM Tests, Mock Data • Android Component Tests, Mock Data Black Box Tests Integration Tests Component Tests Unit Tests
  41. Auto-run Tests for Every PR

  42. GitHub Pull Request Builder Plugin

  43. Capital One Wallet for Android on Jenkins Today Git Commit

    Pull Request (PR) Opened or Updated Unit Tests on PR Branch ✓ PR Merged Assemble APKs and Test APKs Integration Tests Unit Tests Component Tests Blackbox Tests ✓ ✓ ✓ ✓ ✓ ✓
  44. JENKINS AS A PLATFORM

  45. Linked Jobs & Triggers

  46. Newbie – One Job Compile/Build Integration Tests Unit Tests Component

    Tests Blackbox Tests
  47. Better – Four Jobs Run Component Tests Run Unit Tests

    Compile/Build Compile/Build Compile/Build Run Blackbox Tests Run Integration Tests Compile/Build
  48. Better – Four Jobs Run Component Tests Run Unit Tests

    Compile/Build Compile/Build Compile/Build Run Blackbox Tests Run Integration Tests Compile/Build Limitations: • Not great if you want to repeatedly re-run the tests • Wastes time/money if you have less than four slaves/executors
  49. Better – Five Jobs Run Component Tests Run Unit Tests

    Compile/Build Run Blackbox Tests Run Integration Tests
  50. Configuration - Triggering

  51. Better – Five Jobs Run Component Tests Run Unit Tests

    Compile/Build Run Blackbox Tests Run Integration Tests
  52. Configuration – Copy Build Artifacts From Other Job

  53. Continuous Delivery

  54. Building a Release Candidate Compile/Build ✓ Manual Testing and Upload

    Alpha
  55. What if I told you every build that passes your

    automated tests is a release candidate? ”
  56. Continuous Delivery for Clients Compile/Build Integration Tests Unit Tests Component

    Tests Blackbox Tests ✓ ✓ ✓ ✓ ✓ Alpha
  57. Continuous Delivery for All

  58. Server Unit Tests Continuous Delivery for Servers ✓ ✓ Server

    Integration Tests ✓
  59. Your automated client tests are now an API. ”

  60. iOS Stable Integration Tests Server Unit Tests Continuous Delivery for

    Servers ✓ Server Integration Tests Android Stable Integration Tests ✓ ✓ ✓ ✓
  61. Continuous Delivery for Servers 1. Server developer runs unit tests

    locally
  62. Continuous Delivery for Servers 1. Server developer runs unit tests

    locally 2. On pull request, unit tests and all server and client integration tests run on a transient server instance
  63. Continuous Delivery for Servers 1. Server developer runs unit tests

    locally 2. On pull request, unit tests and all server and client integration tests run on a transient server instance 3. On merge, server code gets automatically deployed to shared QA region
  64. Pipeline Views

  65. Pipeline Jenkins Pipelines: https://jenkins.io/doc/pipeline/

  66. None
  67. Manage Scripts in Separate Repository Benefits: • Versioned with great

    tooling to view history
  68. Manage Scripts in Separate Repository Benefits: • Versioned with great

    tooling to view history • Great when you have to update multiple or similar scripts like a library version number
  69. Manage Scripts in Separate Repository Benefits: • Versioned with great

    tooling to view history • Great when you have to update multiple or similar scripts like a library version number • Great when re-building Jenkins
  70. Manage Scripts in Separate Repository Benefits: • Versioned with great

    tooling to view history • Great when you have to update multiple or similar scripts like a library version number • Great when re-building Jenkins
  71. PRO TIPS RUNNING TESTS IN PARALLEL

  72. Recap Run Component Tests Run Unit Tests Compile/Build Run Blackbox

    Tests Run Integration Tests
  73. • Split them across devices and/or executors …or both! 05

    Parallel Tests
  74. Sharding Your Tests Run Component Tests Run Unit Tests Compile/Build

    Run Blackbox Tests Run Integration Tests Blackbox Tests [0] Blackbox Tests [1] Blackbox Tests [2] Blackbox Tests [3]
  75. Parallel Tests Attempt #1

  76. None
  77. None
  78. • The main job uses up an executor while waiting

    for Phases to complete (which each use their own executor) • Phases need to do their own partitioning of the test suite • Test results are not aggregated back into a single report 05 Problems with MultiJob Plugin
  79. 05 Android Instrumentation Sharding (docs)

  80. Matrix Project Plugin

  81. None
  82. 05 Configuring The Matrix

  83. 05 Tell The Job Which Test Shard To Run

  84. • Easier to pass sharding index to build executors •

    Starts parallel jobs for all combinations of configurations 05 Benefits of Matrix Plugin
  85. Supercharging Your Matrix

  86. 05 Android Emulator Plugin • Built in support for Multi-configuration

    (matrix) jobs!
  87. 05 Android Emulator Plugin

  88. THE END

  89. ENCORE

  90. 05 Android Emulator PROS • Multiple screen sizes are easy

    • FREE! • Fast (10x faster install speed then normal devices) CONS • “Snapshot” feature is unreliable • Workaround: Re-create the AVD every time
  91. 05 Genymotion Emulator PROS • Multiple screen sizes are easy

    • Fast • Snapshot feature works pretty well CONS • Requires Licenses • Can be difficult to manage with sharding
  92. 05 Real Devices PROS • They are REAL Devices •

    There are things like Google Test Cloud, Test Droid and Amazon Device Farm CONS • You have to buy all the devices & hook them up • OR you have to pay a lot for cloud based use of devices which can be slow
  93. 05 Jenkins Architecture

  94. THAT’S ALL FOLKS