Slide 1
Slide 1 text
CONTINUOUS
INTEGRATION
FOR ANDROID
USING JENKINS
Slide 2
Slide 2 text
05
About Us – Dan Jarvis & Sam Edwards
• Tech Leads for Capital One Wallet on Android
• Top rated finance app by a US bank
• First US bank with native Android contactless payments
• Each has 12+ years professional experience
• Passionate about automation and knowledge sharing
@HandstandSam
Slide 3
Slide 3 text
• Continuous Integration (CI) fundamentals
• Productivity tips
• Capital One Wallet case study
• Jenkins as a platform
• Pro tips - how to run tests in parallel
05
Agenda
Slide 4
Slide 4 text
CONTINUOUS
INTEGRATION
FUNDAMENTALS
Slide 5
Slide 5 text
What is CI?
Slide 6
Slide 6 text
Continuous Integration (CI) is a development practice
that requires developers to integrate code into a shared
repository several times a day. Each check-in is then
verified by an automated build, allowing teams to
detect problems early.
”
Slide 7
Slide 7 text
Why do CI?
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
Why?
• Deliver faster and higher quality
• Your product is always ready to deploy
• Capture institutional knowledge in tests
• Automation hugely improves productivity
• Multiply your impact!
Slide 10
Slide 10 text
What is Jenkins?
Slide 11
Slide 11 text
A server that executes jobs
based on triggers.
”
Slide 12
Slide 12 text
Jenkins Terminology
•Jobs
•Plugins
•Master, slaves and executors
Slide 13
Slide 13 text
Jenkins Jobs – “A Job is essentially a script”
Slide 14
Slide 14 text
Jenkins Jobs - Configuration
Slide 15
Slide 15 text
Jenkins Jobs – Execution & History
Slide 16
Slide 16 text
No content
Slide 17
Slide 17 text
No content
Slide 18
Slide 18 text
Jenkins Plugins
• Plugins can be written to extend Jenkins
capabilities
Default Install Plugins Added
Slide 19
Slide 19 text
05
Android Emulator Plugin
Slide 20
Slide 20 text
Jenkins Master, Slaves and Executors
• Jobs are configured on the master node
• The master dispatches jobs to run on
executors on slave nodes (which could
be on the same machine as the master)
Slide 21
Slide 21 text
What Makes Jenkins so Great?
• Easy, installation, configuration & a web based interface
• Open source https://github.com/jenkinsci/jenkins
• >1000 plugins
• Distributed and scalable
• Long history and widespread adoption
• Enterprise integrations
Slide 22
Slide 22 text
PRODUCTIVITY
TIPS
Slide 23
Slide 23 text
Real-time Notifications
Slide 24
Slide 24 text
No content
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
Spoon - https://github.com/square/spoon
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
Screenshots on Assertion Failure
Spoon.screenshot(activity,"test_failure");
Slide 31
Slide 31 text
Spoon & Screenshots
Slide 32
Slide 32 text
More on Spoon, Espresso and Screenshots
http://droidcon.nyc/
http://www.devfestdc.org/
Slide 33
Slide 33 text
How to Run Spoon
Slide 34
Slide 34 text
Custom Links to Spoon Results
Slide 35
Slide 35 text
Running Specific Instrumentation Tests (docs)
Run By Instrumentation Argument(s) Code
Method -e class com.FooTest#test1
Class -e class com.FooTest
Standard
Annotations
-e size small
-e size medium
-e size large
@SmallTest
@MediumTest
@LargeTest
Custom
Annotations
-e annotation
-e notAnnotation
@MyCustomAnnotation
Package –e package
Slide 36
Slide 36 text
Android Instrumentation Runner (docs)
Slide 37
Slide 37 text
Spoon Arguments
Slide 38
Slide 38 text
CASE STUDY
WALLET
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
The Testing Pyramid - Wallet Examples
• Android Espresso Tests, Real Servers
• Android Tests, Real Servers
• JVM Tests, Mock Data
• Android Component Tests, Mock Data
Black
Box Tests
Integration Tests
Component Tests
Unit Tests
Slide 41
Slide 41 text
Auto-run Tests for Every PR
Slide 42
Slide 42 text
GitHub Pull Request Builder Plugin
Slide 43
Slide 43 text
Capital One Wallet for Android on Jenkins Today
Git Commit
Pull Request (PR)
Opened or Updated
Unit Tests on PR Branch ✓
PR Merged
Assemble APKs
and Test APKs
Integration Tests
Unit Tests
Component Tests
Blackbox Tests
✓ ✓
✓
✓
✓
✓
Slide 44
Slide 44 text
JENKINS
AS A PLATFORM
Slide 45
Slide 45 text
Linked Jobs & Triggers
Slide 46
Slide 46 text
Newbie – One Job
Compile/Build
Integration Tests
Unit Tests
Component Tests
Blackbox Tests
Slide 47
Slide 47 text
Better – Four Jobs
Run Component Tests
Run Unit Tests
Compile/Build
Compile/Build Compile/Build
Run Blackbox Tests
Run Integration Tests
Compile/Build
Slide 48
Slide 48 text
Better – Four Jobs
Run Component Tests
Run Unit Tests
Compile/Build
Compile/Build Compile/Build
Run Blackbox Tests
Run Integration Tests
Compile/Build
Limitations:
• Not great if you want to repeatedly re-run the tests
• Wastes time/money if you have less than four slaves/executors
Slide 49
Slide 49 text
Better – Five Jobs
Run Component Tests
Run Unit Tests
Compile/Build
Run Blackbox Tests
Run Integration Tests
Slide 50
Slide 50 text
Configuration - Triggering
Slide 51
Slide 51 text
Better – Five Jobs
Run Component Tests
Run Unit Tests
Compile/Build
Run Blackbox Tests
Run Integration Tests
Slide 52
Slide 52 text
Configuration – Copy Build Artifacts From Other Job
Slide 53
Slide 53 text
Continuous Delivery
Slide 54
Slide 54 text
Building a Release Candidate
Compile/Build
✓
Manual Testing
and Upload
Alpha
Slide 55
Slide 55 text
What if I told you every build
that passes your automated tests
is a release candidate?
”
Slide 56
Slide 56 text
Continuous Delivery for Clients
Compile/Build
Integration Tests
Unit Tests
Component Tests
Blackbox Tests
✓
✓
✓
✓
✓
Alpha
Slide 57
Slide 57 text
Continuous Delivery for All
Slide 58
Slide 58 text
Server
Unit Tests
Continuous Delivery for Servers
✓
✓
Server
Integration Tests
✓
Slide 59
Slide 59 text
Your automated client tests are
now an API.
”
Slide 60
Slide 60 text
iOS Stable
Integration Tests
Server
Unit Tests
Continuous Delivery for Servers
✓
Server
Integration Tests
Android Stable
Integration Tests
✓
✓
✓
✓
Slide 61
Slide 61 text
Continuous Delivery for Servers
1. Server developer runs unit tests locally
Slide 62
Slide 62 text
Continuous Delivery for Servers
1. Server developer runs unit tests locally
2. On pull request, unit tests and all server and client integration tests run
on a transient server instance
Slide 63
Slide 63 text
Continuous Delivery for Servers
1. Server developer runs unit tests locally
2. On pull request, unit tests and all server and client integration tests run
on a transient server instance
3. On merge, server code gets automatically deployed to shared QA
region
Slide 64
Slide 64 text
Pipeline Views
Slide 65
Slide 65 text
Pipeline
Jenkins Pipelines: https://jenkins.io/doc/pipeline/
Slide 66
Slide 66 text
No content
Slide 67
Slide 67 text
Manage Scripts in Separate Repository
Benefits:
• Versioned with great tooling to view history
Slide 68
Slide 68 text
Manage Scripts in Separate Repository
Benefits:
• Versioned with great tooling to view history
• Great when you have to update multiple or similar scripts like a library
version number
Slide 69
Slide 69 text
Manage Scripts in Separate Repository
Benefits:
• Versioned with great tooling to view history
• Great when you have to update multiple or similar scripts like a library
version number
• Great when re-building Jenkins
Slide 70
Slide 70 text
Manage Scripts in Separate Repository
Benefits:
• Versioned with great tooling to view history
• Great when you have to update multiple or similar scripts like a library
version number
• Great when re-building Jenkins
Slide 71
Slide 71 text
PRO TIPS
RUNNING TESTS
IN PARALLEL
Slide 72
Slide 72 text
Recap
Run Component Tests
Run Unit Tests
Compile/Build
Run Blackbox Tests
Run Integration Tests
Slide 73
Slide 73 text
• Split them across devices
and/or executors
…or both!
05
Parallel Tests
Slide 74
Slide 74 text
Sharding Your Tests
Run Component Tests
Run Unit Tests
Compile/Build
Run Blackbox Tests
Run Integration Tests
Blackbox Tests [0]
Blackbox Tests [1]
Blackbox Tests [2]
Blackbox Tests [3]
Slide 75
Slide 75 text
Parallel Tests Attempt #1
Slide 76
Slide 76 text
No content
Slide 77
Slide 77 text
No content
Slide 78
Slide 78 text
• The main job uses up an executor while waiting for Phases to
complete (which each use their own executor)
• Phases need to do their own partitioning of the test suite
• Test results are not aggregated back into a single report
05
Problems with MultiJob Plugin
Slide 79
Slide 79 text
05
Android Instrumentation Sharding (docs)
Slide 80
Slide 80 text
Matrix Project Plugin
Slide 81
Slide 81 text
No content
Slide 82
Slide 82 text
05
Configuring The Matrix
Slide 83
Slide 83 text
05
Tell The Job Which Test Shard To Run
Slide 84
Slide 84 text
• Easier to pass sharding index to build executors
• Starts parallel jobs for all combinations of configurations
05
Benefits of Matrix Plugin
Slide 85
Slide 85 text
Supercharging Your Matrix
Slide 86
Slide 86 text
05
Android Emulator Plugin
• Built in support for
Multi-configuration (matrix)
jobs!
Slide 87
Slide 87 text
05
Android Emulator Plugin
Slide 88
Slide 88 text
THE
END
Slide 89
Slide 89 text
ENCORE
Slide 90
Slide 90 text
05
Android Emulator
PROS
• Multiple screen sizes are easy
• FREE!
• Fast (10x faster install speed then normal devices)
CONS
• “Snapshot” feature is unreliable
• Workaround: Re-create the AVD every time
Slide 91
Slide 91 text
05
Genymotion Emulator
PROS
• Multiple screen sizes are easy
• Fast
• Snapshot feature works pretty well
CONS
• Requires Licenses
• Can be difficult to manage with sharding
Slide 92
Slide 92 text
05
Real Devices
PROS
• They are REAL Devices
• There are things like Google Test Cloud, Test Droid and Amazon
Device Farm
CONS
• You have to buy all the devices & hook them up
• OR you have to pay a lot for cloud based use of devices which
can be slow
Slide 93
Slide 93 text
05
Jenkins Architecture
Slide 94
Slide 94 text
THAT’S ALL FOLKS