Continous Testing at Scale

Slide 1

Slide 1 text

Continuous Testing at Scale May 25th 2016, Stockholm

Slide 2

Slide 2 text

Dmitry Buzdin [email protected] @buzdin www.neueda.com

Slide 3

Slide 3 text

Introduction to Continuous Testing

Slide 4

Slide 4 text

Continuous Delivery Pipeline SCM Build & Unit Test Release Test Stage Prod Develop Change Conﬁdence

Slide 5

Slide 5 text

CD Current Focus Containers and virtualisation Build ﬂow engines Provisioning tools

Slide 6

Slide 6 text

What does “Tests” Mean? • Just a box in a CD pipeline • Write tests in your favoured test framework and execute - and you are done!

Slide 7

Slide 7 text

Agile Testing Strategy Developers write some acceptance tests and run them.

Slide 8

Slide 8 text

Have You Considered?… environment live checks smoke testing functional testing security testing performance testing fault-tolerance testing micro-services testing browser compatibility testing mobile testing test environment preparation testing strategy test scheduling test reporting testing process traceability

Slide 9

Slide 9 text

What tend to happen is “Development only CD”

Slide 10

Slide 10 text

Development Only CD sprint 1 sprint 2 sprint 3 Version 1 Version 2 testing 1 testing 2 Manual Testing

Slide 11

Slide 11 text

Testers are lagging behind a continuous delivery development train

Slide 12

Slide 12 text

“Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” Shippable feature DEV OPS TEST https://en.wikipedia.org/wiki/Continuous_testing

Slide 13

Slide 13 text

Testers Test management Test plans Manual testing Non-functional testing Structured approach System level testing Developers Write in TDD No bureaucracy All automated tests Ad-hoc decisions Feature-level testing How do you bring both together?

Slide 14

Slide 14 text

Is Continuous Testing easy? Not at all! Some advices to follow…

Slide 15

Slide 15 text

Story Begins

Slide 16

Slide 16 text

Setting the Context • Telecom company had a software product it developed for past 15 years • Time to do a complete rewrite for 4G/5G • Continuous Delivery model selected

Slide 17

Slide 17 text

100+ Scrum teams 1000+ Components 2000+ Git repos 20000+ Acceptance tests Everything automated 100+ Test Envs

Slide 18

Slide 18 text

Expectations • Quality statement several times a day • Testing has to be compliant to quality standards • Traceability throughout the development process

Slide 19

Slide 19 text

Traceability is the ability to verify the history, location, or application of an item by means of documented recorded identiﬁcation. Boring bureaucratic stuff relevant in non-agile environments only, right?

Slide 20

Slide 20 text

Container Artefact Test Case Requirement Change Relates to Deploys container Covers Build Executes Tests Produces JIRA ID BUILD ID ARTIFACT ID SHA CONTAINER ID ???

Slide 21

Slide 21 text

Test ID Every test should have a unique test id Test case related meta-data

Slide 22

Slide 22 text

Allure Framework https://github.com/allure-framework/ Test Cases Result XMLs Aggregation Interactive Report All major test frameworks supported

Slide 23

Slide 23 text

Feature coverage by tests

Slide 24

Slide 24 text

Drill-down to test step level https://wiki.jenkins-ci.org/display/JENKINS/Allure+Plugin

Slide 25

Slide 25 text

Test Run Summary

Slide 26

Slide 26 text

CD Meets Microservices Build Test Deploy Build Test Deploy Build Test Deploy Service A Service B Service C x1000

Slide 27

Slide 27 text

Service A Service B Service C 1.0 1.1 1.2 1.3 2.1 2.2 2.3 2.4 1.12 1.13 1.14 1.15 1.16 System Version

Slide 28

Slide 28 text

System Level Testing Build Test Deploy End 2 End Tests

Slide 29

Slide 29 text

Team Testware Package Produces Contains Test Suites Test Cases Teams are producing testware packages

Slide 30

Slide 30 text

Testware 1 Testware 2 Testware 3 Test Schedule Test Executor Team A Team B Team C Testware packages are combined with help of test schedules

Slide 31

Slide 31 text

Test Schedule Team A Testware Team B Testware Team C Testware Consistency Check Smoke Testing Environment Cleanup Schedule items are coming from different teams Environment Preparation

Slide 32

Slide 32 text

https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin parallel ({ guard { build(“run-tests”, “testware1”) build(“run-tests”, “testware2”) } rescue { build(“run-tests”, “cleanup”) } }) Tests schedule produces build ﬂow deﬁnition, which is given to Jenkins build farm Visualisation

Slide 33

Slide 33 text

Aggregation Full Execution Report Network Drive Test ware Level Reports Tests

Slide 34

Slide 34 text

How to run tests as quickly as possible to minimise the feedback loop?

Slide 35

Slide 35 text

Distributed Test Executor • Cloud based • Scale-up/scale-down • Health checks • Parallelisation

Slide 36

Slide 36 text

TE based on Swarm Plugin https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin This plugin consists of two pieces: A self-contained CLI client that discovers a nearby Jenkins (via a UDP broadcast) and joins it A plugin that needs to be installed on Jenkins master to accept swarm clients Swarm is embeddable

Slide 37

Slide 37 text

MASTER SLAVE MASTER SWARM Test Lab SLAVE SLAVE SWARM MASTER SWARM SWARM MASTER SWARM SWARM

Slide 38

Slide 38 text

Interpreting Results • 100% green tests pipeline all the time - not realistic at this scale • what is failed test severity? • is it a new test? • was it rerun several times?

Slide 39

Slide 39 text

Failed test != Sometimes

Slide 40

Slide 40 text

Testing Flow Smoke Tests Quick Regression Long Regression Full Regression Component Tests Isolated runs Environment up and running Core use-cases Important use-cases All use-cases Component Tests Component Tests After every step decision is taken if to continue testing pipeline

Slide 41

Slide 41 text

Test Result Analytics Need something like this, but with more information

Slide 42

Slide 42 text

http://neueda4ta.com/test-reporting Test Result Analytics Solution To be open sourced…

Slide 43

Slide 43 text

Achievements • Full traceability • End2End test automation • Continuous testing

Slide 44

Slide 44 text

Lessons Learned

Slide 45

Slide 45 text

There are not too many tools out there to support large scale continuous testing today

Slide 46

Slide 46 text

• Jenkins as a platform - super extensible • Saved us lots of time • May be not the most beautiful tool sometimes • But a very decent choice for building your CT ﬂow

Slide 47

Slide 47 text

Instead of using vendor speciﬁc monolithic tools - aim for extensible independent services and interoperability standards

Slide 48

Slide 48 text

CD Standards • Not too many standards at the moment • JUnit 5 is a good example • Allure reporting API is another example

Slide 49

Slide 49 text

Next 5 years are going to be about CD tool interoperability

Slide 50

Slide 50 text

This slide intentionally left blank

Slide 51

Slide 51 text

Copyright: nisanga / 123RF Stock Photo martinm303 / 123RF Stock Photo dundanim / 123RF Stock Photo enjoyphoto / 123RF Stock Photo pedrosala / 123RF Stock Photo saranya2908 / 123RF Stock Photo kzwwsko / 123RF Stock Photo