Continous Testing at Scale

Continuous Delivery Summit 2016 Stockholm

Transcript

1. Continuous Testing at Scale May 25th 2016, Stockholm
2. Dmitry Buzdin dmitry@buzdin.lv @buzdin www.neueda.com
3. Introduction to Continuous Testing
4. Continuous Delivery Pipeline SCM Build & Unit Test Release Test Stage Prod Develop Change Confidence
5. CD Current Focus Containers and virtualisation Build flow engines Provisioning tools
6. What does “Tests” Mean? • Just a box in a CD pipeline • Write tests in your favoured test framework and execute - and you are done!
7. Agile Testing Strategy Developers write some acceptance tests and run them.
8. Have You Considered?… environment live checks smoke testing functional testing security testing performance testing fault-tolerance testing micro-services testing browser compatibility testing mobile testing test environment preparation testing strategy test scheduling test reporting testing process traceability
9. What tend to happen is “Development only CD”
10. Development Only CD sprint 1 sprint 2 sprint 3 Version 1 Version 2 testing 1 testing 2 Manual Testing
11. Testers are lagging behind a continuous delivery development train
12. “Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” Shippable feature DEV OPS TEST https://en.wikipedia.org/wiki/Continuous_testing
13. Testers Test management Test plans Manual testing Non-functional testing Structured approach System level testing Developers Write in TDD No bureaucracy All automated tests Ad-hoc decisions Feature-level testing How do you bring both together?
14. Is Continuous Testing easy? Not at all! Some advices to follow…
15. Story Begins
16. Setting the Context • Telecom company had a software product it developed for past 15 years • Time to do a complete rewrite for 4G/5G • Continuous Delivery model selected
17. 100+ Scrum teams 1000+ Components 2000+ Git repos 20000+ Acceptance tests Everything automated 100+ Test Envs
18. Expectations • Quality statement several times a day • Testing has to be compliant to quality standards • Traceability throughout the development process
19. Traceability is the ability to verify the history, location, or application of an item by means of documented recorded identification. Boring bureaucratic stuff relevant in non-agile environments only, right?
20. Container Artefact Test Case Requirement Change Relates to Deploys container Covers Build Executes Tests Produces JIRA ID BUILD ID ARTIFACT ID SHA CONTAINER ID ???
21. Test ID Every test should have a unique test id Test case related meta-data
22. Allure Framework https://github.com/allure-framework/ Test Cases Result XMLs Aggregation Interactive Report All major test frameworks supported
23. Feature coverage by tests
24. Drill-down to test step level https://wiki.jenkins-ci.org/display/JENKINS/Allure+Plugin
25. Test Run Summary
26. CD Meets Microservices Build Test Deploy Build Test Deploy Build Test Deploy Service A Service B Service C x1000
27. Service A Service B Service C 1.0 1.1 1.2 1.3 2.1 2.2 2.3 2.4 1.12 1.13 1.14 1.15 1.16 System Version
28. System Level Testing Build Test Deploy End 2 End Tests
29. Team Testware Package Produces Contains Test Suites Test Cases Teams are producing testware packages
30. Testware 1 Testware 2 Testware 3 Test Schedule Test Executor Team A Team B Team C Testware packages are combined with help of test schedules
31. Test Schedule Team A Testware Team B Testware Team C Testware Consistency Check Smoke Testing Environment Cleanup Schedule items are coming from different teams Environment Preparation
32. https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin parallel ({ guard { build(“run-tests”, “testware1”) build(“run-tests”, “testware2”) } rescue { build(“run-tests”, “cleanup”) } }) Tests schedule produces build flow definition, which is given to Jenkins build farm Visualisation
33. Aggregation Full Execution Report Network Drive Test ware Level Reports Tests
34. How to run tests as quickly as possible to minimise the feedback loop?
35. Distributed Test Executor • Cloud based • Scale-up/scale-down • Health checks • Parallelisation
36. TE based on Swarm Plugin https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin This plugin consists of two pieces: A self-contained CLI client that discovers a nearby Jenkins (via a UDP broadcast) and joins it A plugin that needs to be installed on Jenkins master to accept swarm clients Swarm is embeddable
37. MASTER SLAVE MASTER SWARM Test Lab SLAVE SLAVE SWARM MASTER SWARM SWARM MASTER SWARM SWARM
38. Interpreting Results • 100% green tests pipeline all the time - not realistic at this scale • what is failed test severity? • is it a new test? • was it rerun several times?
39. Failed test != Sometimes
40. Testing Flow Smoke Tests Quick Regression Long Regression Full Regression Component Tests Isolated runs Environment up and running Core use-cases Important use-cases All use-cases Component Tests Component Tests After every step decision is taken if to continue testing pipeline
41. Test Result Analytics Need something like this, but with more information
42. http://neueda4ta.com/test-reporting Test Result Analytics Solution To be open sourced…
43. Achievements • Full traceability • End2End test automation • Continuous testing
44. Lessons Learned
45. There are not too many tools out there to support large scale continuous testing today
46. • Jenkins as a platform - super extensible • Saved us lots of time • May be not the most beautiful tool sometimes • But a very decent choice for building your CT flow
47. Instead of using vendor specific monolithic tools - aim for extensible independent services and interoperability standards
48. CD Standards • Not too many standards at the moment • JUnit 5 is a good example • Allure reporting API is another example
49. Next 5 years are going to be about CD tool interoperability
50. This slide intentionally left blank
51. Copyright: nisanga / 123RF Stock Photo martinm303 / 123RF Stock Photo dundanim / 123RF Stock Photo enjoyphoto / 123RF Stock Photo pedrosala / 123RF Stock Photo saranya2908 / 123RF Stock Photo kzwwsko / 123RF Stock Photo