Continuous Testing

Shift Left and Shift Right in Testing, Testing in Production, DevOps

1. @y3key linkedin.com/in/stefan-friese-12205 05/31/2018 https://speakerdeck.com/y3key/ continuous-testing
2. Presentation title | Month 00, 2016 2 © 2016 HERE | HERE Internal Use Only Unit + Component Tests API / Service Layer Tests Service-integration Tests GUI Layer Tests Test Pyramid Exploratory Tests
3. Presentation title | Month 00, 2016 4 © 2016 HERE | HERE Internal Use Only Let’s turn it upside down …
4. Presentation title | Month 00, 2016 5 © 2016 HERE | HERE Internal Use Only [More Information: Search for “Ice Cream Anti-Pattern”] Unit Tests Service-level Tests Service-integration Tests E2E Tests
5. Presentation title | Month 00, 2016 6 © 2016 HERE | HERE Internal Use Only Increased test debt  Test debt occurs due to shortcuts related to testing. It is the test dimension of technical debt.
6. Presentation title | Month 00, 2016 7 © 2016 HERE | HERE Internal Use Only The right ratio of tests – with many unit tests as basis - is important! But how can we mature the test execution?
7. Presentation title | Month 00, 2016 8 © 2016 HERE | HERE Internal Use Only Let us return to the Test Pyramid …
8. Presentation title | Month 00, 2016 9 © 2016 HERE | HERE Internal Use Only … but let’s change the Perspective
9. Presentation title | Month 00, 2016 10 © 2016 HERE | HERE Internal Use Only Continuous Integration & Continuous Delivery Faster Feedback More Stable Less Expensive to fix Later Feedback More Fragile More Expensive to fix commit deploy & release [More Information: Search for “CI&CD”, “Shift Left Testing”]
10. Presentation title | Month 00, 2016 12 © 2016 HERE | HERE Internal Use Only The Shift Left is essential. However, we can’t replace all long running tests with unit tests. How can we reduce the test debt further?
11. Presentation title | Month 00, 2016 14 © 2016 HERE | HERE Internal Use Only development stable increased confidence
12. Presentation title | Month 00, 2016 15 © 2016 HERE | HERE Internal Use Only Shift Right commit deploy incremental release & monitoring [More Information: Search for “Test in Production”]
13. Presentation title | Month 00, 2016 16 © 2016 HERE | HERE Internal Use Only We talked about Test Debt: Why should I test in Production?
14. Presentation title | Month 00, 2016 17 © 2016 HERE | HERE Internal Use Only How well can you predict how your application will be used in production?
15. Presentation title | Month 00, 2016 18 © 2016 HERE | HERE Internal Use Only Let us return to the Test Pyramid …
16. Presentation title | Month 00, 2016 19 © 2016 HERE | HERE Internal Use Only … but let’s change (again) the Perspective
17. Presentation title | Month 00, 2016 20 © 2016 HERE | HERE Internal Use Only Bug Filter (Semantic) Monitoring Some Integration and E2E Tests small risk high risk that bugs are not detected Many Unit Tests! Comprehensive Acceptance Test Set Incremental Release A/B Tests Alpha/Beta Releases Controlled Experiments (e.g. “ChaosMonkey”) Contract Tests Pre- Production Tests Production Tests Based on: https://t.co/ereKYJzyzM
18. Presentation title | Month 00, 2016 21 © 2016 HERE | HERE Internal Use Only Based on: https://danashby.co.uk/2016/10/19/continuous-testing-in-devops Dev Ops Test Here Test Here Test Here
19. Presentation title | Month 00, 2016 22 © 2016 HERE | HERE Internal Use Only That’s Continuous Testing ☺
20. Presentation title | Month 00, 2016 23 © 2016 HERE | HERE Internal Use Only Shadow Deployment Tests* Load Tests Canary Release* Feature Flagging Monitoring A/B Test – lean experimentation Alpha/Beta Tests Chaos Engineering Synthetic Monitoring Pre-Release Production Traffic Tests Release Tests in Production Post-Release Tests in Production See: https://www.infoq.com/articles/architecturally-aligned-testing
21. Presentation title | Month 00, 2016 24 © 2016 HERE | HERE Internal Use Only production instances test instance production- like instance frontend end users load balancer See: https://www.infoq.com/articles/performance-tests Shadow Deployment
22. Presentation title | Month 00, 2016 26 © 2016 HERE | HERE Internal Use Only production instances frontend end users load balancer new version (v1.1) current release (v1.0) current release (v1.0) current release (v1.0) canary instance
23. Presentation title | Month 00, 2016 27 © 2016 HERE | HERE Internal Use Only Shift in Mindset Shift Left • Early and fast feedback. • Comprehensive set of automated pre-production tests. Shift Right • Release often and early. • Expose new features early in a controlled way. • Execute some of the tests in Production.
24. Presentation title | Month 00, 2016 28 © 2016 HERE | HERE Internal Use Only More information: • Dan Ashby (3 May 2018): A Better Testing Pyramid - https://danashby.co.uk/2018/05/03/a-better-testing-pyramid • Ham Vocke (26 Feb 2018): The Practical Test Pyramid - https://martinfowler.com/articles/practical-test-pyramid.html • Noah Sussman (28 Feb 2017): Abandoning the Pyramid Of Testing in favor of a Band- Pass model of risk management - https://twitter.com/noahsussman/status/836612175707930625 • Katrin Clockie (2017): A Practical Guide to Testing in DevOps. Leanpub. • Stefan Friese (2016): Testing the Unexpected: A Shift Right in DevOps Testing. https://www.stickyminds.com/article/testing-unexpected-shift-right-devops-testing • Stefan Friese (2018): https://speakerdeck.com/y3key/continuous-testing
25. Presentation title | Month 00, 2016 29 © 2016 HERE | HERE Internal Use Only