Visualize How Changes Flow to Production

When a change is made to your team's product, what has to happen for it to get deployed to production? How can you improve your deployment pipeline and shorten feedback loops and cycle time? Lisa shares some exercises and conversation starters you and your teams can use to understand your pipelines and optimize them. The talk also covers some terminology to help you talk with your team about continuous delivery and deployment as well as pipelines.

Transcript

1. @lisacrispin Lisa Crispin With material from Abby Bangser, Ashley Hunsberger, Lisi Hocke & more Copyright 2019 Lisa Crispin Visualize How Changes Flow to Production
2. @lisacrispin A little about me Lisa Crispin Testing Advocate at mabl lisa@mabl.com With Janet Gregory
3. @lisacrispin Learning intentions • Understand terminology for DevOps and continuous delivery • Visualize the flow of each new change from commit to prod deploy • Use the Test Suite Canvas for improving automated tests • Ways to shorten feedback loops
4. @lisacrispin Quality is the job of the whole team • Commitment to a level of confidence ◦ Bug prevention over bug detection ◦ Learning from production use, errors ◦ …and responding fast ◦ Focus on what’s valuable to customers • Diverse perspectives, skill sets, biases
5. @lisacrispin It’s all about: • Collaboration • Continuous improvement • Continuous learning Testing is the heart of DevOps
6. @lisacrispin 2018 State of DevOps Survey Report High performing teams have the lowest numbers for: • Deployment frequency - Multiple times/day vs. once/week or month • Lead time for changes - < 1 hour vs. between 1 & 6 months • Time to restore service - < 1 hour vs. between 1 & 6 weeks • Change failure rate - < 15% vs. 46-60%
7. @lisacrispin Imagine this scenario… • Your product has a showstopper bug in production • A pair of developers has a test to repro the bug, and a fix What has to happen for that fix to get out to production? Take a minute to sketch out how this would look for your team Commit Static Code Analysis Unit Tests …
8. @lisacrispin What does the journey look like for your change? Let’s build some common terminology and discuss ways to visualize your deployment pipeline Listens to Auto Trigger Auto Triggers Manual Trigger Code Check-in Stage 1 Stage 2 Stage ... Prod Deploy
9. @lisacrispin Continuous Integration (CI) • Integrate code into a shared repository one or more times per day • On trunk/master or branches (see https://trunkbaseddevelopment.com) • Starts off a deployment pipeline • Each check-in verified by automated test suites Trunk (or master)
10. @lisacrispin Deployment pipelines • Multiple stages designed to provide fast feedback • Central to continuous delivery & deployment • Value streams depend on consistent pipelines • Require analysis & testing by whole team to mitigate risks
11. @lisacrispin Continuous Delivery (CD)
12. @lisacrispin Continuous Deployment (also CD) Every change is deployed if build passes
13. @lisacrispin CD principles Jez Humble and David Farley, continuousdelivery.com: • Build quality in • Work in small batches • Computers perform repetitive tasks, people solve problems • Relentlessly pursue continuous improvement • Everyone is responsible
14. @lisacrispin CD (either one) without automation… Like driving at night without your headlights. It’s possible… but headlights reduce the risk. Automated regression tests are one headlight Operability – monitoring, observability, testing in prod – is the other Props to Ashley Hunsberger for the analogy
15. @lisacrispin To sum up the terms again…
16. Deployment Pipeline Developer Commit Deployed to Production Stage Stage Stage Stage Stage Stage
17. Deployment Pipeline Developer Commit Deployed to Production Code is merged (master/trunk) Stage Stage Stage Continuous Integration (CI) Stage Stage
18. Business approval Compilation/ unit testing Create deployment artifact Testing stage 1 Testing stage 2 Deployment Pipeline Developer Commit Deployed to Production Code is merged (master/trunk) Continuous Integration (CI) Continuous Delivery (CD)
19. Continuous Deployment (confusingly, this is also CD) Busines s approva l Compilation/ unit testing Create deployment artifact Testing stage 1 Testing stage 2 Deployment Pipeline Developer Commit Deployed to Production Code is merged (master/trunk) Continuous Integration (CI) Continuous Delivery (CD)
20. @lisacrispin Shortening your feedback loops Let’s look at some ways to get optimal feedback, and shorten those feedback loops Deliver Learn
21. @lisacrispin Visualize your pipeline • Try getting a cross-functional group of team members together • Devs, testers, product folks, ops • Write your pipeline steps on big stickies (real or virtual) • Arrange them on a table, wall, virtual whiteboard • Talk about it!
22. @lisacrispin How could you deliver faster? • Parallelizing steps? • New tools? More automation? Moving to the cloud? • What regression tests do you need to run for confidence?
23. @lisacrispin The Test Suite Canvas (from Ashley Hunsberger, inspired by Katrina Clokie) https://github.com/ahunsberger/TestSuiteDesign
24. @lisacrispin Let’s explore a few canvas discussion points Confidence
25. @lisacrispin A good place to start is… do we do each step in our current pipeline?
26. @lisacrispin What do we want to learn from each step? What business questions can each step in our pipeline answer? • Integration and build • Static code analysis • Automated test suites • Manual testing Who can benefit from the information? How should they be informed? What risks can we mitigate with each step?
27. @lisacrispin A few real world examples… API Test Suite • Am I getting correct responses? Are we ready to do UI testing? Static code analysis • Does the code follow our accessibility standards? Build installer testing • Does the build install without error? If not – maybe not worth further testing
28. @lisacrispin Dependencies What needs to be in place for a given step to run successfully? • Other systems • Tools • Data, environments...
29. @lisacrispin Constraints • What’s preventing us from optimizing a given step? • For example: is it a manual step we could automate? • Do we have automation, but it’s slow and flaky? • What are our known workarounds?
30. @lisacrispin Triggering each step in your pipeline • What kicks off each step in your pipeline? • Can you parallelize to shorten your feedback loop? • If one prerequisite step fails but another passes – do you run the shared next step? Or stop?
31. @lisacrispin Gates Automatically stop defects from making it any further down stream – fast feedback
32. @lisacrispin Test data How do we manage test data? • Tradeoff of speed vs. simulating production • Unit tests use test doubles - fakes, stubs, mocks • Higher level tests use fixture or canonical data ◦ which simulates prod data • Setup and teardown for each test
33. @lisacrispin Actionable alerts – avoiding “alert fatigue” When there’s a failure… Who will take responsibility to make sure it gets addressed?
35. @lisacrispin There’s more to the canvas • Use it to generate conversations • Identify biggest impediment, experiment to make it smaller • Find ways to make your tests trustworthy so you have confidence for continuous delivery • Be patient, make improvements step-by-step
36. @lisacrispin • Visualize your pipeline(s) together • Collaborate to continuously improve test suites • Requires whole team commitment, engagement • Set goals, experiment, track progress Build confidence with reliable, fast feedback
37. @lisacrispin Questions?
38. @lisacrispin A few resources • Agile Testing, More Agile Testing (Lisa and Janet) • Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation (Jez Humble and David Farley) • A Practical Guide to Testing in DevOps (Katrina Clokie) • Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations (Nicole Forsgren, Jez Humble, Gene Kim) • Test Suite Canvas: https://github.com/ahunsberger/TestSuiteDesign
39. Contact Information lisa@mabl.Com www.lisacrispin.com twitter: @lisacrispin www.agiletester.com www.agiletestingfellow.com