How to Break Software: The art of continuous testing on Azure.

Presented by Suhas Roa at the Global Azure Bootcamp 2019 in Sydney.

Are you curious to know why Microsoft merged its SDE and SDET positions into the software engineer title? As a developer how confident are you about writing code without any bugs? Join me for session on how Azure and DevOps is changing the way testing is done. Understand how quality unit tests to automated functional tests, all integrated with your CI/CD pipelines can help you ship your bug-free code faster!

Transcript

1. Suhas Rao Cloud Solution Architect App Dev Microsoft @suhasaraos suhasaraos How to break software: The art of Continuous Testing Images are copyrighted to their respective creators
2. Books that inspired me…
3. Whose responsibility is software testing? A) Developers B) Business Analysts/Product Owners/Stakeholders C) Quality Analysts (QA)/Test Engineers D) End Users
4. Pace of Business today “We are implementing features and products and using technology that were not invented 18 months ago. No longer can we afford these large monolithic programs that go on for two to three years. We know that what we set out to do at the beginning of that time is not what we will finish out doing. So, we are focusing on very rapid delivery cycles, asking ourselves: How do we mobilize a project very quickly? How do we use the right delivery techniques to work through it quickly? How do we get product into market or to customers or into the business?” - Bronwyn Clere, Executive Director for Capital Planning & Delivery, Telstra Corporation
5. Every company is a software company “Every company is a software company. You have to start thinking and operating like a digital company. It’s no longer just about procuring one solution and deploying one. It’s not about one simple software solution. It’s really you yourself thinking of your own future as a digital company.” Satya Nadella, CEO of Microsoft Source: https://www.skyhighnetworks.com/cloud-security-blog/every-company-is- a-software-company-today/
6. Complexity of Software • Let’s play a game. 2 coins = {HH, HT, TH, TT} 1 coins = {H, T} 3 coins = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT} How many coins for 1000 arrangements? • Software complexity grows in much the same way. • Making a small addition can be rather simple, but can result in an exponential increase in complexity on the entire program. Ex: Suppose you have a settings page with five settings, each setting having two possible states. That’s 32 possible configurations. Adding a sixth setting will result in 64 possible setting configurations. • Its imperative that each one of these configurations must be tested to ensure everything functions as intended.
7. Applications are not in isolation anymore! Can you accurately determine the impact of your change – to all the applications beyond your own?
9. https://www.tricentis.com/resources/software-fail-watch-5th-edition/
10. Are we getting better at testing?
11. Testing is indeed changing… • The responsibility for testing is now being shifted leftward, closer to the beginning of the software development lifecycle and then all the way along it. • Knocking down the wall that has traditionally separated developers and testers, making quality everyone’s responsibility.
12. • 63% of software development delays occur in Test-QA practices across the lifecycle. • 56% of critical dependencies are unavailable • 50% of time is spent looking for test data • 64% of defects occur in the requirements phase • 70% of testing is still manual. Testing is indeed changing…
13. What is continuous testing? • Continuous testing is the practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected. • Continuous testing provides quality gates throughout the pipeline and increases confidence in code long before production. • Automated unit, integration, coded UI, and load tests are common in continuous testing. • Depth of testing often progresses as the environment gets closer to production. • In practice, continuous testing is the most difficult part of a continuous delivery pipeline to keep up to date.
14. School of Continuous Testing • Twelve Disciplines • Examples based on Microsoft Technologies
15. Requirements Engineering & Acceptance Criteria • Include all SDLC stakeholders into the continuous testing journey. • Must utilize better ways to communicate and collaborate on the requirements rather than just emails/word docs. • Take the acceptance criteria for each user story and create the tests to ensure each of those are met. • Ensure test steps to acceptance criteria traceability. • Define detailed acceptance criteria, which requires tests to also be more comprehensive.
16. Unit Tests • The method of testing specific functions and areas (or units) of your code. • Select right tools: • Unit test Framework • Isolation Framework • Code coverage tools • Test Explorer
17. On demand Environments • More Frequent Testing • Creation and teardown environments on demand • Co-ordinate testing through multiple environment owners that may not always have environments up and running (or required versions of it) • Use Cloud and Infra as Code to avoid bottlenecks.
18. Test & Test Data Management • Right set of tests and data • Appropriate diversity of data for positive and negative scenarios • Synthetic data generation is the way to go to achieve continuous testing with the highest levels of confidence.
19. API Testing • Enable alignment of the testing pyramid. • Strive to test as much at the Unit and API levels as possible and minimize reliance on UI testing Test Automation Pyramid
20. Performance/Load Testing • Use the power and scale of Cloud to generate load. • Many free and commercial tools available (Like Apache JMeter) • Note : • Web performance and load test functionality in Visual Studio is deprecated. • Cloud load testing (CLT) functionality in Azure DevOps is deprecated.
21. Security Testing • Application security area is always evolving, its important to stay updated. • Setup security alerts for vulnerable dependencies. • Setup Pipelines to run static analysis, dynamic analysis and software composition analysis • Security testing needs to happen at every step of the lifecycle. • Automate as much of Pen testing as you can in the release pipelines using tools like OWASP ZAP.
22. Security Testing https://docs.microsoft.com/en-us/azure/devops/migrate/security-validation-cicd-pipeline?view=azure-devops
23. Test Automation • When you think test early and test often by using automated testing, you essentially build a robust testing architecture that helps produce higher- quality software. • Follow the Pyramid model to Automate UI, Unit and API tests. • Tools like Selenium can be used for automating UI Tests.
24. Automated Test Generation • Use tools like IntelliTest to explore your .NET code to generate test data and a suite of unit tests. • For every statement in the code, a test input is generated that will execute that statement. • A case analysis is performed for every conditional branch in the code. • Good starting point –must build on it.
25. Pipeline Orchestration • It must be integrated with your automation suite. • Understand how it works, how to interpret results and how to make it scalable. • Must be transparent and everyone must have full visibility into what’s being run through the pipeline.
26. Deployments • A/B Testing (Testing in Production) • A randomized experiment with two variants, A and B • Azure App Service supports Slots to run two or more variants of app simultaneously • Blue-Green Deployments • The staging slot represents “green” deployment and production slot represents “blue” deployment. • Canary Releases (Deployment Rings) • Reduces the risk of introducing a new software version in production by gradually rolling out the change to a small subgroup of users, before rolling it out to everyone
27. Continuous Monitoring • Continuous monitoring across the entire SDLC, not just production, should be used as a compass to help you navigate your continuous testing transformation. • Dashboards in real time. Must be automatic and entire team must have access. • Use automated runners to identify issues that do not manifest on initial deploy & occur after certain time.
28. 12 Key Disciplines of Continuous Testing (Recap) 1. Requirements Engineering and Acceptance Criteria 2. Unit tests 3. On-Demand Environments 4. Test and Test Data Management 5. API Management 6. Performance/Load Testing 7. Security Testing 8. Test Automation 9. Automated Test Generation 10. Pipeline Orchestration 11. Deployments 12. Continuous Monitoring
29. Azure DevOps
30. Azure DevOps Deliver value to your users faster using proven agile tools to plan, track, and discuss work across your teams. Build, test, and deploy with CI/CD that works with any language, platform, and cloud. Connect to GitHub or any other Git provider and deploy continuously. Get unlimited, cloud-hosted private Git repos and collaborate to build better code with pull requests and advanced file management. Test and ship with confidence using manual and exploratory testing tools. Create, host, and share packages with your team, and add artifacts to your CI/CD pipelines with a single click. Azure Boards Azure Repos Azure Pipelines Azure Test Plans Azure Artifacts https://azure.com/devops ➔
31. Get end-to-end traceability. Run tests and log defects from your browser. Track and assess quality throughout your testing lifecycle. Azure Test Plans Capture rich data Capture rich scenario data as you execute tests to make discovered defects actionable. Explore user stories without test cases or test steps. You can create test cases directly from your exploratory test sessions. Test across web and desktop Test your application where it lives. Complete scripted tests across desktop or web scenarios. Test on-premises application from the cloud and vice- versa. Get end-to-end traceability Leverage the same test tools across your engineers and user acceptance testing stakeholders. Pay for the tools only when you need them. https://azure.com/devops ➔
32. Self-Service Dev/Test Environments Azure Lab Services Simplify cloud environment management for developers and testers. Enforce policies and control costs with full visibility Use templates, custom images and formulas to reproduce environments. Orchestrate with Azure Pipelines or integrate using REST API
33. Infrastructure and Configuration as Code Azure Resource Manager, Automation & 3rd Party Integrations Infrastructure as Code, built-in Support for 3rd party and OSS tooling such as Terraform, Ansible, Chef, Puppet & SaltStack Azure Config & Automation
34. Gain full visibility and control of your cloud security state Continuous Security Azure Security Center Leverage ML to Proactively identify and mitigate risks to reduce exposure to attacks Quickly detect and respond to threats with advanced analytics
35. Smarter Insights, Faster Azure Monitor, Application Insights & Log Analytics Pre-defined solutions with smart thresholds Visualize data in intuitive and customizable dashboards Separate the signal from the noise and accelerate root-cause analysis Integrate your existing processes & tools like Service Now
36. Mix and match to create workflows with tools from Microsoft, open source or your favorite 3rd party tools Azure DevOps: Choose the tools and clouds you love Azure DevOps lets developers choose the tools that are right for them Target any cloud, on-prem or both and deploy to the servers you need
37. Summary • Learning the art of testing is like studying for a Ph.D.; the more you learn, the more you realize there is much more to know. • Testing must become integral to the development process (rather than a “hygiene” task tacked on when development is complete) • Don’t re-invent the wheel, rather build on it - what kind of patterns to adopt and where to look for faults/bugs. • Determine the right tests to execute at different stages of the delivery pipeline (smoke testing upon check-in, API/message layer testing after integration, end-to-end testing at the system level, …) • Each set of tests must execute fast enough that it does not create a bottleneck at the associated stage of the software delivery pipeline.
38. © Microsoft Corporation Lets break the software before it breaks in production!
39. © Microsoft Corporation Questions? Please provide Feedback Or aka.ms/suhasr2019