Test in DevOps

Disclaimer: This presentation is to share concepts described in this awesome book for a internal study group and most materials are originated from it. You can find the book [here].

BACKGROUND

• Sole ownership of test strategy and execution • Thinking up our own test ideas • Exploring the SW independently Waterfall

• Testing is an activity own by whole development team • Discuss requirement with DEV & BA thru BDD collaboratively • Acceptance testing in development process • Fast feedback by iterative release with pre-release checks • It changes the scope and pace of testing Agile

• Team manages SW lifecycle: operation infra and monitoring, customer support and analytics • Test in production-like env: docker, k8s • Rapid customer feed back from metrics, beta testing groups, A/B testing • Real-time monitoring dashboards DevOps

• Fostering collaborative culture to achieve rapid and reliable release • Testers struggle to find balance of new feature exploring without slowing down the release • Automation is really a solution to this ?? DevOps is Speed

DevOps is a term used to refer to a set of practices that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably. Definition of DevOps

• Focus on developer and operations • Testing is absence from the the term • Testing is a pervasive activity throughout development (like the air) DevOps and Testing image: Dan Ashby [https://danashby.co.uk/2016/10/19/continuous- testing- in- devops/]

• What exactly is your responsibility? • How should you try to contribute? • Still benefit in distinguishing testing tasks from other types of work? • What weighting should you give to testing? Beyond the Testing Role Q: The difficulty for testers is identifying what their role is, when testing is expected to be a part of everything.

TESTING IN A DEVOPS CULTURE

Test Strategy Retrospective

No content

Collaboration Beyond Development

TESTING IN DEVELOPMENT

• Unit test, integration tests, e2e tests (confirm behaviour) • Non-Functional behaviors: accessibility, security and performance • But no deep or thoughtful insight provided from these tests Feedback in Development - Automation

• Searching for the unknowns • Adopting different personas • …… • Generate ideas, process results, and form opinions Feedback in Development - Exploration image: https://findingdeefex.com/2016/05/20/the-testing-checking-synergy/

What we do in DevOps (shift right) • Testing in production: A/B testing, beta testing, monitoring as testing • Use feature toggles • People who perform testing will grow: bug bashes, crowd testing Test Practices in Development

Test Practices in Dev - Deployment Pipelines Before (dedicated env): After (cloud env):

Think about how your pipeline might fail: • What if pipeline fails between steps / commands, any impact? • Test how to recover/rollback of sth went wrong? • Be more careful if steps target the production How do You Test a Pipeline

Test Practices in Dev - Feature Toggles

Test Practices in Dev - Feature Toggles

Questions: • switch the toggle off -> safety net -> but how do users feel if premium features disappear? • Can I easily determine current toggle status and audit information • Are there active toggles that should be removed? • is the toggle needed at all? Test Practices in Dev - Feature Toggles

A bug bash is an activity where all the people involved in the delivery of software put aside their usual work and participate in a dedicated testing session. Test Practices in Dev - Bug Bash

Questions: If the team are releasing multiple times per day there may not be a window of opportunity to explore a fixed release, still value to bug bash? Test Practices in Dev - Bug Bash

TESTING IN PRODUCTION

Testing monitoring and alerting • When configuring a new monitor, ideally you will test that it works by causing an issue that the monitor should detect. • If you have configured a monitor that generates an alert, you may wish to test that the alert is delivered successfully. Feedback in Production - Monitoring and alerting

No content

No content

What is Analytics? • In a web application: page views • In a telephone network: call durations • In a banking product: transaction volumes Feedback in Production - Analytics Events (Data) Organizations collect all sorts of data to learn about their users’ behaviour and respond to it. Be mindful of what is recorded, and work closely with the analytics team to determine what’s most appropriate to track.

Testing analytics: Testing of analytics will generally focus on the design of information rather than the specific behaviour of individual pieces of data. Ex: should you record ‘quick transfer’ and ‘payments’ actions as separate events? Should you record web and mobile actions separately? Beta and A/B experiments are supported by analytics information. Fast feedback may appear through monitoring, but it is often through deeper analysis of analytics data that final decisions are made. Feedback in Production - Analytics Events (Data)

Developer consider • What type of information are valuable to log • How to categorize messages to: errors, warning, info and debug Testing logging A bug in development can be a good opportunity to test log files. Any problems should create appropriate log messages. Feedback in Production - Logging

Given I run a scenario as a Lawyer And I create a document [And I wait 5 seconds] When I search the logs API Then I should find a recent entry for “AuditRecordCreated” Feedback in Production - Logging

Test practices in production • Traditional development: testing in production after a large release, rarely generates new work • In DevOps: testing in production continuously providing feedback that shapes future iterations Test Practices in Production

Testing and monitoring have traditionally served different purposes. Monitoring as testing blurs these differences, pulling ideas from testing into creating and testing monitors. Instead of waiting passively for a user to interact with a feature, run test automation to generate events in production. You may discover and resolve a bug in production before a user experiences it. Passive validation gives insight into how users behave, by monitoring information for real interactions. Active validation will generate observable data by executing known repeatable scenarios. Test Practices in Production - Monitoring as Testing

• Canary release • Staged rollout • Dogfooding • Dark Launching Testing in Production - Exposure Control

TESTING IN DEVOPS ENVIRONMENTS

Platform evolution (container-based, cloud) Infrastructure as code • VM, OS version, network topology • Vagrant, Terraform • Version controlled Configuration management • Puppet, Chef, Ansible, SaltStack … • Handles complexity of configuring distributed infrastructure resources Container • …… Testing in DevOps Environments [Note] ex: dev can run MySQL, server app on local desktop

Platform evolution (container- based, cloud) Cloud • …… Testing in DevOps Environments [Note] ex: dev can run MySQL, server app on local desktop

Testing Practices for Environments

• Servers are a commodity • Test automation can provision a fresh environment • Reduce “it works on my machine” • More environments (beta, staging, production) have a negative impact - Different versions deployed in diff environments - Data integrity problem - Remember that production env is the ultimate destination Testing Practices - Environments Management

Testing Practices - Environments Management

When infrastructure becomes code : • Static analysis, unit tests, and acceptance tests can all be applied to infrastructure. • Test Kitchen is a tool for automated testing of configuration management code. Testing Practices - Infrastructure Testing Once environments are no longer precious, destructive testing becomes easier: • Kill a network interface • Stop database processes • Flood a web services port • Restart the environment unexpectedly Testing Practices - Destructive Testing

Testing Practices - Destructive Testing

INDUSTRY EXAMPLES

The Guardian - Testing in Production (2016)

Etsy track everything: “number of logins, number of login errors, dollars of transactions through the site, bugs reported on the forum – you name it” Etsy - Monitoring as Testing

TEST STRATEGY IN DEVOPS

Test Strategy in DevOps

TestOps environment, how day-to-day work of a tester might change: 1. The tester moves away from up-front testing and execution of functional test cases. This responsibility shifts towards the developer who “must use every available means to produce code free of low context bugs that can be found by running on his desktop” 2. The tester creates tools that can run tests in the production environment, which may look a lot like monitors. “The value test brings is in moving these monitors from heartbeats and simple scenarios to high context user affecting scenarios” Testing vs Tester

How DevOps is Killing QA – Asaf Yigal “In the end, if there is a layer of people in the middle between development and operations, then you cannot perform CI and CD seamlessly and, therefore, cannot do DevOps. To run a proper DevOps operation, you cannot have QA at all.” Testing vs Tester

DevOps presents a lot of opportunities to think differently about testing. Creating connections between the development and operations teams can influence test coverage, test strategy, and even the infrastructure where testing occurs. The degree of change in your organization will depend on the business drivers for DevOps and the appetite for risk. Conclusion

[Introduced in this book:] Test Strategy Retrospective workshop Risk workshop A structure way while paring Workshops & Collaboration

Q & A

source from: https://www.slideshare.net/noahsussman/continuous-improvement-devops-day-mountain-view-2012

No content

No content