Complexity Measurement in Software Testing

Used for in QA internal sharing session

Transcript

1. Complexity Measurement in Software Testing
2. Software Testing Metrics • Measuring progress and results of the testing is necessary (create test cases, testing, bug findings, etc…) • Often times, we can’t measure what we really care about or at least what we are expecting ◦ Ad-hoc requests ◦ Missing requirements ◦ Design changes, … • Metrics in the context of software testing is proxy
3. Defining Size of Metrics • Size is the most fundamental metrics • We can estimate the proxy that we can’t measure before based on: ◦ Tight schedule ◦ Limited resources ◦ Complexity of the features, … • Or, fundamentally we can go with the most simplest thing such as: ◦ Defect metrics ◦ Test coverage (manual or automated) • Communicate, evaluation and report is the important assessment after defined the metrics
4. Software Testing Complexity • How are we going to measure the complexity of features/programs? ◦ Analyze based on design, business process, requirements, … ◦ Experiences ◦ Rationale Intuitive • If we could measure the features/program early, we could: ◦ Focus on testing and analysis ◦ Create better schedule and plan ◦ Redesign of testing approaches, … • But, the control flow complexity in software testing is a proxy (again)
5. What does even mean with proxy? • Uncertainty • Unpredictability, Randomness • Indirect representation of the programs/systems/features • Potential of errors • How much time the programs should be tested? • Assumption or rough of ideas how we are going to test the programs/systems/features
6. Cyclomatic Complexity
7. What’s that? • One of the software testing engineering concept to measure the complexity, mapping out and identify the control flow • Technically speaking, it’s a quantitative measurement to program’s source code • Why? ◦ Designing proper test case generation ◦ Avoid redundancy and fallacy of errors ◦ Achieve test coverage :: feature correctness and robustness
8. Formulation V(g) = E - N + 2P V(g): scores of cyclomatic complexity E: number of edges or component that has a logic to execute the programs N: number of nodes or control flow to connecting between edges 2P: number of connected components
9. Scenario: Simple Login Flow • Assuming we have simplified login flow (on Maukerja) Start => Open Login Page => Input Form => Handling Input Validation => Valid Input Authentication => Success Login => Redirect to the Homepage => Exit • Number of Nodes: 6 • Number of Edges: 7 • Number of Connected Components: 1 • V(g) = 7 - 6 + 2(1) = 3 • Therefore, at least we need to test 3 scenarios
10. Scenario: Simple Login Flow w/ Independent Scenarios • Assuming we have simplified login flow (on Maukerja) Start => Open Login Page => Input Form => Handling Input Validation => Valid Input Authentication => Success Login => Redirect to the Homepage => Exit => User Retries* => Failed • Number of Nodes: 6 • Number of Edges: 9 • Number of Connected Components: 1 • V(g) = 9 - 6 + 2(1) = 5 • Therefore, at least we need to test 5 scenarios *User retries is one of the example of independent scenarios that wasn’t expected
11. Scenario: Simple Login Flow w/ Interjoint Flow • Assuming we have simplified login flow (on Maukerja) Start => Open Login Page => Input Form => Handling Input Validation => Valid Input Authentication => Success Login => Job Details* => Exit • Number of Nodes: 6 • Number of Edges: 7 • Number of Connected Components: 2 • V(g) = 7 - 6 + 2(2) = 5 • Therefore, at least we need to test 5 scenarios *flow that is not correlated with login flow, it will be considered as connecting components
12. Interpreting Complexity • If V(g) scores 1-20: ◦ Low to Moderate test scenarios ◦ Simple programs • If V(g) scores 20-50: ◦ High test scenarios ◦ Complex programs • If V(g) scores >= 50: ◦ Very high test scenarios ◦ Difficult to test, highly complex programs such as: distributed systems, network protocol, …
13. Interpreting Complexity (2) • Combined with suitable testing techniques, for instances: ◦ V(g) scores 1-20: regression (round-robin testing, priority testing), smoke testing, sanity testing ◦ V(g) scores 20-50: regression (re-test all), acceptance criteria test, heuristic testing ◦ V(g) scores +50: mutation testing, formal method, unit test, … • Make it as a base guideline metrics for OKRs, KPIs, and so on ◦ Objectives, not subjectives ◦ Data triangulation • Make it as a testing prioritization => P0, P1, P2, … • Measurement for automation testing development ◦ V(g) scores 1-20: mandatory or a must to be automated ◦ V(g) scores 20-50: depends on context => time, resources, impact ◦ V(g) scores +50: depends on context => higher scores would be led to flakiness
14. When to use it? Do • Complex programs and you want to mapping out all of the possibilities for the control flow • Designing better test case management or test plan • Proximity of test scenarios • Objectively speaking: software testing metrics Don’t (or at least Depends) • Tight schedule between release and development • Usability • Team experience • Third-party library • Specific cases of logic (such as: machine learning implementation)
15. Wrap-up • Notice that this is just a concept, in reality would be different ◦ Somehow you need to adjusted it or customized it based on your own • It matters to define the software testing metrics • Exhaustive testing isn’t possible, thus the cyclomatic complexity is feasible to implement • Cost, time, resources need to be escalated, thus the cyclomatic complexity is doable
16. Thank You