Slide 1

Slide 1 text

© 2025 All Rights Reserved API Resiliency Testing Applying Property Based Testing and Mutation Testing to APIs Hari Krishnan [email protected] @harikrishnan83

Slide 2

Slide 2 text

© 2025 All Rights Reserved Introduction OpenAPI Specifications as API Contract Tests #NOCODE

Slide 3

Slide 3 text

© 2025 All Rights Reserved Production Architecture Request Response Request Response 1 2 3 4 5 Message Topic Domain Service App BFF Analytics Service

Slide 4

Slide 4 text

© 2025 All Rights Reserved System Under Test Dependencies Test Test Mode Request Response Request Response 1 2 3 4 5 Message Topic Domain Service App BFF Analytics Service

Slide 5

Slide 5 text

© 2025 All Rights Reserved Dependencies System Under Test Test Test Mode – Contract Test Setup Request Response Request Response 1 3 4 5 Message Contract Test Http Stub Kafka mock Broker (In-memory) Schema Validation Topic BFF Set Expectations 2 Set Expectations Verify message count and schema 7 8 6

Slide 6

Slide 6 text

© 2025 All Rights Reserved Live Demo OpenAPI Specifications as API Contract Tests #NOCODE

Slide 7

Slide 7 text

© 2025 All Rights Reserved API Coverage Report Drill Down API Contract Test Report Drill Down Correlate

Slide 8

Slide 8 text

© 2025 All Rights Reserved How do we cover the 4xx scenarios?

Slide 9

Slide 9 text

© 2025 All Rights Reserved Property Based Testing Treating API Specifications as Behavior Definition

Slide 10

Slide 10 text

© 2025 All Rights Reserved Property Based Testing • Define rules or properties that your system should always satisfy • Instead of writing individual test cases • Example: “Addition is commutative: a + b = b + a.” https://scalacheck.org/ QuickCheck https://hackage.haskell.org/package/QuickCheck

Slide 11

Slide 11 text

© 2025 All Rights Reserved API Specifications as API Behavior Definition Mandatory vs Optional Nullability Only gadget, book, food or other Range

Slide 12

Slide 12 text

© 2025 All Rights Reserved Live Demo Exercising range of inputs within the boundaries set by API specification

Slide 13

Slide 13 text

© 2025 All Rights Reserved Mutation Testing Introducing mutations to your code to check the quality of your tests

Slide 14

Slide 14 text

© 2025 All Rights Reserved Mutation Testing Code (System Under Test) if (currVal == newVal) { return currVal } Test Suite if (currVal = newVal) { return currVal }

Slide 15

Slide 15 text

© 2025 All Rights Reserved Mutation Testing • Introduce small deliberate changes called mutations in your code • See if your tests can identify them • Thereby killing the mutation • Mutation Score = Mutants Killed / Total Mutants • Examples: • Java PIT • Python Mutmut • JS / TS Stryker

Slide 16

Slide 16 text

© 2025 All Rights Reserved Mutation Testing as applied API Specification • Instead of mutating code, introduce mutations in inputs. Examples: • Skip sending a mandatory field • Send the wrong data type • Expect the application to handle these spurious inputs

Slide 17

Slide 17 text

© 2025 All Rights Reserved Negative Scenario Testing • Deliberately send invalid inputs • And expect that the response is not a ”200 OK ” • Thereby validating that the input mutation has been successfully killed by input validation in your Component / Service (System Under Test)

Slide 18

Slide 18 text

© 2025 All Rights Reserved Negative Test Example – Mutating Data Type

Slide 19

Slide 19 text

© 2025 All Rights Reserved Live Demo API Resiliency Testing

Slide 20

Slide 20 text

© 2025 All Rights Reserved Issues with 500 response for invalid input • Conveys poor health of application • Thereby even bad input can lead to middleware considering service to be unavailable • Exposes unnecessary technical details

Slide 21

Slide 21 text

© 2025 All Rights Reserved What can be worse than a 500?

Slide 22

Slide 22 text

© 2025 All Rights Reserved Input validation is non-negotiable • “Insufficient Input/Output Validation” is one of the top risks listed by OWASP - https://owasp.org/www-project-mobile-top-10/2023- risks/m4-insufficient-input-output-validation • Can lead to incorrect data being introduced into your data store • Compromise other system and services that depend on this data • And many more high-risk issues

Slide 23

Slide 23 text

© 2025 All Rights Reserved Under the hood The math behind the test generation

Slide 24

Slide 24 text

© 2025 All Rights Reserved Why 77 tests for POST /products ?? Fields Data Type Required Constraints Values name string Yes - - type string Yes Enum book, food, gadget & others inventory number Yes min & max min = 1 & max = 101 Positive Variations Negative Variations Variation Count string int, bool, null 4 4 enum values int, bool, null 7 1, 1

Slide 25

Slide 25 text

© 2025 All Rights Reserved How do we cover a 503 scenario?

Slide 26

Slide 26 text

© 2025 All Rights Reserved Fault Injection Delay simulation and more in dependencies

Slide 27

Slide 27 text

© 2025 All Rights Reserved Delay simulation Stub Data

Slide 28

Slide 28 text

© 2025 All Rights Reserved

Slide 29

Slide 29 text

© 2025 All Rights Reserved Recap Why API Spec driven API Resiliency Testing is indispensable?

Slide 30

Slide 30 text

© 2025 All Rights Reserved #NOCODE #LOWCODE API Resiliency Testing API Resiliency Tests API Specification Service Implementation

Slide 31

Slide 31 text

© 2025 All Rights Reserved Thank You! Hari Krishnan [email protected] @harikrishnan83