API and Data Mocking for Python

Slide 1

Slide 1 text

12.12.2023, Lausanne Testcontainers Faker Synthesized

Slide 2

Slide 2 text

2 * Testcontainers/AtomicJar is a WireMock partner

Slide 3

Slide 3 text

Outline ● Intro to API integration testing ● Testcontainers and WireMock ● Using them together ● And Python examples! (mostly) 3 Slides

Slide 4

Slide 4 text

> whoami @oleg_nenashev oleg-nenashev Dr. Nenashev / Mr. Jenkins Developer tools hacker Community builder & DevRel consultant #RussiansAgainstPutin #StandWithUkraine

Slide 5

Slide 5 text

5 rts.ch/info/regions/neuchatel/12836885-la-population-neuchateloise-en-hausse-malgre-un-recul-dans-les-montagnes.html

Slide 6

Slide 6 text

● Build, Test and CI automation in Jenkins ● Zenoss plugins for database monitoring ● Jupyter playbooks ● MkDocs ● Finding excuses to not use Python :=( 6 censored censored censored /me and Python

Slide 7

Slide 7 text

* Testcontainers/AtomicJar is a WireMock partner

Slide 8

Slide 8 text

8 wiremock.org IF (request_url) THEN (response) * * it gets MUCH more complex WireMock Config JSON: Client library response request Client App Mock API Server HTTP/2

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

12 GROWTH IN WEB APIS SINCE 2005 Over 90% of developers use APIs. Skyrocket growth of APIs JANUARY 2006 JANUARY 2008 JANUARY 2010 JANUARY 2012 JANUARY 2014 JANUARY 2016 JANUARY 2018 MONTH The growth over time of the Programmable Web API API directory to more than 22,000 entries 22000 20000 18000 16000 14000 12000 10000 8000 6000 4000 2000 0 TOTAL API COUNT Programmable Web * Gartner Hype Cycle for APIs, 2022

Slide 13

Slide 13 text

Source: https://kids-first.github.io/kf-airflow- dags/warehouse.html kidsfirstdrc.org

Slide 14

Slide 14 text

Source: https://kids-first.github.io/kf-airflow- dags/warehouse.html Data Provider APIs

Slide 15

Slide 15 text

APIs

Slide 16

Slide 16 text

16 Source: https://pixelfriedhof.com/en/csv-is-not-an-interface/

Slide 17

Slide 17 text

Most of data comes from APIs ● opendata.swiss ● kidsfirstdrc.org/portal/portal-about-data ● vs.inf.ethz.ch/res ● github.com/public-apis/public-apis ● www.kaggle.com/datasets ● data.world 17

Slide 18

Slide 18 text

18 Source: https://medium.com/10-minutes-qa-story/api-testing-from-scratch-api-anatomy-f220856ce9b3

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

20 Build Unit tests Publish Reports Integration tests Publish Reports WAS

Slide 21

Slide 21 text

21 Build Unit tests Publish Reports Integration tests Publish Reports NOW “Shift Left” Fast integrations tests are critical

Slide 22

Slide 22 text

Ways to do integration testing 1. Testing against Production/Staging servers 2. Testing against a simplified/containerized instance (e.g. Testcontainers) 3. Mocking at the API provider level (e.g. WireMock) 4. Mocking at the code level 22 Slow Fast Nope Accu- rate

Slide 23

Slide 23 text

23 Staging & QA Tests Build Integration tests Build Tests Gaining confidence in you software

Slide 24

Slide 24 text

* Testcontainers/AtomicJar is a WireMock partner

Slide 25

Slide 25 text

25 wiremock.org IF (request_url) THEN (response) * * it gets MUCH more complex WireMock Config JSON: Client library response request Client App Mock API Server HTTP/2

Slide 26

Slide 26 text

wiremock.org

Slide 27

Slide 27 text

27 WireMock 101 Open Source tool for building mock APIs Available beyond Java, on Python too You can can: ● Create stable development environments ● Isolate from unstable 3rd party APIs ● Simulate APIs that don't exist yet wiremock.org

Slide 28

Slide 28 text

WireMock Cloud by WireMock Inc. 28 ● WireMock creator is a co-founder of the Inc. ● WireMock Cloud - SaaS for end-to-end API mocking ● Private beta: K8s Edition for managed / on-premises wiremock.io

Slide 29

Slide 29 text

WireMock Ecosystem & Features 29 wiremock.org/docs

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

WireMock for Python wiremock.org/docs/solutions/python wiremock.readthedocs.io ● Python SDK ● REST API Client library ● Pytest and Robot Framework integrations ● Testcontainers Module * Custom logo is approved by the Python Software Foundation 31 More: github.com/wiremock/python-wiremock

Slide 32

Slide 32 text

WireMock Ecosystem & Features 32 wiremock.org/docs

Slide 33

Slide 33 text

33 WireMock Standalone client from wiremock.constants import Config from wiremock.client import * Config.base_url = 'https://mockserver.example.com/__admin/' mapping = Mapping( priority= 100, request=MappingRequest( method=HttpMethods.GET, url= '/hello' ), response=MappingResponse( status= 200, body= 'hi' ), persistent= False, ) mapping = Mappings.create_mapping(mapping=mapping) all_mappings = Mappings.retrieve_all_mappings() More: github.com/wiremock/python-wiremock Client library response request Test Mock API Server HTTP/2

Slide 34

Slide 34 text

Using WireMock in unittest 34 class MyTestClassBase(TestCase): @classmethod def setUpClass(cls): wm = self.wiremock_server = WireMockServer() wm.start() Config.base_url = 'http://localhost:{}/__admin' .format(wm.port) @classmethod def tearDownClass(cls): self.wiremock_server.stop() More: github.com/wiremock/python-wiremock unittest

Slide 35

Slide 35 text

Robot Framework github.com/wiremock/robotframework-wiremock 35

Slide 36

Slide 36 text

Robot Framework 36

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

github.com/wiremock/python-wiremock /tree/master/examples 38

Slide 39

Slide 39 text

* Testcontainers/AtomicJar is a WireMock partner

Slide 40

Slide 40 text

Slide 41

Slide 41 text

41 atomicjar.com/2023/12/atomicjar-is-now-part-of-docker/ Congrats to the team!

Slide 42

Slide 42 text

42 testcontainers.com

Slide 43

Slide 43 text

Why containers? ● Configuration Management ● Fast provisioning ● Clean environments ● Disposability ● Clustering

Slide 44

Slide 44 text

Docker ● Popular container engine ● Developer-friendly ● Huge ecosystem ● DockerHub and Container Registries ● Docker Compose – multi-container apps ● Universal image format (OCI) https://www.docker.com/

Slide 45

Slide 45 text

Slide 46

Slide 46 text

46 SDKs C/C++ testcontainers.com

Slide 47

Slide 47 text

47 testcontainers.com/modules

Slide 48

Slide 48 text

Testcontainers for Python 48 ● Started by Sergey Pirogov, now - Till Hoffmann ● github.com/testcontainers/testcontainers-python ● testcontainers-python.readthedocs.io testcontainers.com

Slide 49

Slide 49 text

Example - MongoDB 49 https://testcontainers-python.readthedocs.io

Slide 50

Slide 50 text

Containers are not always slow! ● On-demand image build ● Caching Docker image builds ● Suspending containers between tests ● Graceful termination ○ github.com/testcontainers/moby-ryuk 50 testcontainers.com

Slide 51

Slide 51 text

* Testcontainers/AtomicJar is a WireMock partner

Slide 52

Slide 52 text

52 WireMock has an official Testcontainers module! testcontainers.com/modules/wiremock

Slide 53

Slide 53 text

53 wiremock.org/docs/solutions/testcontainers

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

WireMock as a Proxy 55 Tests Real API Provider OR ● Recording ● Fault injection ● Protocol Verification

Slide 56

Slide 56 text

WireMock Record & Playback wiremock.org/docs/record-playback ● Record communications ● Capture data ● Replay the requests ● Reverse-engineer protocols to OpenAPI 56

Slide 57

Slide 57 text

WireMock Faker Extension github.com/wiremock/wiremock-faker-extension ● There’re many Data::Faker library ports ○ Python: joke2k/faker ○ Ruby: faker-ruby/faker (logo source!) ○ Java: datafaker-net/datafaker ● We use the Java one ● It can be included in a Testcontainer image 57

Slide 58

Slide 58 text

WireMock Faker Extension wiremock.org/docs/response-templating/#fake-data-helpers 58 "response": { "status": 200, "jsonBody": { "namme": {{random 'Name.first_name'}} "surname": {{random 'Name.last_name'}} "postcode": {{random 'Address.postcode_by_state.AL' }} } }

Slide 59

Slide 59 text

Simple data is not enough? 59 www.synthesized.io

Slide 60

Slide 60 text

Synthesized’ Data offerings 60 Synthesized Scientific Data Kit (SDK) Synthesized Test Data Kit (TDK) Synthesized Fairlens (Data Bias) www.synthesized.io Data sets / Streams *.CSV Relational datasets (for databases)

Slide 61

Slide 61 text

61 Testcontainers Module for Synthesized TDK ● Only TDK - Relational Data ● Only Java Module No Runtime Use at the moment, But NOT a showstopper

Slide 62

Slide 62 text

62 TDK SDK Database Module for Testcontainers App Tests WireMock Module Build On-Demand Your App Tests ● Random Fetch ● CSV => JSON generator

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

github.com/wiremock/wiremock-docker/ tree/main/samples/random 64

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

WireMock Public Roadmap 66 github.com/orgs/wiremock/projects/4

Slide 67

Slide 67 text

Python WireMock - My Wishlist wiremock/python-wiremock/issues ● Feature parity with WireMock Java in the SDK ● More features in the Testcontainers module ● Removing the JVM process implementation ● Integrations with FastAPI and Hug ● Making Python a first class citizen in WireMock docs 67

Slide 68

Slide 68 text

WireMock docs move to MkDocs! PoC: https://oleg-nenashev.github.io/wiremock.org/docs/ 68 * we plan to contribute to a few plugins like Multi-Repo

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

Takeaways 70 ● Shift left your integration testing ● Mock APIs and Data ● There are tools for that, including WireMock and Testcontainers ● WireMock and Testcontainers co-exist well ● They are available in Python!

Slide 71

Slide 71 text

Get Started wiremock.org/docs wiremock.org/docs/solutions/python 71

Slide 72

Slide 72 text

It’s [always] a great time to contribute! slack.wiremock.org wiremock.org/participate 72

Slide 73

Slide 73 text

THANK YOU Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? 73

Slide 74

Slide 74 text

Credits to ● All WireMock contributors ● WireMock Inc. Team ● All Testcontainers contributors and AtomicJar folks ● All FOSS contributors 74