He has 100% code coverage and writes FEWER tests! [SEE HOW]

1. He has 100% code coverage and writes FEWER tests! Sebastian Buczyński @ Pycon PL 30.06.2023 [SEE HOW]
4. Automated tests >>> tedious manual checks def test_dialing_another_phone_makes_it_ring() -> None: phones.register("Alice", "111") phones.register("Bob", "222") phones.of("Alice").dial("Bob") assert phones.of("Bob").is_ringing()
5. Are we writing enough tests?
6. Code Coverage!
7. def determine_queue_position(patient: Patient, queue: Queue) -> int: position = len(queue) if patient.is_pregnant or patient.is_regular_blood_donor: position = 0 return position
8. def determine_queue_position(patient: Patient, queue: Queue) -> int: position = len(queue) if patient.is_pregnant or patient.is_regular_blood_donor: position = 0 return position
9. def determine_queue_position(patient: Patient, queue: Queue) -> int: position = len(queue) if patient.is_pregnant or patient.is_regular_blood_donor: position = 0 return position
10. def test_determine_queue_position() -> None: queue = Queue() patient = Patient(is_pregnant=False, is_regular_blood_donor=False) position = determine_queue_position(patient, queue) assert position == 0
11. def test_determine_queue_position() -> None: queue = Queue() patient = Patient(is_pregnant=False, is_regular_blood_donor=False) position = determine_queue_position(patient, queue) assert position == 0
12. def test_determine_queue_position() -> None: queue = Queue() patient = Patient(is_pregnant=False, is_regular_blood_donor=False) position = determine_queue_position(patient, queue) assert position == 0
13. pip install pytest - cov pytest test_doctor_queue.py -- cov doctor_queue/ —cov - report=term - missing
14. test_doctor_queue_1.py . [100%] ---------- coverage: platform darwin, python 3.11.0-final-0 ---------- Name Stmts Miss Cover Missing --------------------------------------------------------- doctor_queue/ __ init __ .py 0 0 100% doctor_queue/determine.py 11 1 91% 18 --------------------------------------------------------- TOTAL 11 1 91%
18. • Trainer / Consultant @ Bottega IT Minds • TL @ Sauce Labs • Evangelist of software engineering in Python • Substack - pythoneer.substack.com • Fan of Bartosz Walaszek • Apprentice of alternative coffee brewing methods whoami
19. pytest test_doctor_queue.py -- cov doctor_queue/ —cov - report=term - missing test_doctor_queue_1.py . [100%] ---------- coverage: platform darwin, python 3.11.0-final-0 ---------- Name Stmts Miss Cover Missing --------------------------------------------------------- doctor_queue/ __ init __ .py 0 0 100% doctor_queue/determine.py 11 1 91% 18 --------------------------------------------------------- TOTAL 11 1 91% Top 10 Photos Taken Before Tragedy Generated by Midjourney
20. „Let’s rewrite this project!”
21. class SomeClass: ... @property def some_function(self) -> int: return 42 # <-- you don't have coverage for this line, bro „Let’s rewrite this project!”
22. def test_calculating_very_important_stuff() : ... assert True def test_calculating_other_very_important_stuff() : ... # Look ma, no assertion!
23. Code coverage is not that great def test_pregnancy_means_accessing_doctor_without_having_to_wait() : queue = Queue() patient = Patient(is_pregnant=True, is_regular_blood_donor=False) queue_position = determine_queue_position(patient, queue) assert queue_position == 0
24. Code coverage is not that great def test_pregnancy_means_accessing_doctor_without_having_to_wait() : queue = Queue() patient = Patient(is_pregnant=True, is_regular_blood_donor=False) queue_position = determine_queue_position(patient, queue) assert queue_position == 0
25. --cov-fail-under 100 Locally, as well in CI
26. Staying up to date with libraries and Python
27. Safe refactoring (also thanks to testing strategy)
28. Hints during development process (parts of code that are not covered are something to take care about next)
29. Goodhart’s Law „When a measure becomes a target, it ceases to be a good measure”
31. Improving the process is the hard part
32. HELPFUL 100% code coverage without MEANINGLESS effort
33. Vertical slicing Elephant Carpaccio How to be always ready for production release?
34. 🚫 Horizontal slicing 🚫 1. Integrate with vendor’s API 2. Create Backend service with API to browse offers and buy stuff 3. Frontend
35. ✅ Vertical slices ✅ 1. PoC of integration with one of vendor’s API 2. Browsing offers from one vendor 3. Ability to make a purchase on one vendor 4. Adding 2nd vendor with refactoring 5. Adding more vendors
36. ATDD
37. Observable results in tests What to assert? • Where can a user see some change after their action? • Where other types of users can see some effect? • How this action affects other features? • What a user can now do which they couldn’t do before? • What a user can no longer do which they were able to do before?
38. This could’ve been done better
39. def test_generates_files_for_SOME_data_in_db() -> None: put_SOME_data_in_db() run_one_loop_of_program() assert some_files_exist() This could’ve been done better
40. Development process with USEFUL 100% code coverage • Step #1 - Pick up a task and ensure it can be veri f ied • Step #2 - Come up with acceptance testing scenario(s) • Step #3 - Write a test for this scenario (or at least empty test) • Step #4 - Implement the feature • Step #5 - Check code coverage • Repeat steps 2-5 until satis f ied
41. Example #1 A customer can store their payment card
42. Example #1 A customer can store their payment card… and later retrieve it
43. Example #1 A customer can store their payment card… and later retrieve it def test_customer_can_read_stored_payment_card_details() -> None: pass
44. Example #1 A customer can store their payment card… and later retrieve it
45. def test_customer_can_read_stored_payment_card_details( authorized_client: TestClient, ) - > None: add_response = authorized_client.post( "/payment - cards", json={ "card_number": "4111 1111 1111 1111", "names": "John Doe", "cvc": "123", }, )
46. def test_customer_can_read_stored_payment_card_details( authorized_client: TestClient, ) - > None: add_response = authorized_client.post( "/payment - cards", json={ "card_number": "4111 1111 1111 1111", "names": "John Doe", "cvc": "123", }, ) assert add_response.status_code == 201
47. def test_customer_can_read_stored_payment_card_details( authorized_client: TestClient, ) - > None: add_response = authorized_client.post( "/payment - cards", json={ "card_number": "4111 1111 1111 1111", "names": "John Doe", "cvc": "123", }, ) assert add_response.status_code == 201 get_response = authorized_client.get("/payment - cards") assert get_response.status_code == 200 assert get_response.json() == [ { "card_number": "4111 **** **** 1111", "names": "John Doe", "cvc": " ** * ", } ]
48. We check coverage… tests/acceptance/test_storing_payment_card.py . [100%] ---------- coverage: platform darwin, python 3.11.0-final-0 ---------- Name Stmts Miss Branch BrPart Cover ------------------------------------------------------------------ subscriptions/ __ init __ .py 0 0 0 0 100% subscriptions/anonymization.py 4 0 0 0 100% subscriptions/api.py 31 0 8 0 100% subscriptions/db.py 7 0 2 0 100% subscriptions/models.py 9 0 0 0 100% ------------------------------------------------------------------ TOTAL 51 0 10 0 100%
49. We look for more test scenarios …and repeat def test_initially_customer_has_no_saved_payment_cards() -> None: pass def test_customer_can_read_only_payment_cards_they_created() -> None: pass
50. Digression: veri f ication methods There are only 4 • Returned result (from a function / a method) • Raised exception (or lack of it) • State veri f ication - we call one method / endpoint, then we check result using other method / endpoint • Interaction - Mocks • e.g. assert_called_once_with
51. Example #2 Customer’s card is saved in external system
52. @app.post("/payment - cards - in - external - system") def store_payment_card_in_external_system( payload: PaymentCardPayload, user_id: int = Depends(user_id), payments_gateway: PaymentsGateway = Depends(), ) -> Response: httpx.post("https: // example.tech/payment - cards", json={ "card_number": payload.card_number, "names": payload.names, "cvc": payload.cvc, "user_id": user_id, }) return Response(status_code=201)
53. Example #2 Customer’s card is saved in external system
54. class PaymentsGateway: def store_payment_card(self, ... ) -> None: httpx.post("https: // example.tech/payment - cards", json={ "card_number": dto.card_number, "names": dto.names, "cvc": dto.cvc, "user_id": dto.user_id, })
55. @define(frozen=True, repr=False) class PaymentCardDto: user_id: int card_number: str names: str cvc: str class PaymentsGateway: def store_payment_card(self, dto: PaymentCardDto) -> None: httpx.post("https: // example.tech/payment - cards", json={ "card_number": dto.card_number, "names": dto.names, "cvc": dto.cvc, "user_id": dto.user_id, })
56. def test_payment_card_is_stored_in_external_system( authorized_client: TestClient, user_id: int, when: Any, ) -> None: when(PaymentsGateway).store_payment_card( ... ).thenReturn(None) add_response = authorized_client.post( ... ) assert add_response.status_code == 201 verify(PaymentsGateway, times=1).store_payment_card( PaymentCardDto( user_id=1, card_number="4000-0000-0000-0002", names="Janine Doe", cvc="321", ) )
57. tests/acceptance/test_storing_payment_card.py ... [ 75%] tests/acceptance/test_storing_payment_card_with_external_system.py . [100%] ---------- coverage: platform darwin, python 3.11.0-final-0 ---------- Name Stmts Miss Branch BrPart Cover --------------------------------------------------------------------- subscriptions/ __ init __ .py 0 0 0 0 100% subscriptions/anonymization.py 4 0 0 0 100% subscriptions/api.py 38 0 10 0 100% subscriptions/db.py 7 0 2 0 100% subscriptions/models.py 9 0 0 0 100% subscriptions/payments_gateway.py 11 1 2 0 92% --------------------------------------------------------------------- TOTAL 69 1 14 0 99%
58. Is this safe? With strict mypy settings - yes. @app.post("/payment - cards - in - external - system") def store_payment_card_in_external_system( ... ) -> Response: dto = PaymentCardDto( user_id=user_id, card_number=payload.card_number, names=payload.names, cvc=payload.cvc, ) payments_gateway.store_payment_card(dto) return Response(status_code=201)
59. More ways to use Gateway Pattern • In tests we can emulate behaviour of the external system (fake) • We translate errors we want to handle explicitly for example to exceptions
60. Testing strategy with 100% code coverage • Start from acceptance tests • Add more lower-level tests if necessary (fractal testing) • Test(s) end-to-end / synthetic monitoring etc • # pragma: no cover where it makes sense (e.g. abstract methods)
61. My acceptance tests have few hundred lines :( • Your architecture doesn’t support testing • You need more modularization (think smaller components)
62. Evolutionary architecture and fractal tests
63. Code coverage tools are not THAT demanding All we can have is branch coverage
65. Code coverage is not everything
67. Further reading https://commoncog.com/goodharts-law-not-useful/ https://breadcrumbscollector.tech/how-to-use-code-coverage-in-python-with-pytest/ https://lukeonpython.blog/2021/11/why-i- f ind-100-code-coverage-very-useful/
68. Q & A pythoneer.substack.com/