Coding-challenge: AI-test vs PI-test (20260522 @ DevDays-Europe)

Let's be honest: every developer loves writing tests, right? Meaning that we're all diligently practicing Test-Driven Development (TDD), not just preaching it. Or... are we? Maybe that's why we keep asking our AI coding assistants to write tests for us. Just prompt ChatGPT to generate a test for your latest class and voilà: "Of course I can help you write your tests. Here's a complete test suite that achieves 100% line and branch coverage."

But does that really mean the tests are any good?

In this session, Frederieke Scheper will push AI-generated tests to their limits using mutation testing and the PI-Test framework. Frederieke, with attendees, will explore whether these tests truly hold up under scrutiny, or if they just look good on paper.

By the end, attendees will know when to say to their coding assistant, "Thanks, ChatGPT, that was helpful," and when to say, "Thanks, but no thanks. I'll write my own test this time."

Transcript

1. AI-TEST VS PI-TEST Frederieke Scheper Designed by Robin Potze Coding Challenge:
2. About me Coding Challenge: AI-Test vs PI-Test Frederieke Scheper • Java Architect & Codesmith ‣ I ❤ OSS conferences! • Working for Dutch national police • Passion for ‣ Software development ‣ TDD, BDD, DDD ‣ And AI agents … • Contact me on (@fbascheper)
3. 1. Who loves TDD ? 2. Who loves writing tests? 3. Who actually writes them first ? ACT 1 - Warming up Three questions: 🙋
4. 1. Who loves TDD ? 2. Who loves writing tests? 3. Who actually writes them first ? ACT 1 - Warming up Three questions: 🤩 AI to the rescue ? It can write our tests !
5. 1. Who loves TDD ? 2. Who loves writing tests? 3. Who actually writes them first ? ACT 1 - Warming up Three questions: 🤔 Hey, we can test AI… with mutation testing !
6. “Simple as 1, 2, 3 ?” mvn test-compile pitest:mutationCoverage “All the mutants should have been killed 🛸, No survivors allowed ☠ …” PI-test - mutation testing in Java
7. Coding Challenge: AI-Test vs PI-Test PI-test - example mutators if (a < b) { // do something } if (a <= b) { // do something } ➡ becomes “Conditionals boundary mutator” Source code < <= > >= Mutated code <= < >= > ➡ becomes AND THE TEST SHOULD FAIL !
8. Coding Challenge: AI-Test vs PI-Test PI-test - example mutators public int method(int i) { i++; return i; } public int method(int i) { i--; return i; } ➡ becomes “Increments mutator” ➡ becomes Source code i++ i-- i = i+1 i = i-1 Mutated code i-- i++ i = i-1 i = i+1 AND THE TEST SHOULD FAIL (AGAIN) !
9. Coding Challenge: AI-Test vs PI-Test PI-test - example mutators public void aVoidMethod(int i) { // does something } public int foo() { int i = 5; aVoidMethod(i); return i; } public void aVoidMethod(int i) { // does something } public int foo() { int i = 5; /* method removed !! */ return i; } ➡ becomes “Void method calls mutator” AND THE TEST SHOULD FAIL (AGAIN) !
10. Coding Challenge: AI-Test vs PI-Test PI-test - example mutators public String sayHello() { int i = 5; String foo = method(i); return "hello " + foo; } public String sayHello() { int i = 5; String foo = method(i); return null; } ➡ becomes “Null returns mutator” AND THE TEST SHOULD FAIL (AGAIN) !
11. Coding Challenge: AI-Test vs PI-Test PI-test - example mutators public Set<String> strings() { int i = 5; Set<String> foo = method(i); return foo; } public Set<String> strings() { int i = 5; Set<String> foo = method(i); return Collections.emptySet(); } ➡ becomes “Empty returns mutator” AND THE TEST SHOULD FAIL (AGAIN) ! As I said before: “Simple as 1, 2, 3 !”
12. Coding Challenge: AI-Test vs PI-Test PI-test - example mutators public Set<String> strings() { int i = 5; Set<String> foo = method(i); return foo; } public Set<String> strings() { int i = 5; Set<String> foo = method(i); return Collections.emptySet(); } ➡ becomes “Empty returns mutator” AND THE TEST SHOULD FAIL (AGAIN) ! “The decks are ready Let’s spin up the demo.” 🎧 As I said before: “Simple as 1, 2, 3 !”
13. • Today’s domain model ‣ About Dance-Events ‣ Where the DJ guides us through the night 🎧 ACT 2 - Demo time → Introduction
14. — People come together to dance and have fun. — DJs perform MixSessions to guide the crowd. — Music evolves: warm-up → peak → cool-down. Coding Challenge: AI-Test vs PI-Test ACT 2 - Demo time → The venue
15. ACT 2 - Demo time → The model
16. Select Tracks Mixes & Transitions Read the Crowd Respond to Requests Build Atmosphere Type Action Coding Challenge: AI-Test vs PI-Test ACT 2 - Demo time → The DJ at work
17. Coding Challenge: AI-Test vs PI-Test RequestFromAudienceReceived ACT 2 - Demo time → Many CrowdEvents ! CrowdCheered DancefloorFilledUp DancefloorEmptied CrowdEnergyDropped
18. The DJ starts TESTS findTrack MixSession RequestFromAudienceReceived: Play my favourite song! Coding Challenge: AI-Test vs PI-Test
19. RequestFromAudienceReceived ("Radar Love") RequestFromAudienceReceived ("Something like suavemente ?!?") RequestFromAudienceReceived ("Wake Me Up")
20. RequestFromAudienceReceived ("Radar Love") RequestFromAudienceReceived ("Something like suavemente ?!?") RequestFromAudienceReceived ("Wake Me Up")
21. ACT 3 - The dance floor starts filling up CrowdCheered LOW MEDIUM HIGH CrowdCheered: Sheesh! Coding Challenge: AI-Test vs PI-Test
22. The floor is packed DanceFloorFilledUpEvent Sheesh! Turn it up! Pop off! Straight bangers! Coding Challenge: AI-Test vs PI-Test
23. The DJ makes a mistake E-8302: DanceFloorEmptied CrowdEnergyDropped: Boooooo! Coding Challenge: AI-Test vs PI-Test
24. > Who’s actually prompting like this?? Just one question: I certainly don’t … not anymore Because AI changed gears So let’s try something else ! ACT 4 - CrowdEnergyDropped → BOOOOOOO
25. Hey Claude, I have a working legacy application with no unit tests. Test support objects are available in src/test/java — use those as a starting point. - Use ./claude-config-starter as reference examples to create valid .claude config files for this project - NEVER modify anything under src/main - Use mocks to avoid turning unit tests into integration tests - Create a single beads issue for writing all tests, then one beads issue per improvement Plan first, then wait for my approval before writing any code. After writing tests, run PITest mutation testing and show me the results table. Then plan improvements and wait for my approval before implementing — create a beads issue per improvement, implement, then run PITest again to show the delta. ACT 4 - CrowdEnergy Dropped → Switching gears Claude’s screencast ! 🖥
26. SCREENCAST PART 1 ➡ THE PROMPT ⏸ (PAUSE FOR NOW)
27. SCREENCAST PART 2 ➡ PREPARE PLAN ⏸ (PAUSE FOR NOW)
28. SCREENCAST PART 3 ➡ PLAN PHASE 2 ⏸ (PAUSE FOR NOW)
29. SCREENCAST PART 4 ➡ PLAN PHASE 3 ⏸ (PAUSE FOR NOW)
30. SCREENCAST PART 5 ➡ FINISH PLAN & CONFIG ⏸ (PAUSE FOR NOW)
31. SCREENCAST PART 6 ➡ WRITE THE TESTS ⏸ (PAUSE FOR NOW)
32. SCREENCAST PART 7 ➡ RUN PI-TEST ⏸ (PAUSE FOR NOW)
33. SCREENCAST PART 8 ➡ PLAN TO IMPROVE TESTS ⏸ (PAUSE FOR NOW)
34. SCREENCAST PART 9 ➡ CHECK NEW RESULTS ⏸ (PAUSE FOR NOW)
35. SCREENCAST PART 10 ➡ WRAPPING UP 😍 (FINISHED) 😍
36. CrowdCheered: Saved!!! Coding Challenge: AI-Test vs PI-Test robin-potze@live.nl **************** ACT 5 - The DJ recovered
37. Coding Challenge: AI-Test vs PI-Test AI-TEST? PI-TEST ? And the winner is …
38. Coding Challenge: AI-Test vs PI-Test AI-TEST? THE DEV-COMMUNITY ! PI-TEST ? 🍻 And the winner is …
39. THANK YOU Frederieke Scheper Coding Challenge: AI-Test vs PI-Test Designed by Robin Potze
40. AI-TEST VS PI-TEST Frederieke Scheper Designed by Robin Potze Coding Challenge: