TDD: Test-Driven Development vs
TAB-Driven Development
Marit van Dijk
Developer Advocate
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
Question
●Can we do (strict) TDD with AI?
Slide 4
Slide 4 text
Agenda
●What is TDD?
●TDD with AI:
●A series of experiments
●Results
Slide 5
Slide 5 text
Test-Driven Development
Slide 6
Slide 6 text
Test-Driven Development
1. Write a failing test (Red)
Slide 7
Slide 7 text
Failing test
Never trust a test you haven’t seen fail
https://maritvandijk.com/use-testing-to-develop-better-software-faster/
Slide 8
Slide 8 text
Test-Driven Development
2. Write just enough code pass the test (Green)
Slide 9
Slide 9 text
YAGNI
You Ain't Gonna Need It
Slide 10
Slide 10 text
Test-Driven Development
3. Improve the code, without changing its behaviour
(Refactor)
Slide 11
Slide 11 text
TDD is not about testing
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
Think about
• Intended behavior
Slide 14
Slide 14 text
Think about
• Intended behavior
• How to test it
Slide 15
Slide 15 text
No content
Slide 16
Slide 16 text
Code completion is here
● Auto completion while typing
● with syntax highlighting
Slide 17
Slide 17 text
Tab-Driven Development
Cloud completion Local completion
Slide 18
Slide 18 text
Side note
● You can change the key used for completion 🤫
Slide 19
Slide 19 text
Accept partial suggestion
macOS Windows / Linux
Accept suggestion ⇥ TAB
Accept line ⌘→ END
Accept word ⌥→ Ctrl+→
Slide 20
Slide 20 text
Tab-Driven Development
For the purpose of this talk:
Tab-Driven Development = AI assisted programming
Slide 21
Slide 21 text
Generate tests
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
Generate code
• Can we generate code from tests?
Slide 24
Slide 24 text
Tools
Slide 25
Slide 25 text
JetBrains AI Assistant
—
Slide 26
Slide 26 text
Plugin
Slide 27
Slide 27 text
JetBrains Junie
—
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
Plugin
Slide 30
Slide 30 text
Junie
A coding agent
by JetBrains
Slide 31
Slide 31 text
LLM demo
Slide 32
Slide 32 text
A series of experiment
Slide 33
Slide 33 text
Experiments: "TDD" with AI Assistant
• No equivalent "Generate code from tests"
Slide 34
Slide 34 text
Experiments: "TDD" with AI Assistant
• No equivalent "Generate code from tests"
• Find the best way to do so
Slide 35
Slide 35 text
Experiments: "TDD" with AI Assistant
• Problem: FizzBuzz
Tried different options:
1. Strict TDD (Red - Green - Refactor)
2. From Chat
3. Inline
Slide 36
Slide 36 text
Experiment: Strict TDD
• Problem: FizzBuzz
1. Write a test
2. Ask for implementation
3. (Ask for) refactoring
Slide 37
Slide 37 text
Experiment: Strict TDD
• Problem: FizzBuzz
1. Write a test
2. Ask for implementation
3. (Ask for) refactoring
• Code completion kicks in for known problem.
Slide 38
Slide 38 text
Experiment: From Chat
• Problem: FizzBuzz
1. Ask AI Assistant about FizzBuzz
Slide 39
Slide 39 text
Experiment: From Chat
• Problem: FizzBuzz
1. Ask AI Assistant about FizzBuzz
2. Ask for list of test cases
Slide 40
Slide 40 text
Experiment: From Chat
• Problem: FizzBuzz
1. Ask AI Assistant about FizzBuzz
2. Ask for list of test cases
3. Ask for tests in Java
Slide 41
Slide 41 text
Experiment: From Chat
• Problem: FizzBuzz
1. Ask AI Assistant about FizzBuzz
2. Ask for list of test cases
3. Ask for tests in Java
Slide 42
Slide 42 text
Experiment: From Chat
• Problem: FizzBuzz
1. Ask AI Assistant about FizzBuzz
2. Ask for list of test cases
3. Ask for tests in Java
4. Ask for implementation
Slide 43
Slide 43 text
Experiment: Inline
• Problem: FizzBuzz
1. Ask AI Assistant for test cases based on description
Slide 44
Slide 44 text
Experiment: Inline
Slide 45
Slide 45 text
Experiment: Inline
• Problem: FizzBuzz
1. Ask AI Assistant for test cases based on description
2. Ask to implement code that passes the tests
Slide 46
Slide 46 text
Experiment: Inline
• Wrong place in file
• Easy to fix
Slide 47
Slide 47 text
TDD workflow with AI Assistant
1. Create test(s)
2. Ask AI Assistant Chat for code that passes the tests
3. Apply code from chat
• Works with simple problems
Slide 48
Slide 48 text
Result
Can do (strict) TDD with AI Assistant
Can apply code directly (no copy-paste)
Slide 49
Slide 49 text
Experiment: Strict TDD
• Problem: FizzBuzz
• Strict TDD with Junie (Red/Green/Refactor):
1. Write a test
2. Ask Junie for implementation
Slide 50
Slide 50 text
Experiment: Strict TDD
• Provide Junie with Guidelines first
Slide 51
Slide 51 text
Experiment: Strict TDD
Slide 52
Slide 52 text
Experiment: Strict TDD
Slide 53
Slide 53 text
Experiment: Strict TDD
Slide 54
Slide 54 text
Experiment: Strict TDD
Slide 55
Slide 55 text
Experiment: Strict TDD
Slide 56
Slide 56 text
Experiment: Strict TDD
• Problem: FizzBuzz
• DEMO
Slide 57
Slide 57 text
Experiment: Strict TDD
• Problem: FizzBuzz
• Junie creates the right code in the right location
• Junie runs the tests & makes sure they pass
Slide 58
Slide 58 text
TDD workflow with Junie
1. Create tests
2. Ask Junie for code that passes the tests
3. Repeat
• Junie creates the right code in the right location
• Junie runs the tests & makes sure they pass
Slide 59
Slide 59 text
Result
Can do (strict) TDD with Junie
Tips:
• Provide guidelines
• Take turns
Slide 60
Slide 60 text
Toy problems
• One class
• Limited functionality
• What about multiple classes?
Experiment: Multiple files
• Problem: Calculator
• Can generate multiple classes
• but not always...
Slide 63
Slide 63 text
Experiment: Multiple files
• Problem: Calculator
• Provide set of tests
• Ask for implementation
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
Experiment: Multiple files
Slide 66
Slide 66 text
Experiment: Multiple files
Slide 67
Slide 67 text
Experiment: Multiple files
• DEMO
Slide 68
Slide 68 text
Experiment: Multiple files
• Problem: Calculator
• Junie creates the right files in the right location
• Junie runs the tests & makes sure they pass
Slide 69
Slide 69 text
Toy problems
• What about actual apps?
Slide 70
Slide 70 text
Experiment: Build a small app test first
• Ask Junie to generate implementation based on tests
Slide 71
Slide 71 text
Experiment: Build a small app test first
• We need an example
• With tests
• Let's reverse engineer!
Slide 72
Slide 72 text
Example: Build a small app
• Create Spring Boot project
• Ask Junie to implement small app
Slide 73
Slide 73 text
No content
Slide 74
Slide 74 text
No content
Slide 75
Slide 75 text
Example: Build a small app
• Add task
• Mark task done
• Delete task
Slide 76
Slide 76 text
No content
Slide 77
Slide 77 text
The code is tested
• Junie added tests
• Junie ran tests
• How good are the tests?
Slide 78
Slide 78 text
Example: Build a small app
Let's use Junie to check up on Junie...
Slide 79
Slide 79 text
Example: Build a small app
Let's use Junie to check up on Junie...
1. Add coverage with JaCoCo
Slide 80
Slide 80 text
No content
Slide 81
Slide 81 text
Example: Build a small app
Let's use Junie to check up on Junie...
1. Add coverage with JaCoCo
Slide 82
Slide 82 text
Example: Build a small app
Let's use Junie to check up on Junie...
1. Add coverage with JaCoCo
2. Add mutation testing with Pitest
Slide 83
Slide 83 text
No content
Slide 84
Slide 84 text
Example: Build a small app
Let's use Junie to check up on Junie...
1. Add coverage with JaCoCo
2. Add mutation testing with Pitest
Slide 85
Slide 85 text
No content
Slide 86
Slide 86 text
Example: Build a small app
Use Junie to increase test coverage
Slide 87
Slide 87 text
No content
Slide 88
Slide 88 text
Example: Build a small app
Use Junie to increase test coverage
Slide 89
Slide 89 text
Experiment: Rebuild the small app
• Remove implementation
• Ask Junie to build app based on tests
• Might get it right based on project history (?)
Slide 90
Slide 90 text
Experiment: Rebuild the small app
• Create empty project
• Copy in the tests
• Ask Junie to build app based on tests
Slide 91
Slide 91 text
Experiment: Rebuild the small app
Slide 92
Slide 92 text
Experiment: Rebuild the small app
Slide 93
Slide 93 text
No content
Slide 94
Slide 94 text
Experiment: Rebuild the small app
• Tests don't always pass on first try
Slide 95
Slide 95 text
Experiment: Rebuild the small app
• Tests don't always pass on first try
• Junie makes fixes and tries again
Slide 96
Slide 96 text
Experiment: Rebuild the small app
• Working app
• Not always the same
Slide 97
Slide 97 text
Experiment: Rebuild the small app
• Helps to provide guidelines
Slide 98
Slide 98 text
Experiment: Rebuild the small app
• Helps to provide guidelines
Slide 99
Slide 99 text
Experiment: Build small app test-first
• DEMO
Slide 100
Slide 100 text
Results
• Tests are not enough
• Needs guidelines
Slide 101
Slide 101 text
We need more than passing tests
• Does the application start up?
• What about security?
• Performance?
• Code quality?
• Etc
Slide 102
Slide 102 text
A learning curve
• Which requirements
• Which guidelines
• Which prompts
• Which models
Slide 103
Slide 103 text
Can we do (strict) TDD with AI?
• While it's possible to do (strict) TDD with AI
• Generating code based on tests alone does not give
(me) reliable results
Slide 104
Slide 104 text
There is still value in
• Defining requirements & guidelines
• Fast feedback, like tests
• Small(er) increments