TDD: Test-Driven Development vs Tab-Driven Development (JCON)

by Marit van Dijk

Slide 1

Slide 1 text

TDD: Test-Driven Development vs TAB-Driven Development Marit van Dijk Developer Advocate

Slide 2

Slide 2 text

Question ●Can we do (strict) TDD with AI?

Slide 3

Slide 3 text

Agenda ●What is TDD? ●TDD with AI: ●A series of experiments ●Results

Slide 4

Slide 4 text

Test-Driven Development

Slide 5

Slide 5 text

Test-Driven Development 1. Write a failing test (Red)

Slide 6

Slide 6 text

Failing test Never trust a test you haven’t seen fail https://maritvandijk.com/use-testing-to-develop-better-software-faster/

Slide 7

Slide 7 text

Test-Driven Development 2. Write just enough code pass the test (Green)

Slide 8

Slide 8 text

YAGNI You Ain't Gonna Need It

Slide 9

Slide 9 text

Test-Driven Development 3. Improve the code, without changing its behaviour (Refactor)

Slide 10

Slide 10 text

TDD is not about testing

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Think about • Intended behavior

Slide 13

Slide 13 text

Think about • Intended behavior • How to test it

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

What does this mean in the age of AI?

Slide 16

Slide 16 text

Code completion is here ● Auto completion while typing ● with syntax highlighting

Slide 17

Slide 17 text

Tab-Driven Development Cloud completion Local completion

Slide 18

Slide 18 text

Side note You can change the key used for completion 🤫

Slide 19

Slide 19 text

Accept partial suggestion macOS Windows / Linux Accept suggestion ⇥ TAB Accept line ⌘→ END Accept word ⌥→ Ctrl+→

Slide 20

Slide 20 text

Tab-Driven Development For the purpose of this talk: Tab-Driven Development = AI assisted programming

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Generate tests

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

What about the other way around?

Slide 25

Slide 25 text

What about the other way around? • Can we generate code from tests?

Slide 26

Slide 26 text

Tools

Slide 27

Slide 27 text

AI Assistant  —

Slide 28

Slide 28 text

AI Assistant  —

Slide 29

Slide 29 text

Plugin

Slide 30

Slide 30 text

JetBrains Junie  —

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Plugin

Slide 33

Slide 33 text

Junie A coding agent by JetBrains

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

LLM demo

Slide 36

Slide 36 text

A series of experiments

Slide 37

Slide 37 text

Experiments: "TDD" with AI Assistant • No equivalent "Generate code from tests"

Slide 38

Slide 38 text

Experiments: "TDD" with AI Assistant • No equivalent "Generate code from tests" • Find the best way to do so

Slide 39

Slide 39 text

Experiments: "TDD" with AI Assistant • Problem: FizzBuzz Tried different options: 1. Strict TDD (Red - Green - Refactor) 2. From Chat 3. Inline

Slide 40

Slide 40 text

Experiment 1: Strict TDD • Problem: FizzBuzz 1. Write a test 2. Ask for implementation 3. (Ask for) refactoring

Slide 41

Slide 41 text

Experiment 1: Strict TDD • Problem: FizzBuzz 1. Write a test 2. Ask for implementation 3. (Ask for) refactoring • Code completion kicks in for known problem.

Slide 42

Slide 42 text

Experiment 2: From Chat • Problem: FizzBuzz 1. Ask AI Assistant about FizzBuzz

Slide 43

Slide 43 text

Experiment 2: From Chat • Problem: FizzBuzz 1. Ask AI Assistant about FizzBuzz 2. Ask for list of test cases

Slide 44

Slide 44 text

Experiment 2: From Chat • Problem: FizzBuzz 1. Ask AI Assistant about FizzBuzz 2. Ask for list of test cases 3. Ask for tests in Java

Slide 45

Slide 45 text

Experiment 2: From Chat • Problem: FizzBuzz 1. Ask AI Assistant about FizzBuzz 2. Ask for list of test cases 3. Ask for tests in Java

Slide 46

Slide 46 text

Experiment 2: From Chat • Problem: FizzBuzz 1. Ask AI Assistant about FizzBuzz 2. Ask for list of test cases 3. Ask for tests in Java 4. Ask for implementation

Slide 47

Slide 47 text

Experiment 2: From Chat • Problem: FizzBuzz 1. Ask AI Assistant about FizzBuzz 2. Ask for list of test cases 3. Ask for tests in Java 4. Ask for implementation

Slide 48

Slide 48 text

Experiment 3: Inline • Problem: FizzBuzz 1. Ask AI Assistant for test cases based on description

Slide 49

Slide 49 text

Experiment 3: Inline

Slide 50

Slide 50 text

Experiment 3: Inline • Problem: FizzBuzz 1. Ask AI Assistant for test cases based on description 2. Ask to implement code that passes the tests

Slide 51

Slide 51 text

Experiment 3: Inline • Wrong place in file • But: Easy to fix • Move statement up/down

Slide 52

Slide 52 text

TDD workflow with AI Assistant 1. Create test(s) 2. Ask AI Assistant for code that passes the tests 3. Apply code from chat / Generate code inline • Works with simple problems

Slide 53

Slide 53 text

Result Can do (strict) TDD with AI Assistant Can apply code directly (no copy-paste)

Slide 54

Slide 54 text

Experiment: Strict TDD • Problem: FizzBuzz • Strict TDD with Junie (Red/Green/Refactor): 1. Write a test 2. Ask Junie for implementation

Slide 55

Slide 55 text

Experiment: Strict TDD • Provide Junie with Guidelines first

Slide 56

Slide 56 text

Experiment: Strict TDD

Slide 57

Slide 57 text

Experiment: Strict TDD

Slide 58

Slide 58 text

Experiment: Strict TDD

Slide 59

Slide 59 text

Experiment: Strict TDD

Slide 60

Slide 60 text

Experiment: Strict TDD • Problem: FizzBuzz • DEMO

Slide 61

Slide 61 text

Experiment: Strict TDD • Problem: FizzBuzz • Junie creates the right code in the right location • Junie runs the tests & makes sure they pass

Slide 62

Slide 62 text

TDD workflow with Junie 1. Create tests 2. Ask Junie for code that passes the tests 3. Repeat • Junie creates the right code in the right location • Junie runs the tests & makes sure they pass

Slide 63

Slide 63 text

Experiment: Strict TDD - Result Can do (strict) TDD with Junie Tips: • Provide guidelines • Take turns

Slide 64

Slide 64 text

Toy problems • One class • Limited functionality • What about multiple classes?

Slide 65

Slide 65 text

Experiment: Multiple files • Problem: Calculator • Tests require multiple classes + interface

Slide 66

Slide 66 text

Experiment: Multiple files • Problem: Calculator • Can generate multiple classes • but not always...

Slide 67

Slide 67 text

Experiment: Multiple files • Problem: Calculator • Can generate multiple classes • Have to apply each separately

Slide 68

Slide 68 text

Experiment: Multiple files • Problem: Calculator • Can generate multiple classes • Have to apply each separately • Tests pass • but not always...

Slide 69

Slide 69 text

Experiment: Multiple files • Problem: Calculator • Provide set of tests • Ask for implementation

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

Experiment: Multiple files

Slide 72

Slide 72 text

Experiment: Multiple files

Slide 73

Slide 73 text

Experiment: Multiple files • Problem: Calculator • DEMO

Slide 74

Slide 74 text

Experiment: Multiple files • Problem: Calculator • Junie creates the right files in the right location • Junie runs the tests & makes sure they pass

Slide 75

Slide 75 text

Toy problems • What about actual apps?

Slide 76

Slide 76 text

Experiment: Build a small app test first • Ask Junie to generate implementation based on tests

Slide 77

Slide 77 text

Experiment: Build a small app test first • We need an example • With tests • Let's reverse engineer!

Slide 78

Slide 78 text

Example: Build a small app • Create Spring Boot project • Ask Junie to implement small app

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

Example: Build a small app • Add task • Mark task done • Delete task

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

The code is tested • Junie added tests • Junie ran tests • How good are the tests?

Slide 84

Slide 84 text

Example: Build a small app Let's use Junie to check up on Junie...

Slide 85

Slide 85 text

Example: Build a small app Let's use Junie to check up on Junie... 1. Add coverage with JaCoCo

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

Example: Build a small app Let's use Junie to check up on Junie... 1. Add coverage with JaCoCo

Slide 88

Slide 88 text

Example: Build a small app Let's use Junie to check up on Junie... 1. Add coverage with JaCoCo Tip: Ask to make sure the report is generated and to add how to generate the report to the README.

Slide 89

Slide 89 text

Example: Build a small app Let's use Junie to check up on Junie... 1. Add coverage with JaCoCo 2. Add mutation testing with Pitest

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

Example: Build a small app Let's use Junie to check up on Junie... 1. Add coverage with JaCoCo 2. Add mutation testing with Pitest

Slide 92

Slide 92 text

Example: Build a small app Let's use Junie to check up on Junie... 1. Add coverage with JaCoCo 2. Add mutation testing with Pitest Tip: Ask to make sure the report is generated and to add how to generate the report to the README.

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

Example: Build a small app Use Junie to increase test coverage

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

Example: Build a small app Use Junie to increase test coverage

Slide 97

Slide 97 text

Experiment: Rebuild the small app • Remove implementation • Ask Junie to build app based on tests • Might get it right based on project history (?)

Slide 98

Slide 98 text

Experiment: Rebuild the small app • Create empty project • Copy in the tests • Ask Junie to build app based on tests

Slide 99

Slide 99 text

Experiment: Rebuild the small app

Slide 100

Slide 100 text

No content

Slide 101

Slide 101 text

Experiment: Rebuild the small app • Tests don't always pass on first try

Slide 102

Slide 102 text

Experiment: Rebuild the small app • Tests don't always pass on first try • Junie makes fixes and tries again

Slide 103

Slide 103 text

Experiment: Rebuild the small app • Working app • Not always the same • Plain HTML

Slide 104

Slide 104 text

Experiment: Rebuild the small app • Helps to provide guidelines

Slide 105

Slide 105 text

Experiment: Rebuild the small app • Helps to provide guidelines

Slide 106

Slide 106 text

Experiment: Rebuild the small app • Helps to provide guidelines Tip: Ask AI Assistant to generate visual guidelines (based on existing implementation).

Slide 107

Slide 107 text

Experiment: Build small app test-first • Application: Todo-app • DEMO

Slide 108

Slide 108 text

Experiment: Build small app test-first • Application: Bookmarks • DEMO

Slide 109

Slide 109 text

Results • Tests might not be enough • Might need guidelines

Slide 110

Slide 110 text

We need more than passing tests • Does the application start up? • What about security? • Performance? • Code quality? • Etc

Slide 111

Slide 111 text

No content

Slide 112

Slide 112 text

A learning curve • Which requirements • Which guidelines • Which prompts • Which models

Slide 113

Slide 113 text

No content

Slide 114

Slide 114 text

No content

Slide 115

Slide 115 text

No content

Slide 116

Slide 116 text

Generating

Slide 117

Slide 117 text

No content

Slide 118

Slide 118 text

Can we do (strict) TDD with AI? • While it's possible to do (strict) TDD with AI • Generating code based on tests alone does not always give (me) reliable results

Slide 119

Slide 119 text

There is still value in • Defining requirements & guidelines • Fast feedback, like tests • Small(er) increments

Slide 120

Slide 120 text

No content

Slide 121

Slide 121 text

Debugging • Debugging is harder if you didn't build / understand it

Slide 122

Slide 122 text

No content

Slide 123

Slide 123 text

https://x.com/karpathy/status/1617979122625712128

Slide 124

Slide 124 text

Slides & More https://maritvandijk.com/presentations/tdd-test-vs-tab/