Using TDD to Get Better Results From LLMs/AI

Using TDD to Get Better Results From LLMs/AI @ClareSudbery

YOUR NAMES? @ClareSudbery

@ClareSudbery YOU MIGHT NOT STAY IN YOUR SEAT Be prepared
to move to a different seat!

@ClareSudbery ALL LINKS FROM TODAY (including a link to these
slides)

@ClareSudbery Orientation

BREAK INTRO + PAIRS Exercise Part 1 Agenda 14:00 –
14:10 14:10 – 14:40 14:40 – 14:50 Exercise Part 1 (Continued) 14:50 – 15:00 @ClareSudbery Exercise Part 2 Let me tell you a story… 15:00 – 15:10 15:10 – 15:15 Final discussion 15:15 – 15:30

@ClareSudbery JManc Unconference, July 10th Manchester

@ClareSudbery RAISED HAND MEANS QUIET PLEASE, ATTENTION NEEDED! 00:00 –
05:00 MANY

How many people? @ClareSudbery

LLM experience? @ClareSudbery

@ClareSudbery Windsurf – free until you reach your token limit
(which could happen during the day!) Cursor – free up to a limit, or if you have your own API keys from OpenAI or Anthropic (Claude) VSCode with GitHub CoPilot – free up to a limit Codex (ChatGPT) Claude (reported performance degaradation recently) Download a tool (if you haven’t already) One that will work in an integrated IDE

@ClareSudbery 2 minutes Write your most optimistic LLM-coding goal on
a postit What would be fun to build using LLM-augmented coding? Write clearly – people need to read it Add your name and remember what you wrote!

@ClareSudbery This half of the room, put your sticky notes
up Spread them out along the wall

@ClareSudbery YOU ARE LIKELY TO BE ABOUT TO MOVE SEATS
Be prepared! And remember to introduce yourself… Name + role

@ClareSudbery Other half of the room, take one sticky note
each One that you like the look of

@ClareSudbery Walk around, waving your sticky note Find the person
who wrote it Form a pair / three with them Introduce yourselves! Name + role

TDD • Know what it is? • Used it before?
• Use it regularly? @ClareSudbery

@ClareSudbery Test Driven Development

bit.ly/making-tech-better • Geepaw Hill – Test Driven Development I hosted
a podcast… @ClareSudbery

Exercise, Part One @ClareSudbery

Exercise, Part One Create a music sequencer. The user will
be shown a simple grid. When they click on squares in the grid, musical notes are played. Notes can be toggled on and off. When they click a Play button, the notes in the grid are played in sequence. @ClareSudbery

The Catch This half of the room will use TDD,
and enforce the AI’s use of TDD. The other half are BANNED from any mention of tests. If the AI suggests tests, follow its suggestions but don’t intervene. @ClareSudbery

Exercise, Part One Create a music sequencer. The user will
be shown a simple grid. When they click on squares in the grid, musical notes are played. Notes can be toggled on and off. When they click a Play button, the notes in the grid are played in sequence. @ClareSudbery

Exercise, Part Two @ClareSudbery

Exercise, Part Two Join with another pair. Prove to them
that your app does what it should. @ClareSudbery

Let me tell you a story… @ClareSudbery

@ClareSudbery Things breaking repeatedly

@ClareSudbery Enough. We need tests!

@ClareSudbery "We need tests" is not enough

@ClareSudbery Diminishing returns. Uh-huh.

@ClareSudbery So addictive!

@ClareSudbery Training an LLM to do TDD

@ClareSudbery Independent specification

@ClareSudbery The LLM writes its own processes

@ClareSudbery LLM-AUGMENTED CODING: GOLDEN RULES 1. Encourage the LLM to
ask clarifying questions 2. Move in small steps. Big steps feel fast but they quickly slow you down. 3. Always start with tests. - a. Ask the LLM to write tests before code. - b. Check and refine the tests. - c. Ask the LLM to make the tests pass by implementing the solution. - d. Keep running all tests to make sure nothing is broken. 4. Ask the LLM to summarise conversations. Persist those summaries in files. 5. Use a starter character in each process file. 6. Throw things away! Start again from scratch. Beware the sunk cost fallacy. 7. Regularly start a new chat context. 8. Be polite. Use "please" and "thank you".

@ClareSudbery LLM-AUGMENTED CODING: GOLDEN RULES 9. Give the LLM examples
that will help it to understand 10. Don't write code (first draft) if the LLM can do it for you 11. Beware confirmation bias! 12. Try having two or more LLMs working in parallel 13. Ask the LLM for multiple options, with explanations of pros and cons

DISCUSSION @ClareSudbery

RESULTS • Who completed part 1? • Who completed the
extension? • What problems did you encounter? • Who's happy with the result? What did you like? • If you were successful, why do you think that is? @ClareSudbery

@ClareSudbery

slides)

Context windows @ClareSudbery Context window = num of tokens the
LLM can handle Tokens are chunks of text All training text is broken down into tokens LLMs work by predicting the next most likely token Context windows are large - often millions of tokens Every time we start a new chat, we start filling up the effective context

DEMO @ClareSudbery

Effective context @ClareSudbery Every time we start a new chat,
we start filling up the effective context This seems to be more like 10,000 - 20,000 tokens

Parallel contexts & multiple agents @ClareSudbery

How do you keep up with it all? @ClareSudbery

DON’T PANIC Nobody is up to date with all this
stuff! @ClareSudbery

THE HYPE @ClareSudbery

HUMAN COGNITIVE ABILITIES @ClareSudbery

PRODUCTIVITY & JOB LOSSES @ClareSudbery

THE ECONOMY & THE BUBBLE @ClareSudbery

ETHICS @ClareSudbery

SECURITY & RELIABILITY @ClareSudbery

THE PLANET @ClareSudbery

EFFECTIVE LLM USE @ClareSudbery

Landing in a neat pair-synchronized way. https://www.instagram.com/p/BsY1qBeg5 Qg @ClareSudbery

@ClareSudbery THANK YOU! FEEDBACK

slides)

Using TDD to Get Better Results From LLMs/AI

Using TDD to Get Better Results From LLMs/AI

More Decks by Clare Sudbery

Featured

Transcript