QA and Testing Knowledge for Vibe Coders

Vibe Coders Tokyo #11 - Good Quality Apps

https://luma.com/eh7b46r1

Transcript

1. Vibe Coders Tokyo #11 - Good Quality Apps · Oct 14, 2026 QA & Testing Knowledge for Vibe Coders Yoshiki Ito
2. About this talk QA and software testing knowledge that every vibe coder should know 2
3. About me Yoshiki Ito QA engineer from Japan 10+ years in software testing and quality Started my career as a tester — no experience as a developer 3
4. Today 1. QA and vibe coding 2. Quality 3. Testing 4. Wrap-up 4
5. Today 1. QA and vibe coding 2. Quality 3. Testing 4. Wrap-up 5
6. What is Quality Assurance? Making sure your product has good quality Measuring and improving quality — through reviews, testing, better team communication, and more 6
7. What is quality? Quality is value to some person. — Gerald M. Weinberg "To some person" is the key. In the end, users decide if the quality is good or bad. 7
8. Quality and testing matter for vibe coders, too Some of you want to ship products that solve someone's problem — not just handy tools for yourself When other people use your app, quality matters. That is the difference between a hobby and a product That is why I want you to know about QA 8
9. Today 1. QA and vibe coding 2. Quality 3. Testing 4. Wrap-up 9
10. Know the name, hand it to AI You don't need to master the theory. Use old QA ideas as keywords Set priorities See more angles Which feature to build first Quality beyond "it works" Kano model ISO 25010 Build it safely "Production-ready" One phrase for security Give the AI the keyword. The AI fills in the details 10
11. The Kano model You want to build many features. Where do you start? → Think in 3 types. Type Must-be Satisfied Attractive Not done Performance Must-be Done well Dissatisfied What it is, and an example (booking app) Expected. If missing, users are very unhappy e.g. Bookings are saved correctly better it is, the happier users are Performance The e.g. Fast pages, fewer steps expected, but delights users Attractive Not e.g. AI suggests open time slots For example: get must-be quality right first, then improve attractive quality. Reference: What Is the Kano Model? - ASQ 11
12. Quality characteristics When we build apps, we tend to think only about "Does it work?" (functionality). But quality has many more parts ISO/IEC 25010 12
13. Try this: the "ISO 25010 prompt" After the AI builds a feature, make it a habit to ask: "Review the feature you just built from these ISO 25010 quality characteristics. Find anything that is missing." 1. Reliability: Is there error handling and retry? 2. Usability: Does it prevent mistakes? Does it show loading states? 3. Security: Are inputs checked? Is any authentication missing? 13
14. Are vibe-coded apps secure? 91% of apps had at least one vulnerability About 6.5 per app on average. About 2 in 3 were Critical or High severity Most common: seeing other people's data / weak checks on bad input / broken login Top cause: the AI does not follow security rules nobody stated The AI does not protect what you did not ask for → ask for security explicitly, and check before you launch Source: Deng, Fan, Meng "Understanding the (In)Security of Vibe-Coded Applications" (arXiv:2606.23130, 2026). 200 publicly deployed vibe-coded apps were audited 14
15. Security: what should you do? When asking While building Say "production-ready" Read the warnings Before launch Ask for a self-check "Build it ready for When fixing errors, don't let Postponed tasks, production" the AI weaken login checks permissions, input, and The most effective phrase in secrets the paper 💬 "Before launch, check postponed tasks, login and permissions, input checks, and how secrets are handled" Still not zero → treat it as a prototype, and check it yourself 15
16. Today 1. QA and vibe coding 2. Quality 3. Testing 4. Wrap-up 16
17. What is testing? One way to measure quality Examples Regression testing: checks for unexpected side effects Exploratory testing: hunts for bugs on purpose 17
18. Regression testing Checks that what used to work still works after a change When something breaks, we call it a "regression" A fix in one place can break something that looks unrelated You repeat it after every change, so start with the most important features When you find a bug, add it to your checklist for next time Reference: What is regression testing? - MagicPod Blog (Japanese) 18
19. Real case: a fixed bug came back Twitter (now X), 2010 1. Twitter fixed a bug: text in a tweet could run as code on other people's screens 2. Later, an unrelated site update brought the bug back, without anyone noticing 3. Hovering over a tweet could show pop-ups, or retweet it without consent 4. It took about 4 hours to fix August 2010 Found and fixed a security bug ✓ Later An unrelated site update September 21, 2010 The fixed bug came back ✕ A fixed bug can come back with an unrelated change → add fixed bugs to your tests Source: Twitter Blog "All about the 'onMouseOver' incident" (September 21, 2010) 19
20. So, what should you do? STEP 1 STEP 2 STEP 3 Pick key flows Prepare the tests Run every time Choose the flows that hurt most if they break e.g. See open slots → book → see it in the list → cancel Start with a checklist of steps and expected results Add fixed bugs to it, too Whenever you change something, and before every release Check the key flows with the same tests every time → if something breaks, you notice before users do 20
21. Exploratory testing Instead of following fixed steps, you decide what to test as you go Thinking and testing at the same time Not random clicking: you aim at where bugs are likely to be Set a time limit, like 30 minutes, and note what you tried and found Finds problems that prepared test cases often miss Reference: What is exploratory testing? - MagicPod Blog (Japanese) 21
22. Example: a 15-minute exploration Goal: "Use unusual input and actions on the booking form, and find bookings that should not be possible" What I tried Result What I thought of next Book a date in the past It worked (bug!) "Then, a closed day?" Book on a closed day It worked (bug!) "Two people, same slot?" Same slot in two tabs at once Double booking "Click Submit many times?" Click Submit many times 3 identical bookings (Note it, move to the next goal) Also try: submit empty fields, enter a very long name, go back or reload halfway, check if another account can see your bookings Look at the result, then decide what to try next. That is how exploratory testing works 22
23. Try to break your own app When you build something, you tend to try only the "right" way to use it You know how it should work, so you miss things When testing, switch to a tester's mindset The goal is not "check that it works" but "find bugs" Cause problems on purpose. Sometimes, try to break it You can only do so much alone. Borrow other people's eyes and AI's eyes 23
24. Help from others: bug bash A set time when everyone uses the app at once to find bugs They find uses and bugs outside the builder's assumptions How to run one Set 30–60 minutes. Share which features or goals to focus on Look back at the end, and fix the most important ones first Try bug-bashing each other's apps with other vibe coders! 24
25. Help from AI: exploratory testing An AI agent can use your app in a browser and help with exploratory testing Example: an agent like Claude Code + Playwright MCP As a partner: "How could this app break? List things to try" As a tester: give it a goal, and let it run the app and take notes Example: "Try the booking form with impossible dates and inputs. Find bugs" You make the final call. Do not just trust "no problems found" — check the report 25
26. Today 1. QA and vibe coding 2. Quality 3. Testing 4. Wrap-up 26
27. Wrap-up Quality is value to some person. If others use your app, quality matters Learn ways to think about quality, and tell the AI explicitly Kano model: build must-be quality first Quality characteristics: decide what matters, and ask the AI to self-review from those viewpoints Security: say "production-ready," and ask for a self-check before launch Measure quality with testing Regression testing: prepare tests for key flows, and run them on every change and before release Exploratory testing: try to break your app, looking at each result. Get help from others and AI Use time-tested QA wisdom — quickly, with AI 27
28. If you start tomorrow, try these 3 1. Add "production-ready" when you ask the AI 2. Make one checklist of your key flows 3. Spend 15 minutes trying to break your own app Tools needed: none You do not have to do everything. Start with just one 28
29. Thank you! Yoshiki Ito · QA Engineer