WeAreDevs NA 2026 - Reinventing Testing Practices in the AI Era

From https://www.wearedevelopers.com/world-congress-north-america/agenda/sessions/reinventing-testing-practices-in-the-ai-era-1289026

AI-infused applications demand a rethinking of our testing practices. Developers face a new class of challenge as LLMs become standard integration points in modern applications: non-deterministic behavior that traditional testing approaches were never designed to handle. The current wave of distributed, orchestrated, agentic AI systems is evolving fast and, if we're being honest, it smells a lot like the early days of microservices.

In this session, we'll explore how your DevOps and testing practices must evolve when you wire AI into your applications. Not all AI failures look the same, and recognizing the difference is the first step toward building systems you can actually trust. We'll walk through practical testing and observability strategies, using open source tools that give you confidence in AI-infused applications at every layer of the stack.

You'll leave with a concrete mental model for reasoning about AI failures and one grounding question: What if AI was just an API call?

Transcript

2. Who am I? • Java Champion • 27+ years software development experience • Works on Open Source projects Quarkus LangChain4j, Quarkus LangChain4j Docling Java, Quarkus Docling Langfuse Java, Quarkus Langfuse Spring Boot, Spring Framework, Spring Security Testcontainers Wiremock Microcks • Boston Java Users ACM Chapter Vice Chair & Board Member • Published Author • Cat lover @edeandrea
3. @edeandrea
4. @edeandrea
7. What’s the difference between these? CRUD application Application Database Microservice Application Service AI-Infused application Application Model @edeandrea
8. What’s the difference between these? CRUD application Database Application Microservice Application AI-Infused application Application Integration Points Service Model @edeandrea
9. What’s the difference between these? CRUD application Database Application Microservice Application AI-Infused application Application Integration Points Service What do we Model do? @edeandrea
10. What’s the difference between these? CRUD application Application Microservice Application AI-Infused application Application Database Integration Points Service Model @edeandrea
11. high realism high value end-to-end tests integration tests unit tests low effort @edeandrea
12. https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw @edeandrea
13. e v a h e s e h n? t o o d m t a m o h W in c https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw @edeandrea
14. What does failure look like? The system worked! @edeandrea
15. r e w s n ! a g e n o h r T w s a w What does failure look like? The system worked! @edeandrea
16. Traditional tests tell us if the system worked. @edeandrea
17. o i t a s e r u l i a f n Traditional tests tell us if the system worked. t In r g e @edeandrea
18. Evaluations tell us whether it worked well. @edeandrea
19. c i t n a m e S Evaluations tell us whether it worked well. s e r u l i fa @edeandrea
20. What happens when we do this? @edeandrea
21. What happens when we do this? Is a s i th ? e r u l i fa @edeandrea
22. What’s an evaluation? Unit test assertEquals(expected, actual) exact match pass / fail same input, same answer @edeandrea
23. What’s an evaluation? Unit test Evaluation assertEquals(expected, actual) “How good was this answer?” exact match semantic similarity, correctness, domain rules pass / fail score + explanation same input, same answer same input, different answers @edeandrea
24. high realism end-to-end tests tests with application server test REST endpoints integration tests tests using AI unit tests low effort @edeandrea
25. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. @edeandrea
26. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. Tier 2: Per-session one full conversation End-of-session analysis of full user journeys measuring sentiment and goal success. @edeandrea
27. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. Tier 2: Per-session Tier 3: Drift detection End-of-session analysis of full user journeys measuring sentiment and goal success. Automated gates comparing production metrics against historical baselines. one full conversation all traffic, over time @edeandrea
28. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. Regression No intentional changes (model update, infrastructure, etc) Tier 2: Per-session Tier 3: Drift detection End-of-session analysis of full user journeys measuring sentiment and goal success. Automated gates comparing production metrics against historical baselines. one full conversation all traffic, over time Change Validation Intentional changes (prompt, business logic, data, etc) @edeandrea
29. I changed something. Did I make it worse? @edeandrea
30. Rescoring - Evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces an actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation @edeandrea
31. Rescoring - Evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces an actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation @edeandrea
32. Demo time! @edeandrea
33. Takeaways • Remember the testing pyramid — use the right tool at each level • Don’t build observability into your apps — build it around them • Test in production. It’s the only place the real inputs are. • AI is just an API call — the answer isn’t. • Traditional tests tell you if the system worked. Evaluations tell you whether it worked well. @edeandrea
34. Thank You! bit.ly/4xLY9mf @edeandrea