Bucharest Tech Week 2026 - Reinventing testing practices in the AI era

AI-infused applications demand a rethinking of our testing practices.

Developers face a new class of challenge as LLMs become standard integration points in modern applications: non-deterministic behavior that traditional testing approaches were never designed to handle. The current wave of distributed, orchestrated, agentic AI systems is evolving fast and, if we're being honest, it smells a lot like the early days of microservices.

In this session, we'll explore how your DevOps and testing practices must evolve when you wire AI into your applications. Not all AI failures look the same, and recognizing the difference is the first step toward building systems you can actually trust.

We'll walk through practical testing and observability strategies, using open source tools that give you confidence in AI-infused applications at every layer of the stack.

You'll leave with a concrete mental model for reasoning about AI failures and one grounding question: What if AI was just an API call?

Transcript

2. @edeandrea
3. @edeandrea
4. @edeandrea
5. @edeandrea
6. @edeandrea Because we are not data scientists We integrate existing models Do you really want to do these in Python? • Transactions • Security • Scalability • Observability into enterprise- grade systems and applications Java??? 😯 … no seriously … why not Python? 🤔
7. @edeandrea I don’t care if it works on your Jupyter notebook We are not shipping your Jupyter notebook
8. @edeandrea • Java Champion • 27+ years software development experience • Works on Open Source projects Quarkus LangChain4j, Quarkus LangChain4j Docling Java Langfuse Java, Quarkus Langfuse Spring Boot, Spring Framework, Spring Security Testcontainers Wiremock Microcks • Boston Java Users ACM Chapter Vice Chair & Board Member • Published Author • Cat lover Who am I?
9. @edeandrea • Showcase & explain Quarkus, how it enables modern Java development & the Kubernetes- native experience • Introduce familiar Spring concepts, constructs, & conventions and how they map to Quarkus • Equivalent code examples between Quarkus and Spring as well as emphasis on testing patterns & practices 9 https://red.ht/quarkus-spring-devs
11. @edeandrea What are you hoping to learn here? What are you hoping to learn here? What are you going to leave with?
14. @edeandrea What’s happening in industry? • Standardization ◦ Or lack thereof (lots of competing standards)? • Distributed • Orchestrated • Agentic • Agents • Agentic Agents • Autonomous Agents • Autonomous Agentic Agents Smells like microservices?
15. @edeandrea
16. @edeandrea How does your DevOps evolve when you infuse your applications with AI?
17. @edeandrea
18. @edeandrea DevOps Evolution Dev Ops Release Deploy Operate Monitor Plan Code Build Test Train Evaluate Deploy Collect Evaluate Curate Analyze Data ML
19. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application What’s the difference between these?
20. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these?
21. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these? What do we do?
22. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these?
23. @edeandrea @edeandrea end-to-end tests unit tests integration tests low effort high realism high value
24. @edeandrea Testing AI Interactions https://youtu.be/S3XFFeTnILM https://github.com/edeandrea/non-deterministic-no-problem/tree/langfuse/parasol-app
25. @edeandrea Testing AI Interactions 25
26. @edeandrea https://library.wiremock.org/catalog/api/o/openai.com/openai-com https://mockgpt.wiremock.io https://docs.quarkiverse.io/quarkus-wiremock/dev
27. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points Observability (metrics, tracing, logs, auditing) Fault Tolerance (timeout, bulkhead, circuit breaker, rate limiting, fallbacks, …) What’s the difference between these?
28. @edeandrea This isn’t the answer!
29. @edeandrea @edeandrea Is that enough?
30. @edeandrea What does failure look like? What do we need to do differently?
31. @edeandrea https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw
32. @edeandrea What happens when we do this?
33. @edeandrea What happens when we do this? Is this a failure?
34. @edeandrea @edeandrea end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints tests using AI
35. @edeandrea Observability Collect metrics - Exposed as Prometheus - Track token usage & cost OpenTelemetry Tracing - Trace interactions with the LLM Auditing - Track of interactions with the LLM - Ability to replay & re-score interactions Continuous evaluation - Evaluate interactions in real time
36. @edeandrea @edeandrea Re-Evaluations
37. @edeandrea Rescoring - Evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
38. @edeandrea Rescoring - Evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
39. @edeandrea Rescoring - Evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
40. @edeandrea @edeandrea Takeaways
41. @edeandrea • Naming things is still the hardest thing in computer science • Java is still relevant • Remember the testing pyramid! Use appropriate tools at each level! • LangChain4j & Quarkus are awesome! They provide foundational building blocks! • Don’t build observability into your apps - build it around your apps • Test in production! • Write tests, expect change and failure, deploy often • AI is just an API call Actual takeaways
43. @edeandrea @edeandrea Thank You! Slides https://bit.ly/ro-ai-testing