7-9-26 - Boston Java Users ACM Chapter - Reinventing testing practices in the AI era

From https://www.meetup.com/nejug1/events/315171444/

Testing is hard, which is why developers tend to avoid it. Testing non-deterministic things is even harder, which is unfortunate, since we're all writing AI-infused applications, and AI models are notoriously non-deterministic. What happens when the applications start using advanced features, such as RAG, tools, and agents? How do you test these applications? There must be some tools, technologies, and practices out there that can help, while not costing your organization lots of money!

Join Eric as he visits a topic he’s been chasing for well over a year. The AI landscape changes at a breathtaking pace, so what new capabilities and strategies are available?

Hopefully by the end of the presentation you will be able to answer the question "If I change my model/prompt/application, did I get better or worse"?

Transcript

2. @edeandrea What are you hoping to learn here? What are you hoping to learn here? What are you going to leave with?
3. @edeandrea
4. @edeandrea
5. @edeandrea
6. @edeandrea
7. @edeandrea Because we are not data scientists We integrate existing models Do you really want to do these in Python? • Transactions • Security • Scalability • Observability into enterprise- grade systems and applications Java??? 😯 … no seriously … why not Python? 🤔
8. @edeandrea I don’t care if it works on your Jupyter notebook We are not shipping your Jupyter notebook
9. @edeandrea • Java Champion • 27+ years software development experience • Works on Open Source projects Quarkus LangChain4j, Quarkus LangChain4j Docling Java Langfuse Java, Quarkus Langfuse Spring Boot, Spring Framework, Spring Security Testcontainers Wiremock Microcks • Boston Java Users ACM Chapter Vice Chair & Board Member • Published Author • Cat lover Who am I?
10. @edeandrea • Showcase & explain Quarkus, how it enables modern Java development & the Kubernetes- native experience • Introduce familiar Spring concepts, constructs, & conventions and how they map to Quarkus • Equivalent code examples between Quarkus and Spring as well as emphasis on testing patterns & practices 1 0 https://red.ht/quarkus-spring-devs
13. @edeandrea What’s happening in industry? • Standardization ◦ Or lack thereof (lots of competing standards)? • Distributed • Orchestrated • Agentic • Agents • Agentic Agents • Autonomous Agents • Autonomous Agentic Agents Smells like microservices?
14. @edeandrea
15. @edeandrea How does your DevOps evolve when you infuse your applications with AI?
16. @edeandrea
17. @edeandrea DevOps Evolution Dev Ops Release Deploy Operate Monitor Plan Code Build Test Train Evaluate Deploy Collect Evaluate Curate Analyze Data ML
18. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application What’s the difference between these?
19. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these?
20. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these? What do we do?
21. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these?
22. @edeandrea @edeandrea end-to-end tests unit tests integration tests low effort high realism high value
23. @edeandrea Testing AI Interactions https://youtu.be/S3XFFeTnILM https://github.com/edeandrea/non-deterministic-no-problem/tree/langfuse/parasol-app
24. @edeandrea Testing AI Interactions 24
25. @edeandrea https://library.wiremock.org/catalog/api/o/openai.com/openai-com https://mockgpt.wiremock.io https://docs.quarkiverse.io/quarkus-wiremock/dev
26. @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points Observability (metrics, tracing, logs, auditing) Fault Tolerance (timeout, bulkhead, circuit breaker, rate limiting, fallbacks, …) What’s the difference between these?
27. @edeandrea This isn’t the answer!
28. @edeandrea @edeandrea Is that enough?
29. @edeandrea What does failure look like? What do we need to do differently?
30. @edeandrea https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw
31. @edeandrea @edeandrea end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints tests using AI
32. @edeandrea 3-Tier Evaluation Strategy Tier 1: Per-trace Real-time checks ensuring immediate response quality.
33. @edeandrea 3-Tier Evaluation Strategy Tier 1: Per-trace Real-time checks ensuring immediate response quality. Tier 2: Per-session Post-hoc analysis of full user journeys measuring sentiment and goal success.
34. @edeandrea 3-Tier Evaluation Strategy Tier 1: Per-trace Real-time checks ensuring immediate response quality. Tier 2: Per-session Post-hoc analysis of full user journeys measuring sentiment and goal success. Tier 3: Drift detection Automated gates comparing production metrics against historical baselines.
35. @edeandrea 3-Tier Evaluation Strategy Tier 1: Per-trace Real-time checks ensuring immediate response quality. Tier 2: Per-session Post-hoc analysis of full user journeys measuring sentiment and goal success. Tier 3: Drift detection Automated gates comparing production metrics against historical baselines. Regression No intentional changes (model update, infrastructure, etc) Change Validation Intentional changes (prompt, business logic, data, etc)
36. @edeandrea What happens when we do this?
37. @edeandrea What happens when we do this? Is this a failure?
38. @edeandrea @edeandrea Demo time!
39. @edeandrea Observability Collect metrics - Exposed as Prometheus - Track token usage & cost OpenTelemetry Tracing - Trace interactions with the LLM Auditing - Track of interactions with the LLM - Ability to replay & re-score interactions Continuous evaluation - Evaluate interactions in real time
40. @edeandrea @edeandrea Change Validation
41. @edeandrea Rescoring - Evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
42. @edeandrea Rescoring - Evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
43. @edeandrea Rescoring - Evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
44. @edeandrea @edeandrea Takeaways
45. @edeandrea • Naming things is still the hardest thing in computer science • Java is still relevant • Remember the testing pyramid! Use appropriate tools at each level! • LangChain4j & Quarkus are awesome! They provide foundational building blocks! • Don’t build observability into your apps - build it around your apps • Test in production! • Write tests, expect change and failure, deploy often • AI is just an API call Actual takeaways
46. @edeandrea @edeandrea Thank You!