Did you really get better?

From https://www.jfokus.se/talks.html?showid=2701

Testing is hard, which is why developers tend to avoid it. Testing non-deterministic things is even harder, which is unfortunate, since we're all writing AI-infused applications, and AI models are notoriously non-deterministic. What happens when the applications start using advanced features, such as RAG, tools, and agents? How do you test these applications? There must be some tools, technologies, and practices out there that can help, while not costing your organization lots of money!

Join Java Champions Oleg & Eric in this session as they revisit a topic they debuted at JFokus last year. The AI landscape changes at a breathtaking pace, so what new capabilities and strategies have come along in the last year?

Hopefully by the end of the presentation you will be able to answer the question "If I change my model/prompt/application, did I get better or worse"?

Transcript

1. @shelajev @edeandrea Eric Deandrea, Sr. Principal Software Engineer Oleg Šelajev, AI Developer Relations Did you really get better?
2. @shelajev @edeandrea • Java Champion • 27+ years software development experience • Works on Open Source projects Quarkus LangChain4j (& Quarkus LangChain4j) Docking Java (Project lead) Spring Boot, Spring Framework, Spring Security Wiremock Testcontainers • Boston Java Users ACM Chapter Vice Chair • Published Author • Black belt in martial arts • Cat lover About Us
3. @shelajev @edeandrea • Showcase & explain Quarkus, how it enables modern Java development & the Kubernetes-native experience • Introduce familiar Spring concepts, constructs, & conventions and how they map to Quarkus • Equivalent code examples between Quarkus and Spring as well as emphasis on testing patterns & practices 3 https://red.ht/quarkus-spring-devs
4. @shelajev @edeandrea • Also a Java Champion • Java developer • ~12 years Developer Advocate • Loves to stare at Open Source projects • Allergic to cats About Us
5. @shelajev @edeandrea What are you hoping to learn here? What are you hoping to learn here? What are you going to leave with?
6. @shelajev @edeandrea @shelajev @edeandrea Did we get better or worse with this release? (& can we figure it out before we release?)
7. @shelajev @edeandrea
8. @shelajev @edeandrea Nature is healing… JFokus 2025 JFokus 2026
9. @shelajev @edeandrea What’s changed in the last year? • Standardization ◦ Or lack thereof (lots of competing standards)? • Distributed • Orchestrated • Agentic • Agents • Agentic Agents • Autonomous Agents • Autonomous Agentic Agents Smells like microservices?
10. @shelajev @edeandrea @shelajev @edeandrea “Types” of AI
11. @shelajev @edeandrea AI replacing humans
12. @shelajev @edeandrea AI replacing software
13. @shelajev @edeandrea
14. @shelajev @edeandrea How does your DevOps evolve when you infuse your applications with AI?
15. @shelajev @edeandrea
16. @shelajev @edeandrea DevOps Evolution Dev Ops Release Deploy Operate Monitor Plan Code Build Test Train Evaluate Deploy Collect Evaluate Curate Analyze Data ML
17. @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem
18. @shelajev @edeandrea Chat Bot Web Socket Claim AI Assistant Claim Status Notification Tool invocation Generate Email AI Assistant Output Guardrails Politeness AI Assistant AI replacing humans AI replacing software https://github.com/edeandrea/non-deterministic-no-problem Code I write Voodoo magic Legend RAG Retrieval Input Guardrails Could be an agent? Could be an agent?
19. @shelajev @edeandrea AI: String -> String Agent: String -> IO String For you, Haskell aficionados:
20. @shelajev @edeandrea
21. @shelajev @edeandrea @shelajev @edeandrea Vanilla AI
22. @shelajev @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application What’s the difference between these?
23. @shelajev @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points What’s the difference between these?
24. @shelajev @edeandrea Testing AI Replacing Humans
25. @shelajev @edeandrea Testing AI Replacing Humans 25
26. @shelajev @edeandrea Rethink your approach to tests
27. @shelajev @edeandrea Signal from tests: ❌ - stuff needs fixing ✅ - confident to release
28. @shelajev @edeandrea Signal from tests: - stuff needs fixing - confident to release Purpose of tests: ❌ - prevent breaking prod ✅ - continuously improve your app
29. @shelajev @edeandrea https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw
30. @shelajev @edeandrea What does failure look like? What do we need to do differently?
31. @shelajev @edeandrea This isn’t the answer!
32. @shelajev @edeandrea Application Database Application Service CRUD application Microservice Application Model AI-Infused application Integration Points Observability (metrics, tracing, logs, auditing) Fault Tolerance (timeout, bulkhead, circuit breaker, rate limiting, fallbacks, …) What’s the difference between these?
33. @shelajev @edeandrea https://library.wiremock.org/catalog/api/o/openai.com/openai-com https://mockgpt.wiremock.io https://docs.quarkiverse.io/quarkus-wiremock/dev
34. @shelajev @edeandrea What happens when we do this?
35. @shelajev @edeandrea What happens when we do this? Is this an error?
36. @shelajev @edeandrea @shelajev @edeandrea end-to-end tests unit tests integration tests low effort high realism high value
37. @shelajev @edeandrea @shelajev @edeandrea end-to-end tests unit tests integration tests low effort high realism tests with application server test REST endpoints tests using AI
38. @shelajev @edeandrea
39. @shelajev @edeandrea
40. @shelajev @edeandrea Stupidity will always find a way…
41. @shelajev @edeandrea Stupidity Prompt: Please return a JSON document in the following format: { “name: “String”, “countryOfOrigin”: “String”} Response: Sure I’d love to give you some JSON! Here it is: ```json { “name”: “Eric”, “countryOfOrigin”: “USA” } ```
42. @shelajev @edeandrea Prompt Engineering and team topologies I said JSON!
43. @shelajev @edeandrea Guardrails - Out of the box in LangChain4j & Quarkus! - Functions used to validate the input and output of the model - Detect invalid input or output - Detect prompt injection - Detect hallucination - Chain of guardrails - Sequential - Stop at first failure
44. @shelajev @edeandrea Retry and Reprompt Output guardrails can have 4 different outcomes: - Success - Response is passed to the caller or next guardrail - Fatal - Stop and throw an exception - Retry - Call the model again with the same context we never know ;-) - Reprompt - Call the model again with another message in the model indicating how to fix the response
45. @shelajev @edeandrea
46. @shelajev @edeandrea @shelajev @edeandrea
47. @shelajev @edeandrea
48. @shelajev @edeandrea Stupidity will always find a way…
49. @shelajev @edeandrea Observability
50. @shelajev @edeandrea Observability Collect metrics - Exposed as Prometheus - Track token usage & cost OpenTelemetry Tracing - Trace interactions with the LLM Auditing - Track of interactions with the LLM - Ability to replay & re-score interactions
51. @shelajev @edeandrea Observability
52. @shelajev @edeandrea @shelajev @edeandrea Continuous Evaluation
53. @shelajev @edeandrea
54. @shelajev @edeandrea Continuous Scoring Application doesn’t know Its being scored!
55. @shelajev @edeandrea Continuous Scoring
56. @shelajev @edeandrea Continuous Scoring @shelajev @edeandrea
57. @shelajev @edeandrea Continuous Scoring (interaction capture) @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem/blob/main/parasol-app/src/main/java/ai/scoring/InteractionPublisher.java
58. @shelajev @edeandrea Continuous Scoring - Communication @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem/blob/main/openapi/ai-interactions.yml
59. @shelajev @edeandrea Continuous Scoring - Client Communication @shelajev @edeandrea https://docs.quarkiverse.io/quarkus-openapi-generator/dev/client.html
60. @shelajev @edeandrea Continuous Scoring - Scorer Communication
61. @shelajev @edeandrea Continuous Scoring - Scorer Communication @shelajev @edeandrea
62. @shelajev @edeandrea Continuous Scoring @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/scoring/InteractionScorer.java
63. @shelajev @edeandrea Continuous Scoring @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/scoring/InteractionScoringInterceptor.java
64. @shelajev @edeandrea @shelajev @edeandrea Rescoring
65. @shelajev @edeandrea Continuous Scoring
66. @shelajev @edeandrea Rescoring
67. @shelajev @edeandrea Continuous Scoring @shelajev @edeandrea
68. @shelajev @edeandrea Rescoring @shelajev @edeandrea
69. @shelajev @edeandrea Rescoring - Communication @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem/blob/main/openapi/ai-interactions.yml Same as Continuous Scoring
70. @shelajev @edeandrea Rescoring @shelajev @edeandrea https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/scoring/InteractionScorer.java https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/evaluation/InteractionEvaluator.java
71. @shelajev @edeandrea @shelajev @edeandrea Evaluations
72. @shelajev @edeandrea Rescoring - Evaluation @shelajev @edeandrea https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
73. @shelajev @edeandrea Rescoring - Evaluation @shelajev @edeandrea https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
74. @shelajev @edeandrea Rescoring - Evaluation @shelajev @edeandrea https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces and actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation
75. @shelajev @edeandrea Rescoring - Loading Samples https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_custom_sample_loaders https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/evaluation/InteractionsSampleLoader.java
76. @shelajev @edeandrea Rescoring - Evaluation Strategy https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_built_in_evaluation_strategies https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/evaluation/InteractionEvaluator.java
77. @shelajev @edeandrea Rescoring - Scoring https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_fluent_builder_api https://github.com/edeandrea/non-deterministic-no-problem/blob/main/ai-scorer/src/main/java/ai/scoring/evaluation/InteractionEvaluator.java
78. @shelajev @edeandrea @shelajev @edeandrea Putting it all together
79. @shelajev @edeandrea What happens when we do this?
80. @shelajev @edeandrea @shelajev @edeandrea What’s still missing?
81. @shelajev @edeandrea RAG Evaluation: Two Surfaces • RAG introduces new surfaces for necessary evaluation • Evaluate the retrieval of relevant context documents • Evaluate the generation based on the retrieved context • Semantic similarity checks are crucial for RAG evaluation
82. @shelajev @edeandrea @shelajev @edeandrea Takeaways
83. @shelajev @edeandrea Selection decisions are not application based https://artificialanalysis.ai/models
84. @shelajev @edeandrea • Naming things is still the hardest thing in computer science • LangChain4j & Quarkus are awesome! They provide foundational building blocks! • Don’t build observability into your apps - build it around your apps • Don’t forget your craft: DevOps process is there to help • Write tests, expect change and failure, deploy often • AI is just an API call Actual takeaways
85. @shelajev @edeandrea https://quarkus.io @quarkusio https://quarkusio.zulipchat.com @quarkus.io
86. @shelajev @edeandrea @shelajev @edeandrea Thank You! Slides https://bit.ly/jf26-did-you-get-better Rate the talk (honest feedback)! https://bit.ly/jf26-edos-rate