Quarkus Insights #265: Observability and Evaluation with Langfuse

AI-infused applications demand a rethinking of our testing practices. Developers face a new class of challenge as LLMs become standard integration points in modern applications: non-deterministic behavior that traditional testing approaches were never designed to handle.

Join Eric as he talks about testing strategies and evaluations in Quarkus and LangChain4j, as well as integrations with other open source tools.

Transcript

2. @edeandrea
3. @edeandrea
4. @edeandrea
5. Java??? 😯 … no seriously … why not Python? 🤔 We integrate existing models Because we are not data scientists Do you really want to do these in Python? into enterprisegrade systems and applications • • • • Transactions Security Scalability Observability @edeandrea
6. I don’t care if it works on your Jupyter notebook We are not shipping your Jupyter notebook @edeandrea
9. What’s the difference between these? CRUD application Application Database Microservice Application Service AI-Infused application Application Model @edeandrea
10. What’s the difference between these? CRUD application Database Application Microservice Application AI-Infused application Application Integration Points Service Model @edeandrea
11. What’s the difference between these? CRUD application Database Application Microservice Application AI-Infused application Application Integration Points Service What do we Model do? @edeandrea
12. What’s the difference between these? CRUD application Application Microservice Application AI-Infused application Application Database Integration Points Service Model @edeandrea
13. Traditional tests tell us if the system worked. @edeandrea
14. o i t a s e r u l i a f n Traditional tests tell us if the system worked. t In r g e @edeandrea
15. high realism high value end-to-end tests integration tests unit tests low effort @edeandrea
16. https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw @edeandrea
17. e v a h e s e h n? t o o d m t a m o h W in c https://www.upworthy.com/prankster-tricks-a-gm-dealership-chatbot-to-sell-him-a-76000-chevy-tahoe-for-1-rp3 https://www.cbsnews.com/news/aircanada-chatbot-discount-customer https://www.bbc.com/news/technology-35902104 https://www.spiceworks.com/tech/artificial-intelligence/news/meta-blender-bot-3-controversy https://www.linkedin.com/posts/stephanjanssen_princoming-activity-7285987635628507136-9Ubw @edeandrea
18. What does failure look like? The system worked! @edeandrea
19. r e w s n ! a g e n o h r T w s a w What does failure look like? The system worked! @edeandrea
20. Did it work well? @edeandrea
21. c i t n a m e S Did it work well? s e r u l i fa @edeandrea
22. What happens when we do this? @edeandrea
23. What happens when we do this? Is a s i th ? e r u l i fa @edeandrea
24. What’s an evaluation? Unit test assertEquals(expected, actual) exact match pass / fail same input, same answer @edeandrea
25. What’s an evaluation? Unit test Evaluation assertEquals(expected, actual) “How good was this answer?” exact match semantic similarity, correctness, domain rules pass / fail score + explanation same input, same answer same input, different answers @edeandrea
26. high realism end-to-end tests tests with application server test REST endpoints integration tests tests using AI unit tests low effort @edeandrea
27. Angie Jones: AI Enablement for Engineering @ Block https://engineering.block.xyz/blog/testing-pyramid-for-ai-agents @edeandrea
28. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. @edeandrea
29. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. Tier 2: Per-session one full conversation End-of-session analysis of full user journeys measuring sentiment and goal success. @edeandrea
30. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. Tier 2: Per-session Tier 3: Drift detection End-of-session analysis of full user journeys measuring sentiment and goal success. Automated gates comparing production metrics against historical baselines. one full conversation all traffic, over time @edeandrea
31. 3-Tier Evaluation Strategy Tier 1: Per-trace one request, one response Real-time checks ensuring immediate response quality. Regression No intentional changes (model update, infrastructure, etc) Tier 2: Per-session Tier 3: Drift detection End-of-session analysis of full user journeys measuring sentiment and goal success. Automated gates comparing production metrics against historical baselines. one full conversation all traffic, over time Change Validation Intentional changes (prompt, business logic, data, etc) @edeandrea
32. I changed something. Did I make it worse? @edeandrea
33. Anatomy of an Evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces an actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation @edeandrea
34. Anatomty of an Evaluation 1. Sample ◦ The test case containing input parameters & expected output. 2. Function under test ◦ The function being evaluated. Receives input parameters & produces an actual output. 3. Evaluation Strategy ◦ Logic that determines if the actual output is acceptable based on the expected output. 4. Evaluation Result ◦ Outcome (pass/fail), score, explanation, and metadata from the evaluation https://docs.quarkiverse.io/quarkus-langchain4j/dev/testing.html#_evaluation @edeandrea
35. Demo time! https://docs.quarkiverse.io/quarkus-langfuse/dev https://github.com/edeandrea/non-deterministic-no-problem @edeandrea
36. Under the covers @edeandrea
37. @edeandrea
38. Tiers in Practice https://github.com/edeandrea/non-deterministic-no-problem @edeandrea
39. Tier 1 https://github.com/edeandrea/non-deterministic-no-problem @edeandrea
40. Tier 2 https://github.com/edeandrea/non-deterministic-no-problem @edeandrea
41. Tier 3 https://github.com/edeandrea/non-deterministic-no-problem @edeandrea
42. What’s next? @edeandrea
43. @edeandrea
45. Takeaways • Remember the testing pyramid — use the right tool at each level • Don’t build observability into your apps — build it around them • Test in production. It’s the only place the real inputs are. • AI is just an API call — the answer isn’t. • Traditional tests tell you if the system worked. Evaluations tell you whether it worked well. @edeandrea
46. Thank You! @edeandrea