datadog dash 2025 LLM observability for reliability and stability

2025/06/11に開催された「DASH by Datadog」に、IVRyのMoriyaが登壇しました。

■リンク集
・IVRyコーポレートサイト：https://ivry.jp/company/
・IVRy採用ページ：https://ivry-jp.notion.site/
・IVRyイベント一覧：https://ivry.connpass.com/event/
・IVRy Tech Xアカウント：https://x.com/IVRy_tech

Transcript

1. LLM Observability for Reliability & Stability DASH 2025 Hiroyuki Moriya AI Engineer IVRy.Inc. 2025/06/11
2. 2 LLMs can do anything. Easily build amazing products. Perfect self-driving cars within a few years. Automate everything so humans don’t need to work anymore.
3. 3 Easily build amazing products. Perfect self-driving cars within a few years. Automate everything so humans don’t need to work anymore. LLMs can do anything. Not really.
4. 4 What is necessary for providing LLM-powered service in production? Today’s Topic
5. 5 About me Develop products that integrate LLM APIs Monitor & optimize LLM APIs for reliability and performance Hiroyuki Moriya AI engineer / SRE Speaker Intro
6. 6 About IVRy Challenges Solutions Recap & Tips Agenda
7. 7 Founded/HQ: 2019. Tokyo, Japan Number of employees: 200+ people Product: AI/LLM based phone communication service Reach: 30,000+ accounts & 40 million+ incoming calls in total IVRy inc. Company Info “Revolutionizing the telephone experience and boosting productivity for businesses ”
8. 8 AI-powered automated phone service Our Product
9. 9 Simplified system architecture
10. 10 Simplified system architecture
11. 11 Simplified system architecture
12. 12 Phone calls are still important communication tools in Japan Source: Rakuten Communications, “Survey on Call Handling at Small and Medium-Sized Businesses.”
13. 13 Trusted across industries
14. 14 Medical appointments Restaurant reservations Hotel bookings FAQ inquiries We power phone communication with AI for businesses of all sizes IVRy in action
15. 15 About IVRy Challenges Solutions Recap & Tips Agenda
16. 16 Three key challenges for AI phone service Robust fault detection & recovery Challenge #3 Minimizing hallucinations Challenge #1 Ensuring natural conversation pace Challenge #2
17. 17 About IVRy Challenges Solutions Recap & Tips Agenda
18. 18 Three solutions for AI phone service Robust fault detection & recovery Solution #3 Ensuring natural conversation pace Solution #2 Minimizing hallucinations Solution #1
19. 19 LLMs can hallucinate Problem
20. 20 Divide and conquer Solution #1-1
21. 21 Example AI workflow Break down a task into multiple specialized AI components. → Beer validation and error analysis, leading to more stable & reliable results.
22. 22 An example of AI workflow in action
23. 23 Outputs from LLM APIs can change due to silent model updates Problem Output has changed
24. 24 Trust, but verify Solution #1-2
25. 25 Monitor LLM API consistency every day Solution 1. Test cases 2. Run consistency tests 3. Notify / record results
26. 26 Automated phone E2E test
30. 30 Executing phone E2E tests after code merge Merge code Deploy latest code Execute automated phone E2E tests Monitor on Datadog LLM Observability
31. 31 Monitoring with Datadog LLM observability
32. 32 Categorizing topics of test cases
33. 33 Reservation Cancellation Question Categorizing topics of test cases
34. 34 To minimize hallucinations, 1 Divide and conquer Divide one task into multiple, easier steps. Trust, but verify Verify LLM API responses regularly. 2 Summary 34
35. 35 Three solutions for AI phone service Minimizing hallucinations Solution #1 Ensuring natural conversation pace Solution #2 Robust fault detection & recovery Solution #3
36. 36 Slow dialogue could miss oportunities Problem
37. 37 Done is beer than perfect Solution #2-1
38. 38 Fast, stable, and cheap Slower, more $$$ We choose fast, proven models over cuing-edge but slow ones—beer latency, fewer rate limits, lower cost. Stability & performance > latest models
39. 39 See the forest for the tree Solution #2-2
40. 40 Monitor metrics with Datadog Inferred Services
41. 41 Metrics are shown on inferred services page
42. 42 To ensure natural conversation pace, 1 Done is beer than perfect Choose the model that aligns with your case. See the forest for the tree See the overall metrics for each client. 2 Summary 42
43. 43 Three solutions for AI phone service Robust fault detection & recovery Solution #3 Minimizing hallucinations Solution #1 Ensuring natural conversation pace Solution #2
44. 44 System failure could cause fatal issues Problem
45. 45 LLM APIs connection is not stable. Connectivity issues happen frequently. LLM API Status in one day
46. 46 Prepare for the worst Solution #3
47. 47 Created monitoring alerts using custom metrics.
48. 48 Built a robust fallback system using multiple LLMs. It routes requests based on API statuses. LLM fallback strategy
49. 49 LLM APIs are called from LiteLLM proxy sidecars ECS Cluster LLM APIs
50. 50 Using LiteLLM proxy for other applications
51. 51 Emergency phone transfer
52. 52 To implement the robust fault detection and recovery, Prepare for the worst Think the worst scenario and implement the robust recovery system. Summary 52
53. 53 Key lessons for operating LLM APIs Divide and conquer / Trust, but verify 01 for minimizing hallucinations Done is beer than perfect / See the forest for the tree 02 for ensuring natural conversation pace Prepare for the worst 03 for robust fault detection & recovery
54. Thank you! Hiroyuki Moriya AI Engineer IVRy.Inc.