Observability for LLM Application lifecycle

by 株式会社IVRy（社員登壇資料）

Slide 1

Slide 1 text

Observability for LLM Application lifecycle JDDUG #12 Hiroyuki Moriya AI Engineer IVRy.Inc. 2025/08/20

Slide 2

Slide 2 text

2 LLMs can do anything. Easily build amazing products. Perfect self-driving cars within a few years. Automate everything so humans don’t need to work anymore.

Slide 3

Slide 3 text

3 Easily build amazing products. Perfect self-driving cars within a few years. Automate everything so humans don’t need to work anymore. LLMs can do anything. Not really.

Slide 4

Slide 4 text

4 What is necessary for developing LLM-powered service? Today’s Topic

Slide 5

Slide 5 text

5 About me Develop products that integrate LLM APIs Monitor & optimize LLM APIs for reliability and performance Hiroyuki Moriya AI engineer Speaker Intro

Slide 6

Slide 6 text

6 About IVRy Challenges Solutions Recap & Tips Agenda

Slide 7

Slide 7 text

7 Founded/HQ: 2019. Tokyo, Japan Number of employees: 200+ people Product: AI/LLM based phone communication service Reach: 30,000+ accounts & 40 million+ incoming calls in total IVRy inc. Company Info “Revolutionizing the telephone experience and boosting productivity for businesses ”

Slide 8

Slide 8 text

8 AI-powered automated phone service Our Product

Slide 9

Slide 9 text

9 Simpliﬁed system architecture

Slide 10

Slide 10 text

10 About IVRy Challenges Solutions Recap & Tips Agenda

Slide 11

Slide 11 text

11 Three steps for LLM product lifecycle Stg Prd Dev

Slide 12

Slide 12 text

12 Three steps for LLM product lifecycle Stg Prd Dev Difficult to manage many experiments Difficult to enhance output consistency Difficult to stabilize LLM product

Slide 13

Slide 13 text

13 About IVRy Challenges Solutions Recap & Tips Agenda

Slide 14

Slide 14 text

14 Development

Slide 15

Slide 15 text

15 Stg Prd Dev Diicult to manage many experiments Problem

Slide 16

Slide 16 text

16 Diicult to manage many experiments Problem

Slide 17

Slide 17 text

17 Management by Datadog Experiments (Though we don’t utilize this tool not so much) Solution

Slide 18

Slide 18 text

18 Staging

Slide 19

Slide 19 text

19 Stg Prd Dev Problem Diicult to enhance output consistency

Slide 20

Slide 20 text

20 Outputs from LLM APIs can change for many reasons Problem Output has changed

Slide 21

Slide 21 text

21 Automated e2e testing Solution

Slide 22

Slide 22 text

22 Executing phone E2E tests after code merge Merge code Deploy latest code Execute automated phone E2E tests Monitor on Datadog LLM Observability

Slide 23

Slide 23 text

23 Monitoring with Datadog LLM observability

Slide 24

Slide 24 text

24 Production

Slide 25

Slide 25 text

25 Stg Prd Dev Problem Diicult to stabilize LLM product’s latency

Slide 26

Slide 26 text

26 Slow dialogue could miss oportunities Problem

Slide 27

Slide 27 text

27 Monitor metrics with Datadog Inferred Services Solution

Slide 28

Slide 28 text

28 To develop LLM based applications, Datadog is very useful! Summary

Slide 29

Slide 29 text

29 MLOps 勉強会 #55 8/22 19:00 ~ 20:00

Slide 30

Slide 30 text

Thank you! Hiroyuki Moriya AI Engineer IVRy.Inc.