Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya Hiroyuki Moriya
December 11, 2024
420
1

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya

Hiroyuki Moriya

December 11, 2024

More Decks by Hiroyuki Moriya

See All by Hiroyuki Moriya
少人数プロダクト開発において SRE業務を増やさない工夫
Avatar for Hiroyuki Moriya gekko0114
0
1k
音声データ解析パイプラインの Software Engineering / Context Engineering
Avatar for Hiroyuki Moriya gekko0114
0
330
LLM Observabilityによる 対話型音声AIアプリケーションの安定運用
Avatar for Hiroyuki Moriya gekko0114
2
510
kueueに新しいPriorityClassを足した話
Avatar for Hiroyuki Moriya gekko0114
0
800
JobSet超入門
Avatar for Hiroyuki Moriya gekko0114
1
1.2k

Featured

See All Featured
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
870
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
250
Navigating the moral maze — ethical principles for Al-driven product design
Avatar for Skipper Chong Warson skipperchong
2
400
Designing for Performance
Avatar for Lara Hogan lara
611
70k
The #1 spot is gone: here's how to win anyway
Avatar for Tamara Novitovic tamaranovitovic
3
1.1k
30 Presentation Tips
Avatar for Ian Lurie portentint
PRO
1
340
brightonSEO & MeasureFest 2025 - Christian Goodrich - Winning strategies for Black Friday CRO & PPC
Avatar for Christian Goodrich cargoodrich
3
750
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
Conquering PDFs: document understanding beyond plain text
Avatar for Ines Montani inesmontani
PRO
4
2.9k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
191
11k
<Decoding/> the Language of Devs - We Love SEO 2024
Avatar for Nikki Halliwell nikkihalliwell
1
270

Transcript

  1. confidencial LLM؂ࢹͷ࠷લઢ IVRy ΤϯδχΞ๨೥LTେձ 2024/12/11 Moriya Hiroyuki

  2. confidencial 2 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ

  3. confidencial 3 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ

  4. confidencial 4 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ

  5. confidencial 5 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

  6. confidencial 6 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

    ؾ͕͍ͭͨΒɺԶͷϓϩμΫτ͸ղ໿͕૬࣍͗ɺձࣾ͸౗࢈ͯ͠͠·͍ͬͯͨ...

  7. confidencial 7 ׬

  8. confidencial 8 ࠓ೔͸ɺ޻౻৽Ұ܅͕ɺ͜Μͳ݁຤Λܴ͑ͳ͍ͨΊʹͰ͖Δ͜ͱΛ͓࿩͠͠·͢ɻ

  9. confidencial ࣗݾ঺հ 2024/08 ೖࣾ SWEɾػցֶशΤϯδχΞͳͲΛܦݧ LLM͕ίΞʹͳΓͦ͏ͳαʔϏεͩͱࢥͬͯIVRyʹೖࣾ Moriya Hiroyuki 9 AI

    engineer

  10. confidencial IVRyͰͷLLMΛར༻ͨ͠AIର࿩ 10 WebsocketΛར༻͠ΤϯυϢʔβʔͱLLM͕ϦΞϧλΠϜʹ΍ΓऔΓ͍ͯ͠Δ

  11. confidencial LLM Fallback 11 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚

  12. confidencial LLM Fallback 12 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚ ؂ࢹ͢Ε͹ ྑ͍ͷ͡Ό

  13. confidencial ํ๏ 1ɿDataDog LLM observability 13 DataDog͕Ӷҙ։ൃதͷLLM؂ࢹʹಛԽͨ͠ػೳɻ Latency, token਺, promptͳͲΛऔಘͰ͖Δɻ

  14. confidencial 14 ʮ͜ΕͰɺOpenAIͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  15. confidencial 15 ͋ΕΕʙɺ͓͔͍͠Αʙ

  16. confidencial 16 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻ΒͷϓϩμΫτ͸ɺfallbackػߏΛ ࣮૷͍ͯ͠Δͷʹɺ OpenAIͷlatency͔͠؂ࢹͰ͖ͯͳ͍Αʙ

  17. confidencial ํ๏ 2ɿOpenLIT (OpenTelemetry) 17 OpenTelemetryن֨ʹଇͬͨɺLLM؂ࢹʹಛԽͨ͠πʔϧɻ ༷ʑͳLLMΛ؂ࢹ͢Δ͜ͱ͕Ͱ͖Δɻ

  18. confidencial 18 ʮ͜ΕͰɺ৭ʑͳmodelͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  19. confidencial 19 ͋ΕΕʙɺ͓͔͍͠Αʙ

  20. confidencial 20 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺ৭ʑͳϞσϧΛ࢖͏Μ͔ͩΒɺ provider͝ͱʹɺlatencyΛܭଌ͢Δඞཁ͕͋Δͷʹ LiteLLMશମͰͷlatency͔͠औΕͯͳ͍Αʙ

  21. confidencial ํ๏ 3ɿDataDog Inferred services 21 DataDogʹ౥ࡌ͞ΕͨɺApp֎΁ͷϦΫΤετΛ؂ࢹͯ͘͠ΕΔػߏ

  22. confidencial 22 ʮ͜ΕͰɺLiteLLMͰ࢖͍ͬͯΔ͢΂ͯͷmodelΛ؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  23. confidencial 23 ͋ΕΕʙɺ͓͔͍͠Αʙ

  24. confidencial 24 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺGeminiɺOpenAIͰ̍ͭͷmodelΛ ࢖͏ͱ͸ݶΒͳ͍ͷʹɺ ݸผͷmodelͷlatencyΛऔಘ͢Δ͜ͱ͸ Ͱ͖ͯͳ͍Αʙ

  25. confidencial ·ͱΊ LLM؂ࢹ͸ɺ·ͩ·ͩൃల్্Ͱ஌ݟ͕͋Γ·ͤΜʂ AIɾLLMΛ࢖͍͜ͳͯ͠ϓϩμΫτʹೖΕ͍ͯ͘աఔͰɺ ࣗΒ͕੾Γ։͍͍ͯ͘ඞཁ͕͋Γ·͢ɻ ͥͻҰॹʹAI؂ࢹΛ΍͍͖ͬͯ·͠ΐ͏ʂ 25