Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya Hiroyuki Moriya
December 11, 2024
1
350

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya

Hiroyuki Moriya

December 11, 2024
Tweet

More Decks by Hiroyuki Moriya

See All by Hiroyuki Moriya
kueueに新しいPriorityClassを足した話
Avatar for Hiroyuki Moriya gekko0114
0
740
JobSet超入門
Avatar for Hiroyuki Moriya gekko0114
1
980

Featured

See All Featured
The Invisible Side of Design
Avatar for Vitaly Friedman smashingmag
302
51k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
95
14k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
31
2.7k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
234
18k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
210k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
29
5.6k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
220k
What's in a price? How to price your products and services
Avatar for Michael Herold michaelherold
246
12k
The Language of Interfaces
Avatar for Des Traynor destraynor
162
25k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
49
14k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.2k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
192
56k

Transcript

  1. confidencial LLM؂ࢹͷ࠷લઢ IVRy ΤϯδχΞ๨೥LTେձ 2024/12/11 Moriya Hiroyuki

  2. confidencial 2 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ

  3. confidencial 3 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ

  4. confidencial 4 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ

  5. confidencial 5 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

  6. confidencial 6 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

    ؾ͕͍ͭͨΒɺԶͷϓϩμΫτ͸ղ໿͕૬࣍͗ɺձࣾ͸౗࢈ͯ͠͠·͍ͬͯͨ...

  7. confidencial 7 ׬

  8. confidencial 8 ࠓ೔͸ɺ޻౻৽Ұ܅͕ɺ͜Μͳ݁຤Λܴ͑ͳ͍ͨΊʹͰ͖Δ͜ͱΛ͓࿩͠͠·͢ɻ

  9. confidencial ࣗݾ঺հ 2024/08 ೖࣾ SWEɾػցֶशΤϯδχΞͳͲΛܦݧ LLM͕ίΞʹͳΓͦ͏ͳαʔϏεͩͱࢥͬͯIVRyʹೖࣾ Moriya Hiroyuki 9 AI

    engineer

  10. confidencial IVRyͰͷLLMΛར༻ͨ͠AIର࿩ 10 WebsocketΛར༻͠ΤϯυϢʔβʔͱLLM͕ϦΞϧλΠϜʹ΍ΓऔΓ͍ͯ͠Δ

  11. confidencial LLM Fallback 11 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚

  12. confidencial LLM Fallback 12 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚ ؂ࢹ͢Ε͹ ྑ͍ͷ͡Ό

  13. confidencial ํ๏ 1ɿDataDog LLM observability 13 DataDog͕Ӷҙ։ൃதͷLLM؂ࢹʹಛԽͨ͠ػೳɻ Latency, token਺, promptͳͲΛऔಘͰ͖Δɻ

  14. confidencial 14 ʮ͜ΕͰɺOpenAIͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  15. confidencial 15 ͋ΕΕʙɺ͓͔͍͠Αʙ

  16. confidencial 16 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻ΒͷϓϩμΫτ͸ɺfallbackػߏΛ ࣮૷͍ͯ͠Δͷʹɺ OpenAIͷlatency͔͠؂ࢹͰ͖ͯͳ͍Αʙ

  17. confidencial ํ๏ 2ɿOpenLIT (OpenTelemetry) 17 OpenTelemetryن֨ʹଇͬͨɺLLM؂ࢹʹಛԽͨ͠πʔϧɻ ༷ʑͳLLMΛ؂ࢹ͢Δ͜ͱ͕Ͱ͖Δɻ

  18. confidencial 18 ʮ͜ΕͰɺ৭ʑͳmodelͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  19. confidencial 19 ͋ΕΕʙɺ͓͔͍͠Αʙ

  20. confidencial 20 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺ৭ʑͳϞσϧΛ࢖͏Μ͔ͩΒɺ provider͝ͱʹɺlatencyΛܭଌ͢Δඞཁ͕͋Δͷʹ LiteLLMશମͰͷlatency͔͠औΕͯͳ͍Αʙ

  21. confidencial ํ๏ 3ɿDataDog Inferred services 21 DataDogʹ౥ࡌ͞ΕͨɺApp֎΁ͷϦΫΤετΛ؂ࢹͯ͘͠ΕΔػߏ

  22. confidencial 22 ʮ͜ΕͰɺLiteLLMͰ࢖͍ͬͯΔ͢΂ͯͷmodelΛ؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  23. confidencial 23 ͋ΕΕʙɺ͓͔͍͠Αʙ

  24. confidencial 24 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺGeminiɺOpenAIͰ̍ͭͷmodelΛ ࢖͏ͱ͸ݶΒͳ͍ͷʹɺ ݸผͷmodelͷlatencyΛऔಘ͢Δ͜ͱ͸ Ͱ͖ͯͳ͍Αʙ

  25. confidencial ·ͱΊ LLM؂ࢹ͸ɺ·ͩ·ͩൃల్্Ͱ஌ݟ͕͋Γ·ͤΜʂ AIɾLLMΛ࢖͍͜ͳͯ͠ϓϩμΫτʹೖΕ͍ͯ͘աఔͰɺ ࣗΒ͕੾Γ։͍͍ͯ͘ඞཁ͕͋Γ·͢ɻ ͥͻҰॹʹAI؂ࢹΛ΍͍͖ͬͯ·͠ΐ͏ʂ 25