Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya Hiroyuki Moriya
December 11, 2024
1
360

IVRyエンジニア忘年LT大会2024 LLM監視の最前線

Avatar for Hiroyuki Moriya

Hiroyuki Moriya

December 11, 2024
Tweet

More Decks by Hiroyuki Moriya

See All by Hiroyuki Moriya
kueueに新しいPriorityClassを足した話
Avatar for Hiroyuki Moriya gekko0114
0
770
JobSet超入門
Avatar for Hiroyuki Moriya gekko0114
1
1k

Featured

See All Featured
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
274
41k
Unlocking the hidden potential of vector embeddings in international SEO
Avatar for Frank van Dijk frankvandijk
0
140
Collaborative Software Design: How to facilitate domain modelling decisions
Avatar for Kenny Baas-Schwegler baasie
0
110
Prompt Engineering for Job Search
Avatar for Mfonobong Umondia mfonobong
0
140
Breaking role norms: Why Content Design is so much more than writing copy - Taylor Woolridge
Avatar for UX Y'all uxyall
0
120
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
50
14k
Building the Perfect Custom Keyboard
Avatar for Naoto Takai takai
2
670
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.6k
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
0
1.9k
It's Worth the Effort
Avatar for 3n 3n
187
29k
Getting science done with accelerated Python computing platforms
Avatar for Jacob Tomlinson jacobtomlinson
0
88
JAMstack: Web Apps at Ludicrous Speed - All Things Open 2022
Avatar for David Neal reverentgeek
1
300

Transcript

  1. confidencial LLM؂ࢹͷ࠷લઢ IVRy ΤϯδχΞ๨೥LTେձ 2024/12/11 Moriya Hiroyuki

  2. confidencial 2 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ

  3. confidencial 3 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ

  4. confidencial 4 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ

  5. confidencial 5 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

  6. confidencial 6 Զͷ໊લ͸ɺߴߍੜ୳ఁ޻౻৽Ұ AIΤϯδχΞͱͯ͠ಇ͖࢝ΊͨԶ͸ɺLLMΛ࢖ͬͯɺͨ͘͞Μ͓ۚΛՔ͍Ͱ͍Δε λʔτΞοϓΛ໨ܸͨ͠ɻ ʮͪΐͪΐͬͱ։ൃͨ͠ΒϘϩṶ͚Ͱ͖Δ΍Μʂʯͱؾ͍ͮͨԶ͸ɺىۀͯ͠ɺͻͨ ͢ΒPoCϓϩδΣΫτΛΫϥΠΞϯτʹఏڙ͢Δ͜ͱʹͨ͠ɻ Զ͸ɺഎޙ͔Β͍ۙͮͯ͘Δrate limit੍ݶͱɺLatencyͷѱԽʹؾ͕͍͍ͭͯͳ͔ͬ ͨɻ

    ؾ͕͍ͭͨΒɺԶͷϓϩμΫτ͸ղ໿͕૬࣍͗ɺձࣾ͸౗࢈ͯ͠͠·͍ͬͯͨ...

  7. confidencial 7 ׬

  8. confidencial 8 ࠓ೔͸ɺ޻౻৽Ұ܅͕ɺ͜Μͳ݁຤Λܴ͑ͳ͍ͨΊʹͰ͖Δ͜ͱΛ͓࿩͠͠·͢ɻ

  9. confidencial ࣗݾ঺հ 2024/08 ೖࣾ SWEɾػցֶशΤϯδχΞͳͲΛܦݧ LLM͕ίΞʹͳΓͦ͏ͳαʔϏεͩͱࢥͬͯIVRyʹೖࣾ Moriya Hiroyuki 9 AI

    engineer

  10. confidencial IVRyͰͷLLMΛར༻ͨ͠AIର࿩ 10 WebsocketΛར༻͠ΤϯυϢʔβʔͱLLM͕ϦΞϧλΠϜʹ΍ΓऔΓ͍ͯ͠Δ

  11. confidencial LLM Fallback 11 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚

  12. confidencial LLM Fallback 12 ෳ਺ͷLLMΛར༻͢Δ͜ͱΛલఏʹFallbackػߏΛߏங APIͷStatus, Ratelimit΍σʔλ੍໿(஍ཧ੍໿)Λ΋ͱʹৼΓ෼͚ ؂ࢹ͢Ε͹ ྑ͍ͷ͡Ό

  13. confidencial ํ๏ 1ɿDataDog LLM observability 13 DataDog͕Ӷҙ։ൃதͷLLM؂ࢹʹಛԽͨ͠ػೳɻ Latency, token਺, promptͳͲΛऔಘͰ͖Δɻ

  14. confidencial 14 ʮ͜ΕͰɺOpenAIͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  15. confidencial 15 ͋ΕΕʙɺ͓͔͍͠Αʙ

  16. confidencial 16 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻ΒͷϓϩμΫτ͸ɺfallbackػߏΛ ࣮૷͍ͯ͠Δͷʹɺ OpenAIͷlatency͔͠؂ࢹͰ͖ͯͳ͍Αʙ

  17. confidencial ํ๏ 2ɿOpenLIT (OpenTelemetry) 17 OpenTelemetryن֨ʹଇͬͨɺLLM؂ࢹʹಛԽͨ͠πʔϧɻ ༷ʑͳLLMΛ؂ࢹ͢Δ͜ͱ͕Ͱ͖Δɻ

  18. confidencial 18 ʮ͜ΕͰɺ৭ʑͳmodelͷlatency͕؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  19. confidencial 19 ͋ΕΕʙɺ͓͔͍͠Αʙ

  20. confidencial 20 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺ৭ʑͳϞσϧΛ࢖͏Μ͔ͩΒɺ provider͝ͱʹɺlatencyΛܭଌ͢Δඞཁ͕͋Δͷʹ LiteLLMશମͰͷlatency͔͠औΕͯͳ͍Αʙ

  21. confidencial ํ๏ 3ɿDataDog Inferred services 21 DataDogʹ౥ࡌ͞ΕͨɺApp֎΁ͷϦΫΤετΛ؂ࢹͯ͘͠ΕΔػߏ

  22. confidencial 22 ʮ͜ΕͰɺLiteLLMͰ࢖͍ͬͯΔ͢΂ͯͷmodelΛ؂ࢹͰ͖ΔΑ͏ʹͳͬͨͥʂʯ

  23. confidencial 23 ͋ΕΕʙɺ͓͔͍͠Αʙ

  24. confidencial 24 ͋ΕΕʙɺ͓͔͍͠Αʙ ๻Β͸ɺGeminiɺOpenAIͰ̍ͭͷmodelΛ ࢖͏ͱ͸ݶΒͳ͍ͷʹɺ ݸผͷmodelͷlatencyΛऔಘ͢Δ͜ͱ͸ Ͱ͖ͯͳ͍Αʙ

  25. confidencial ·ͱΊ LLM؂ࢹ͸ɺ·ͩ·ͩൃల్্Ͱ஌ݟ͕͋Γ·ͤΜʂ AIɾLLMΛ࢖͍͜ͳͯ͠ϓϩμΫτʹೖΕ͍ͯ͘աఔͰɺ ࣗΒ͕੾Γ։͍͍ͯ͘ඞཁ͕͋Γ·͢ɻ ͥͻҰॹʹAI؂ࢹΛ΍͍͖ͬͯ·͠ΐ͏ʂ 25