Leveraging SRE and Observability Techniques for the Wild World of Building on LLMs

LLMs can provide a quick injection of magic into an existing product (or product concept)! Most of us looking to build on LLMs aren't ML engineers or AI experts, after all, and this new wave of LLM offerings makes it easy for any of us to build something delightful.

But once that product or feature is shipped, in production, in front of users, the problems all collapse back into something that feels awfully familiar: performance challenges, questionable accuracy, and unhappy or confused users.

This talk will assert that building on LLMs is just like buliding on top of any other sort of black box in our architecture (APIs, DBs, etc)—this one just happens to be inherently unpredictable and probablistic.

We'll cover techniques for instrumentation, how to leverage observability practices, and even incorporate SLOs to ensure your team continues to deliver a great service to your users.

Transcript

1. Leveraging SRE and Observability Techniques for the Wild World of Building on LLMs @cyen @honeycombio
3. LLMs ≈ like APIs we know and 💛 App Auth REST API Payments REST API Telephony REST API ‣ Well-formed inputs according to a spec ‣ Cleaned-up user inputs ‣ Well-formed outputs according to a spec ‣ Standard protocols (e.g. HTTP, SMTP) testable mockable =
4. LLMs != like APIs we know and 💛 App Auth REST API Payments REST API Telephony REST API LLMs REST API predictable ∴ testable ∴ mockable
5. LLMs != like APIs we know and 💛 Normal APIs LLMs can conceivably scope the range of inputs intentionally invites free-form, natural-language input from users unit tests reproducible 
 (AKA mockable) deterministic + 
 (ideally) idempotent subject to change ("drift" in model behavior) via public API access explainable 
 (AKA debuggable) based on spec, can understand how a change in input → change in output prompting can yield very different responses through small, subtle changes to prompt
6. LLMs: even more unpredictability App LLMs REST API Extra context
7. LLMs: how do we de fi ne "correct"? App LLMs REST API unit tests "early access" staging env integration tests ☹
8. observability
9. observability AKA: an understanding the behavior of a system based on knowledge of its external outputs.
10. observability expected actual (especially in prod!)
11. Observability: what’s in the box? App Payments REST API user_id endpoint params roundtrip_ms response_status_code pricing_plan_id price_usd_cents payment_source error_code user_id roundtrip_ms
12. LLMs Observability: what’s in the box? App REST API user_id endpoint params roundtrip_ms response_status_code app_metadata user_id LLM_response error_code prompt_version roundtrip_ms prompt_text
13. Observability: ∞ feedback loops → OBSERVE TEST OBSERVE IDEATE → WRITE → TEST → RELEASE
14. Why believe me?
15. Query Assistant: timeline May 2023 6 weeks of development 8 weeks of iteration
16. Query Assistant: goals "What’s the 95th percentile latency on the /checkout endpoint?"
17. Query Assistant: goals 🤔
18. Laws of building on LLMs ‣ Failure will happen—it’s a question of when, not if. ‣ Users will do things you can’t possibly predict. ‣ You will ship a "bug fi x" that breaks something else. ‣ You can’t really write unit tests for this (nor practice TDD) ‣ Latency is often unpredictable ‣ Early access programs won’t help you https://honeycomb.io/blog/hard-stuff-nobody-talks-about-llm
19. How do we go forward? OK, so
20. Instrumentation ~= docs and tests App capture data for your hypotheses
21. Instrumentation ~= docs and tests capture data for your hypotheses
22. Instrumentation for LLMs ‣ user/team IDs ‣ full user input string ‣ add’l product context for prompt ‣ token usage ‣ LLM latency ‣ full LLM response ‣ parse and/or validation errors ‣ user feedback
23. Instrumentation for LLMs ‣ user/team IDs ‣ full user input string ‣ add’l product context for prompt ‣ token usage ‣ LLM latency ‣ full LLM response ‣ parse and/or validation errors ‣ user feedback
24. Instrumentation for LLMs ‣ user/team IDs ‣ full user input string ‣ add’l product context for prompt ‣ token usage ‣ LLM latency ‣ full LLM response ‣ parse and/or validation errors ‣ user feedback
25. Instrumentation for LLMs ‣ user/team IDs ‣ full user input string ‣ add’l product context for prompt ‣ token usage ‣ LLM latency ‣ full LLM response ‣ parse and/or validation errors ‣ user feedback
26. Instrumentation > EXCEPTIONS ‣ user/team IDs ‣ full user input string ‣ add’l product context for prompt ‣ token usage ‣ LLM latency ‣ full LLM response ‣ parse and/or validation errors ‣ user feedback
27. Instrumentation for LLMs ‣ user/team IDs ‣ full user input string ‣ add’l product context for prompt ‣ token usage ‣ LLM latency ‣ full LLM response ‣ parse and/or validation errors ‣ user feedback
28. Emerging behaviors Tapping into
29. DEV 🔥 WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT → WRITE → TEST → COMMIT
30. write lots of code service ownership developers on call test in production DEV
31. DEV PROD identify levers impacting logical branches in code 
 (debuggability + reproducibility) instrument code with intention compare expected vs actual inspect results after changes go live; watch for deviations fail fast / fail fi rst; 
 embrace fast feedback loops TDD ship to prod quickly (CI/CD); expect to iterate o11y
32. A truth in all software systems, but never more true than with LLMs: Software behaves in unpredictable, emergent ways, 
 and the important part is observing your code 
 as it’s running in production, while users are using it.
33. Service Level Objectives Let’s zoom in on
34. SLOs: a quick de fi nition Service Level Objectives codify what it means to "deliver great service" ‣ "Key user fl ows like cart checkout should complete quickly and reliably" ‣ "99.9% of shopping cart checkout attempts complete error-free in < Xs"
35. Laws of building on LLMs ‣ Failure will happen—it’s a question of when, not if. ‣ Users will do things you can’t possibly predict. ‣ You will ship a "bug fi x" that breaks something else. ‣ You can’t really write unit tests for this (nor practice TDD) ‣ Latency is often unpredictable ‣ Early access programs won’t help you ‣ LINK TO: hard things about hard things blog post Remember this? Degradation will happen. SLOs can help.
36. SLOs for developing with LLMs
37. From others in the wild Some more stories
38. LLMs App REST API user_id roundtrip_ms … llm_roundtrip_ms …
39. app_id user_id roundtrip_time endpoint params upstream_time feature_flag_x feature_flag_y LLMs App REST API prompt_version prompt_text model_version algorithm_version time_to_first_token time_to_first_usable_token prompt_input_x prompt_input_y
40. So in the end: ‣ Incorporating LLMs breaks many of our existing tools for ensuring correctness + a good user experience ‣ Observability can help! Instrument + observe from the outside in ‣ Capture all the metadata to be able to debug and analyze unexpected behavior in LLMs ‣ Embrace the unpredictability of user input + LLMs: run in production and plan to iterate fast
41. thanks! q? @cyen @honeycombio For resources + a copy of these slides: