key, and where is it used? • Usage: who is using what, across which providers? • Billing: separate dashboards, invoices, and surprise costs 💰💰💰. • Every new provider adds another thing to chase.
principle: centralize governance, decentralize UX • Architecture & tools: LiteLLM, Langfuse, Onyx, Sourcebot • The hard part: usage outside the gateway • Governance as guardrails, not gates • What we learned
context Answers, not AI inside the • Claude Code, • Chat in web & Slack • LLM APIs for Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage product features • Agents & internal services • Reliable, with fallback
& context Answers, not AI inside the Cost & control • Claude Code, • Chat in web & Slack • LLM APIs for • Spend by team & • Agents & internal • Data safety & policy Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage product features services • Reliable, with fallback app • Proof of value
& context Answers, not AI inside the Cost & control • Claude Code, • Chat in web & Slack • LLM APIs for • Spend by team & • Agents & internal • Data safety & policy Cursor, agents • Direct API access • Code-aware search APIs • Company knowledge • No subscription to manage product features services • Reliable, with app • Proof of value fallback Same company. Different entry points. Who owns the guardrails?
LITTLE CONTROL → chaos • Approval queue per use case • API keys scattered in repos & laptops • One mandated tool for everyone • Nobody knows the monthly spend • Platform team becomes the • Sensitive data in random tools bottleneck • Shadow AI grows anyway • No idea what actually happened
LITTLE CONTROL → chaos • Approval queue per use case • API keys scattered in repos & laptops • One mandated tool for everyone • Nobody knows the monthly spend • Platform team becomes the • Sensitive data in random tools bottleneck • No idea what actually happened • Shadow AI grows anyway The goal: guardrails built into the platform, not an approval process.
KNOWLEDGE Who can use what? Which provider and What internal data can AI model? see? OBSERVABILITY COST What actually happened? How much are we spending, by who?
KNOWLEDGE Who can use what? Which provider and What internal data can AI model? see? OBSERVABILITY COST DEVELOPER EXPERIENCE What actually happened? How much are we Can people use it without spending, by who? opening a ticket? Capabilities first. Tools come later.
Observability ENGINEERS NON-ENGINEERS PRODUCTS Claude Code, Cursor Web chat Internal services Sourcebot, APIs Slack bot Different tools — the same guardrails underneath. Agents · APIs
in repos, single key used multiple places, spend nobody can see. ROLE AI Gateway WHAT WE GET • One endpoint, many providers • Virtual keys per team, app, workflow • Budgets, rate limits, model allow-lists • Spend tracked per key and team • Logs and Traces (minimal version) NOT RAG · observability UI · a user-facing chatbot
the controls around them Teams Keys grouped by team, app and workflow Budgets Spend caps per team and per key Alerts Know before a budget runs out Models hub Approved models across providers
the controls around them Teams Keys grouped by team, app and workflow Budgets Spend caps per team and per key Alerts Know before a budget runs out Models hub Approved models across providers MCP & Skills hubs Shared tools and skills for agents
happening, but do we know what actually happened inside them? ROLE LLM Observability WHAT WE GET • Traces & spans per request • Tokens, cost & latency • Prompt / response visibility • Prompt versioning • Datasets & Evaluation NOT a gateway · a policy engine
Engineers and agents need answers about our whole codebase, not just the repo that's open. ROLE Code search & context WHAT WE GET • Fast, repo-aware code search • Ask questions about the codebase • Agents query it via MCP • Self-hosted — code stays with us NOT a knowledge base · an IDE agent
API & CLI • Chat in browser & Slack • IDE & agents — Claude Code, Cursor • Search across Jira & Confluence • Sourcebot code search • Custom agents per team • MCP tools • No API key, no personal subscription
API & CLI • Chat in browser & Slack • IDE & agents — Claude Code, Cursor • Search across Jira & Confluence • Sourcebot code search • Custom agents per team • MCP tools • No API key, no personal subscription Governance shouldn't require teaching every employee how to use an API.
People need company knowledge — not API keys Employee · web or Slack or personal AI subscriptions. Onyx ROLE Knowledge & chat interface Jira Confluence LiteLLM Slack GitHub Permission-aware access — required, not optional
People need company knowledge — not API keys Employee · web or Slack or personal AI subscriptions. Onyx ROLE Knowledge & chat interface WHAT WE GET • Chat & search on web + Slack bot • Connectors to our SaaS tools • Custom agents per team Jira Confluence LiteLLM Slack GitHub Permission-aware access — required, not optional • LLM calls still go via LiteLLM • Retrieval exposed to engineers via MCP NOT code search · a way around source permissions
we can’t observe Claude usage — and we’re not big enough for the Enterprise plan. BUT MANAGEMENT STILL NEEDS Utilization Token spend Budget planning Who uses it, how much Where tokens go Next year’s AI subscriptions
we can’t observe Claude usage — and we’re not big enough for the Enterprise plan. BUT MANAGEMENT STILL NEEDS Utilization Token spend Budget planning Who uses it, how much Where tokens go Next year’s AI subscriptions So we needed an extra hack.
Managed settings 2 Traces go straight in 3 Push OTLP telemetry config to every laptop Managed settings · logs example Langfuse ingests spans via its OTel endpoint { "env": { "CLAUDE_CODE_ENABLE_TELEMETRY": "1", "OTEL_LOGS_EXPORTER": "otlp", "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL": "http/json", "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT": "https://bridge.example.com/v1/logs" } Logs need a bridge Content arrives as OTel logs, which Langfuse drops traces (protobuf) → OTel endpoint Claude Code logs (http/json) Langfuse Bridge ingestion API }
Platform setup Use AI Every new use case waits in line PLATFORM GOVERNANCE User AI Guardrails applied automatically • Identity: who is calling • Model access: allow-lists per key • Budget & rate limits • Audit trail • Observability: traces & cost The default path is already governed
everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry. 03 Observability is part of governance. You can't govern what you can't see.
everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry. 03 Observability is part of governance. You can't govern what you can't see. 04 Gateway visibility is not total AI visibility. Label data: authoritative vs. estimated.
everyone. IDE for engineers, chat for everyone else. 02 Centralize control points, not every interaction. Identity, keys, budgets, telemetry. 03 Observability is part of governance. You can't govern what you can't see. 04 Gateway visibility is not total AI visibility. Label data: authoritative vs. estimated. 05 Assemble mature OSS before building it yourself. Own the integration, policy and DX.
linedevth
.png){kind=avatar}
codatum
ymotongpoo
kaminashi
camiha
y0hgi
terapyon
takanorip
takahashim
yamanoku
kentarofujii
daidaidai
pfn
baasie
codingconduct
aleyda
addyosmani
reverentgeek
raygrieselhuber
katarinadahlin
chriscoyier
productmarketing
marcoroth
leimatthew05
