Enterprise AI Economics: Understanding and Controlling AI Cost

Speaker: Srivatsa RV & Biswa Prakash Mishra - SRE, One2N

Session Overview

Enterprise AI Economics explores how AI costs evolve in production, beyond simple token pricing. It covers the key drivers of AI spend across prompts, retrieval, agents, and tools, along with techniques like caching, model routing, and context optimization to improve efficiency. The session also highlights how mature organisations use observability to track usage and shift from measuring cost per token to cost per successful business outcome.

Transcript

1. Token Economics. “Jesse, we need to meterˮ Paisa, Prompts and Production Biswa Prakash Mishra · SRE, One2N Srivatsa RV · SRE, One2N PRAGMATIC ENGINEERING, RELEASED TO PRODUCTION 01
2. WHO WE ARE, WHAT WE DO 02 About One2N. A hands on engineering team of 70, based in Pune. FOUNDED 2020, Pune. Clients in India, South East Asia, the US, UK and Europe We started by building software, then specialised in running it reliably at scale. ENGAGEMENTS 30+ companies, from seed to enterprise across 7 different industries MODEL Embedded AI, SRE & Backend engineers PRACTICES AI · SRE · Platform · Backend · Data That journey took us from backend engineering into SRE, platform engineering and distributed systems. Today, we apply the same engineering principles to production AI. WHO WE PRIMARILY WORK WITH Companies whose business is growing faster than their technology can handle.
3. 03 The problem. Enterprise AI spend is already a FinOps topic. 98% 80% 36% of FinOps respondents now manage AI spend report that AI spend increased believe their organisation is overspending on AI FinOps Foundation, 2026 Flexera, 2026 Flexera, 2026 • • • • How many of you are responsible for AI spend in your organization? How many of you are responsible for AI cost optimisation? Do you believe your organization is overspending? And are you an CLI person, or a IDE person?
4. 04 What is a token and tokenizer? Token - A chunk of data or text (a word, part of a word, or punctuation) used as a basic building block by LLMs Tokenizer - Breaks text into smaller units and assign them unique identifiers IDs TEXT TOKENIZED OUTPUT Before you send a prompt to the model, the runtime assembles the full context: system rules, prior turns, retrieved documents and tool schemas. The tokenizer then breaks all of that text into tokens, and the provider bills you for every token it processes, whether it was freshly read or served from the cache. Before you send a prompt to the model, the runtime assembles the full context: system rules, prior turns, retrieved documents and tool schemas. The tokenizer then breaks all of that text into tokens, and the provider bills you for every token it processes, whether it was freshly read or served from the cache. 73 53 310 tokens words characters Top 5 most frequent tokens Top 5 most expensive words Word Tokens Token Freq. schemas. 3 the 6 tokens, 3 , 5 processes, 3 token 3 assembles 2 you 2 tokenizer 2 and 2
5. TOKEN CACHING 101 05 Request A A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. User: Hello! Prompt ROUND 2 Prompt ROUND 1 Human: What's AI? Human: What's AI? LLM Result ROUND 1 LLM Artificial Intelligence, or AI, is cached Request B LLM Artificial Intelligence, or AI, is A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. User: How are you? computer technology that lets machines computer technology that lets machines learn, think, and solve problems like people learn, think, and solve problems like people Human: Cool, thanks! LLM Result ROUND 2 Request C A chat between a curious user and an artificial intelligence assistant. The assistant speaks French. User: Bonjour! System / instructions LLM Request LLM No problem! Input Tokens Cache Tokens Output Tokens Tokens sent to the model Repeated prompt prefix reused Tokens generated by the model User prompt Conversation history / retrieved context Cache write Storing a new prefix Cache read / cache hit Reusing a stored prefix Visible response Reasoning / thinking tokens What the user sees Provider-dependent Useful when the stable prefix repeats PRICING IDEA Total cost = input tokens + cache read/write tokens + output tokens
6. TOKEN CACHING 101 - CONTD 06 Optimisation Techniques Practical controls available to the API caller What to optimise 1 Stable prefix Stable prompt prefix Place static content (tool definitions, system instructions, context, examples) at the beginning of your prompt. Mark the end of the reusable content for caching using the cache_control parameter. 2 Request design Cache breakpoint System / developer instructions → Tool / MCP definitions Cache breakpoint Shared examples → / reusable context Place reusable content before the breakpoint and changing input after it. 3 Current user input → Dynamic data / latest context Changes every request Conversation history Send only the history needed for the next request. 4 Tool / MCP prefix stability 5 Generation controls LLM Request Prevent tool definitions from continuously changing the prefix Set output limits and use only the reasoning level you need. Higher cache hits Lower fresh input More prefix reuse Less recomputation Important Automatic prefix caching and explicit cache breakpoints are provider-specific mechanisms. The optimisation principle is the same, but the API implementation differs.
7. TOKEN CACHING 101 - CONTD 07 Who Controls What? Hosted APIs expose some optimisation controls. Self-hosting adds control over the inference engine itself. API / Application Provider Internals Controls exposed to your workload 1 2 3 4 5 Prefix + breakpoints cache_control • prompt-cache breakpoint Cache lifecycle TTL • pre-warm • cache key Conversation history history • tool results Tool / MCP prefix stability defer-loading • allowed, tools Generation controls effort • output • model Hosted inference implementation 1 2 3 4 5 Prefix matching lookback • longest match Cache storage placement • persistence Eviction expiry • memory pressure Inference scheduling batching • hardware Cache isolation tenant • security Self-Hosted / Model Serving Additional controls when you operate inference 1 2 3 4 5 KV-cache sizing memory • precision Continuous batching scheduling • concurrency Chunked Prefill prefill • decode Quantisation FP8  INT8  INT4 Speculative decoding draft • verify Important Hosted API. control the workload and exposed cache policy. Self-hosted: control those plus the inference implementation.
8. INSTRUMENTING LLM TELEMETRY FOR AGENTIC APPLICATIONS 08 What to Measure Cost, latency and generation measures for AI workloads 1 3 Cost per Task 2 Time to first token · TTFT What does one completed task cost? How quickly does generation begin? wᵢ = workload weighting First token may be reasoning for reasoning models. Output tokens are further split into output + reasoning tokens, priced separately. Output speed How quickly is the response generated? Output tokens / second 4 End-to-end response time How long does the complete task take? Input + reasoning + generation TTFT: Measures how quickly the system gets through queueing and prefill to produce the first token. Not same as RTT we conventionally know. CPST: Includes cached/fresh tokens, output, retries and failed attempts, discounting that with a provider rate card only can become costly Together: TTFT tells you how quickly work starts; CPST tells you whether that work is economically efficient.
9. CACHE ECONOMICS 09 The 90% cache-hit payoff, in real dollars. optimise your cache! Bedrock rates for Opus 5, Sonnet 5 and Haiku 4.5 - with prefix cache hit as variable 100M input · 25M output tokens / mo MODEL Cache read = 0.1× input rate MONTHLY  20% CACHE HIT Bedrock Global endpoint MONTHLY  90% CACHE HIT SAVED / MONTH Claude Opus 5 $1,035.00 $720.00 $315.00 Claude Sonnet 5 $414.00 $288.00 $126.00 Claude Haiku 4.5 $207.00 $144.00 $63.00 30.4% savings The differences is real, using cache and not using cache But, cache control and optimisation techniques is an area definitely worth investing effort into. Providers have a rate card - primarily focused on - model | input | cached input | output • • • • Anthropic models on Amazon Bedrock, Global endpoint, Aug 2026. Regional/multi-region endpoints add a flat 10% across input, output and cache - relative savings hold at 30.4%. Provider rate cards are split by token type. Fresh input, cached input, cache writes and output tokens can all be priced differently. Cache reuse changes both cost and latency. Cache reads are cheaper than creating or rewriting the cache, and repeated prefix reuse reduces input processing work. Output still dominates the user experience. Caching can improve TTFT, but reasoning and output generation still determine much of the final response time and cost.
10. WE LOVE SPEAKING, AND CONTRIBUTING Where to go next. WHAT WE PUBLISH MEETUPS WE RUN IN THE WORKS One2N Bits, and the blog mitramandal.ai, Command Line Heroes Live on Production, our podcast Two series we run ourselves, and an office in Pune we are happy to lend to any community that needs a room. Failure modes, incident stories and what it takes to run data at scale, told by the engineers who were on call. We love writing, and talking about our work, this includes engineering practices, hiring, technical deep dives. WHAT WE WRITE OUR WORK HOW WE OPERATE LEARN WITH US MEETUPS one2n.io/blog one2n.io/case-studies one2n.io/playbook one2n.io/sre-bootcamp one2n.io/meetups TALK TO US partners@one2n.io