AI ENGINEER WORLD'S FAIR 2026 - We Cut 94% of Our AI Coding Tokens With a Local Code Index

AI ENGINEER WORLD'S FAIR 2026 - We Cut 94% of Our AI Coding Tokens With a Local Code Index

Transcript

1. AI ENGINEER WORLD'S FAIR June 30 – July 2, 2026 · San Francisco SEARCH & RETRIEVAL TRACK We Cut 94% of Our AI Coding Tokens With a Local Code Index Here's the architecture. 94% 0.4ms 0.90 Token Reduction Search Latency Recall@10 RS Rajkumar Sakthivel github.com/elara-labs/codecontext-engine
2. THE ASSUMPTION Every AI coding tool we tried had the same assumption: send as much context as possible. WHAT AGENTS SEND WHAT'S ACTUALLY USEFUL 45,000 tokens per query → ~5,000 tokens per query We didn't notice until we saw the cost and latency impact. ◢ elara-labs/code-context-engine
3. WHAT WE GOT WRONG We optimized the model. We should have optimized the context. Better prompts "Be concise." "Only return relevant code." The model still received 45k tokens of input. Model settings Temperature, top-p, max_tokens control output shape. The 45k input was already sent and billed. Output compression "Talk like a caveman." Saves 75% of output (10% of bill). Net impact: ~8%. Wrong 10%. A retrieval layer between codebase and agent Search an index, return only relevant chunks. 94% fewer tokens. ◢ elara-labs/code-context-engine
4. WHY INPUT MATTERS Where your tokens actually go Output compression Saves 75% of output tokens = ~8% off total bill 90% is input Input retrieval Saves 94% of input tokens = ~61% off total bill Input tokens (file reads, search, context) Output tokens (agent replies, code) ◢ elara-labs/code-context-engine Both help. But if you're only doing one, do the one that targets 90% of your spend.
5. ARCHITECTURE A local retrieval layer between codebase and agent Tree-sitter Chunking › Hybrid Retrieval › Chunk Compression › Code Graph › Confidence Scoring AST-aware splits Vector+BM25+RRF Signatures + docs CALLS · IMPORTS Threshold gate 10 langs 94% 89% related filter Everything runs locally. No cloud, no API calls. sqlite-vec + FTS5 + graph in three SQLite files. ◢ elara-labs/code-context-engine
6. DEEP DIVE Why not just vector search? 🎯 🔤 ⚡ Vector Search FTS5 (BM25) RRF Fusion bge-small-en-v1.5 (384d). Finds conceptually related code even with different naming. Exact keyword matching. Catches function names and identifiers vector search fuzzes over. Reciprocal Rank Fusion (k=60) merges both. Confidence blends similarity, keywords, recency. Recall: 0.78 Recall: 0.72 Recall: 0.90 Neither retriever is good enough alone. Together they cover each other's blind spots. ◢ elara-labs/code-context-engine
7. THE HARD PART The hardest problem wasn't retrieval. It was knowing when retrieval was wrong. LLM-based scoring Confidence scoring blend Asked the model to rate relevance. Accurate but +2-3s latency and cost per query. Similarity 50% Fixed thresholds Keywords 30% cosine > 0.7 = relevant. Broke on short queries and long queries alike. Recency 20% Simple heuristic won 50% similarity + 30% keyword + 20% recency. Adaptive. 0.4ms, no API calls. ◢ elara-labs/code-context-engine Lesson: don't reach for an LLM when a weighted average will do.
8. BENCHMARK FastAPI: 53 files, 20 real questions, reproducible Full file baseline 83,681 tok/q After retrieval 4,927 tok/q 94% retrieval savings After compression Recall@10 ◢ elara-labs/code-context-engine 523 tok/q 0.90 No cherry-picking. No synthetic queries. 20 questions a developer would actually ask. $ python benchmarks/run_benchmark.py --repo fastapi/fastapi --source-dir fastapi
9. TRADE-OFFS What we're honest about 94% is against full-file reads Monorepos dilute recall Claude Code already uses grep and partial reads. Real-world savings vs normal behavior are lower. Full-file is our reproducible baseline. On Go's fiber (396 files), recall dropped to 0.07@10. One-featureper-file repos hit R=1.00. Focused files retrieve best. Embedding model matters What actually worked bge-small-en-v1.5 (384d) is fast, not SOTA. Bigger models lift recall but add latency. We chose speed; <1s re-index at 96% cache. Simple heuristics over ML. SQLite over specialized DBs. Hybrid over pure vector. Local-first. The boring choices compounded. ◢ elara-labs/code-context-engine
10. MULTI-AGENT One index. Every agent. Shared memory. Works with every major AI coding tool via MCP protocol Claude Code Cursor Gemini CLI VS Code / Copilot Tabnine Codex CLI OpenCode MCP Protocol Shared Index Cross-session 5 tools + session memory Per-project, not per-agent Decisions persist across tools Decisions made in Claude Code surface in Codex. Memory is per-project, not per-agent. ◢ elara-labs/code-context-engine
11. KEY TAKEAWAY The biggest optimization in AI coding isn't the model. It's the context. $ uvx --from "code-context-engine[local]" cce init 94% local MIT fewer input tokens no data leaves your machine free, open source Try it now Scan to open the repo. Star it, fork it, run the benchmark yourself. github.com/elara-labs/code-context-engine Thank you · Rajkumar Sakthivel