From Hours to Minutes - An AI Case Study with Sheriff

Transcript

1. From Hours to Minutes An AI Case Study with Sheriff Angular London Meetup, 04. December 2025 Murat Sari Rainer Hahnekamp
3. About Me... https://www.youtube.com/ @RainerHahnekamp https://www.ng-news.com • Rainer Hahnekamp • ANGULARarchitects.io • NgRx Core Team • Developer / Trainer / Speaker @RainerHahnekamp Open Source Projects NgRx Toolkit Testronaut NgRx Sheriff
4. Murat Sari & Rainer Hahnekamp Sheriff: Modularity in TypeScript • Module Encapsulation • Dependency Rules
5. Murat Sari & Rainer Hahnekamp Sheriff: Modularity in TypeScript • Module Encapsulation • Dependency Rules • Lightweight • Convention over Configuration • Zero Dependencies • For all TypeScript Projects
6. Murat Sari & Rainer Hahnekamp From simple configurations…
7. Murat Sari & Rainer Hahnekamp …to something else
8. Murat Sari & Rainer Hahnekamp Motivation • Creating a config could be overwhelming ◦ How to split - Architecture Styles ◦ Transform the desired rules into a config ◦ How to verify the config • Maybe AI can automate it? ◦ Ask to analyze the project ◦ Suggest a style or domain cut ◦ Verify the architecture ◦ Visualize the architecture
9. Murat Sari & Rainer Hahnekamp MCP - Approach • Model context protocol solves the NxM integration problem
10. Murat Sari & Rainer Hahnekamp "08/15" Approach: MCP • Prompts (Predefined templates / instructions) ◦ sheriff_config_assistant - System prompt for Sheriff config assistance ◦ generate_sheriff_config - Guided workflow to analyze project and generate config • Resources (Structured data to give more context) ◦ sheriff://configuration-reference - Sheriff documentation reference (markdown) • Tools (Functions which the model can execute) ◦ Calls are model-controlled
11. Murat Sari & Rainer Hahnekamp Challenges Quality Latency Data Privacy Context Size Token Consumption
12. Murat Sari & Rainer Hahnekamp Features without AI Data Condensation UI* *Foundation is static
13. Murat Sari & Rainer Hahnekamp LLM only MCP Agents
14. Murat Sari & Rainer Hahnekamp Architecture Detection • Compress huge import graphs ◦ Sheriff dependency graph (Dir are nodes and imports are weighted edges) ◦ Detect Communities (Louvain algorithm to find clusters) ◦ Directory-level analysis scales logarithmically • Current impl. Provides level of detail (Full Matrix vs Caped by depth) ◦ Provide tools to validate the rules against the full matrix • Ongoing implementation ◦ But we need to keep the token count low
15. Murat Sari & Rainer Hahnekamp MCP Tools – Usage Workflow 1. AI assistance (Prompts / Resources) ◦ pre-built prompts and documentation to guide LLMs through config generation workflows 2. Project exploration & analysis (Explore) ◦ explore directory structure, analyze project architecture, and understand module organization 3. Dependency & module analysis (Analyze) ◦ visualize module hierarchies, compute dependency matrices, inspect tags, and determine module access rules 4. Configuration management (Generate/Validate) ◦ read existing configs, validate syntax, preview changes, and write validated configurations
16. Murat Sari & Rainer Hahnekamp MCP Results • Really depends on the model we are using ◦ GPT5-Codex – Great ◦ Sonnet 4.5 – Good ◦ Haiku – Not that great • End of the day MCP do not make the process robust itself – but enable it
17. Murat Sari & Rainer Hahnekamp MCP Results • Sheriff-MCP reads and writes files ◦ Prompt Injection ◦ Path Traversal Attacks • User needs to start the server on his machine ◦ As we need access to analyze the code • MCP is not a silver bullet still good prompting is needed ◦ When you use an agent which has access to your FS than there is a great chance it will do it TODO → CONTEXT
18. Murat Sari & Rainer Hahnekamp Local LLMs with MCP’s • For example, LMStudio is supporting MCP Servers • When using local llms you never know how the agent uses it (as shown in the image) • Some smaller models do not even support tool calling • All tool response are folded into our context ◦ Context rot
19. Murat Sari & Rainer Hahnekamp Approach 1 - ICL Prompting (No MCP / No Tools) • How can we use AI to produce a valid Sheriff configuration as a reliable starting point? • Structured prompting with In-Context Learning (ICL) ◦ The idea is to embedding structured context (documentation, examples and constraints) within the system prompt. ◦ Here is our example https://hackmd.io/@wolfmanfx/SyKDmNveWx ◦ Demo
20. Murat Sari & Rainer Hahnekamp Approach 2 - State Machine (No MCP but tools) • In approach 1 we included the project structure into the prompt which is inconvenient • We decompose our task in manageable state • On the server we preserve the context which is not the full conversation ◦ We care about structured data
21. Murat Sari & Rainer Hahnekamp Approach 2 - Multi-state agent system • Router agent - analyzes user input and decides • Each phase / state is specialized ◦ Modular context engineering ◦ Direct tools calls (no mcp overhead) ◦ Structured outputs • Final agent is doing natural language presentation
22. Murat Sari & Rainer Hahnekamp Approach 2 • State machine controlled by an “ROUTER SYSTEM PROMPT” ◦ acts as a controller that routes requests to sub handler • Each state consist of a short lived context and specific goal ◦ We do not blow up our context as we always include only the data we need in each specific sub state (each state has an isolated context) • DEMO
23. Murat Sari & Rainer Hahnekamp
24. Murat Sari & Rainer Hahnekamp Local LLMs vs Frontier LLMs • Proprietary Frontier models (GPT, Gemini, Llama3 and Claude) ◦ Trained on huge data volumes ◦ Trained for Tool Use and Function Calling (Intensive instruction tuning) ◦ High inference cost / Limited domain specific knowledge ◦ Best for strategic low volume and high complexity tasks • Local model (Fine-Tuned LLM) ◦ Repetitive productions tasks (high volume) ◦ Security / Data Governance
25. Murat Sari & Rainer Hahnekamp Approach 3 - Own LLM • Current LLMs (Frontier / Local) do not know sheriff ◦ Leads to full hallucination (if no ICL prompting is applied or MCP is used) • Idea train the LLM to be a “Sheriff configuration expert” ◦ Should prevent hallucinate incorrect answers / configs • Traditional Fine-Tuning Problem ◦ Full model: ~1.1 billion parameters (TinyLlama-1.1B-Chat-v1.0) ◦ Training: Update all 1.1B parameters • LoRA - Low-Rank Adaption ◦ Freeze base model: 1.1B parameters (locked) ◦ Train small adapters: 110M parameters (10%)
26. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA • Step 1 - Create Examples (Manual / Automated) • Step 2 - Prepare Training Data • Step 3 - Train Model • Step 4 - Test Model • Step 5 - Convert for Deployment (GGUF / MLX)
27. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA / Step 1 • We have created the examples in markdown ◦ Following this structure: ▪ Question - “Generate a basic Sheriff config for…) ▪ Additional input (Project structure) ▪ Than a requirements section - “Each domain can only access…” ▪ Most important the expected result (Sheriff config) ◦ Why markdown ▪ Easy to use ▪ Human-readable and editable ◦ Created ~15 manual examples and extrapolated to ~250 examples using “Ai”
28. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA / Step 2 • Data Preparation ◦ Input our markdown folder ◦ Output JSONL in ChatML format • JSONL (JSON lines) ◦ It means we store one valid JSON per line in the file ◦ How we store our examples on disk • ChatML (Always check the model how it expects the training data)
29. Murat Sari & Rainer Hahnekamp Approach 3 - Full Fine Tuning / LORA • We use Transformers lib from hugging face (AutoModelForCausalLM)
30. Murat Sari & Rainer Hahnekamp
31. Murat Sari & Rainer Hahnekamp Demo Model • Video
32. Murat Sari & Rainer Hahnekamp Outlook
33. Murat Sari & Rainer Hahnekamp Summary • AI makes a significant contribution • AI as a helper ◦ All tasks can also be done without AI ◦ No dependency to AI ◦ Tooling to verify the outcome (non-deterministic behavior) • Where AI can't help us ◦ Specific UI ◦ Raw Import Graph • Mixed approaches ◦ No MCP ◦ MCP with controlled tooling access ◦ Full MCP ◦ State Machine
34. Murat Sari & Rainer Hahnekamp Thank you!
35. Murat Sari & Rainer Hahnekamp Fallback Videos • In-Context Learning ◦ https://share.cleanshot.com/8Ww9lGXr • State Machine ◦ https://share.cleanshot.com/WZdHXqFq • MCP ◦ https://share.cleanshot.com/wsDfTZQw