Upgrade to Pro — share decks privately, control downloads, hide ads and more …

llamadart on-device agent

권태뽕
November 13, 2026

llamadart on-device agent

Deck delivered at Flutter Korea 2026

권태뽕

November 13, 2026

Transcript

50 slides
  1. Flutter Korea 2026 Starting with llamadart: Building On-Device Agents feat:

    tool call parser Taehyung Kwon · Korea University / Flutter Seoul 1 / 43
  2. Past talk - Flutter Seoul OpenStage Daegu 2026 Add AI

    to Your Flutter App for Free 3 / 43
  3. Hello, Flutter Korea 2026! Taehyung Kwon @TaeBbong Korea University /

    Flutter Seoul • '20 Korea University, School of Cybersecurity • '20 ~ '25 Offensive Security • '25 ~ '27 AI Engineering in CTI Domain • '19 ~ Flutter dev & open source 4 / 43
  4. Hello, Flutter Korea 2026! Taehyung Kwon @TaeBbong Korea University /

    Flutter Seoul • '20 Korea University, School of Cybersecurity • '20 ~ '25 Offensive Security • '25 ~ '27 AI Engineering in CTI Domain • '19 ~ Flutter dev & open source 4 / 43
  5. Contents Agenda 1 2 3 4 Agent Core Concepts Structured

    Output, Tool Call, Parser On-Device AI and llamadart How Tool Calls are handled on-device llamadart Contributions handler checks, bug fixes, Tool Call loop llamadart Examples In-app Copilot, Laya Tetris llamadart 5 / 43
  6. LLM-Era Agentic AI: Core Tech? ➀ Concepts Prompt Engineering LoRA

    Multimodal Few-shot RAG Vector DB Fine-tuning Structured Output Chain-of-Thought Context Window Reasoning Embedding Quantization Hallucination MCP Guardrails Agent Tool Call Evaluation Memory 6 / 43
  7. LLM Nondeterminism ➀ Concepts · Structured Output Response 1: Flutter

    Korea is held on November 7. Response 2: Event: Flutter Korea ∕ Date: 2026.11.7 Response 3: {"event": "Flutter Korea", "when": "11∕7"} Same request, different format every time → unparseable 7 / 43
  8. Schema in the Prompt ➀ Concepts · Structured Output Respond

    only in the JSON format below. {"name": string, "date": string, "participants": string[]} Prompts can't guarantee the format 100% 8 / 43
  9. constrained decoding ➀ Concepts · Structured Output Schema {"func": string,

    "param": array} · output so far {"func": "add", ▌ Candidate token Prob. Schema "param" 0.52 ✓ Allowed "note" 0.21 ✗ Unknown key Here 0.15 ✗ Not JSON } 0.07 ✗ Missing key Filtering every token keeps the format 100% valid 9 / 43
  10. Function Call Schema ➀ Concepts · Tool Call Structured Output

    schema App code { result = json.loads(llm_response) "func": "func_name", "param": [ ... ] } LLM output {"func": "add", "param": [10, 30]} → func_map = {"add": add, "multiply": multiply} func = func_map[result["func"]] output = func(*result["param"]) # add(10, 30) → 40 LLM returns the function and args; the app runs the call 10 / 43
  11. LLM Providers & Engines ➀ Concepts · Serving Who runs

    inference, and where? LLM Provider Inference engine Frontier models via an external API Run open-weight models locally Google Gemini OpenAI Anthropic vLLM llama.cpp Ollama 12 / 43
  12. Mobile On-Device Options ➀ Concepts · On-Device Type OS models

    Examples (iOS · Android · Both) Apple Foundation Models Gemini Nano ML Kit Prompt API Platform ML Core AI / Core ML framework LiteRT-LM Inference engine llama.cpp · ExecuTorch · MLX Usable models OS models Open models Open models In Flutter platform channel flutter_edge_ai (builtin) flutter_edge_ai (ex-flutter_gemma) llamadart · flutter_edge_ai OS models are easy but fixed / engines run any open model 13 / 43
  13. Mobile On-Device Options ➀ Concepts · On-Device Type OS models

    Examples (iOS · Android · Both) Apple Foundation Models Gemini Nano ML Kit Prompt API Platform ML Core AI / Core ML framework LiteRT-LM Inference engine llama.cpp · ExecuTorch · MLX Usable models OS models Open models Open models In Flutter platform channel flutter_edge_ai (builtin) flutter_edge_ai (ex-flutter_gemma) llamadart · flutter_edge_ai OS models are easy but fixed / engines run any open model 14 / 43
  14. Meet llamadart! ② llamadart • Maintainer: leehack (Flutter & Dart

    GDE) • Binds the llama.cpp engine via FFI and exposes it as a Dart API • Shared in the Flutter Seoul group chat, early 2026 github.com/leehack/llamadart 15 / 43
  15. Tool Call Reliability ➁ llamadart · Context Frontier models Claude

    Opus 4.6 · GPT-5.2 2026.02 Tool Call reliability near 100% Open-weight models Qwen3 series Many models fail at Tool Calls 1) The model can't tell when a Tool Call is needed, 2) it lacks training on the Tool Call format, 3) or the Tool Call output isn't parsed correctly 17 / 43
  16. Tool Call Reliability ➁ llamadart · Context Frontier models Claude

    Opus 4.6 · GPT-5.2 2026.02 Tool Call reliability near 100% Open-weight models Qwen3 series Many models fail at Tool Calls 1) The model can't tell when a Tool Call is needed, 2) it lacks training on the Tool Call format, 3) or the Tool Call output isn't parsed correctly 18 / 43
  17. llamadart and Tool Calls ➁ llamadart · Motivation Tested Tool

    Calls on several open models Many models didn't work properly Common: no dedicated template or parser Took on the issue with leehack At v0.4.x, llamadart required OpenAI-style JSON from all models 19 / 43
  18. llamadart and Tool Calls ➁ llamadart · Motivation Model family

    Output for "weather in Seoul" Qwen · Hermes <tool_call>{"name": "get_weather", "arguments": {"city": "Seoul"}}</tool_call> Llama 3.1 <|python_tag|>{"name": "get_weather", "parameters": {"city": "Seoul"}} Mistral [TOOL_CALLS][{"name": "get_weather", "arguments": {"city": "Seoul"}}] FunctionGemma <start_function_call>call:get_weather{city:<escape>Seoul<escape>} Qwen3 Coder <tool_call><function=get_weather><parameter=city>Seoul ... Models learn different Tool Call formats; JSON alone isn't enough 20 / 43
  19. Tool Call Flow in llamadart ② llamadart · Tool Call

    Step Owner v0.5.1 code Detect format ChatTemplateEngine detectChatFormat(templateSource) Render tool list into prompt handler HermesHandler.render Constrain output if required handler HermesHandler.buildGrammar Generate tokens llama.cpp LlamaEngine.generate Text → function name & args handler HermesHandler.parse Run tool, return result App tool.invoke · LlamaToolResultContent template and parser → a single handler, one handler per model 21 / 43
  20. 1) create() : new session ② llamadart · Tool Call

    ChatSession.create Adds user msg to history // lib/src/core/engine/engine.dart · LlamaEngine.create final result = await chatTemplate(messages, tools: effectiveTools, toolChoice: toolChoice); LlamaEngine.create Passes tools, toolChoice chatTemplate Builds prompt from GGUF chat template generate llama.cpp decodes, yields chunks ChatTemplateEngine.parse Parses full output when done final tokenStream = generate(result.prompt, params: params.copyWith( stopSequences: stops, grammar: effectiveGrammar)); await for (final token in tokenStream) { buffer.write(token); ∕∕ yield content ∕ thinking chunks right away } final parsed = ChatTemplateEngine.parse( result.format, buffer.toString()); if (parsed.hasToolCalls) { yield LlamaCompletionChunk(... delta: LlamaCompletionChunkDelta(toolCalls: parsed.toolCalls)); } 22 / 43
  21. 2) handler: pick a handler by format ② llamadart ·

    Tool Call Detected from template, not model name // lib/src/core/template/chat_format.dart ChatFormat detectChatFormat(String? templateSource) { if (templateSource == null || templateSource.isEmpty) { return ChatFormat.contentOnly; } ∕∕ ... check DeepSeek, Granite, GLM, etc. in order ∕∕ Qwen3 Coder XML: contains <tool_call>, so before Hermes if (templateSource.contains('<tool_call>') && templateSource.contains('<function=')) { ∕∕ actually 5 conditions Model file handler Qwen Instruct HermesHandler Qwen Coder Qwen3CoderXmlHandler Unrecognized template GenericHandler (JSON) handlerFor(format) picks the handler instance; 28 handlers as of v0.5.1 return ChatFormat.qwen3CoderXml; } // Hermes 2 Pro / Qwen 2.5 / Qwen 3 if (templateSource.contains('<tool_call>')) { return ChatFormat.hermes; 23 / 43
  22. 3) handler.render: prompt from template ② llamadart · Tool Call

    Rendered: part of Qwen2.5 system prompt ∕∕ handlers∕hermes_handler.dart · render final template = Template(templateSource); ∕∕ Jinja inside the GGUF # Tools var prompt = template.render({ 'messages': messages.map((m) => m.toJson()).toList(), You may call one or more functions to assist with the 'add_generation_prompt': addAssistant, user query. 'tools': tools?.map((t) => t.toJson()).toList(), }); return LlamaChatTemplateResult( prompt: prompt, You are provided with function signatures within <tools></tools> XML tags: <tools> {"type": "function", "function": {"name": grammar: buildGrammar(tools), "get_weather", ...}} additionalStops: getStops(hasTools: hasTools), </tools> grammarTriggers: [ GrammarTrigger(type: 0, value: '<tool_call>')], For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags 24 / 43
  23. 4) handler.parse: parse Tool Calls ② llamadart · Tool Call

    // handlers/hermes_handler.dart · parse final toolCallRegex = RegExp( r'<tool_call>\s*(.*?)\s*</tool_call>', dotAll: true); Extract by regex <tool_call> text between tags for (var i = 0; i < matches.length; i++) { final match = matches.elementAt(i); JSON decode try { final json = jsonDecode(match.group(1)!) as Map<String, dynamic>; toolCalls.add(LlamaCompletionChunkToolCall( index: i, id: 'call_$i', type: 'function', function: LlamaCompletionChunkFunction( Sent in the final chunk Parsed once after generation; tool calls arrive whole name: json['name'], ...))); } catch (_) { 25 / 43
  24. Bug 1: Ministral Parsing ➂ Contributions 1) [THINK]...[∕THINK][TOOL_CALLS][{"name": "get_weather", "arguments":

    {...}}] 2) [THINK]...[∕THINK][TOOL_CALLS]get_weather[ARGS]{"location": {"city": "Paris"}} 3) get_weather{"location": "Paris"} ∕∕ short form, no marker Regex truncated nested objects Short form misfires on prose {"location": {"city": "Paris"}} ← last } missing hello{"key": "val"} → tool call? Replaced with a balanced-brace parser that counts Require an explicit marker like [TOOL_CALLS] depth and skips braces inside strings 26 / 43
  25. Bug 1: Ministral Parsing - Fix ➂ Contributions magistral_handler.dart ·

    parse _extractJsonObject() - if (!trimmed.startsWith('[TOOL_CALLS]')) { + String? _extractJsonObject(input, offset) { + if (!trimmed.contains('[TOOL_CALLS]')) { + var depth = 0, inString = false; + for (var i = start; i < input.length; i++) { return ChatParseResult(content: trimmed, ...); } + final afterMarker = trimmed + .substring(markerIdx + '[TOOL_CALLS]'.length).trim(); + // Format 1: Ministral name[ARGS]{...} + final ministralPattern = RegExp(r'([\w-]+)\[ARGS\]'); + if (inString) { ... continue; } + if (c == '"') { inString = true; continue; + } + if (c == '{') depth++; if (c == '}' && --depth == 0) + + return input.substring(start, i + 1); } + } + for (final match in ministralPattern.allMatches(afterMarker)) + { + final jsonObj = _extractJsonObject(afterMarker, match.end); ... + } + // Format 2: Mistral [TOOL_CALLS][{...}, ...] Thinking-aware parse rules + better brace handling 27 / 43
  26. Bug 2: Qwen Double Braces ➂ Contributions {{"name": "get_weather", "arguments":

    {{"city": "Seoul"}}}} Fix 1 Strip outer braces Fix 2 Collapse all double braces → Inner args still had double braces → Fails if only outer braces are doubled 28 / 43
  27. Bug 2: Qwen Double Braces - Fix ➂ Contributions hermes_handler.dart

    · parse _normalizeDoubleBraces() - final json = jsonDecode(match.group(1)!) as Map<String, dynamic>; + String _normalizeDoubleBraces(String input) + final jsonStr = match.group(1)!.trim(); + ∕∕ first check if all are doubled + for (...) { + Map<String, dynamic> json; + + try { if (single brace) return input; + } + // {{ → {, }} → } + } on FormatException { + ... + if (!(jsonStr.startsWith('{{') && jsonStr.endsWith('}}'))) rethrow; + } + try { + json = jsonDecode(jsonStr); + + ∕∕ ➁ peel one outer layer json = jsonDecode(jsonStr.substring(1, jsonStr.length - 1)); } on FormatException { + + ∕∕ ➀ parse as-is ∕∕ ➂ collapse all json = jsonDecode(_normalizeDoubleBraces(jsonStr)); } + } Branches for each double-brace case 29 / 43
  28. Ext. 1: Example App Tool Call Loop ➂ Contributions example/basic_app/lib/services/llama_service.dart

    · chatStream - return _session.create([LlamaTextContent(text)], params: params, tools: _tools) .map((chunk) => chunk.choices.first.delta.content ?? ''); ∕∕ tools declared, never run + final chunks = await _session.create([...], tools: _tools, toolChoice: toolChoice) + .toList(); + final toolCalls = chunks.expand((c) => c.choices.first.delta.toolCalls ?? []).toList(); + for (final call in toolCalls) { + final tool = _tools!.firstWhere((t) => t.name == functionName, + orElse: () => throw Exception('Tool not found: $functionName')); + final result = await tool.invoke(args); ∕∕ app runs it + _session.addMessage(... LlamaToolResultContent(id: call.id, ...)); + } + final finalChunks = _session.create([], params: params); (Before) Only declared the available tools (After) Runs them end to end → 30 / 43
  29. Ext. 1 / Review 1: Tool Call Loop ➂ Contributions

    OpenAI streaming: one call spans multiple deltas chunk 1 { index: 0, id: "call_1", name: "get_weather" } chunk 2 { index: 0, arguments: '{"loc' } chunk 3 { index: 0, arguments: 'ation": "Seoul"}' } ✗ addAll 1 call split into 3 ✓ accumulator joins by index into 1 6da21b06 · llama_service.dart - final toolCalls = <LlamaCompletionChunkToolCall>[]; + final toolCallAccumulators = <int, _ToolCallAccumulator>{}; await for (final chunk in _session.create(...)) { toolCalls.addAll(chunk.choices.first.delta.toolCalls ?? []); + for (final call in delta.toolCalls ?? const []) { + final acc = toolCallAccumulators.putIfAbsent( + call.index, _ToolCallAccumulator.new); ∕∕ index is the key + if (call.id != null) acc.id = call.id; + if (fn?.name != null) acc.name = fn!.name; + if (fn?.arguments != null) acc.arguments.write(fn!.arguments); + } } Lesson: build widely used features on the standard spec 31 / 43
  30. Ext. 1 / Review 2: Tool Call Loop ➂ Contributions

    6da21b06 · llama_service.dart - if (functionName == null) continue; - final tool = _tools!.firstWhere((t) => t.name == functionName, orElse: () => throw Exception('Tool not found: $functionName')); ∕∕ aborts via exception - final args = jsonDecode(argsJson) as Map<String, dynamic>; ∕∕ throws on broken JSON + if (functionName == null || functionName.isEmpty) { + _addToolResultMessage(..., result: 'Error: Tool call missing function name.'); continue; + } + final tool = _findToolByName(functionName); + if (tool == null) { _addToolResultMessage(..., result: 'Error: Unknown tool: ...'); continue; } + try { args = _decodeToolArgs(argsJson); } + on FormatException { _addToolResultMessage(..., 'Error: Invalid arguments ...'); continue; } + try { result = await tool.invoke(args); } + catch (e) { result = 'Error: Tool execution failed for $functionName: $e'; } Failures as tool results, loop continues → model may recover 32 / 43
  31. Example 1: In-app Copilot ➃ Examples · In-app Copilot Query

    1 "Last month's delivery spend?" 🔧 find_expenses(category: "delivery", from: "2026-10-01", to: "2026-10-31") → Delivery: ₩472,700 Query 2 "Was that taxi logged twice? Remove dupes." 🔧 find_expenses(category: "transport") 🔧 delete_expenses(ids: [117]) → Approval dialog → Transport ₩34,200 → ₩19,900 On-device agent added to an existing budget app 34 / 43
  32. Example 1: In-app Copilot ➃ Examples · In-app Copilot Query

    1 "Last month's delivery spend?" 🔧 find_expenses(category: "delivery", from: "2026-10-01", to: "2026-10-31") → Delivery: ₩472,700 Query 2 "Was that taxi logged twice? Remove dupes." 🔧 find_expenses(category: "transport") 🔧 delete_expenses(ids: [117]) → Approval dialog → Transport ₩34,200 → ₩19,900 On-device agent added to an existing budget app 35 / 43
  33. Example 1: In-app Copilot - Tools ➃ Examples · In-app

    Copilot copilot_tools.dart Tool Purpose Type find_expenses Filter expenses Read ToolDefinition( name: 'add_expense', description: 'Records an expense.', parameters: [ spending_summary Spend by period Read ToolParam.integer('amount', description: 'Whole won.', required: true), ToolParam.string('category', description: categories, required: true), budget_status Category budgets Read ToolParam.string('merchant', required: true), ToolParam.string('date', description: 'YYYY-MM-DD. Defaults to today.'), add_expense Log expense Write set_budget Set budget Write delete_expenses Delete expense ], handler: (params) async { final amount = params.getRequiredInt('amount'); if (amount <= 0) throw ArgumentError('amount must be positive.'); Write · Confirm final expense = ledger.add(amount: amount, ...); ∕∕ existing app function return {'added': expense.toJson()}; }, ), Existing app functions as tools 37 / 43
  34. Example 1: In-app Copilot - Tool Call Loop ➃ Examples

    · In-app Copilot copilot.dart final result = await _session.sendWithTools( request, tools: _tools, maxRounds: 6, enableThinking: false, params: const GenerationParams(maxTokens: 512, temp: 0), onMessageAdded: (message) { for (final part in message.parts) { if (part is LlamaToolCallContent) { ∕* call: show in panel *∕ } if (part is LlamaToolResultContent) { ∕* result: mark done *∕ } } }, ); A simple Tool Call Loop with sendWithTools 38 / 43
  35. Example 1: In-app Copilot - Retries ➃ Examples · In-app

    Copilot Bad args · copilot_tools.dart Category? _category(String? name) { if (name == null || name.trim().isEmpty) return null; return Category.parse(name) ?? (throw ArgumentError( 'Unknown category "$name". Use one of ' '${Category.values.map((c) => c.name).join(', ')}.', )); Send error to the model → retry coffee: sees category list, calls again category is a string, not enum, checked in handler User approval · delete_expenses handler if (matches.isEmpty) { throw ArgumentError('No expense has the ids ${wanted.join(', ')}.'); } if (!await confirm(question)) { return {'deleted': 0, 'note': 'The user declined.'}; Approval dialog first Declines go back to the model too } return {'deleted': [for (final e in ledger.delete(wanted)) e.id]}; Errors and declines as tool results → model decides 39 / 43
  36. Example 2: Laya Tetris ➃ Examples · Laya Engine DecisionEngine

    supported by llamadart state Board and candidate placements query Best placement? (A~F, 6 choices) action Play the highest-probability choice example/laya_tetris 40 / 43
  37. Example 2: Laya Tetris ➃ Examples · Laya huggingface.co/convaiinnovations/laya ·

    421M params huggingface.co/leehack/laya-tetris-head · 6-choice accuracy 41 / 43
  38. Takeaways Wrap-up 01 The key concept in agents is handling

    Tool Calls llamadart brings llama.cpp to Dart and makes Tool Call work easy to start 02 Analyzing and contributing to llamadart deepened my grasp of on-device agents Just handling each model's Tool Call format and fixing bugs can raise accuracy 03 llamadart keeps getting updates! Now has DecisionEngine and many fun examples 42 / 43

More Decks by 권태뽕

Other Decks in Programming