Squeezing the most out of Foundational Models on-device LLM

In the summer 2025 Apple released their OS updates shipping with on-device LLMs. While quite limited, you can still get quite a bit of milage out of them. This talk is going through multiple patterns that allow to mitigate many shortcomings:
1. Short context window → Recompact chat history to create illusion of infinite chat.
2. Routing → Make your own multimodal model without waiting for Apple to ship it.
3. RAG → Ground model in your private knowledge.
4. Majority voting → Improve quality of answers by choosing the best one with judge LLM.
5. Memory → Preserve user information across sessions allowing LLM to read and write memories.
6. Semantic caching → Save cycles on generating expensive content.
7. Agentic setup → Use Apple Foundation Models to build Perplexity-like agent searching internet for you.

Bonus:
How to set up evals using Swift unit testing framework preventing sudden quality degradation if Apple updates Foundation Models

Source code for the companion app https://github.com/zats/LLMPatterns

Transcript

1. NSSpain'25, @zats
2. NSSpain'25, @zats
3. I. Gluttony Make 4k context window last Recompacting NSSpain'25, @zats
4. NSSpain'25, @zats
5. NSSpain'25, @zats
6. II. Lust Text isn't always what you want Routing NSSpain'25, @zats
7. NSSpain'25, @zats
8. NSSpain'25, @zats
9. III. Pride Know where your knowledge cutoff is RAG NSSpain'25, @zats
10. NSSpain'25, @zats
11. NSSpain'25, @zats
12. NSSpain'25, @zats
13. NSSpain'25, @zats
14. IV. Envy There's always a better answer out there Majority voting NSSpain'25, @zats
15. NSSpain'25, @zats
16. NSSpain'25, @zats
17. V. Wrath Make me repeat that one more time! Memory NSSpain'25, @zats
18. NSSpain'25, @zats
19. NSSpain'25, @zats
20. NSSpain'25, @zats
21. VI. Sloth When being lazy is good Semantic caching NSSpain'25, @zats
22. NSSpain'25, @zats
23. NSSpain'25, @zats
24. VII. Greed Single LLM isn't all you need Agentic stuff NSSpain'25, @zats
25. NSSpain'25, @zats
26. NSSpain'25, @zats
27. NSSpain'25, @zats
28. Atonement Future crime prevention Evals NSSpain'25, @zats
29. NSSpain'25, @zats
30. Thank You NSSpain'25! NSSpain'25, @zats