the Development Environment and adherence of the developers to their “local developer experience” in particular for testing and debugging Convenience & Simplicity Direct Access to Hardware Ease of Integration Simplify the integration of the model with existing systems and applications that are already running locally. For Organizations Data Privacy and Security Data is the fuel for AI, and a differentiator factor (quality, quantity, qualification). Keeping data on-premises ensures sensitive information doesn’t leave the local environment → crucial for privacy-sensitive applications Cost Control While there is an initial investment in hardware and setup, running locally can potentially reduce ongoing costs of cloud computing services and alleviate the vendor-locking played by Amazon, MSFT, Google Regulatory Compliance Some industries have strict regulations about where and how data is processed Customization & Control Easily train or fine-tune your own model, from the convenience of the developer’s local machine.
locally, offline, and privately ▸ Extensible: Basic model customization (Modelfile) and importing of fine-tuned LLMs ▸ Cons: Loads in ALL accelerators/libraries, no clear path to production Tool #1: Ollama https://ollama.com
serve models • Debug Mode: See what’s happening in the background • Ability to customize runtime for best performance • NOT Open Source ☹ Tool #2: LM Studio https://lmstudio.ai/
RAG, Agentic, Summarizers ▸ Curated Models: Easily access Apache 2.0 open-source options. ▸ Container Native: Easy app integration and movement from local to production. ▸ Interactive Playgrounds: Test & optimize models with your custom prompts and data. Tool #3: Podman AI Lab https://podman-desktop.io/docs/ai-lab
want to tackle & how ”Open Source” it should be. ▸ DeepSeek, gpt-oss, Gemma4 models excel in reasoning tasks and complex problem-solving. ▸ Qwen, GLM, Devstral, Minimax are good coding assistant models. ▸ IBM’s Granite models are great for general tasks using minimal resources, RAG with docling, and fine-tuning So, which local model should you select?
text-to-image text-to-text image-to-text image-to-image text-to-code Text Image Audio Video Multimodal any-to-any ✓ Single data input ✓ Less resources ✓ Single modality ✓ Limited depth and accuracy ✓ Multiple data inputs ✓ More resources ✓ Multiple modality ✓ Better understanding and accuracy OR
various architectures! Also! There’s a naming convention ibm-granite/granite-4.0-8b-base Family name Model architecture and version Number of parameters Model fine-tuned to be a baseline Mixtral-8x7B-Instruct-v0.1 Family name Model version Number of parameters Model fine-tuned for instructive tasks Architecture type
numerical precision. ▸ Converts high-precision weights (FP32) into lower-bit formats (FP16, INT8, INT4). ▸ Reduces size and memory footprint, making models easier to deploy. It’s a way to compress models, think like a .zip or .tar Most models for local usage are quantized!
programmer, to generate and explain your codebase. Tools: Continue, Roo Code, Cline, OpenCode, Devoxx Genie … How to use local, disconnected (?) code assistants Fortunately, many tools exist for this too!
so a few adjustments help: • Be specific. Instead of "fix the bug," point to the file and describe the issue. • Work in small steps. Ask for one change at a time rather than large refactors across multiple files. • Use capable models. For complex code tasks, the 26B+ parameter models perform significantly better than smaller ones. (if your system can handle it) • Keep context focused. Smaller context windows mean your assistant may struggle with very large files. Break big files into smaller modules when possible. Tips for local models
compute / GPUs ➔ Good for small tasks ➔ Can be slow ➔ Need to craft prompts carefully ➔ Not the best for more complex work ➔ Able to work disconnected Premium: ➔ Pretty much works on any hardware ➔ Faster and typically much better at doing complex work ➔ No/unstable network or out of tokens? game over ➔ Can become expensive ➔ Support
models locally ▸ Pick the right model for the right use case ▸ Make sure the model comes from a reputable source (!) ▸ Local code assistants work… ish ▸ You might need to ask for hardware upgrades 😅 ▸ Developing local Agentic AI apps with Java is definitely possible ▸ Quarkus has a lot of productivity AI tricks up its sleeve Wrapping it up
.jpg){kind=small}
.jpg){kind=avatar}
.jpg){kind=avatar}
kdubois
musan
oniak3ibm
hjmkth
nakahara
shogogg
chobishiba
handlename
manfredsteyer
masuda220
rollbear
javiergs
niftycorp
gr2m
asonas
notwaldorf
watany
tmiket
marcoroth
chikuwait
.png){kind=avatar}
helenjbeal
rasagy
inesmontani
mfonobong
.jpg){kind=avatar}
cargoodrich
