1 Java + LLMs: A hands-on guide to building LLM Apps in Java with Jakarta Syed M Shaaf Developer Advocate @ Red Hat Technical Editor @ InfoQ Bazlur Rahman Java Champion 🏆 Staff Software Developer at DNAstack 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev ● Quick introduction ● Prompts ● Tools ● RAG (Retrieval Augmented Generation) ● Model Context Protocol What we will cover today 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev ● Systems do not speak Natural language, can’t translate and lack context outside of system boundaries. (e.g. sentiment) ● Generating content is costly and sometimes hard. ● Rapid data growth ● Rising Expectations: Customers demand instant, personalized solutions. ● Inefficiency: Manual processes increase costs and slow operations. ● Skill Gaps: Limited expertise in AI adoption. Systems, Data, Networks and a Solution? 

No content

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Langchain4J 

DEMO 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev A Simple chat bot - Basic htmx - Chat window - Backend sends question to the LLM. - Streaming is also an option 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Whats an AI Service? - AI Services, tailored for Java - similar to Spring Data JPA or Retrofit - handle the most common operations 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Prompts System prompt - Define the task - Set the expectations - Provide examples User prompt - Specific to the input When to use system vs user? What is a good prompt!? - E.g. Structure your input and output, (different LLMs behave differently) ** Try not to migrate prompts across models 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Reasoning - Chain of Thought - TOT reasoning - Tree of Thought (Thinking, Organizing, Translating) 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Few-Shot , Zero Shot Zero-Shot - No data collection needed - Better accuracy with minimal examples - Lower accuracy on complex tasks Few-Shot - Fast implementation - Adaptable to niche tasks - Sensitive to example quality/order 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Chat and Memory ● Eviction policy ● Persistence ● Special treatment of SystemMessage and Tools 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Function calling / Tools @Tool double squareRoot(double x) { return Math.sqrt(x); } - Call other services or functions to enhance the response. - E.g. Web APIs, internal system requests 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev “feed relevant pieces of information (chunks) from your knowledge base to an LLM along with the user's query”... Retrieval Augmented Generation 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation What is the representation of the data? How do I want to split? Per document Chapter Sentence How many tokens do I want to end up with? How much overlap is there between segments? 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation Paragraph and Sentence splitters aim for semantic coherence but have variable chunk sizes. Character (especially recursive) and Word splitters offer size control but risk breaking semantic meaning. Line splitters are for specific line-oriented formats. Regex splitters provide maximum flexibility for known structures. 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev MCP - Model Context Protocol - Standardized Context Format - Improved Interoperability - Richer Context Representation - Enhanced Model Grounding https://github.com/modelcontextprotocol 

https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev ● Quick introduction ● Prompts ● Tools ● RAG (Retrieval Augmented Generation) ● Model Context Protocol What we covered…. 

1 9 Thank you! Syed M Shaaf Developer Advocate Red Hat Bazlur Rahman Java Champion 🏆 Empowering Developers through Speaking 🗣 Writing ✍ Mentoring 🤝 & Community Building 🌍 Published Author 📖 Contributing Editor at InfoQ and Foojay.IO fosstodon.org/@shaaf sshaaf https://www.linkedin.com/in/shaaf/ shaaf.dev https://bsky.app/profile/shaaf.dev https://x.com/bazlur_rahman rokon12 https://www.linkedin.com/in/bazlur/ https://bazlur.ca/ https://bsky.app/profile/bazlur.ca Source for the demo https://github.com/learnj-ai/llm-jakarta https://docs.langchain4j.dev/ LangChain4J