TechTalk - Java + LLMs: A hands-on guide to building LLM Apps in Java with Jakarta

Transcript

1. 1 Java + LLMs: A hands-on guide to building LLM Apps in Java with Jakarta Syed M Shaaf Developer Advocate @ Red Hat Technical Editor @ InfoQ Bazlur Rahman Java Champion 🏆 Staff Software Developer at DNAstack
2. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Quick introduction • Prompts • Tools • RAG (Retrieval Augmented Generation) • Model Context Protocol What we will cover today
3. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Systems do not speak Natural language, can’t translate and lack context outside of system boundaries. (e.g. sentiment) • Generating content is costly and sometimes hard. • Rapid data growth • Rising Expectations: Customers demand instant, personalized solutions. • Inefficiency: Manual processes increase costs and slow operations. • Skill Gaps: Limited expertise in AI adoption. Systems, Data, Networks and a Solution?
5. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Langchain4J
6. DEMO
7. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev A Simple chat bot - Basic htmx - Chat window - Backend sends question to the LLM. - Streaming is also an option
8. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Whats an AI Service? - AI Services, tailored for Java - similar to Spring Data JPA or Retrofit - handle the most common operations
9. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Prompts System prompt - Define the task - Set the expectations - Provide examples User prompt - Specific to the input When to use system vs user? What is a good prompt!? - E.g. Structure your input and output, (different LLMs behave differently) ** Try not to migrate prompts across models
10. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Reasoning - Chain of Thought - TOT reasoning - Tree of Thought (Thinking, Organizing, Translating)
11. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Few-Shot , Zero Shot Zero-Shot - No data collection needed - Better accuracy with minimal examples - Lower accuracy on complex tasks Few-Shot - Fast implementation - Adaptable to niche tasks - Sensitive to example quality/order
12. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Chat and Memory • Eviction policy • Persistence • Special treatment of SystemMessage and Tools
13. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Function calling / Tools @Tool double squareRoot(double x) { return Math.sqrt(x); } - Call other services or functions to enhance the response. - E.g. Web APIs, internal system requests
14. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev “feed relevant pieces of information (chunks) from your knowledge base to an LLM along with the user's query”... Retrieval Augmented Generation
15. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation What is the representation of the data? How do I want to split? Per document Chapter Sentence How many tokens do I want to end up with? How much overlap is there between segments?
16. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation Paragraph and Sentence splitters aim for semantic coherence but have variable chunk sizes. Character (especially recursive) and Word splitters offer size control but risk breaking semantic meaning. Line splitters are for specific line-oriented formats. Regex splitters provide maximum flexibility for known structures.
17. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev MCP - Model Context Protocol - Standardized Context Format - Improved Interoperability - Richer Context Representation - Enhanced Model Grounding https://github.com/modelcontextprotocol
18. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Quick introduction • Prompts • Tools • RAG (Retrieval Augmented Generation) • Model Context Protocol What we covered….
19. 1 9 Thank you! Syed M Shaaf Developer Advocate Red Hat Bazlur Rahman Java Champion 🏆 Empowering Developers through Speaking 🗣 Writing ✍ Mentoring 🤝 & Community Building 🌍 Published Author 📖 Contributing Editor at InfoQ and Foojay.IO fosstodon.org/@shaaf sshaaf https://www.linkedin.com/in/shaaf/ shaaf.dev https://bsky.app/profile/shaaf.dev https://x.com/bazlur_rahman rokon12 https://www.linkedin.com/in/bazlur/ https://bazlur.ca/ https://bsky.app/profile/bazlur.ca Source for the demo https://github.com/learnj-ai/llm-jakarta https://docs.langchain4j.dev/ LangChain4J