Quarkus meets AI: Build your own LLM-powered application

In today’s dynamic AI landscape, the seamless integration of Large Language Models (LLMs) into applications is a key focus for developers. While many initiatives have emerged to facilitate the integration of LLMs, the world of Java has seen limited options.

Enter Langchain4j, a powerful library designed to seamlessly integrate Java applications with LLMs. The excitement amplifies Langchain4j into Quarkus, a framework designed for building Cloud-Native applications in Java. Quarkus is tuned for Kubernetes environments boasting faster startup times and reduced memory usage compared to traditional Java applications. When Quarkus meets Langchain4j, the process of building a Java LLM-powered application becomes an enjoyable experience.

In this talk, we’ll delve into building AI applications powered by LLMs, using Quarkus and Langchain4j. We’ll leverage existing features from the ecosystem to create effective strategies for data ingestion. We’ll demonstrate how to seamlessly bring in information from a broader set of resources, with the power of Apache Camel.

Transcript

1. Quarkus meets AI Build your own LLM-powered applications Dimitris Andreadis Engineering Director Twitter @dandreadis Devoxx.ma, Oct 2024 Zineb Bendhiba Principal Engineer Twitter @zinebbendhiba
2. Red Hat 2 War Games (1983) https://www.imdb.com/title/tt0086567/
3. Red Hat 30 Years Later … 3
4. Red Hat AI was like 4 Turing Test (1950) ELIZA, early chatbot (1966) Expert Systems Coded in Prolog (1970s - 1980s) Deep Blue vs. Kasparov (1997)
5. Red Hat Or more like 5
6. Red Hat But then things started getting interesting… 6 AlphaGo (2016) / AlphaZero (2017) By Dllu - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=64517567 Waymo robotaxi (2018) Boston Dynamics Parkour Atlas (2018) https://www.youtube.com/watch?v=LikxFZZO2sk IBM Watson wins Jeopardy! (2011)
7. Red Hat …and then it got crazy with Generative AI 7 By Left intentionally blank - Colorado State Fair, Public Domain, https://commons.wikimedia.org/w/index.php?curid=137219696 Théâtre D'opéra Spatial - Jason M. Allen / Midjourney (2022)
8. Red Hat Generative AI for the masses! 8 https://en.wikipedia.org/wiki/ChatGPT
9. Red Hat Large Language Models - 101 9 Model Pre-training (self-supervised) • Prepare/configure the billion parameters* • Collect/tokenize trillions of data* • Let the model self-train on that data • Model learns to predict the next word ◦ E.g. "once upon a" ⇒ "time" (99%) ◦ But it can also make up stuff! • Requires $M(MM) in GPU/processing power and days to weeks of training Model Fine-tuning (supervised) • Curate hundreds of high quality chats (Q&As) • Let the model retrain on the new data set • Model learns to mimic the chat behaviour ◦ Understands instructions ◦ Using the pre-trained knowledge! • [Retrain to make it Helpful, Honest, Harmless] ◦ Reinforcement Learning from Human Feedback (RHFL) • Takes a day of processing for each cycle Turn Base Model into a Chat Assistant Model Evaluate, Deploy, Monitor, Re-train
10. Red Hat This is Data Science stuff! 10 What about the Developers?
11. Red Hat The "Model" is only part of your Architecture 11 It's just another API
12. Red Hat You can do *a lot* with a generic model 12 …and enough context (docs, emails, other data)
13. LangChain LangChain4j
14. Components of LangChain4j Prompt Templates Language Models Output Parsers Memory Document Splitters Document Loaders Embedding Models Embedding Stores Basics RAG Image Models New! LangChain4j Chains AI Services Tools
15. Red Hat • Build time ◦ Build time warnings ◦ Compile to native with GraalVM • Production Enhancements ◦ Unified Configuration ◦ Metrics, Tracing, Auditing • Programming Model ◦ Seamless integration with CDI ◦ Declarative AI Services (@RegisterAiService) • Developer Joy ◦ 'quarkus dev' enables iterative testing and Prompt tuning. ◦ Dev UI allows to view AI services, tools, config, add/search embeddings, test prompts, generate images • Performance Enhancements ◦ Optimized Quarkus REST/JSON libs ◦ Reduced library footprint Why LangChain4j with Quarkus? 15 +
16. Red Hat 16 Show me the Code!
17. Red Hat 17
18. Red Hat • Sentiment analysis on reviews, posts, etc. • Structured info from unstructured data • Transformation, summaries, explansions • Code suggestions, explanations • Language Translation / Localization • Data Analysis and Reporting • … • Customer service & support automation ◦ Chatbots, Booking assistants ◦ Educational assistants ◦ Learning and training • Marketing / Content creation ◦ Articles, blogs, social media updates ◦ Product recommendations ◦ Personalized content • Human Resources / Recruitment ◦ Job descriptions, CV screening, etc. • … Now your turn! 18 +
19. Red Hat Data Ingestion Pipeline to Store Embeddings 19 for Retrieval-Augmented Generation AWS S3 <data storage> Qdrant <vector database> Collect Data Transform Generate Embedding s Store Embedding s Apache Camel <data ingestion pipeline> New data <TXT> Embeddings <Point record> from("aws2-s3://my-bucket") .convertBodyTo(String.class) .to("langchain4j-embeddings:test&embedding-model=#embeddingModel") .transform(new DataType("qdrant:embeddings")) .setHeader(Qdrant.Headers.ACTION, constant(QdrantAction.UPSERT)) .to("qdrant:my-collection"); New AI-related components: • AI tools: Chatscript, DJL, LangChain4j-* • AI services: AWS Bedrock • Other vector databases: Milvus, Pinecone
20. Red Hat 20 Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more
21. Red Hat 21 Use your Super Powers to do Good!
22. https://quarkus.io @quarkusio quarkusio.zulipchat.com If you like Quarkus, star it on GitHub! https://github.com/quarkusio/quarkus @quarkusio