Slide 1

Slide 1 text

Supersonic, Subatomic intelligent applications. Infusing Generative AI in your Java Apps with LangChain4j Kevin Dubois(@kevindubois) Principal Developer Advocate

Slide 2

Slide 2 text

Kevin Dubois ★ Principal Developer Advocate at Red Hat ★ Java Champion ★ Based in Belgium 󰎐 ★ 🗣 Speak English, Dutch, French, Italian ★ Open Source Contributor (Quarkus, Camel, Knative, ..) @[email protected] youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Left / Right of the model

Slide 5

Slide 5 text

Set goals App developer IT operations Data engineer Data scientists ML engineer Gather and prepare data Develop model Integrate models in app dev Model monitoring & management Retrain models Business leadership AI as a team initiative

Slide 6

Slide 6 text

Some Definitions

Slide 7

Slide 7 text

Boring Stuff Artificial Intelligence Artificial Intelligence “Ability of a machine to imitate human intelligent behaviour”

Slide 8

Slide 8 text

Machine Learning Artificial Intelligence “Application of AI to automatically learn and improve from experience” Machine Learning

Slide 9

Slide 9 text

Boring Stuff Deep Learning Artificial Intelligence “Vast volumes of data and complex algorithms to train a model using Neural Networks.” Machine Learning Deep Learning

Slide 10

Slide 10 text

Boring Stuff Generative AI Artificial Intelligence “Focuses on creating new and original content ” Machine Learning Deep Learning Generative AI

Slide 11

Slide 11 text

▸ Transformers (recognize, predict and generate human language) ▸ Trained in a huge amount of data (trillions of tokens) ▸ Relationship between words and phrases ▸ Fine-tuned for specific domains What are Large Language Models (LLMs)?

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

LLM Langchain4j

Slide 14

Slide 14 text

Dependency io.quarkiverse.langchain4j quarkus-langchain4j-openai 0.16.2

Slide 15

Slide 15 text

Prompts ▸ Interacting with the model for asking questions ▸ Interpreting messages to get important information ▸ Populating Java classes from natural language ▸ Structuring output

Slide 16

Slide 16 text

@RegisterAiService interface Assistant { String chat(String message); } -------------------- @Inject private final Assistant assistant; quarkus.langchain4j.openai.api-key=sk-... Configure API key Define Ai Service Create the instance

Slide 17

Slide 17 text

Prompting @SystemMessage("You are a professional poet") @UserMessage(""" Write a poem about {topic}. The poem should be {lines} lines long. """) String writeAPoem(String topic, int lines); Add context to the calls Main message to send Placeholder

Slide 18

Slide 18 text

class TransactionInfo { @Description("full name") public String name; @Description("IBAN value") public String iban; @Description("Date of the transaction") public LocalDate transactionDate; @Description("Amount in dollars of the transaction") public double amount; } interface TransactionExtractor { @UserMessage("Extract information about a transaction from {{it}}") TransactionInfo extractTransaction(String text); } Marshalling objects

Slide 19

Slide 19 text

Chains & Memory ▸ Create conversations ▸ Refer to past answers ▸ Manage concurrent interactions

Slide 20

Slide 20 text

@RegisterAiService(chatMemoryProviderSupplier = BeanChatMemoryProviderSupplier.class) interface AiServiceWithMemory { String chat(@UserMessage String msg); } --------------------------------- @Inject private AiServiceWithMemory ai; String userMessage1 = "Can you give a brief explanation of Kubernetes?"; String answer1 = ai.chat(userMessage1); String userMessage2 = "Can you give me a YAML example to deploy an app for this?"; String answer2 = ai.chat(userMessage2); Possibility to customize memory provider Remember previous interactions

Slide 21

Slide 21 text

@RegisterAiService(/*chatMemoryProviderSupplier = BeanChatMemoryProviderSupplier.class*/) interface AiServiceWithMemory { String chat(@MemoryId Integer id, @UserMessage String msg); } --------------------------------- @Inject private AiServiceWithMemory ai; String answer1 = ai.chat(1,"I'm Frank"); String answer2 = ai.chat(2,"I'm Betty"); String answer3 = ai.chat(1,"Who Am I?"); default memory provider Refers to conversation with id == 1, ie. Frank keep track of multiple parallel conversations

Slide 22

Slide 22 text

Tools ▸ Mixing business code with model ▸ Delegating to external services

Slide 23

Slide 23 text

@RegisterAiService(tools = EmailService.class) public interface MyAiService { @SystemMessage("You are a professional poet") @UserMessage("Write a poem about {topic}. Then send this poem by email.") String writeAPoem(String topic); @ApplicationScoped public class EmailService { @Inject Mailer mailer; @Tool("send the given content by email") public void sendAnEmail(String content) { mailer.send(Mail.withText("[email protected]", "A poem", content)); } } Describe when to use the tool Register the tool Ties it back to the tool description

Slide 24

Slide 24 text

Embedding Documents ▸ Adding terms of use rules into the model ▸ Asking questions of legal documents ▸ Natural queries

Slide 25

Slide 25 text

@Inject RedisEmbeddingStore store; EmbeddingModel embeddingModel; public void ingest(List documents) { EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder() .embeddingStore(store) .embeddingModel(embeddingModel) .documentSplitter(recursive(500, 0)) .build(); ingestor.ingest(documents); } Document from CSV, spreadsheet, text.. Ingested documents stored in Redis Ingest documents

Slide 26

Slide 26 text

@ApplicationScoped public class DocumentRetriever implements Retriever { private final EmbeddingStoreRetriever retriever; DocumentRetriever(RedisEmbeddingStore store, EmbeddingModel model) { retriever = EmbeddingStoreRetriever.from(store, model, 20); } @Override public List findRelevant(String s) { return retriever.findRelevant(s); } } CDI injection Augmentation interface

Slide 27

Slide 27 text

@RegisterAiService(retrieverSupplier = BeanRetrieverSupplier.class) public interface MyAiService { (..) } Look for the Retriever bean

Slide 28

Slide 28 text

Easier way to retrieve docs: Easy RAG! $ quarkus extension add langchain4j-easy-rag quarkus.langchain4j.easy-rag.path=src/main/resources/catalog Path to documents

Slide 29

Slide 29 text

Local Models ▸ Use models on-prem ▸ Evolve a model privately ▸ Sentiment Analysis of private data

Slide 30

Slide 30 text

Run LLMs locally and build AI applications podman-desktop.io Download now at: Supported platforms: Podman AI Lab

Slide 31

Slide 31 text

Fault Tolerance ▸ Gracefully handle model failures ▸ Retries, Fallback

Slide 32

Slide 32 text

@RegisterAiService() public interface AiService { @SystemMessage("You are a Java developer") @UserMessage("Create a class about {topic}") @Fallback(fallbackMethod = "fallback") @Retry(maxRetries = 3, delay = 2000) public String chat(String topic); default String fallback(String topic){ return "I'm sorry, I wasn't able create a class about topic: " + topic; } } Handle Failure $ quarkus ext add smallrye-fault-tolerance Add MicroProfile Fault Tolerance dependency Retry up to 3 times

Slide 33

Slide 33 text

Observability ▸ Collect metrics about your AI-infused app ▸ Trace through requests to see how long they took, and where they happened

Slide 34

Slide 34 text

$ quarkus ext add micrometer opentelemetry micrometer-registry prometheus

Slide 35

Slide 35 text

35 Free Developer e-Books!

Slide 36

Slide 36 text

@kevindubois Thank you! @thekevindubois kdubois @kevindubois