Slide 1
Slide 1 text
Georgios Andrianakis
Principal Software Engineer, Quarkus
Holly Cummins
Senior Principal Software Engineer, Quarkus
Leverage LLMs in Java with
LangChain4j and Quarkus
Open Conf
November 22, 2024
Slide 2
Slide 2 text
The landscape
Slide 3
Slide 3 text
THE 2024 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE
© Matt Turck (@mattturck) , Aman Kabeer (@AmanKabeer11) & FirstMark (@firstmarkcap)
Version 1.0 - March 2024
Blog post: mattturck.com/MAD2024 Interactive version: MAD.firstmarkcap.com Comments? Email [email protected]
AI MODELS
AI FRAMEWORKS, TOOLS & LIBRARIES
DATA & AI CONSULTING
MLOPS & AI INFRA
ESG
LOCATION INTELLIGENCE
DATA SOURCES & APIs
AIR / SPACE / SEA
FINANCIAL & MARKET DATA
PEOPLE / ENTITIES
OPEN SOURCE INFRASTRUCTURE
QUERY / DATA FLOW
STREAMING &
MESSAGING STAT TOOLS &
LANGUAGES
COLLABORATION
FORMATS
DATA MANAGEMENT
OLAP
DATABASES
SEARCH
LOCAL AI
VISUALIZATION
LOGGING & MONITORING
ORCHESTRATION
PRIVACY
& SECURITY
FULLY
MANAGED
GRAPH DBs
MPP DBs
DATA GOVERNANCE
& CATALOG
COMPUTE
GPU CLOUD /
INFRA EDGE AI
CLOSED
SOURCE MODELS
MGMT / MONITORING
NewSQL DATABASES
DATA INTEGRATION
DATA
WAREHOUSES
DATA LAKES /
LAKEHOUSES STREAMING /
IN-MEMORY
ORCHESTRATION
REVERSE ETL
REAL TIME
DATABASES GPU
DATABASES
VECTOR
DATABASES
MULTI-
MODEL
DATABASES &
ABSTRACTIONS
APPLICATIONS — INDUSTRY
APPLICATIONS — HORIZONTAL
HUMAN
CAPITAL DECISION &
OPTIMIZATION
MARKETING
SALES
CUSTOMER EXPERIENCE
FINANCE &
INSURANCE
PARTNERSHIPS
FINANCE
AUTOMATION
& OPERATIONS
TEXT AUDIO & VOICE IMAGE PRESENTATION &
DESIGN
CODE &
DOCUMENTATION
LEGAL
REGTECH &
COMPLIANCE
DATA SCIENCE
NOTEBOOKS DATA SCIENCE
PLATFORMS
COMPUTER
VISION
SPEECH / VOICE NLP
COMMERCIAL AI RESEARCH
NONPROFIT
AI RESEARCH
ENTERPRISE ML/AI PLATFORMS
AI OBSERVABILITY
AI SAFETY & SECURITY
DATA GENERATION
& LABELING
MLOPS
AI DEVELOPER PLATFORMS
AI HARDWARE
AGRICULTURE
HEALTHCARE
INDUSTRIAL &
LOGISTICS
LIFE SCIENCES
CROSS-
INDUSTRY
AEROSPACE,
DEFENSE & GOV’T
VIDEO EDITING
SEARCH
/ CONVER-
SATIONAL AI
VIDEO GENERATION
ANIMATION
& 3D / GAMING
TRANSPORTATION
CUSTOMER DATA
PLATFORMS
LOG ANALYTICS ENTERPRISE SEARCH /
KNOWLEDGE ANALYTICS
BI PLATFORMS
DATA ANALYST PLATFORMS
PRODUCT
ANALYTICS
VISUALIZATION
DATA MARKETPLACES
& DISCOVERY
DATA FRAMEWORKS
NoSQL DATABASES
ETL / ELT /
DATA TRANSFORMATION
RDBMS
STORAGE
DATA QUALITY &
OBSERVABILITY
INFRASTRUCTURE
APPLICATIONS — ENTERPRISE
MACHINE LEARNING & ARTIFICIAL INTELLIGENCE
ANALYTICS
INFRA-
STRUCTURE
AU LARGE
3
3
The landscape
Slide 4
Slide 4 text
THE 2024 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE
© Matt Turck (@mattturck) , Aman Kabeer (@AmanKabeer11) & FirstMark (@firstmarkcap)
Version 1.0 - March 2024
Blog post: mattturck.com/MAD2024 Interactive version: MAD.firstmarkcap.com Comments? Email [email protected]
AI MODELS
AI FRAMEWORKS, TOOLS & LIBRARIES
DATA & AI CONSULTING
MLOPS & AI INFRA
ESG
LOCATION INTELLIGENCE
DATA SOURCES & APIs
AIR / SPACE / SEA
FINANCIAL & MARKET DATA
PEOPLE / ENTITIES
OPEN SOURCE INFRASTRUCTURE
QUERY / DATA FLOW
STREAMING &
MESSAGING STAT TOOLS &
LANGUAGES
COLLABORATION
FORMATS
DATA MANAGEMENT
OLAP
DATABASES
SEARCH
LOCAL AI
VISUALIZATION
LOGGING & MONITORING
ORCHESTRATION
PRIVACY
& SECURITY
FULLY
MANAGED
GRAPH DBs
MPP DBs
DATA GOVERNANCE
& CATALOG
COMPUTE
GPU CLOUD /
INFRA EDGE AI
CLOSED
SOURCE MODELS
MGMT / MONITORING
NewSQL DATABASES
DATA INTEGRATION
DATA
WAREHOUSES
DATA LAKES /
LAKEHOUSES STREAMING /
IN-MEMORY
ORCHESTRATION
REVERSE ETL
REAL TIME
DATABASES GPU
DATABASES
VECTOR
DATABASES
MULTI-
MODEL
DATABASES &
ABSTRACTIONS
APPLICATIONS — INDUSTRY
APPLICATIONS — HORIZONTAL
HUMAN
CAPITAL DECISION &
OPTIMIZATION
MARKETING
SALES
CUSTOMER EXPERIENCE
FINANCE &
INSURANCE
PARTNERSHIPS
FINANCE
AUTOMATION
& OPERATIONS
TEXT AUDIO & VOICE IMAGE PRESENTATION &
DESIGN
CODE &
DOCUMENTATION
LEGAL
REGTECH &
COMPLIANCE
DATA SCIENCE
NOTEBOOKS DATA SCIENCE
PLATFORMS
COMPUTER
VISION
SPEECH / VOICE NLP
COMMERCIAL AI RESEARCH
NONPROFIT
AI RESEARCH
ENTERPRISE ML/AI PLATFORMS
AI OBSERVABILITY
AI SAFETY & SECURITY
DATA GENERATION
& LABELING
MLOPS
AI DEVELOPER PLATFORMS
AI HARDWARE
AGRICULTURE
HEALTHCARE
INDUSTRIAL &
LOGISTICS
LIFE SCIENCES
CROSS-
INDUSTRY
AEROSPACE,
DEFENSE & GOV’T
VIDEO EDITING
SEARCH
/ CONVER-
SATIONAL AI
VIDEO GENERATION
ANIMATION
& 3D / GAMING
TRANSPORTATION
CUSTOMER DATA
PLATFORMS
LOG ANALYTICS ENTERPRISE SEARCH /
KNOWLEDGE ANALYTICS
BI PLATFORMS
DATA ANALYST PLATFORMS
PRODUCT
ANALYTICS
VISUALIZATION
DATA MARKETPLACES
& DISCOVERY
DATA FRAMEWORKS
NoSQL DATABASES
ETL / ELT /
DATA TRANSFORMATION
RDBMS
STORAGE
DATA QUALITY &
OBSERVABILITY
INFRASTRUCTURE
APPLICATIONS — ENTERPRISE
MACHINE LEARNING & ARTIFICIAL INTELLIGENCE
ANALYTICS
INFRA-
STRUCTURE
AU LARGE
3
3
The landscape
😵💫
Slide 5
Slide 5 text
THE 2024 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE
© Matt Turck (@mattturck) , Aman Kabeer (@AmanKabeer11) & FirstMark (@firstmarkcap)
Version 1.0 - March 2024
Blog post: mattturck.com/MAD2024 Interactive version: MAD.firstmarkcap.com Comments? Email [email protected]
AI MODELS
AI FRAMEWORKS, TOOLS & LIBRARIES
DATA & AI CONSULTING
MLOPS & AI INFRA
ESG
LOCATION INTELLIGENCE
DATA SOURCES & APIs
AIR / SPACE / SEA
FINANCIAL & MARKET DATA
PEOPLE / ENTITIES
OPEN SOURCE INFRASTRUCTURE
QUERY / DATA FLOW
STREAMING &
MESSAGING STAT TOOLS &
LANGUAGES
COLLABORATION
FORMATS
DATA MANAGEMENT
OLAP
DATABASES
SEARCH
LOCAL AI
VISUALIZATION
LOGGING & MONITORING
ORCHESTRATION
PRIVACY
& SECURITY
FULLY
MANAGED
GRAPH DBs
MPP DBs
DATA GOVERNANCE
& CATALOG
COMPUTE
GPU CLOUD /
INFRA EDGE AI
CLOSED
SOURCE MODELS
MGMT / MONITORING
NewSQL DATABASES
DATA INTEGRATION
DATA
WAREHOUSES
DATA LAKES /
LAKEHOUSES STREAMING /
IN-MEMORY
ORCHESTRATION
REVERSE ETL
REAL TIME
DATABASES GPU
DATABASES
VECTOR
DATABASES
MULTI-
MODEL
DATABASES &
ABSTRACTIONS
APPLICATIONS — INDUSTRY
APPLICATIONS — HORIZONTAL
HUMAN
CAPITAL DECISION &
OPTIMIZATION
MARKETING
SALES
CUSTOMER EXPERIENCE
FINANCE &
INSURANCE
PARTNERSHIPS
FINANCE
AUTOMATION
& OPERATIONS
TEXT AUDIO & VOICE IMAGE PRESENTATION &
DESIGN
CODE &
DOCUMENTATION
LEGAL
REGTECH &
COMPLIANCE
DATA SCIENCE
NOTEBOOKS DATA SCIENCE
PLATFORMS
COMPUTER
VISION
SPEECH / VOICE NLP
COMMERCIAL AI RESEARCH
NONPROFIT
AI RESEARCH
ENTERPRISE ML/AI PLATFORMS
AI OBSERVABILITY
AI SAFETY & SECURITY
DATA GENERATION
& LABELING
MLOPS
AI DEVELOPER PLATFORMS
AI HARDWARE
AGRICULTURE
HEALTHCARE
INDUSTRIAL &
LOGISTICS
LIFE SCIENCES
CROSS-
INDUSTRY
AEROSPACE,
DEFENSE & GOV’T
VIDEO EDITING
SEARCH
/ CONVER-
SATIONAL AI
VIDEO GENERATION
ANIMATION
& 3D / GAMING
TRANSPORTATION
CUSTOMER DATA
PLATFORMS
LOG ANALYTICS ENTERPRISE SEARCH /
KNOWLEDGE ANALYTICS
BI PLATFORMS
DATA ANALYST PLATFORMS
PRODUCT
ANALYTICS
VISUALIZATION
DATA MARKETPLACES
& DISCOVERY
DATA FRAMEWORKS
NoSQL DATABASES
ETL / ELT /
DATA TRANSFORMATION
RDBMS
STORAGE
DATA QUALITY &
OBSERVABILITY
INFRASTRUCTURE
APPLICATIONS — ENTERPRISE
MACHINE LEARNING & ARTIFICIAL INTELLIGENCE
ANALYTICS
INFRA-
STRUCTURE
AU LARGE
3
3
The landscape
😵💫 😖
Slide 6
Slide 6 text
But I’m a Java developer.
I do not want whitespace to have semantic significance.
Slide 7
Slide 7 text
Our goal: enabling
developers to use AI
models
4
Slide 8
Slide 8 text
A simplified landscape
Left / Right of the Model
Slide 9
Slide 9 text
A simplified landscape
Left / Right of the Model
Slide 10
Slide 10 text
A simplified landscape
Left / Right of the Model
Slide 11
Slide 11 text
A simplified landscape
Left / Right of the Model
Slide 12
Slide 12 text
A simplified landscape
Left / Right of the Model
Slide 13
Slide 13 text
A simplified landscape
Left / Right of the Model
Slide 14
Slide 14 text
A simplified landscape
Left / Right of the Model
Slide 15
Slide 15 text
A simplified landscape
Left / Right of the Model
Slide 16
Slide 16 text
A simplified landscape
Left / Right of the Model
Slide 17
Slide 17 text
Langchain4j
Slide 18
Slide 18 text
Dependency
io.quarkiverse.langchain4j
quarkus-langchain4j-openai
0.16.4
Slide 19
Slide 19 text
It all starts with prompts
Slide 20
Slide 20 text
Demo time
🎸
LangChain4j
Slide 21
Slide 21 text
@RegisterAiService
interface Assistant {
String chat(String message);
}
--------------------
@Inject
private final Assistant assistant;
quarkus.langchain4j.openai.api-key=sk-...
Configure an API key
Define Ai
Service
Use DI to instantiate
Assistant
Slide 22
Slide 22 text
@SystemMessage("You are a professional poet")
@UserMessage("""
Write a poem about {topic}. The poem should be {lines} lines long.
""")
String writeAPoem(String topic, int lines);
Add context to the calls
Main message to send
Placeholder
Slide 23
Slide 23 text
Demo time
🎸
AIService API
Slide 24
Slide 24 text
Interaction flows need memory
Slide 25
Slide 25 text
@RegisterAiService(chatMemoryProviderSupplier =
BeanChatMemoryProviderSupplier.class)
interface AiServiceWithMemory {
String chat(@UserMessage String msg);
}
---------------------------------
@Inject
private AiServiceWithMemory ai;
String userMessage1 = "Can you give a brief explanation of Kubernetes?";
String answer1 = ai.chat(userMessage1);
String userMessage2 = "Can you give me a YAML example to deploy an app for this?";
String answer2 = ai.chat(userMessage2);
Possibility to customize memory provider
(Quarkus provides a default)
Remember previous interactions
Slide 26
Slide 26 text
@RegisterAiService(/*chatMemoryProviderSupplier = BeanChatMemoryProviderSupplier.class*/)
interface AiServiceWithMemory {
String chat(@MemoryId Integer id, @UserMessage String msg);
}
---------------------------------
@Inject
private AiServiceWithMemory ai;
String answer1 = ai.chat(1,"I'm Frank");
String answer2 = ai.chat(2,"I'm Betty");
String answer3 = ai.chat(1,"Who Am I?");
default memory
provider
Refers to conversation with id == 1, ie. Frank
keep track of multiple parallel conversations
Slide 27
Slide 27 text
Demo time
🎸
AIService API
Slide 28
Slide 28 text
Moving beyond text
Slide 29
Slide 29 text
Moving beyond text to objects
Slide 30
Slide 30 text
class TransactionInfo {
@Description("full name")
public String name;
@Description("IBAN value")
public String iban;
@Description("Date of the transaction")
public LocalDate transactionDate;
@Description("Amount in dollars of the transaction")
public double amount;
}
interface TransactionExtractor {
@UserMessage("Extract information about a transaction from {{it}}")
TransactionInfo extractTransaction(String text);
}
Marshalling
objects
Slide 31
Slide 31 text
Demo time
🎸
AIService API
Slide 32
Slide 32 text
Moving beyond text to actions
Slide 33
Slide 33 text
Moving beyond text to logic
Slide 34
Slide 34 text
@RegisterAiService(tools = EmailService.class)
public interface MyAiService {
@SystemMessage("You are a professional poet")
@UserMessage("Write a poem about {topic}. Then send this poem by email.")
String writeAPoem(String topic);
@ApplicationScoped
public class EmailService {
@Inject
Mailer mailer;
@Tool("send the given content by email")
public void sendAnEmail(String content) {
mailer.send(Mail.withText("[email protected]", "A poem", content));
}
}
Describe when to use the tool
Register the tool
Ties it back to the
tool description
Slide 35
Slide 35 text
Demo time
🎸
AIService API
Slide 36
Slide 36 text
Thank you!
red.ht/quarkus-langchain4j-tutorial
http://hollycummins.com/llms-and-quarkus-openconf/