Bringing LangChain4j to Kotlin

Presentation at Devclub.ee 2025-03-26

### **Summary of the Presentation: Bringing LangChain4j to Kotlin**

#### **Speaker:**
- **Konstantin Pavlov**, Software Engineer at Twilio.

#### **Overview:**
- The presentation discusses integrating **LangChain4j** with **Kotlin** to build AI-driven applications.
- Focus on using **Kotlin DSL (Domain-Specific Language)** for **concise, expressive, and safe** AI development.

---

### **Key Topics Covered:**

#### **1. LangChain4j in Kotlin**
- **LangChain4j** enables AI models to interact with various **data sources, memory, tools, and instructions**.
- Provides a **framework** for using **Large Language Models (LLMs)** within Kotlin applications.

#### **2. AI Components & Workflow**
- AI **character (virtual agent)** interacts with:
- 🌍 **World Knowledge**
- 📖 **Personal Knowledge**
- 📝 **Instructions (System Prompt)**
- 💬 **Chat Memory**
- 🛠 **Tools (Functions)**
- Queries are processed by an **LLM**, which returns responses.

#### **3. Retrieval-Augmented Generation (RAG)**
- **Indexing & Retrieval** for **efficient AI responses**.
- **Embedding models** used to store and fetch relevant data chunks.

#### **4. OpenAI & Model Context Protocol**
- Uses **OpenAI APIs** for response generation.
- **Model Context Protocol (MCP)** standardizes AI interactions.

#### **5. Running AI Workloads on Java**
- Java is chosen for **historical reasons** and **enterprise compatibility**.
- Kotlin offers advantages like **concise syntax, null safety, and better async handling**.

#### **6. Kotlin DSL for AI Development**
- **Cleaner and more readable** way to interact with AI models.
- Example:
```kotlin
model.chat {
messages(
userMessage("Tell me a joke about LLM"),
)
}
```
- Allows **extension functions** and **function types with receivers**.

#### **7. Testing AI Models with Mocks**
- Challenges:
- Non-deterministic responses.
- API costs and network dependencies.
- Complexity in **CI/CD**.
- **AI-Mocks**:
- Provides **fast, deterministic, offline tests**.
- Simulates **OpenAI completions** to test AI behavior.

#### **8. Running LLMs in Production**
- **Challenges**:
- High **latency** in LLM responses.
- **Cloud reliability** issues.
- **Security**: Prevent exposing sensitive data.
- **Monitoring**: Track **token usage & costs**.

---

### **Resources & Links**
- **Demo Code**: [GitHub - Elven Project](https://github.com/kpavlov/elven)
- **AI Mocks**: [AI Mocks Documentation](https://kpavlov.github.io/ai-mocks)
- **OpenAI Responses API**: [OpenAI Docs](https://platform.openai.com/docs/guides/responses-vs-chat-completions)
- **Speaker Contact**:
- **GitHub**: [@kpavlov](https://github.com/kpavlov)
- **Website**: [kpavlov.me](https://kpavlov.me)
- **LinkedIn**: [in/kpavlov](https://www.linkedin.com/in/kpavlov)

---

### **Conclusion**
The presentation explores **bringing LangChain4j to Kotlin**, emphasizing **Kotlin DSL, AI integration, testing with mocks, and production challenges**. It provides a **practical approach** to running **AI-driven applications efficiently** on the JVM. 🚀

Transcript

1. Bringing LangChain4j to Kotlin Konstantin Pavlov Software Engineer, Twilio
2. Demo github.com/kpavlov/elven
3. AI Character (Virtual Agent) 🌍 World Knowledge 📖 Personal Knowledge 📝 Instructions (System Prompt) 💬 Chat Memory 🛠 Tools (Functions) Query Response 💾 State 🤖 LLM
4. Reply Query 🤖 Large Language Model Query LLM Response
5. Reply Query 😵 Large Language Model Query LLM Response
6. Reply Query 😵 Large Language Model Query LLM Response 🤖 Large Language Model 2 Query LLM Response
7. Reply Query 🤖 Large Language Model 2 Query LLM Response 🤖 Large Language Model 1 Query LLM Response Framweork
8. Reply Query 🧠 AI Services 🤖 Large Language Model Query + Parameters LLM Response
9. Reply Query 🧠 AI Services 🤖 Large Language Model Query + Parameters LLM Response 🌍 World Knowledge 📖 Personal Knowledge
10. https://tokens-lpj6s2duga-ew.a.run.app Video: "RAG: from dumb implementation to serious results" https://youtu.be/6_wUUYKBdE0
11. RAG (Retrieval-Augmented Generation): Indexing
12. RAG (Retrieval-Augmented Generation): Retrieval
13. Embedding Store Reply Query 🤖 Embedding Model Query 🧩 Chunks 🧠 AI Services 🤖 Large Language Model Query + Parameters 🧩 LLM Response
14. New: OpenAI Responses API https://platform.openai.com/docs/guides/responses-vs-chat-completions Blog post: "New tools for building agents"
15. Embedding Store Reply Query 🤖 Embedding Model Query 🧩 Chunks 🧠 AI Services 🤖 Chat Language Model 💬 Chat Memory Messages + Parameters 🧩 💬 Messages LLM Response
16. Embedding Store Reply Query 🤖 Embedding Model Query 🧩 Chunks 🧠 AI Services 🛠 Tools (Functions) 🤖 Chat Language Model 💬 Chat Memory Messages + Parameters 🧩 💬 🛠 Messages LLM Response 🌡
17. Server Process Server Process Host Transport Layer Transport Layer MCP Client MCP Client MCP Server MCP Server https://modelcontextprotocol.io/ https://github.com/modelcontextprotocol/servers
18. Reference Servers Third-Party Servers
19. Embedding Store Reply Query 🤖 Embedding Model Query 🧩 Chunks 🧠 AI Services 🛠 Tools (Functions) 🤖 Chat Language Model 💬 Chat Memory Messages + Parameters 🧩 💬 🛠 Messages LLM Response 🌡 🤖 Moderation Model Is Query OK?
20. Why Run AI Workloads on Java? "Historical reasons" Can’t run on Python/JavaScript JVM is the Organization choice
21. Java LLM Frameworks LLM Client "Many patterns can be implemented in a few lines of code"
22. Java LLM Frameworks Spring AI When: Spring Multiple LLMs Complex workflow Observability
23. LLM Integration on JVM LangChain4J No Spring Quarkus Spring (yes, there is langchain4j-spring) Multiple LLMs Complex workflow Uncommon LLMs / Vector DBs
24. Why Kotlin? 📝 Concise and expressive syntax 🛡 Built-in Null safety 🔄 Write asynchronous code that feels synchronous
25. Kotlin DSL // Java-style model.chat( ChatRequest .builder() .messages( userMessage("Tell me a joke about LLM"), ).build(), )
26. Kotlin DSL // Kotlin DSL-style model.chat { messages( userMessage("Tell me a joke about LLM"), ) }
27. Kotlin DSL // Function type with receiver fun chatRequest(block: ChatRequestBuilder.() -> Unit): ChatRequest { val builder = ChatRequestBuilder() block.invoke(builder) return builder.build() }
28. Kotlin DSL fun chatRequest(block: ChatRequestBuilder.() -> Unit): ChatRequest { val builder = ChatRequestBuilder() block.invoke(builder) return builder.build() } // Extension function fun ChatLanguageModel.chat(block: ChatRequestBuilder.() -> Unit): ChatResponse { val model = this return model.chat(chatRequest(block)) }
29. Kotlin DSL // Usage: model.chat { messages( userMessage("Tell me a joke about LLM"), ) }
30. Examples
31. Testing Challenges Non-deterministic responses API costs add up quickly Network dependencies CI/CD complexity and fragility Simulating edge cases
32. Simple LC4J Mock val model = ChatModelMock.thatAlwaysResponds( "Yes, Sir", ) // ChatModelMock.thatAlwaysThrowsException() // ChatModelMock.thatAlwaysThrowsExceptionWithMessage(String) val service: JokeService = AiServices.create( JokeService::class.java, model, ) println(service.joke("John")) // "Yes, Sir"
33. AI-Mocks Black-box testing Fast derterministic tests Zero API costs Works offline and on CI/CD Simulates negative scenarios Supports Server-Sent Events (SSE) kpavlov.github.io/ai-mocks
34. Simulating OpenAI Completion Response val mockOpenAi = MockOpenai() mockOpenAi.completion { userMessageContains("Tell me a joke about LLM") } responds { assistantContent = "Why did LLM cross road? Hallucination." } val model: JokeService = OpenAiChatModel.builder() .baseUrl(mockOpenAi.baseUrl()) // other .build()
35. Running LLM Integration on Production Challenges ⏱️ High LLM latencies 🔄 Cloud LLMs may not be reliable ✅ Validate user input and LLM output 🔒 Avoid sharing sensitive data with LLMs 📊 Monitor token usage and costs
36. Thank You! Questions? github.com/kpavlov/lc4j-kotlin- demo/blob/main/LINKS.md in/kpavlov github.com/kpavlov https://kpavlov.me