Spring AI vs LangChain4J

Transcript

1. AI for Java Developers Spring AI vs LangChain4J Jean-Vincent Gennade / Michael Isvy
2. Who we are • Jean-Vincent Gennade - Zenika • Michael Isvy - cynapse.ai M
3. What we will talk about • The AI Space • Prompting with Spring AI and LangChain4J • Retrieval Augmented Generation • Vector Databases M
4. The AI space
5. Artificial Intelligence: which Way? Conventional AI Generative AI LLM, ChatGPT, DALL-E, Gemini, Mistral, Ollama… Programming: mostly API calls Based on custom-trained models Programming: requires AI engineers M
6. Conventional AI example: Licence Plate Recognition • Find a base model online Typically on Github or HuggingFace • Evaluate the model Identify gaps (example: doesn’t work with Singapore truck license plates) • Prepare a fine-tuning dataset • Spend 3-4 days training the model
7. Conventional AI model = MobileNetV2(weights='imagenet') # Prepare a sample input image img = image.load_img('path_to_image.jpg', target_size=(224, 224)) # … # Run inference predictions = model.predict(img_array) # … print(f"Predicted class: {predicted_class[1]}") Load the model Prediction Conventional AI is typically done in Python or C++ Python code
8. Generative AI: usually an API call! • Example: API call to ChatGPT #!/bin/bash API_KEY="sk-proj-7devtvnBIsYXVHJuHBQAT3BlbkFJNBB4uz8Iog5F2y" curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" -H "Authorization: Bearer $API_KEY" \ -d '{ "model": "gpt-4o", "messages": [ {"role": "user", "content": "Tell me a joke." } ] }' Linux/OSX In GenAI, models are much more complex. But most of the time you don’t need to build them
9. Quiz 1 • In the below example, there is something you should never do. What is it? #!/bin/bash API_KEY="sk-proj-7devtvnBIsYXVHJuHBQAT3BlbkFJNBB4uz8Iog5F2y" curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" -H "Authorization: Bearer $API_KEY" \ -d '{ "model": "gpt-4o", "messages": [ {"role": "user", "content": "Tell me a joke." } ] }' Linux/OSX
10. Generative AI • General-Purpose models • Use-cases ◦ Chatbot ◦ Image recognition (read invoices…) ◦ Search a large number of documents ▪ And ask questions from chatbot ◦ …
11. Quiz 2 • I work for a Payment company and we need to setup a system for Fraud detection. We will have a lot of custom rules in order to identify fraud patterns • This is critical to our business • Should I use Conventional AI or Generative AI for that?
12. In the Cloud On premise / on your laptop The Generative AI landscape OpenAI ChatGPT Mistral AI Mistral Anthropic Claude Google Gemini Spin off from OpenAI Ollama Llama3.1 Mixtral LLava tinyllama … Choose your local model
13. Which Skill Set? Software Engineer AI Engineer prepares datasets Trains models Knows how Neural Networks work Good with Python and sometimes C++ Often runs a laptop with a heavy GPU! Handles large codebases Apply clean code and TDD Works with Distributed Systems This is a general trend we have observed. Feedback welcome!
14. Do you need to be an AI Engineer? Software Engineer AI Engineer Build a ChatBot based on OpenAI Run LLM models on-prem in production Fine-tune local LLM model Parse PDF and Excel files and ask questions
15. AI in Java
16. Conventional AI with Java • Java has a tiny ecosystem in Conventional AI • NVIDIA libraries offer native support for Python and C++ only
17. Generative AI with Java • Java is getting a lot of momentum for GenAI ◦ Since it’s mostly about calling API and using (Vector) databases • 2 main options ◦ Spring AI ▪ Created by Mark Pollack and Chris Tzolov (Spring team) ◦ LangChain4J ▪ Created by Dmytro Liubarskyi (Red Hat) ▪ Adaptation of LangChain to Java
18. Our approach • We first learned Spring AI • We then learned LangChain4J and realised how close they are • Most of our experiments were done with OpenAI or local models (with Ollama)
19. How we will conduct this session • We will show examples with Spring AI first • We will then show LangChain4J and how it differs • In the end, we will give guidelines on how to choose
20. TL;DR (Too Long; Didn’t Read) Spring AI LangChain4J Sponsor Spring team @Broadcom Red Hat Simplicity and Documentation Great Good Integration with Spring Great Good Support for Entities Great Hard to use Logging support Basic Good Support for models no Google Gemini yet All models supported RAG and Vector db support Good Great (lots of options)
21. Spring AI
22. The Spring AI ecosystem • Current version: Spring AI 1.0.0-M3 ◦ Not in final release version yet! • Based on Spring and Spring Boot
23. Using Spring AI • Create a Spring Boot Project • Add the Spring AI dependencies • Add your API key • Use Spring AI to prompt queries to your model
24. Generating a Spring AI project • Use start.spring.io
25. Spring AI dependencies <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId> spring-ai-openai-spring-boot-starter </artifactId> </dependency> </dependencies> <dependencyManagement> <dependencies> <dependency> <groupId>org.springframework.ai</groupId> <artifactId> spring-ai-bom </artifactId> <version>1.0.0-M3</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement> pom.xml Spring Boot starter (brings in all needed dependencies) defines versions for all Spring AI dependencies (Bill Of Materials)
26. • Best practice: store the API key is an env variable application.properties spring.application.name=spring-ai-samples spring.ai.openai.api-key=${OPENAI_API_KEY} spring.ai.openai.chat.options.model=gpt-4o application.properties IntelliJ Community Edition
27. Making a call @Service public class MusicService { private final ChatClient chatClient; public MusicService(ChatClient.Builder builder) { this.chatClient = builder.build(); } public String findBestSongs() { return this.chatClient.prompt() .user("which were the best songs in 1993?") .call().content(); } } • Use ChatClient’s fluent API Generic (does not depend on the LLM implementation) Demo: MovieService
28. • Give generic guidance to the prompt Adding a system prompt @Service public class MusicService { private final ChatClient chatClient; public MusicService(ChatClient.Builder builder) { this.chatClient = builder.build(); } public String findBestSongs() { return this.chatClient.prompt() .system(" You are a helpful assistant writing in English from the 1990s ") .user("which were the best songs in 1993?") .call().content(); } }
29. Calling from a JUnit test • Run configuration should include the environment variable @SpringBootTest class MusicServiceTest { @Autowired private MusicService musicService; private static final Logger logger = LoggerFactory.getLogger(MusicService.class); @Test void shouldFindBestSongs() { String response = this.musicService.findBestSongs(); logger.info(response); } }
30. public String recommendMovie(String topic) { return this.chatClient.prompt() .user( userSpec -> userSpec.text("Can you recommend a movie about about {topic}") .param("topic", topic)) .call() .content(); } Using a prompt with Parameters var response = this.movieService.recommendMovie("computers"); this.logger.info(response); Certainly! One highly regarded film that delves into the world of computers is "The Imitation Game" (2014). This biographical drama stars Benedict Cumberbatch as Alan Turing, a pioneering computer scientist and mathematician. Demo: MovieService
31. Mapping a prompt to an entity @Service public class ChatService { private final ChatClient chatClient; public ChatService(ChatClient.Builder builder) { this.chatClient = builder.build(); } public ActorFilms generateResponse() { return this.chatClient.prompt() .user("Generate the 10 most popular movies starring Bruce Willis") .call() .entity(ActorFilms.class); } } public record ActorFilms(String actor, List<String> movies) {} works with Java records Demo: MovieService
32. JSON schema under the hood public ActorFilms generateResponse() { return this.chatClient.prompt() .user("Generate the 10 most popular movies starring Bruce Willis") .call() .entity(ActorFilms.class); } public record ActorFilms(String actor, List<String> movies) {} Demo Do not include any explanations, only provide an RFC8259 compliant JSON response … { \"$schema\" : \"https://json-schema.org/draft/2020-12/schema\", \"type\" : \"object\", \"properties\" : { \"actor\" : { \"type\" : \"string\" }, \"movies\" : { \"type\" : \"array\", \"items\" : { \"type\" : \"string\" } } }
33. Working with images @Service class ImageService { @Value("classpath:images/scientist.jpg") private Resource imageResourceScientist; //… public String describeScientist() { return this.chatClient.prompt() .user( userSpec -> userSpec.text("can you describe this person? And what is written on top of his head?") .media(MimeTypeUtils.IMAGE_PNG, this.imageResourceScientist) ).call().content(); } } Import image file as a Resource Optical Character Recognition Demo: ImageService
34. LangChain4J
35. The LangChain4J ecosystem • Current version: 0.35.0 ◦ Not 1.0.0 yet! • Based on Spring Boot or Quarkus
36. Simple call with LangChain4J @Service class MusicService { private final ChatLanguageModel chatLanguageModel; public MusicService(ChatLanguageModel chatLanguageModel) { this.chatLanguageModel = chatLanguageModel; } public String findMostPopularSongs() { return this.chatLanguageModel.generate("5 best songs in year 2023"); } } Demo: BookRecommendationService and ImageService
37. Spring AI vs LangChain4J syntax return this.chatClient .prompt() .user("which were the best songs in 1993?") .call().content(); } • Spring AI Fluent API call • LangChain4J return this.chatLanguageModel .generate("5 best songs in year 2023"); Template style General trend in the Spring team: Client over Template See: RestClient over RestTemplate, JdbcClient over JdbcTemplate etc.
38. Using local models with Ollama
39. What is Ollama • Desktop software (Linux, Windows, OSX) • Allows to run LLM models locally • Exposes an API so models can be queried • Ollama is different from Llama ◦ Ollama is a platform that runs local models ◦ llama is a series of open source models from Meta
40. Ollama is inspired by Docker • Ollama has been founded by 2 former Docker employees • Aims to run models in the way Docker runs containers • Uses Modelfile (similar to Dockerfile) • Also written in Go
41. Ollama installation • Download from https://ollama.com/download • Once installed, Ollama runs on port 11434:
42. Ollama basic commands • List models michael@macbook-air % ollama list NAME ID SIZE MODIFIED tinyllama:latest 2644915ede 0.6 GB 5 days ago llama3.1:latest f66fc8dc39ea 4.7 GB 5 days ago mistral:latest f974a74358d6 4.1 GB 2 weeks ago llama3.1:8b 91ab477bec9d 4.7 GB 3 weeks ago michaelisvy2@macbook-air documentation % • Install a model michael@macbook-air % ollama pull tinyllama pulling manifest pulling 2af3b81862c6... 100% ▕█████████████████████████████▏ 637 MB verifying sha256 digest writing manifest success
43. Ollama - running models • Language model • Vision model michael@macbook-air % ollama run tinyllama >>> tell me a joke Sure, here's a joke for you: Q: What does an AI robot have in common with a smartphone? A: It's always looking at its screen. michael@macbook-air % ollama run llava >>> what is the weather like in Singapore? /Users/michael//spring-ai/lab-files/lab-02/singapore-weather.png Added image '/Users/michael/singapore-weather.png' The image displays a weather forecast for Singapore on a particular day. According to the forecast, it will be 34 degrees Celsius (93.2 degrees Fahrenheit) with humidity at 76%.
44. Ollama in Spring AI • OpenAI config • Ollama config <dependency> <groupId>org.springframework.ai</groupId> <artifactId> spring-ai-openai-spring-boot-starter </artifactId> </dependency> pom.xml <dependency> <groupId>org.springframework.ai</groupId> <artifactId> spring-ai-ollama-spring-boot-starter </artifactId> </dependency> pom.xml spring.ai.openai.api-key=${OPENAI_API_KEY} spring.ai.openai.chat.options.model=gpt-4o application.properties spring.ai.ollama.chat.model=tinyllama application.properties Demo: MovieServiceTest, tinyllama and llama3.2
45. Let’s have a break!
46. Retrieval Augmented Generation
47. Retrieval Augmented Generation • Bring your own data to the prompt • Give a lot of context to the prompt ◦ Text content, excel, pdf, etc. ◦ ChatGPT does it as well! (custom prompts) JV
48. Why bringing your own data • Models have only be trained on what is available on Internet • Models are not real time ◦ They all have a cutoff date ◦ Example: ChatGPT-4o has been trained on data up to September 2023 • LLM’s Hallucinations
49. Prompt data takes precedence over training data
50. How does it work ?
51. Chunking strategy in RAG Fix size Chunk Recursive chucking Document based chunking
52. Step 1 - Loading an st file into a Service class @Service public class OlympicsService { @Value("classpath:/olympics/context.st") private Resource queryTemplate; } Structured text file Use the following pieces of context to answer the question at the end. {context} Question: {question} context.st org.springframework.core.io.Resource M
53. Step 2 - Using the prompt @Value("classpath:/olympics/context.st") private Resource queryTemplate; public String findOlympicSports() throws IOException { return this.chatClient.prompt() .user( userSpec -> userSpec.text(this.queryTemplate) .param( "context" , "Archery, athletics, badminton, basketball , boxing") .param( "question" ,"How many sports are being included in the 2024 Summer Olympics?") ) .call().content(); } Structured text file Use the following pieces of context to answer the question at the end. Archery, athletics, badminton, basketball , boxing Question: How many sports are being included in the 2024 Summer Olympics? Demo
54. Vector databases
55. Quiz 5 • Using Google Gemini’s prompt context window, up to how many Harry Potter books can I fit at most? (7 books, 600 pages in average) ◦ 10% of a book ◦ 2 books ◦ All of them Context window Cost for providing such a large context: between $0.20 and $1 dependending on LLM provider
56. Going beyond the prompt • How to do when: ◦ Your data is too big to fit into the prompt context window? ◦ You’re spending too much because of the prompt context Context window
57. Solution: Vector databases • Split your data into chunks, and encode each chunk into numbers that the ML model can understand {0.345, 0.465, 0.856, …, 0.1543} {0.545, 0.665, 0.056, …, 0.3543} {0.645, 0.765, 0.156, …, 0.4543} “My house is black” “My garden is big” “My dog is playful” AI Model Each Vector is an array of 1,536 numbers
58. Definition of a Vector • A Vector is just a type of data ◦ Typically an array of 1,536 decimal numbers ◦ Value between -1 and 1 CREATE TABLE paragraph ( id SERIAL PRIMARY KEY, paragraph_text TEXT, vector VECTOR(1536) ); example with pgvector {-0.345, 0.465, 0.856, …, 0.1543} Each Vector is an array of 1,536 numbers “Vector” and “Embedding” are similar concepts. For simplicity, we use the word “Vector” whenever possible in this course
59. How are Vectors created? • Vectors typically represent the output generated by Machine Learning models {0.345, 0.465, 0.856, 0.1543} {0.545, 0.665, 0.056, 0.3543} {0.645, 0.765, 0.156, 0.4543} “My house is black” “My garden is big” “My dog is playful” AI Model The above example is simplified and uses random numbers It is based on text AI models. Vectors may also be used with Computer Vision AI models or audio-based AI models
60. How to search vectors: Similarity Search ◦ Selects the closest Vector(s) ◦ OpenAI recommends using cosine similarity {0.345, -0.465, 0.856, 0.1543} {-0.436, 0.578, 0.935, 0.2193} {-0.445, 0.565, 0.956, 0.2543} {0.545, 0.665, 0.056, 0.3543} {0.645, 0.765, 0.156, -0.4543} {0.745, 0.865, 0.256, 0.5543} {0.845, 0.965, -0.356, 0.6543} SELECT id, name, vector <=> '[0.436, 0.578, 0.935, 0.2193]' AS distance FROM items ORDER BY distance LIMIT 10; sample SQL query with pgvector
61. Going further with Vector Databases Company Data DataOpenAI`s Ada 002
62. Vector database providers • Most SQL and NoSQL databases are working on their Vector support ◦ PostgreSQL (pgvector), MongoDB, ElasticSearch, Cassandra, … • Some databases are specialised Vector databases ◦ Chroma, Milvus, …
63. SimpleVectorStore • Spring AI comes with a file-system based VectorStore implementation ◦ To be used for Educational purpose only VectorStore PGVectorStore ChromaVectorStore SimpleVectorStore … M
64. SimpleVectorStore example with OpenAI @Configuration class VectorStoreConfiguration { @Bean SimpleVectorStore simpleVectorStore(EmbeddingModel embeddingModel) throws IOException { var simpleVectorStore = new SimpleVectorStore(embeddingModel); //... return simpleVectorStore; } } { "aec18bbc-21dc-4763-b93e-2f2ee49f9024" : { "embedding" : [ -0.0671, -0.0342, -0.0103, …], "content" : "He raised the guitar, and Henri …", "id" : "aec18bbc-21dc-4763-b93e-2f2ee49f9024", "metadata" : { "source" : "crime-in-paris.txt" } } sample vector.json file OpenAIEmbeddingModel is injected
65. Example: encoding a text into a Vector database • Step 3: Similarity Search public String answerQuestion(String question) { return chatClient.prompt() .user(question) .call() .content(); } 1. Calls OpenAI Model in order to encode question 2. Compares question against all vectors inside database 3. Returns list of closest vectors 4. Sends Vector to ChatGPT together with question Demo
66. Text splitting strategies
67. Source: Guillaume Laforge - https://github.com/datastaxdevs/conference-2024-devoxx
68. Source: Guillaume Laforge - https://github.com/datastaxdevs/conference-2024-devoxx
69. Source: Guillaume Laforge - https://github.com/datastaxdevs/conference-2024-devoxx
70. Vector DB strategies • LangChain4J supports multiple chunking strategies out of the box • Spring AI only supports chunking by sentence ◦ Other strategies can still be achieved with some code customisation
71. Conclusion
72. Spring AI or LangChain4J • Use Spring AI if you’re a Spring Boot fan • Use LangChain4J if you need to put Vectors in production with best accuracy • Keep an eye open as things change super-fast Disclaimer: all of the above is our perception as of October 2024. It will soon be outdated
73. Cost of 3 months learning Spring AI / LangChain4J • OpenAI ◦ $1.36 • Ollama ◦ $0.00 https://platform.openai.com/usage
74. Our favorite Resources online • https://www.youtube.com/@DanVega (Dan Vega) • https://www.youtube.com/@springinaction (Craig Walls) • RAG from dumb implementation to serious results (Guillaume Laforge) • Our demos: ◦ https://github.com/michaelisvy/demo-spring-ai ◦ https://github.com/michaelisvy/demo-langchain4j 79
75. Thank you!