2024 FlutterMTL January Meetup - Mastering Dart Langchain by Eckarath

Flutter Montréal organized a meetup in Montréal. (https://fluttermtl.dev/)
Eckarath Khounsombath gave a talk about Mastering Dart Langchain

#dart #flutter #gdg #gdgmontreal #fluttermtl #AI #Langchain #LLM

Transcript

1. Mastering Dart Langchain Building a Flutter App for Document Summarization and Discussion
3. Life, an eternal journey of learning Human beings, like artificial intelligences, learn from data Principal Solution Architect / Tech Advisor Freelance - ECKA Studio Eckarath Khounsombath
4. One of the most common use cases for GenAI in businesses is • text summarization • and the ability to query company documents. Typically for a company, GenAI is employed to create an AI capable of answering closed-domain questions, such as those related to a company's knowledge base. GenAI Common Usage Context Photo de Steve Johnson sur Unsplash Lack of Familiarity with Custom Data LLMs use deep learning on extensive datasets but are typically trained on broad public data, limiting their understanding of specific, proprietary information. This results in static LLMs that may give inaccurate or outdated answers when dealing with untrained data. Custom Data Utilization for Effective AI Applications For tailored responses, organizations require LLMs to grasp their domain, providing specific answers from unique datasets. For example, customer support and internal Q&A bots must offer company-specific solutions, posing a challenge to build without retraining models Technical limitations Maximum tokens, hallucinations, Complex Prompting Main Issues
5. -1- Fine-Tuning or Pre-Training with our data This tailored solution offers granular control and high specialization, but implementation can be challenging due to the need for labeled data, computational costs, and intensive resource requirements Quelles solutions ? Context Photo de Steve Johnson sur Unsplash -2- Implement the RAG pattern RAG stands for retrieval-augmented generation. This technique passively analyzes the latest user input and conversation context, using embeddings or search techniques to enhance the AI model's context with specialized knowledge. RAGs enable dynamic context updates, improved accuracy, increased prompt length, and efficient inference computation
6. 4 Being efficient and cost-effective Compared to other approaches to customizing LLMs with domain-specific data, RAG is simple and cost-effective. Organizations can deploy RAG without needing to customize the model. This is especially beneficial when models need to be updated frequently with new data. What is RAG? Key benefits 2 Reducing inaccurate responses, or hallucinations By grounding the LLM model's output on relevant, external knowledge, RAG attempts to mitigate the risk of responding with incorrect or fabricated information (also known as hallucinations). Outputs can include citations of original sources, allowing human verification. 1 Providing up-to-date and accurate responses RAG ensures that the response of an LLM is not based solely on static, stale training data. Rather, the model uses up-to-date external data sources to provide responses. 3 Providing domain-specific, relevant responses Using RAG, the LLM will be able to provide contextually relevant responses tailored to an organization's proprietary or domain-specific data. Retrieval Augmented Generation
7. What is RAG? Embeddings are representations of various forms of data as lists of numbers. This numerical representation captures the meaningful properties and relationships of the objects. Vactor Database allows you to store embeddings vector and performs vector search based on similarity Retriever helps you search and retrieve information from your indexed documents based on an unstructured query Architecture Pattern
8. How do I implement a RAG ? So now what …
9. With LangChain LangChain is a framework for developing applications powered by language models. It enables applications : To be context-aware to connect a language model to sources of context (prompt instructions, few shot examples, content to ground its response in, etc.) To reason by relying on a language model to reason (about how to answer based on provided context, what actions to take, etc.) LangChain provides standard, extendable interfaces and integrations for Models, Retrievals, Tools and utilities for prompting, document treatment
10. LangChain used Python as primary language. But they also support Javascript. LangChain Integration supported ? ~100 LLMs, ~30 Chat Models, ~100 Documents Loaders & Transformers, ~ 30 Embeddings Engine, ~50 Vector Stores, and so on … What about the other language ? As LangChain is a framework that expose standard API and pattern. Community has declined LangChain in another language. Dart LangChain LangChain for Flutter developer What about Dart LangChain ? Development started 6 months ago and it supports all core functionalities Integration supported ? Support OpenAI, Google and majors Vector Store
11. What are we goind to build ? An application that will summarize and let us chat with an upload document on-the-fly Key Features Load - Upload an document from disk or grab content from an URL Summarize - Provide a short summary of the content and extract key points Converse - Chat with the content uploaded Summarize & Converse Let’s build a FlutterApp for that
12. Summarize & Converse How to build the app ? Flutter Build the interface with Flutter framework Dart Langchain Use the framework to load, summarize and converse with content Google Vertex AI Use Google LLM & Embeddings engine
13. # Upload document const url = 'https://www.theverge.com/2024/1/16/24040562/samsung-unpacked-galaxy-ai-s24'; const loader = WebBaseLoader([url]); final documents = await loader.load(); # Split into chunk const textSplitter = CharacterTextSplitter( chunkSize: 800, chunkOverlap: 100, ); final texts = textSplitter.splitDocuments(documents); # Add metadata to chunks final textsWithSources = texts .mapIndexed( (final i, final d) => d.copyWith( metadata: { ...d.metadata, 'source': '$i-pl', }, ), ) .toList(growable: false); Document Loaders & Text Splitters
14. static Future<BaseLLM> getLLM() async { return VertexAI( httpClient: await GoogleApiUtils.getAuthHttpClient(), project: GoogleApiUtils.getProjectId(), defaultOptions: const VertexAIOptions( temperature: 0.2, //model: 'gemini-pro' ), ); } static Future<Embeddings> getEmbeddings() async { return VertexAIEmbeddings( httpClient: await GoogleApiUtils.getAuthHttpClient(), project: GoogleApiUtils.getProjectId(), ); } Future<VectorStore> _createVectorStore(List<Document> textsWithSources) async { // Create a Vector Datastore (in memory) and upload documents data return MemoryVectorStore.fromDocuments(documents: textsWithSources, embeddings: getEmbeddings(),); } LLM & Text Embeddings & Vector Storage
15. static String SHORT_SUMMARY_PROMPT_TEMPLATE = ''' Provide a short summary base only on the given text? "{context}" The length of the summary should be appropriate to capture the main points of the text, without including unnecessary information and contains 100 words maximum: SHORT SUMMARY: '''; List<Document> documents = await _splitDocuments(chunkSize, chunkOverlap); BaseCombineDocumentsChain summarizeChain = SummarizeChain.mapReduce( llm: llm, combinePrompt: PromptTemplate.fromTemplate( SHORT_SUMMARY_PROMPT_TEMPLATE) ); summary = await summarizeChain.run(documents); Summarize Chain
16. static String CHAT_PROMPT_TEMPLATE = """ Answer the question based only on the following context: {context} --- Answer the question based on the above context: {question} """; RetrievalQAChain _getQAChain(VectorStore search) { // Prepare a prompt template final prompt = PromptTemplate.fromTemplate(CHAT_PROMPT_TEMPLATE); final LLMChain llmChain = LLMChain(prompt: prompt, llm: llm); final docPrompt = PromptTemplate.fromTemplate( 'Content: {page_content}\nSource: {source}', ); StuffDocumentsChain chain = StuffDocumentsChain(llmChain: llmChain, documentPrompt: docPrompt); return RetrievalQAChain(retriever: search.asRetriever(), combineDocumentsChain: chain); } RetrievalQAChain retrievalQA = _getQAChain(vectorStore); Map? qaResult = await retrievalQA!(query); String text = qaResult[RetrievalQAChain.defaultOutputKey]; Chat Chain
17. DEMO TIME
18. THANK YOU! AND NOW … LET’S RAG !!!! v