Harnessing LLMs for in-depth document analysis #PyConUG

PDF documents are a common source of data, but they can be hard to analyze and query. Are you ready to unleash the power of Large language Models & generative AI for analysis of PDF documents? In this session we show you how Microsoft Fabric and SynapseML can help you do just that! You will learn how to use these tools to ask natural language questions and get fluent answers from your PDF data. Whether you want to scale up your Q&A process, or seamlessly connect your data sources, Microsoft Fabric and SynapseML have got you covered. Don't miss this opportunity to discover how LLMs can transform your Q&A and data analytics experience with PDF documents!

Transcript

1. Harness the Power of LLMs for In-depth Document analysis J O S H N D E M E N G E & B E T H A N Y J E P C H U M B A
2. Agenda • Introduction • Background on LLMs • Understanding word embedding • Introduction to LangChain • Using SynapseML & LangChain to build the document analysis pipeline • Q&A • Conclusion 9 / 2 1 / 2 0 2 3 L L M S , M I C R O S O F T F A B R I C 2
3. Bethany Jepchumba • Cloud Advocate @Microsoft • Artificial Intelligence • @bethanyjep 9 / 2 1 / 2 0 2 3 3
4. Josh Ndemenge • Cloud Advocate @Microsoft • Data • @jcardif254 9 / 2 1 / 2 0 2 3 4
5. What is Artificial Intelligence? A I I S T H E C R E A T I O N O F S O F T W A R E T H A T I M I T A T E S H U M A N B E H A V I O R S A N D C A P A B I L I T I E S . 9 / 2 1 / 2 0 2 3 5
6. Introduction 9 / 2 1 / 2 0 2 3 6
7. The Rising Speed of Technological Adoption source: visualcapitalist.com
8. A bit of history 1966 ELIZA Typewritten chatbot simulating a psychiatrist 1980s – 1990s Return of AI research and birth of new statistical models 2010 - 2015 Modern assistants like Siri, Cortana, Google Assistant Today Generative AI based on Transformer architecture
9. What is AI? • Machine learning - This is often the foundation for an AI system, and is the way we "teach" a computer model to make predictions and draw conclusions from data. • Anomaly detection - The capability to automatically detect errors or unusual activity in a system. • Computer vision - The capability of software to interpret the world visually through cameras, video, and images. • Natural language processing - The capability for a computer to interpret written or spoken language, and respond in kind. • Knowledge mining - The capability to extract information from large volumes of often unstructured data to create a searchable knowledge store.
10. Background to Large Language Models (LLMs) 9 / 2 1 / 2 0 2 3 10
11. How a NN “understands” a sentence He had tea and she drank iced coffee. 27 24 30 31 23 28 29 25 1st: Split sentence into tokens and turns tokens into numbers. 26
12. How a NN “understands” a sentence He had tea and she drank iced coffee. 27 24 30 31 23 28 29 25 2nd: Standardize the length of your embedding space. 27 24 30 31 23 28 29 25 26 0 26 10-dimensional vector
13. How a NN “understands” a sentence He had tea and she drank iced coffee. 27 24 30 31 23 28 29 25 3rd: Train your model to learn the semantic weights of those relationships. . 27 24 30 31 23 28 29 25 26 0 26 10-dimensional vector Weighted parameters x W he she iced coffee tea had drank . =
14. Context matters She had coffee She had strength Semantic space Encoder: encode a sequence of words into a complex idea Compute with that idea in SEMANTIC SPACE Decoder: decode back into your desired input Lei ha preso il caffé Lei ha forza
15. What’s new in Transformers model? Mary already had had coffee that morning, so she didn’t accept a cup of iced coffee his colleague offered her. Q. Why didn’t Mary accept the coffee? Recurrent NNs are based on windows of words and the encoder’s parameters can be as smart as the sequence it sees. The bigger the windows size the more the cost of training, the higher the complexity and probability of errors.
16. What’s new in Transformers model? Mary already had had coffee that morning, so she didn’t accept a cup of iced coffee his colleague offered her. ATTENTION lets deep networks learn meaning without proximity The transformer model can look at different combinations at the same time, leading to powerful encoders, smarter and time/energy efficient.
17. LLM Models • GPT • LLAMA • Hugging Face models 9 / 2 1 / 2 0 2 3 17
18. GPT model generations aka.ms/AOAI-models GPT-1 GPT-2 GPT-3 GPT-4 Billions of parameters 0.1B 1.5B 175B 100T
19. | Key Terms Prompt—Text input that provides some context to the engine on what is expecting. Completion—Output that the model generates based on the prompt. some context Token — partial or full words processed and produced by the GPT models
20. Prompt Write a tagline for an ice cream shop. Prompt Table customers, columns = [CustomerId, FirstName, LastName, Company, Address, City, State, Country, PostalCode] Create a SQL query for all customers in Texas named Jane query = Prompt A ball of fire with vibrant colors to show the speed of innovation at our media and entertainment company Response We serve up smiles with every scoop! Response SELECT * FROM customers WHERE State = 'TX' AND FirstName = 'Jane' Response Prompt I’m having trouble getting my Xbox to turn on. Response There are a few things you can try to troubleshoot this issue … … Prompt Thanks! That worked. What games do you recommend for my 14-year-old? Response Here are a few games that you might consider: … GPT-3.5 and GPT-4 Text ChatGPT Conversation Codex Code DALL·E 2 Images
21. Learn the Importance of Prompt Engineering • Prompt Engineering is the art of crafting your input prompt to the model to coax the best response out. With more and more data consumed in training, fine-tuning is required less frequently than in the past. The below video illustrates the importance of context: • Few-shot vs zero-shot • Remembering history • Incorporating fresh content (since the model was trained)
22. Prompt Engineering Demo 9 / 2 1 / 2 0 2 3 22
24. Understanding GPT models for natural language generation • Generative pre-trained transformer (GPT) models are excellent at both understanding and creating natural language. Task Prompt Summarizing text "Summarize this text into a short blurb" Classifying text "What genre of book is this?" Generating names or phrases "Write a tagline for my flower company" Translation "Translate 'How are you' to French" Answering questions "What does Azure OpenAI do?" Suggesting content "Give me the five best weddings songs"
25. Do not humanize AI! Always ensure to use AI output responsibly. Image created by DALL E2 Generative AI is not: Intelligent Deterministic Grounded Generative AI can not: Understand language or maths Understand manners or emotions Know facts that are not in its training dataset
26. How to mitigate hallucinations Effective prompt engineering Tweak parameters (temperature/max length) Content filtering Provide validated data sources Ask for citations
27. Understanding Word Embeddings 9 / 2 1 / 2 0 2 3 27
28. Introduction to LangChain 9 / 2 1 / 2 0 2 3 28
29. Understanding LangChain • Open-source framework for developing LLM applications • It allows you to create, train and deploy Natural Language models for tasks such as summarization, question answering and more • Key terms: tools & agents 9 / 2 1 / 2 0 2 3 29
30. Semantic Kernel • This is an open source SDK that integrates LLMs allowing you to define LLM plugins • Sematic Kernel allows you to bring AI into your own existing applications • The main difference between LangChain and Semantic Kernel is the languages supported and LangChain has more popularity and, agents and tools. 9 / 2 1 / 2 0 2 3 30
31. Retrieval Augmented Generation (RAG) • RAG is an approach that enables you to harness the power of LLMs with your own data. • RAG combines LLMs capabilities with search or retrival mechanisms using internal data • RAG facilitates periodic data updates without the need for fine-tuning, thereby streamlining the integration of LLMs into businesses. 9 / 2 1 / 2 0 2 3 31
32. Using SynapseML & LangChain to build the document analysis pipeline. 9 / 2 1 / 2 0 2 3 32
33. Document Analysis Pipeline 33 User Question PDF Documents Azure AI Services Split chunks Azure Cognitive Search Azure Open AI Service Answer
34. Gear Up! •Sign up to Azure: https://aka.ms/GetAzure4Free •Get started with Artificial Intelligence: https://aka.ms/StartYourAIJourney •Do you have an idea? Join Founders Hub and start your AI entrepreneurial journey: https://aka.ms/MfS-FoundersHub Resources Microsoft Fabric / Resources 34