[EuRuKo 2024] Intro to AI Agents

The author of Langchain.rb will walk you through current capabilities of LLMs and what can be built today. We will build a business process automation AI agent in Ruby and discuss the common pitfalls and misconceptions. We’ll discuss what might be emerging as a new LLM-powered software stack.

Generative AI has been taking the world by storm. The Coatue AI (Nov 2023) report is putting AI models at the centerpiece of all modern tech stacks going forward that Application Developers will be using to build on top of. It would not be controversial to say that the Ruby ecosystem lacks in its support and understanding of the AI, ML and DS landscape. If we’d like to stay relevant in the future, we need to start building the foundations now. We’ll look at what Generative AI is, what kind of applications developers in other communities are building and how Ruby can be used to build similar applications today. We’ll cover Retrieval Augmented Generation (RAG), vector embeddings and semantic search, prompt engineering, and what the state of art (SOTA) in evaluating LLM output looks like today. We will also cover AI Agents, semi-autonomous general purpose LLM-backed applications, and what they’re capable of today. We’ll make a case why Ruby is a great language to build these applications because on its strengths and its incredible ecosystem. After the slides, I’ll walk the attendees through building an AI Agent in 15 min with Langchain.rb.

Transcript

1. Intro to AI Agents ƻ EuRuKo 2024 Friday, September 13th, 2024 by Andrei Bondarev
2. Work Source Labs LLC Software-development firm Clients: VC-backed startups and Enterprises We <3 Rails Patterns AI Applied AI research organization Open-source work We <3 Ruby
3. GenAI Impact Before: 1 month Label data 3 months Train custom model 3 months Deploy (optimize) After: Few days Prompt engineering Few weeks Basic RAG (if needed) Few days Deploy
4. Common ML tasks Classification Named Entity Recognition Summarization Translation
5. Capabilities, an API call away Adoption Cost
6. (Re-)Rise of AI Agents 1950s 1970s — 1980s 1990s — 2000s Intelligent Machines Expert Systems 2010s Software Agents 2020s Chatbots LLMs as Agents
7. The Vision
8. AI Agent ƻ Definition: An autonomous software system capable of perceiving its environment, making decisions, and taking actions to achieve specific goals. ♻ Environment awareness 2 Decision-making Ƣ Action-taking
9. Agent vs Assistant Assistant Conversational system that continuously takes directions from a human Agent Autonomous system that independently executes a task (like a background job)
10. Agent vs Assistant Conversational Assistant Free-for-all input from user Autonomous Agent Guided-input from user
11. Use-cases Automating business processes Mundane low-IQ tasks Personal assistant (co-pilot) Tasks in a consulting business: Creating invoices from timesheets Categorizing business expenses Writing project proposals (incl. service offering, meeting notes) Writing job descriptions. Writing JIRA tickets.
12. Components of an AI agent 1. Planning & Reasoning 2. Role Playing 3. Environment Perception 4. Tool Calling 5. Memory 6. Evaluations (Evals)
13. Planning Plan formulation Decomposing a top-level task into numerous sub- tasks. Plan reflection Leveraging feedback mechanism to reflect upon a plan and evaluate its merits.
14. Reasoning Cornerstone for problem-solving, decision-making and critical analysis. Deductive, inductive, abductive are the primary forms of reasoning. Reasoning capacity is crucial for solving complex tasks.
15. Chain-of-Thought (CoT) Forcing the AI to explain it's reasoning. Without Chain-of-Thought prompting With Chain-of-Thought prompting
16. Role playing Forcing the AI to adopt certain personality, character, and behavior, via prompt engineering Ɗ Strict Manager % Relaxed Manager Dungeon Master Ƽ Helpful AI assistant
17. Environment perception "Today is September 13, 2024 "
18. Tool/Function Calling Structured Outputs Response adhere to a predefined JSON schema External Tools Intent detection
19. Tool Calling Use tools to do the following: Get data from external sources (APIs) Get real-time data Take actions Execute deterministic tasks1 Without Tools Using the Tool (Code Interpreter)
20. Tool Calling
21. Memory ("remembering") Saving the environment, progress, tool calling to memory
22. Memory => Retrieval Augmented Generation (RAG)
23. Common problems ♻ Hallucinations 2 Poor reasoning Ƣ Unreliable tool calling
24. Evaluations Benchmarks Comparing to a large dataset of question-answer pairs. "LLM as a Judge" Asking LLM whether the answer fits a list of criteria.
25. Benchmarks huggingface gretelai/gsm8k-synthetic-diverse-405b · Datasets at Hugging Face We ʼ re on a journey to advance and democratize artificial intelligence through open source and open science .
26. Reasoning (Next Frontier) Training models specifically on reasoning data but… No good training data
27. langchainrb ⭐ Ruby framework for building LLM-powered applications
28. Demo
29. Nerds & Threads Selling comfortable nerdy t-shirts for software engineers that work from home Services ú Customer Management ✉ Email Service  Payment Gateway Service Order Management Inventory Management Shipping Service
30. Diagram
31. Code
32. Demo AI Assistant Chat
33. Why would you use this? Changing requirements on the fly Text-to-SQL using the Database tool
34. Recap
35. Thank you! ɉ @rushing_andrei @andreibondarev in/andreibondarev andrei@sourcelabs.io Discord
36. References "Tool Use with Open-Source LLMs" by Rick Lamers (Groq) 1. https://arxiv.org/abs/2309.07864 2. https://www.promptingguide.ai/techniques/cot 3. https://docs.google.com/presentation/d/1EH11pXLanLMoLGEXd_YduYxEEk3X2Y63KOOFsysq1LY 4. https://www.ibm.com/think/topics/ai-agents 5. Berkeley Function-Calling Leaderboard 6. https://garymarcus.substack.com/p/math-is-hard-if-you-are-an-llm-and 7. https://obie.medium.com/my-kids-and-i-just-played-d-d-with-chatgpt4-as-the-dm-43258e72b2c6 8.
37. langchain.rb contributors