Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Multiplayer AI – A Practical Guide to Agentic W...

Multiplayer AI – A Practical Guide to Agentic Workflows

In the rapidly evolving landscape of AI-powered web apps, agentic assistants are emerging as a game-changer. This talk will demystify agentic workflows and provide a practical guide to implementing multi-step, multi-agent workflows. Drawing from real-world experiences, this talk will explore how a diverse set of AI agents can collaborate to enhance user experiences, streamline processes, and solve complex problems more efficiently. Attendees will gain insights into the technical foundations, best practices, and key considerations for working with multi-agent workflows, equipping them with the knowledge to leverage this cutting-edge approach in their projects. Whether you’re a seasoned developer or new to AI integrations, this session will provide you with valuable insights for your future development projects.

Cody Bromley

October 23, 2024
Tweet

Other Decks in Technology

Transcript

  1. “Agents are systems that use an LLM as a reasoning

    engine to determine which actions to take and what the inputs to those actions should be. The results of those actions can then be fed back into the agent and it determine whether more actions are needed, or whether it is okay to finish.” LangChain Conceptual Guide
  2. Models are like calculator apps Agents are like entire operating

    systems They both use the processor (LLM) but have different goals and outputs
  3. Rent Space in a Data Center Buy or Rent Servers

    Set Up the Server Software Configure Domains & Networking Deploy your App to the Server Monitor & Maintain Add More Servers Later if Needed 1-Click Deploy to Cloud vs. ye olde way to deploy servers & apps this is the way Consider how much easier the cloud made it to deploy apps & servers Agentic AI could do the same for workflows
  4. Agentic AI isn’t more AI for the sake of AI

    The technique already outperforms popular models in automatic & human evaluations And when the next big general models drop agentic techniques will make them even stronger
  5. To build a viable solution, you first need to run

    a discovery process to find the bottlenecks
  6. Agentic AI is a big opportunity for organizations with complex,

    multi-step problems with rules and logic… … because it can free up teams to do the valuable work that actually propels the organization forward
  7. Choose an AI Game Mode > Single Player > Multiplayer

    > DLC Here’s what I’ll cover today
  8. illustration / dall•e That’s ChatGPT, Claude and Gemini. They’re pretty

    good, but not perfect. Imagine if you had a friend who read every book
  9. Translate 'Hello' into Spanish Zero-Shot Prompt Hola 1: "Translate 'Hello'

    into Spanish." → "Hola." 2: "Translate 'Thank you' into Spanish." → "Gracias." Translate 'Goodbye' into Spanish Few-Shot Prompt Adiós I had five tennis balls. I bought two more. Now how many do I have? Chain of Thought You now have seven. The cat had 4 kittens, 2 were adopted. How many are left? There are two left. Prompt Engineering can only help so much (carefully crafting your inputs)
  10. Three Key Failure Modes Imprecision In con sis tent Inattention

    These are a just a few of the challenges with the way General Knowledge LLMs work. It’s helpful to understand these so we can effectively solve for them with Agents cy
  11. 1. Austin, Texas (~6.5 hours) Known for its vibrant live

    music scene, quirky culture, and outdoor activities, Austin offers great food, parks, and fun experiences like Barton Springs and Zilker Park. 2. Santa Fe, New Mexico (~9 hours) Famous for its art, history, and distinctive adobe architecture, Santa Fe is perfect for exploring art galleries, museums, and enjoying the surrounding nature. 3. Memphis, Tennessee (~5.5 hours) Explore the home of blues and rock 'n' roll, including Graceland, Sun Studio, and Beale Street. Memphis also has a rich history, with attractions like the National Civil Rights Museum. 4. Denver, Colorado (~9.5 hours) Denver offers a mix of outdoor adventure and urban excitement, from hiking in the nearby mountains to exploring craft breweries and art districts in the city. 5. Hot Springs, Arkansas (~4 hours) A relaxing getaway with historic bathhouses, beautiful nature trails, and Hot Springs National Park, this is a great destination for a spa and nature retreat. 6. Branson, Missouri (~3.5 hours) Branson is known for its family-friendly entertainment, live shows, Silver Dollar City theme park, and beautiful lakeside views at Table Rock Lake. 7. Dallas, Texas (~4.5 hours) A bustling city with diverse cultural attractions like the Dallas Museum of Art, the Sixth Floor Museum at Dealey Plaza, and the Dallas Arboretum and Botanical Garden. 8. New Orleans, Louisiana (~8.5 hours) A city full of culture, music, and food, New Orleans offers French Quarter strolls, Creole cuisine, and famous jazz clubs. 9. Eureka Springs, Arkansas (~3 hours) A quaint town nestled in the Ozarks, known for its preserved Victorian architecture, art galleries, and natural springs. 10. St. Louis, Missouri (~6 hours) St. Louis is rich in history and attractions, including the Gateway Arch, Forest Park, and the St. Louis Zoo. It’s a good mix of city experiences and nearby outdoor activities. These spots offer a variety of experiences, from city life to nature retreats, and all are easily accessible by car from Tulsa! Looks reasonable, but hold on a sec…
  12. Destination GPT-4o’s Estimate Google Maps w/ Toll Roads Within 30

    minutes? Austin, TX 6.5 6.75 Yes Santa Fe, NM 9 9 Yes Memphis, TN 5.5 5.75 Yes Denver, CO 9.5 9.5 Yes Hot Springs, AR 4 4.5 Yes Branson, MO 3.5 3.25 Yes Dallas, TX 4.5 4.25 Yes New Orleans, LA 8.5 10.25 No Eureka Springs, AR 3 3 Yes St. Louis, MO 6 5.5 Yes ~90% Accurate* Not a huge surprise without live data, but is 90% good enough for actual apps? *Assuming testing is possible, but what if you can’t test every possible option in advance…
  13. As an AI language model, I don't have physical form

    or the ability to actually roll like a die. However, I can simulate a dice roll for you using random number generation. Let's pretend I'm a standard six-sided die: rolling The result is: 4 This simulated roll produced a 4. Is there anything else you'd like me to simulate or explain about dice? As an AI language model, I don't have physical form or the ability to actually roll myself. However, I can simulate a dice roll for you. Let's assume I'm a standard six-sided die (d6). Imaginary rolling sounds The result of my simulated roll is: 4 Is there anything else you'd like me to simulate or explain about dice rolls? As an AI language model, I don't have physical form or the ability to actually roll myself. However, I can simulate a dice roll for you. Let's assume I'm a standard six-sided die: shakes and rolls The result of the simulated roll is: 4 Is there anything else you'd like me to simulate or explain about dice? • • • • • • • • • • • • Prompt: Pretend you’re a dice and roll yourself I asked Claude 3.5 Sonnet to answer this three separate times
  14. As an AI language model, I don't have physical form

    or the ability to actually roll like a die. However, I can simulate a dice roll for you using random number generation. Let's pretend I'm a standard six-sided die: rolling The result is: 4 This simulated roll produced a 4. Is there anything else you'd like me to simulate or explain about dice? As an AI language model, I don't have physical form or the ability to actually roll myself. However, I can simulate a dice roll for you. Let's assume I'm a standard six-sided die (d6). Imaginary rolling sounds The result of my simulated roll is: 4 Is there anything else you'd like me to simulate or explain about dice rolls? As an AI language model, I don't have physical form or the ability to actually roll myself. However, I can simulate a dice roll for you. Let's assume I'm a standard six-sided die: shakes and rolls The result of the simulated roll is: 4 Is there anything else you'd like me to simulate or explain about dice? 26% Unique 12% Unique 6% Unique Verdict: The same core answer with slightly different text
  15. • • • • • • • • Fore! 4

    • • • • * * * * 4 • • • • 4 …. 4 🎲 🎲 🎲 🎲 4 • • • • • • • • f o u r 4 4 🪩 🪩 🪩 🪩 • • • • • • • • • • • • • • • • • • • • Running this 25 more times produced 25 more 4’s with unique text LLMs are pretty consistent in some ways and wildly inconsistent in others But this time, I used GPT-4o mini. Different model, same number!
  16. What if we tried adjusting some settings in the LLM’s

    API or model playground? Temperature Affects the variety of responses - Higher: random and creative - Lower: focused and predictable Top P Limits output to most likely options - Higher: more possibilities - Lower: fewer possibilities
  17. Let’s try GPT 4o-mini again with Temperature & Top P

    set to zero • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • "Rolls the dice... You rolled a 4!" "Rolling the dice... You rolled a 4!" "Alright! Rolling the dice... You rolled a 4!" "Alright! Rolling the dice... You rolled a 4! What would you like to do next?" Still not getting just one answer (and probably never will)
  18. Okay, I reset back to default settings, and now let’s

    try a more complex example knowing we’ll want to use the model’s response in some code Prompt: How far away is the moon? I need to predict the ways the LLM might respond to parse the output in my code
  19. some models can “enforce” JSON syntax / schema* * This

    effectively asks the LLM to act agentically and perform an additional task or using a tool Luckily,
  20. Run Approx (mi) Approx (km) Apogee (mi) Apogee (km) Perigee

    (mi) Perigee (km) 1 238855 384400 252088 405500 225623 363300 2 238855 384400 252088 405500 225623 363300 3 238855 384400 252088 405500 225623 363300 4 238855 384400 252088 405500 226000 363300 5 238855 384400 252088 405500 225623 363300 6 238855 384400 252088 405500 225623 363300 7 238855 384400 252088 405500 226000 363300 8 238855 384400 252088 405696 225623 363104 9 238855 384400 252088 405500 225623 363300 10 238855 384400 252088 405500 225623 363300 11 238855 384400 252088 405500 225623 363300 12 238855 384400 252088 405500 225623 363300 13 238855 384400 252088 405500 225623 363300 14 238855 384400 252088 405500 225623 363300 15 238855 384400 252088 405500 225623 363300 16 238855 384400 252088 405696 225623 363104 17 238855 384400 252088 405500 225623 363300 18 238855 384400 252088 405696 225623 363104 19 238855 384400 252088 405500 225623 363300 20 238855 384400 252088 405500 225623 363300 21 238855 384400 252088 405696 225623 363104 22 238855 384400 252088 405500 226000 363300 23 238855 384400 252088 405500 225623 363300 24 238855 384400 252088 405500 225623 363300 25 238855 384400 252088 405696 225623 363104 HOWEVER… while enforcing syntax / schema helped make the data’s format predictable, that did nothing to keep the data itself consistent Here’s the final output of my application. It called GPT-4o 25 times and because the JSON syntax was consistent, it successfully built this table.
  21. I'm a mid-30’s man with a large build. My preferred

    sizing is 3XT in most brands, but fit can vary. I have a client facing role at an agency with a casual dress code, but I also need versatile options for weekends and social events. My color preferences lean towards darker shades and neutrals, with a particular fondness for various shades of blues, oranges and greens. I'm prefer solids to patterns, but subtle patterns may be okay. I'm in urgent need of new shoes - both for work and casual wear. My shoe size is 13, and comfort is a priority due to being on my feet often. I'm particularly interested in boots for the fall/winter season. I have a passion for vests, both for layering and as standalone pieces. I find they're flattering for my body type and I love the extra pockets. It’s October and I need several outfits suitable for cooler weather. I live in a region that can get cold, snowy winters, so warmth is essential. However, I also tend to run hot, so breathable layers are ideal. It might push my budget too much, but I could use: - 3 business casual outfits for work - 2 smart casual outfits for social events - 1 formal outfit for occasional upscale events - 2 casual, comfortable weekend outfits My budget is flexible, but I prefer investing in quality pieces that will last. I'm willing to spend up to $750 for this seasonal update to my wardrobe. In terms of style, I appreciate classic looks with modern touches. I want to appear polished and professional, but not stuffy. Comfort is key, given my active lifestyle and body type. Accessories-wise, I wear glasses and a watch daily. I'm open to suggestions for scarves, gloves, and hats that would complement my outfits and provide warmth. Lastly, I have a sensitivity to down, so while I love vests, any down stuffed items are best avoided. Please suggest a comprehensive fall/winter wardrobe based on these specifications, including mix-and-match options to maximize versatility. Please use information from the LLBean.com website to inform your choices. Let’s provide some details about preferences More information now will mean better, more accurate recommendations later… right?
  22. Hey look! ChatGPT used a tool to get data from

    a website! This is Agentic design!
  23. No Products No Products No Products Tops & Pants Tops,

    Pants & Beanies Oh, but hold on. Most of these pages don’t have any products. And none of them have boots.
  24. Business Casual Outfits: 1. Men’s Signature Cotton Waffle Sweater ($79)

    - This piece in a dark navy or olive green provides both warmth and breathability, perfect for layering in a casual office setting. 2. Men’s BeanFlex® Stretch Chinos ($79) - Comfortable and durable with a bit of stretch, these chinos will keep you looking sharp while staying comfortable during long workdays. 3. Men’s L.L.Bean Wrinkle-Free Kennebunk Sport Shirt ($59.95) - The traditional plaid or check options fit well with your fondness for subtle patterns. Pair with chinos for an easy, polished look. Repeat similar combinations with different colors for variety, keeping darker neutrals or blues as your core palette. Smart Casual Outfits: 1. Men’s Signature Heritage Moleskin Shirt Jac ($149) - A rugged yet refined option for social events, this piece adds warmth and structure, making it a perfect layering option over a lighter shirt. 2. Men’s Lakewashed Stretch Khakis ($69.95) - These pants, available in classic colors like charcoal or tan, provide a smart, tapered fit that is comfortable for social outings. Pair these outfits with the versatile Bean's Sweater Fleece Vest ($64.99), which adds style and functionality without the bulk. Formal Outfit: 1. Men’s Comfort Stretch Blazer ($249) - A modern blazer in navy or charcoal will suit any upscale event. It can easily pair with the chinos or a more formal pair of Wrinkle-Free Classic Oxford Cloth Shirt ($64.95) for a sleek yet comfortable look. Shoes: For the fall/winter season, I recommend two types of footwear: 1. Men’s Bean Boots, Gore-Tex/Thinsulate ($239) - These iconic boots are perfect for cold, snowy days. They are waterproof and insulated, making them ideal for your region’s winter. 2. Men’s Katahdin Iron Works Engineer Boots ($199) - These sturdy, stylish boots are versatile enough for work and casual wear, providing excellent comfort for long hours on your feet. Accessories: • Wool-Blend Scarf ($39.95) - Available in neutral colors to pair well with your wardrobe. • Men’s Leather Gloves with PrimaLoft ($99) - Warm and breathable gloves that provide insulation without overheating. • Classic Ragg Wool Hat ($39.95) - A warm, cozy option for the colder months. Not on a linked page On a linked page
  25. • Balancing occasional formal attire with everyday casual wear •

    My preferred color palette and preference for simple, subtle patterns • At least 1 vest • My footwear needs and interest in boots • Considerations for fall/winter • My desire for quality, lasting pieces • My goal of not getting hot / being comfy • An attempt to find seasonal accessories • I had a budget • The correct retailer (L.L.Bean) and several real products and prices • My sizing information • My sensitivity to down-linings It Mentioned It Did Not Mention Set the response itself aside for second. Of the 25 different things I asked about… Less objectively, it didn’t feel very personalized
  26. Client-facing role Casual work dress code Color preferences Pattern preferences

    Affinity for vests Work and casual shoe needs Interest in boots Cold & winter weather Need for warmth and breathability Prefer quality, lasting pieces Classic style w/ modern touches Polished but not stuffy Comfort focused Seasonal Accessories Budget LL Bean.com Sensitivity to Down Sizing Information Trying to do everything at once stretched the model thin
  27. To understand what’s happening, let’s look at the influential research

    paper that introduced the “Transformer” architecture generative AI models are based on Transformer architecture… ROLL OUT!
  28. The quick brown fox jumps over the lazy dog. No,

    puedo explicar! Google researchers were working on how to improve the way neural networks perform machine translation El… rapido… marrón?
  29. El rapido marrón zorro salta sobre el perezoso perro. It’s

    hard because a fluent translation requires more context than swapping one word at a time No. Rapido (adjective) modifies zorro (noun). Español places adjective after nouns.
  30. The quick brown fox jumps over the lazy dog El

    zorro marrón rápido salta sobre el perro perezoso 1 2 3 4 5 6 7 8 9 1 2 3 5 6 7 8 9 4 In short, machine translations were wrong because they weren’t looking ahead to get context.
  31. To solve this, the researchers developed the “Self-Attention Mechanism” which

    analyzes the whole input by applying “weights” to focus on key parts. Multiple attention “heads” scan the input to rank and connect ideas for better responses
  32. YOU ARE TEARING ME APART ATTENTION MECHANISM! The technique works

    remarkably well, but can struggle for large inputs with a bunch of unrelated details
  33. Have large, but dated knowledge LLMs on their own Agents

    could help because they Are hard to control or predict Can gather new info with tools Can play a specialized role Have limited focus / specialty Can think critically before output To summarize so far..
  34. Agent 1 Task 1 Agent 2 Task 2 Agent 3

    Task 3 Agent 4 Task 4 Work can be an easy, straightforward line
  35. Agent 1 Task 1 Agent 2 Task 2 Agent 3

    Task 4 Agent 4 Task 5 Agent 1 Task 3 Agent 3 Task 6 Or something more complex that passes work back and forth between agents with multiple tasks or stages
  36. Agent 1 Agent 2 Agent 3 Agent 4 Manager Agent

    Agent 1 do Task X then pass off to Agent 3 to do Task Y Agents 2 & 4 work on Task Z then report back to me Agent 3 report back to me when finished Manager agents can also be introduced to handle the planning and handoffs
  37. X X X X O O O O Your playbook

    determines how fast or slow a process is Careful planning can make it possible for some tasks to run in parallel
  38. Document Processing Agents Scanner Agent Use OCR to convert JPEGs

    to Text Sorting Agent Is this a contract, brief or invoice? Extraction Agent Put key information in a database Summary Agent Send out an easy-to-read email Imagine a law firm using AI to handle its daily flood of paperwork
  39. Document Processing Agents Scanner Agent Scan Complete Sorting Agent Is

    this a contract, brief or invoice? Extraction Agent Put key information in a database Summary Agent Send out an easy-to-read email Manager Agent Optionally, a Manager agent could help handle errors and exceptions. Like, what if an agent encounters a new document type? I don’t know how to sort this item Classify this as “Form” and then continue as normal
  40. illustration / dall•e Imagine a team of agents in an

    open-plan office who just swivel chair over into ad-hoc teams
  41. Remember Peter, with incredibly powerful AI agent flexibility comes great

    responsibility Real-time collaboration is great for complex, unclear tasks, but that can make it difficult to define “done”, leading to agents collaborating when they don’t need to or unwanted compromises because they worked by consensus.
  42. Collaborative Writing Agents Manager We need a blog post on

    AI in Healthcare. @Researcher please collect some data. Researcher I compiled data on the rise of AI in healthcare and placed my report in the project folder. @Writer let me know if you need more information from me. Writer I created a first draft but I still need a good quote for it. @Reseacher, can you find one for me? Researcher I found a one from Dr. Ainsley MacLean. I made a new copy of your draft in the project folder. @Manager, approved?
  43. Foster Productive Debate Reach Consensus, Make a Decision or Repeat

    Present Arguments Evaluate Trade-Offs This is a great approach for those tasks that don’t necessary have a single right or wrong answer.
  44. Investment Committee Agents Aggressive We’re missing out on NVIDIA and

    other hot AI stocks right now. Risk Averse I’d prefer we avoided such volatile stocks. What about treasury bonds? They have steady, predictable returns with low risk. Values What if we did a Clean Energy-focused ETF? Less risk than just 1 company, but still lets us bet on the future of that industry. Then it’s up to a human evaluator or another agent to decide what to do with this.
  45. “We’re going to see 10-person companies with billion-dollar valuations pretty

    soon…in my little group chat with my tech CEO friends there’s this betting pool for the first year there is a one-person billion-dollar company, which would’ve been unimaginable without AI. And now [it] will happen.” Sam Altman, CEO, OpenAI
  46. An AI Agent Company Sales Marketing Support Product Finance Human

    CEO The increasingly digital nature of work makes this feel almost possible, if very improbable.
  47. illustration / dall•e Fully agent-powered companies could undercut competitors due

    to their significantly lower overhead Given infinite scalability, Agent-run companies could effectively monopolize entire industries It sounds very scary, but it’s all still theoretical. For now… It’s a race for the bottom our legal & regulatory systems aren’t ready for Organizations that wait to adopt agentic AI solutions risk getting steamrolled
  48. Do Less Code Do More Problem Discovery Process Mapping UX

    / UI Designing Building effective agentic solutions is so much easier with clarity about the process. The code can wait.
  49. Agentic Frameworks When you’re ready to start building there’s more

    choices than ever to handle the heavy lifting
  50. CrewAI CrewAI has some of the clearest, easiest to follow

    docs. They deeply abstract concepts and can be great for sequential tasks.
  51. LangGraph Studio LangGraph comes from the LangChain team, and their

    new Studio tool offers an incredible way to build visually.
  52. Microsoft AutoGen (+ Studio) AutoGen excels at the kind of

    “swivel-chair” real-time collaboration, and their relatively new AutoGen Studio webapp can help you get started without code
  53. Don’t Over or Under Engineer It Remember, frameworks are someone

    else's opinionated abstraction of the technology and can obscure what's happening underneath. This can sometimes make them overkill for production projects.
  54. Ready, Players 1 through 4o ~ New Game + ~

    Let’s go back and fix those earlier examples with an agentic approach
  55. Agentic Road Trip Planner Tool: Google Maps Navigation API Task:

    Confirm Travel Times Role: Ensure Driveability Distance Agent Task: Suggest Destinations Role: Create compelling recommendations like an experienced travel writer Destination Agent Adding a data gathering tool lets us avoid the pitfalls of an LLM’s general knowledge, and separating work into roles gives us a chance to specialize our work
  56. Agentic Road Trip Planner Distance New Orleans is more than

    10 hours. All others are fine. Destination What about Denver, Austin, Nashville, Santa Fe or New Orleans? Destination Good to know. What about Little Rock instead then? Distance I can confirm Little Rock meets the criteria. Destination Perfect. I will get started writing.
  57. Agentic Dice Roller Tool: Custom Number Generator Task: Get Random

    Dice Roll Results Role: Deliver real results Roller Agent Task: Standardize the outputs Role: Rewrite the output into a consistent template. Writer Agent Example: The die landed on a 4. Splitting the work leaves room for critical thinking, but the right tools and roles can create consistency
  58. Agentic Dice Roller *using tool to simulate six-sided die* Human

    Roll 2 dice and give me the total as a Roman numeral. Roller [6, 3] Writer The total of the die is IX. *using tool to simulate six-sided die*
  59. Agentic Personal Stylists Style Preference Seasonal Fashion Outfit Planner Head

    Style Agent Product Data (Web Scraper) Product Info Size and Fit Budget (Calculator) Shared Memory One agent plans and manages the process while a separate agent ensures shared memory only contains real products
  60. What Comes Next? ~ Part 4o-mini ~ If I haven’t

    convinced you to get started yet, just know that the longer you wait, the more complex Agents will get. There’s still time to get in on the ground floor!
  61. The AI Agentic Future illustration / flux 1.1 Everything is

    moving faster. Expect faster & bigger models, Agent marketplaces and ‘Service as Software’ companies
  62. An Oklahoma-based startup just raised $3 million to bring AI

    agent services to the insurance industry
  63. AGENTECH Proof of Concept in Weeks HOW THEY GOT THERE

    1. Identify the human decision making criteria Understand the decision making process that a human goes through in order to better identify how that translates to LLM driven decision making. 2. Build a Zero-Shot Prototype Gain familiarity with how the LLM responds to the problem by simply asking it to solve it entirely in a single prompt. If successful, then look for areas to break the problem down and create more specialized LLM agents. 3. Create the Agents Break the problem down into multiple steps/stages and define the LLM prompts and interactions for each specific stage. Creating structured data I strongly recommend trying this yourself. I can’t guarantee you’ll raise $3m if you do, but it could be the first step.
  64. ThunderPlains 2024 Multiplayer AI: A Practical Guide to Agentic Workflows

    codybrom.com > links to my GitHub / LinkedIn / Mastodon / Blog @codybrom on Threads