Oct 2025 - AI Agents: From Hype to Reality

The AI agent revolution promises to transform products, but reality is more nuanced. This session brings together three perspectives: where the hype originates, what users actually need vs what’s promised, and the technical limitations we face today. Through startup investment, product management, and AI architecture lenses, we’ll bridge the gap between market excitement and practical implementation. Attendees will learn how to build AI agents that solve real problems rather than chase technological possibilities.

Video here - https://www.youtube.com/watch?v=W1eR4hsnWh0

Transcript

1. AI Agents: From Hype to Reality
2. Speakers Angeley Mullins Founder & CEO Aetheris Ventures Olena Gorbatiuk, Product Manager Independent Atita Arora Director of AI Voyager Search
3. Women of Search (est 2021) • 300+ members across 10+ countries 💪 • A vibrant community dedicated to empowering & celebrating women and allies in search & related tech fields • Provide a platform for networking, mentorship, and knowledge-sharing • Join us: women-of-search.org or #women-of-search on relevancy.slack.com
4. Women of Search @ Haystack EU & US
5. Aug 2025 In 2025 so far, around $2.8 billion in VC funding has flowed into AI agent startups, with projections reaching $6.7 billion by year-end, fueling copilots for law, finance, healthcare, and beyond. https://linas.substack.com/p/agents20 The $6.7 Billion Question !
6. Major agentic launches this year!
7. Where is the Hype Coming From?
8. Where is the Hype Coming From?
9. Where is the Hype Coming From?
10. Investors & CEOs are Pushing for Consolidation
11. Investors & CEOs are Pushing for Consolidation
12. Human authenticity will become the new luxury
14. “The most critical technology changes will happen in the next 5 years. Understanding how to navigate the AI world will be the differentiator for success” -Angeley Mullins
15. And AI agents need search!
16. New technology doesn't work if …
17. Agents need a new kind of search For decades, we have been tuning and optimising our search engines for humans, and today’s trend shows that agents are going to do more of the searching!
18. Current search is not optimised for agents Humans AI Agents • Few searches • Short queries (2-4 words in google search) • Top results • 2–3 engines • What they search • Endless searches • Long detailed queries (simple prompts: 10-20 words, complex 40-70+) • All relevant data • 20–30 engines • Why they search
19. From intent optimisation to task completion Traditional Search (last 30 years): Query → Search → Human evaluation • Designed for humans as the primary user • Goal: Find information • Eval: Did we return relevant results? • Optimised for: speed, relevance (top-3 results), UX AI Agent Era (next 5-10 years?): Prompt → Reasoning → Search → Task • Agents as the new search user • Goal: Task completion • Eval: Was task completed successfully? • Optimised for: recall, context, data openness
20. How we measure search will need to change The search success was defined for human users and now we need to change the definition of success for AI agents. The success metrics shift from “are humans satisfied with search results?” to “are agents reliably completing tasks using our search?”.
21. This is already happening.... ….don't get left behind!
22. Build the road to the future
23. "EierlegendeWollMilchSau" (egg-laying wool milk pig)
24. Agents are Just like “Her” 2013 ChatGPT-4o launch
25. AI Agents IRL “The Awkward Teenager” They're capable of impressive things but lack the maturity and judgement for full independence. They need structure, supervision, and clear boundaries – but within those constraints, they can already deliver remarkable value.
26. What’s actually working ?
27. AI Agents promise - AI as a Tool -> AI as a Teammate • Superhuman speed • Never forgets instructions (within their context window ;) ) • Work 24 x 7 • Interact with any API or tool you give access to • Write code , analyze data , generate content (etc) BUT …..
28. But .. • (May) confidently give wrong information! • Can’t truly understand context the way humans do • (May) do unexpected things if instructions are ambiguous • Needs constant supervision for critical tasks • No real judgement about what’s appropriate or safe without explicit guardrails
29. Current Agents Landscape
30. Waking up to reality ! Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil - https://www.youtube.com/watch?v=d5EltXhbcfA The standardized, cost-aware, and third-party leaderboard for evaluating agents. https://openai.com/index/why-language-models -hallucinate/ Holistic Agent Leaderboard
31. The Uncomfortable Truth and the Exciting Reality
32. Final Thought : A question of When , Not if We're not at the beginning of the agent revolution, and we're not at the end. We're at the end of the beginning !! Don't wait for agents to be perfect. They never will be. Instead, find one process, one workflow, one repetitive task in your organization where an 80% solution would still deliver value. Start there. Learn. Iterate. Build your agent muscle! Because in the age of AI agents, the biggest risk isn't moving too fast – it's not moving at all !
33. Let’s stay in touch ! Email us - info@women-of-search.org Monthly Meetings - First Friday, 9AM PST Discussion - #women-of-search Slack channel Events - Join our Google Group to get added to the calendar of events