A HANDS-ON AI AGENT GUIDE with Gemma, Ollama and more!

🥑 Developer Relations Engineer at deepset 󰎲 🏗 Open source LLM Framework: Haystack 🎓 Sabanci University B.Sc. 🧠 KU Leuven M.Sc. 📍 Istanbul, Turkey ABOUT ME @BILGEYCL /IN/BILGE-YUCEL BILGEYUCEL Bilge Yucel

AGENDA ● AI AGENTS ● BUILDING TOOLS ● 󰟲 BUILDING TIME ● Q&A

AI AGENTS An agent in an LLM-powered system is an autonomous entity that uses a language model as its core to plan, reflect, store memory, and leverage external tools for executing complex tasks and interacting with external data sources efficiently

AI AGENTS Overview of a LLM-powered autonomous agent system. Source: https://lilianweng.github.io/posts/2023-06-23-agent/

● Can you mail me the latest Hacker News articles? ○ Reason + Act = ReAct ○ Tool calls, function calling ■ find latest Hacker News articles ■ generate an email ■ send it AI AGENTS EXAMPLES

● Generate JSON from {this} text ○ Self-reflection: Generates a JSON, verifies it with the second iteration AI AGENTS EXAMPLES

● Who is Taylor Swift? + When was she born? ○ Memory: “When was Taylor Swift born?” AI AGENTS EXAMPLES

BUILDING TOOLS ● Gemma 2 ● Ollama ● Haystack

GEMMA 2 ● An open language model by Google ● 2b, 9b, 27b ● 8192 tokens ● wide range of tasks: question answering, commonsense reasoning, mathematics, science, and coding ● Gemma License ● More info in Gemma 2: Improving Open Language Models at a Practical Size

OLLAMA OLLAMA ● OSS to run models locally on macOS, Linux, and Windows ● Models come with Ollama ● Mostly 4-bit quantization ● GPU -> CPU ● Mostly for PoC

OLLAMA OLLAMA

OLLAMA How to use OLLAMA ● Install Ollama to your computer ● >> ollama pull ● >> ollama serve

OLLAMA HAYSTACK ● Haystack is an open source Python framework for building production-ready LLM applications ● RAG, Semantic search, Agent… ● Prototyping, evaluation, deployment, monitoring… ● Building blocks: Components & Pipelines

OLLAMA Web-Enhanced Self-Reflecting Agent ● Retrieval Augmented Generation (RAG) ● If not enough context, goes to web to enrich the context 🌐 ● If enough context, finishes ✅

OLLAMA Web-Enhanced Self-Reflecting Agent

OLLAMA Web-Enhanced Self-Reflecting Agent

OLLAMA Web-Enhanced Self-Reflecting Agent

OLLAMA PROMPT user {% if web_documents %} You were asked to answer the following query given the documents but the context was not enough. Here is the user question: {{ query }} Context: {% for document in documents %} {{document.content}} {% endfor %} {% for document in web_documents %} URL: {{document.meta.link}} TEXT: {{document.content}} — {% endfor %} Answer the question based on the given context. If you have enough context to answer this question, return your answer with the used links. If you don't have enough context to answer, say 'N0_ANSWER'. {% else %} Answer the following query based on the documents retrieved from Haystack's documentation. Documents: {% for document in documents %} {{document.content}} {% endfor %} Query: {{query}} If you have enough context to answer this question, just return your answer If you don't have enough context to answer, say 'N0_ANSWER'. {% endif %} model

OLLAMA PROMPT user {% if web_documents %} You were asked to answer the following query given the documents but the context was not enough. Here is the user question: {{ query }} Context: {% for document in documents %} {{document.content}} {% endfor %} {% for document in web_documents %} URL: {{document.meta.link}} TEXT: {{document.content}} — {% endfor %} Answer the question based on the given context. If you have enough context to answer this question, return your answer with the used links. If you don't have enough context to answer, say 'N0_ANSWER'. {% else %} Answer the following query based on the documents retrieved from Haystack's documentation. Documents: {% for document in documents %} {{document.content}} {% endfor %} Query: {{query}} If you have enough context to answer this question, just return your answer If you don't have enough context to answer, say 'N0_ANSWER'. {% endif %} model

OLLAMA PIPELINE

OLLAMA LET’S RUN

OLLAMA LET’S RUN

OLLAMA Q&A @BILGEYCL /IN/BILGE-YUCEL BILGEYUCEL Web-Enhanced Self-Reflecting Agent