Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Linus Beckhaus @linusbeckhaus Gen AI Developer 

- Low-latency - Audio native multimodal interaction - Specialized Models - Full feature support - Websocket or WebRTC Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Realtime STT TTS Text Model Speech-to-Speech Model Traditional Voice Pipeline Native Multimodal (Realtime) 

Usecases - "Actual" interactive chatbots - Hands-free application - Digital Assistant - Robotics - Audio native systems (e.g., call center) Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Realtime 

Conversation Session Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Realtime API DEMO 

Conversation lifecycle OpenAI Realtime API Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Client Server Sending of response (including transcript) is finished ••• User Input response.done Events related to user input (speech, text, data) 

Conversation lifecycle OpenAI Realtime API Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Client Server ••• User Input response.done Events happening in the background conversation.item.added conversation.item.done response.created response.output_item.added conversation.item.added response.content_part.added output_audio_buffer.started response.output_audio.done response.content_part.done conversation.item.done response.output_item.done (additional out of order transcription events) 

Input Events (WebRTC) OpenAI Realtime API Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Server has detected speech in audio buffer Server has detected end of speech input_audio_buffer.speech_started input_audio_buffer.speech_stopped input_audio_buffer.committed Client Server Input audio buffer has been committed 

Input Events (Text/Data) OpenAI Realtime API Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Add data into the Conversation context (messages, function call, function call response, full audio) Trigger response/processing conversation.item.create response.create Client Server 

Voice Agents - Typescript client for OpenAI Realtime API - Abstraction of transport layer and session - Full integration into agentic framework Python version support for Realtime in beta Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme OpenAI Agents SDK https://openai.github.io/openai-agents-js/guides/voice-agents/ 

RealtimeAgent RealtimeSession Key Components Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Voice Agents RealtimeSession RealtimeAgent Interface to session Transport Context History Configuration Agent Tools Instructions Settings Guardrails Handoffs RealtimeAgent RealtimeAgent Interface to agent 

Basic Setup Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Voice Agents 1. Create Realtime Agent 2. Create RealtimeSession and set initial agent 3. Establish connection 4. Start server npm run dev and get talking Use ephemeral key - Time limited - Safe - Configurable 

Agents SDK - Combination of Session and Agent - Initial session configuration - Agents have their own configuration - Update used Agent of RealtimeSession Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Configuration session.updateAgent session.updated 

Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Tool Calling Tool Call 1 2 5 6 3 4 Code, Web, Database Agents, API, … Extend LLM capabilities - Ground with code - Perform deterministic actions - Interface with other Systems 

Custom tools - Executable code with metadata - tool wrapper - Structured output - Zod integration Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Tool Calling DEMO 

Using Agents as Tools - Use an (sub-)agent in a Tool - Blocking as all Tool Calls - Agent can run server side - Not limited to RealtimeAgents Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Tool Calling https://openai.github.io/openai-agents-js/guides/voice-agents/build/#delegation-through-tools 

Configuration - Tools set on RealtimeAgent - Create new Agent - Update Agent Config - tools attribute - Tool use behavior set on RealtimeSession - config.toolChoice attribute - Updates through underlying transport session.transport.updateSessionConfig(…) Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Tool Calling session.updateAgent session.updated 

Human-in-the-Loop - Sensitive actions - Pause execution until approval/rejection - needsApproval option - boolean - async function using context - Rejection handeled automatically - Triggers response by default Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Tool Calling Requires Approval 1 Tool Call 2 3 4 3 4 5 6 7 6 DEMO 

Handling Rejection Preventing response on rejection - Lower level sendFunctionCallOutput - Manually send result Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Tool Calling 

Many more features - Handoffs - Guardrails - MCP - Tracing - Turn detection Realtime API und Voice Agents Die nächste Generation interaktiver KI-Systeme Agents SDK 

Thank you for your attention Linus Beckhaus @linusbeckhaus Gen AI Developer