“Hello, AI?!” Real-time interactions with language models Christian Liebel @christianliebel Consultant 

Google Developer Expert Angular & Web W3C WebML CG & WG X: @christianliebel Bluesky: @christianliebel.com Email: [email protected] Hello, it’s me. “Hello, AI!?“ Real-time interactions with language models Christian Liebel 

Overview “Hello, AI!?“ Real-time interactions with language models Generative AI Text OpenAI GPT Mistral … Audio/Music Musico Soundraw … Images DALL·E Firefly … Video Sora Runway … Speech Whisper tortoise-tts … 

Overview “Hello, AI!?“ Real-time interactions with language models Generative AI Text OpenAI GPT Mistral … Audio/Music Musico Soundraw … Images DALL·E Firefly … Video Sora Runway … Speech Whisper tortoise-tts … 

“Hello, AI!?“ Real-time interactions with language models Large Language Models 

“Hello, AI!?“ Real-time interactions with language models Multimodal Models DEMO 

“Hello, AI!?“ Real-time interactions with language models DEMO 

“Hello, AI!?“ Real-time interactions with language models Multimodal Realtime Models DEMO 

“Hello, AI!?“ Real-time interactions with language models DEMO 

– Process speech input and output natively (transcription optional) – Multiple languages and output voices are supported – Tool/function calling are supported – Voice Activity Detection (VAD) activated automatically (model waits for a period of silence before responding) – Model can be interrupted – Use cases: Phone agents, ticket machines, alternative input methods for accessibility and other speech-based user experiences “Hello, AI!?“ Real-time interactions with language models Realtime Models 

Gemini Live API (Preview) Half-cascade (better for tools) – Gemini Live 2.5 Flash – Gemini 2.0 Flash Live 001 Native audio dialog (reasoning) – Gemini 2.5 Flash OpenAI Realtime API (Beta) – GPT-4o Realtime – GPT-4o mini Realtime “Hello, AI!?“ Real-time interactions with language models Realtime Models 

Gemini Live API (Preview) – 40+ lanugages – Supports speech, text and video input – Supports speech and text output – Supports WebSockets – No JS SDK yet, integration is ~1300 LOC OpenAI Realtime API (Beta) – 57+ languages – Supports speech and text input – Supports speech and text output – Supports WebRTC and WebSockets – No JS SDK yet, WebRTC integration is ~50 LOC “Hello, AI!?“ Real-time interactions with language models APIs 

– Bi-directional communication protocol based on TCP – Reduces overhead by eliminating repeated HTTP headers “Hello, AI!?“ Real-time interactions with language models WebSockets https://ai.google.dev/gemini-api/docs/live 

getUserMedia() – JavaScript APIs for accessing media devices – Captures video and/or audio input – W3C Candidate Recommendation – Supported by all major browsers for several years (Chrome 21, Edge 12, Safari 11, Firefox 17) “Hello, AI!?“ Real-time interactions with language models Media Capture & Streams API 

“Hello, AI!?“ Real-time interactions with language models Gemini Live Messages Client Server setup model, system message, modalities, voice configuration, tools realtimeInput setupComplete 

“Hello, AI!?“ Real-time interactions with language models Gemini Live Events Client Server audio interrupted toolcall 

“Hello, AI!?“ Real-time interactions with language models Function calling 

– Realtime models unlock new, exciting opportunities for natural language interfaces beyond chat boxes – Bidirectional, multilingual, minimum latency – All available models (OpenAI Realtime/Gemini Live) in beta or preview – Quality is good, but not perfect – Pricing seems quite high – Fun! – No science fiction, try it today! “Hello, AI!?“ Real-time interactions with language models Summary 

Thank you for your kind attention! Christian Liebel @christianliebel [email protected]