“Hello, AI?!” - Real-time interactions with language models

Large Language Models (LLMs) have changed how we design software: Instead of clicks and GUIs, natural language now dominates— not just via keyboard and text, but also by voice. In this session, you will learn how to integrate voice-enabled AI models directly into your application and control them in real time with your voice. Thanks to Gemini Live and tool calling, new possibilities for natural language interfaces are emerging, with minimal latency, bidirectional, and multilingual.

In practical demos, Christian Liebel from Thinktecture will show you how to address selected LLMs by voice, link your functionalities to them, and develop smart, conversational interfaces. This is not science fiction.

Caution: Interactive voice AIs can be addictive.

Transcript

1. “Hello, AI?!” Real-time interactions with language models Christian Liebel @christianliebel Consultant
2. Google Developer Expert Angular & Web W3C WebML CG & WG X: @christianliebel Bluesky: @christianliebel.com Email: christian.liebel@thinktecture.com Hello, it’s me. “Hello, AI!?“ Real-time interactions with language models Christian Liebel
3. Overview “Hello, AI!?“ Real-time interactions with language models Generative AI Text OpenAI GPT Mistral … Audio/Music Musico Soundraw … Images DALL·E Firefly … Video Sora Runway … Speech Whisper tortoise-tts …
4. Overview “Hello, AI!?“ Real-time interactions with language models Generative AI Text OpenAI GPT Mistral … Audio/Music Musico Soundraw … Images DALL·E Firefly … Video Sora Runway … Speech Whisper tortoise-tts …
5. “Hello, AI!?“ Real-time interactions with language models Large Language Models
6. “Hello, AI!?“ Real-time interactions with language models Multimodal Models DEMO
7. “Hello, AI!?“ Real-time interactions with language models DEMO
8. “Hello, AI!?“ Real-time interactions with language models Multimodal Realtime Models DEMO
9. “Hello, AI!?“ Real-time interactions with language models DEMO
10. – Process speech input and output natively (transcription optional) – Multiple languages and output voices are supported – Tool/function calling are supported – Voice Activity Detection (VAD) activated automatically (model waits for a period of silence before responding) – Model can be interrupted – Use cases: Phone agents, ticket machines, alternative input methods for accessibility and other speech-based user experiences “Hello, AI!?“ Real-time interactions with language models Realtime Models
11. Gemini Live API (Preview) Half-cascade (better for tools) – Gemini Live 2.5 Flash – Gemini 2.0 Flash Live 001 Native audio dialog (reasoning) – Gemini 2.5 Flash OpenAI Realtime API (Beta) – GPT-4o Realtime – GPT-4o mini Realtime “Hello, AI!?“ Real-time interactions with language models Realtime Models
12. Gemini Live API (Preview) – 40+ lanugages – Supports speech, text and video input – Supports speech and text output – Supports WebSockets – No JS SDK yet, integration is ~1300 LOC OpenAI Realtime API (Beta) – 57+ languages – Supports speech and text input – Supports speech and text output – Supports WebRTC and WebSockets – No JS SDK yet, WebRTC integration is ~50 LOC “Hello, AI!?“ Real-time interactions with language models APIs
13. – Bi-directional communication protocol based on TCP – Reduces overhead by eliminating repeated HTTP headers “Hello, AI!?“ Real-time interactions with language models WebSockets https://ai.google.dev/gemini-api/docs/live
14. getUserMedia() – JavaScript APIs for accessing media devices – Captures video and/or audio input – W3C Candidate Recommendation – Supported by all major browsers for several years (Chrome 21, Edge 12, Safari 11, Firefox 17) “Hello, AI!?“ Real-time interactions with language models Media Capture & Streams API
15. “Hello, AI!?“ Real-time interactions with language models Gemini Live Messages Client Server setup model, system message, modalities, voice configuration, tools realtimeInput setupComplete
16. “Hello, AI!?“ Real-time interactions with language models Gemini Live Events Client Server audio interrupted toolcall
17. “Hello, AI!?“ Real-time interactions with language models Function calling
18. – Realtime models unlock new, exciting opportunities for natural language interfaces beyond chat boxes – Bidirectional, multilingual, minimum latency – All available models (OpenAI Realtime/Gemini Live) in beta or preview – Quality is good, but not perfect – Pricing seems quite high – Fun! – No science fiction, try it today! “Hello, AI!?“ Real-time interactions with language models Summary
19. Thank you for your kind attention! Christian Liebel @christianliebel christian.liebel@thinktecture.com