Describes voice interactions in iOS apps with AVAudiorecoder, AVAudioEngine, Speech framework, SiriKit, Voice command for Siri Shortcuts and AVSpeechSynthesizer.
VOICE INTERACTIONS CURRENT USE CASES ▸ Siri interaction ▸ Integration in iOS with system apps ▸ Text dictation ▸ Telephonie Recording of voice messages ▸ VoiceOver
VOICE USER INTERFACES CHALLENGES ▸ Voice-only user interface are hidden ▸ Voice inputs are public ▸ Many utterances to describe the same thing ▸ Everyone says it differently ▸ Queries for all needed values
VOICE USER INTERFACE - HOW IT WORKS VOICE ASSISTANT SPEECH RECOGNITION DOMAIN SELECTION INTENT SELECTION ENTITY EXTRACTION FULFILLMENT Siri Alexa Google Assistant Cortana Bixby Magenta Natural Language Understanding Content Logic App Selection
VOICE INPUT INTERFACE - EXAMPLE VOICE ASSISTANT SPEECH RECOGNITION DOMAIN SELECTION INTENT SELECTION ENTITY EXTRACTION FULFILLMENT How will be the weather tomorrow in Cologne?
TEXT AVAUDIOENGINE ▸ Interaction with raw pcm audio ▸ Access during voice input private let audioEngine = AVAudioEngine() let inputNode = audioEngine.inputNode let recordingFormat = inputNode.outputFormat(forBus: 0) inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { (buffer: AVAudioPCMBuffer, when: AVAudioTime) in }
VOICE INPUT SPEECH FRAMEWORK ▸ Speech to text ▸ Audio file or Microphone ▸ Used by iOS text dictation ▸ Results ▸ can be continuously ▸ Transcriptions + alternative interpretation ▸ Confidence levels
INTENT & ENTITY EXTRACTION DIALOGFLOW ▸ Service to create conversational interfaces ▸ Custom interface with intents & entities ▸ Supports 14 languages ▸ Supports training of conversational model ▸ Interaction history
INTENT HANDLER SIRIKIT ▸ Predefined intents for specific services ▸ Restaurant, Payments, Ride Booking, … ▸ No interaction with user voice or text input ▸ Added as extension ▸ Intents: Handling request from SiriKit ▸ Intents UI: Displaying
TEXT SIRI SHORTCUTS ▸ Shortcuts for already performed actions ▸ App-specific ▸ Intent & entities are donated to Siri Shortcuts ▸ Shortcuts can be executed by voice
DONATION SIRI SHORTCUTS ▸ Donation via NSActivity or Intent ▸ NSActivity: forward to app ▸ Intent: inside Siri let intent = PizzaOrderIntent() intent.pizza = "Tonno" intent.suggestedInvocationPhrase = "Bestelle eine Tonno Pizza" let interaction = INInteraction(intent: intent, response: nil) interaction.donate { _ in }
TEXT AVSPEECHSYNTHESIZER ▸ Generate speech output in app ▸ Siri voices (AVSpeechSynthesisVoice.speechVoices()) ▸ Control and monitoring of ongoing speech ▸ String & NSAttributedString ▸ Custom pronunciation ▸ International Phonetic Alphabet (IPA)
TEXT CONCLUSION ▸ Other approach than Alexa, Google Assistant ▸ Custom conversational interface ▸ Apples voice interface is more predictable for user ▸ SiriKit: Same interface for equal services ▸ Shortcuts: Own Voice command ▸ Speech / AVAudioEngine: In app with additional gui ▸ Will Siri Shortcuts & SiriKit are the next 3D Touch?