Talking to Swift - Speaker Deck

Slide 1

Slide 1 text

Talking to Swift @alexisgallagher

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

"Just use the keyboard." "A keyboard, how quaint..."

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Bots are here ?!

Slide 9

Slide 9 text

Conversational UI 1. What is it? Why the hype? Is it real? 2. How to design it? 3. How to build it?

Slide 10

Slide 10 text

What is it? Trend 1: Messaging

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Message Thread: an evolving interaction model

Slide 16

Slide 16 text

What is it? Trend 2: Chatbots

Slide 17

Slide 17 text

The DREAM is not new

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

The REALITY is not new either

Slide 22

Slide 22 text

$ emacs -q -f doctor

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

But now, we really use them! ! • Siri (iOS, macOS, watchOS, tvOS) • Google (Search, Maps, Google Home) • Amazon Alexa • XFinity X1 • Slackbot

Slide 27

Slide 27 text

But now, we really use them! ! • Voice-based customer service phone tree • Text chat with automated customer service agent • Automated telephone survey • IM messaging spam • Email spam

Slide 28

Slide 28 text

SiriKit?

Slide 29

Slide 29 text

SiriKit.

Slide 30

Slide 30 text

chatbots = restricted domain bots

Slide 31

Slide 31 text

What is it? Trend 3: Voice

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Progress in speech recognition (within the last month!)

Slide 37

Slide 37 text

Progress in speech synthesis (within the last month!)

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

What is it? Conversational UI Reality Check(list): • Messaging? Evolving new interaction model. No magic AI. • Chatbots? Still quite limited. No magic AI. (afaict). • Voice? Yes! Signiﬁcant breakthroughs thanks to neural nets.

Slide 40

Slide 40 text

How to Design it 1. Develop a character

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

What If? by Anne Bernays & Pamela Painter • Exercises for characterization • What does the character want? • Remember most from childhood? • Usually feel two hours after lunch? • What are they in denial of?

Slide 43

Slide 43 text

How to Design it 2. Restrict the domain

Slide 44

Slide 44 text

Voice UI is opposite of Message Thread UI Thread Voice history accessible no history identity visible no identity rich media text only guided input freeform text

Slide 45

Slide 45 text

Example character: Rochefoucauld domain: sorrows of life

Slide 46

Slide 46 text

Rochefoucauld • François VI, Duc de la Rouchefoucauld, Prince de Marcillac • Born 1613, Died 1680 • Wealthy, noble, good-looking • Unhappily married, imprisoned, libeled, exiled, shot in the eye

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

Rochefoucauld • Maxims and other works • 500+ aphorisms on love, ambition, self-delusion, ennui, envy, esteem, evils, the exchange of secrets, etc. • character: cynical, worldly wise, witty • domain: life's sorrows

Slide 49

Slide 49 text

We all have strength enough to bear the misfortunes of others

Slide 50

Slide 50 text

To say that one never ﬂirts is in itself a form of ﬂirtation

Slide 51

Slide 51 text

How to Build it

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

import AVFoundation

Slide 54

Slide 54 text

Speech Synthesis API class MyDelegate : AVSpeechSynthesisDelegate { public func speechSynthesizer(_ synthesizer: AVSpeechSynthesizer, didFinish utterance: AVSpeechUtterance) { print("Just finishe saying hello") } } let synth = AVSpeechSynthesizer() let delegate = MyDelegate() synth.delegate = delegate let utterance = AVSpeechUtterance(string:"Hello, world") synth.speak(utterance)

Slide 55

Slide 55 text

import Speech

Slide 56

Slide 56 text

Once, create and/or conﬁgure • AVAudioEngine, AVAudioSession, SFSpeechRecognizer Per utterance, request stream of live partial results • AVAudioNodeTapBlock, SFSpeechAudioBufferRecognitionRequest, SFSpeechAudioBufferRecognitionTask During utterances, handle callbacks with results • SFSpeechRecognitionResult • but SFSpeechRecognitionResult.isFinal is never

Slide 57

Slide 57 text

Problem: "endpointing"

Slide 58

Slide 58 text

Solution: Asynchronous Operation

Slide 59

Slide 59 text

/** Operation which recognizes speech and finishes after it has reached an "endpoint", an interval over which no speech has been recognized (e.g., a period of silence or of noise). */ public class SpeechRecognitionOperation: Operation, SFSpeechRecognitionTaskDelegate { open override var isAsynchronous: Bool { return true } public var output:String? = nil private var endpointTimer:Timer? private var request:SFSpeechAudioBufferRecognitionRequest? // ... public init(engine e:AVAudioEngine, recognizer r:SFSpeechRecognizer) { /* ... */ } // ... }

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

// Represents a bot, e.g., Eliza, Rochefoucauld, etc.. protocol Interlocutor : class { init() func respondTo(saying:String) -> String } // wraps any Interlocutor in a speech rec/synth UI public class VoiceChatter : NSObject { public init(interlocutor:Interlocutor) { /* ... */ } } // receives a callback when a line of dialog is recognized/spoken. protocol VoiceChatterDelegate { func engineDidUpdateDialog(engine:VoiceChatter, dialogLines:[String]) }

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Tree, where every node is a Q&A or a set of quotations: let rocheTree:ConversationNode = .Question( question: "Is your problem with your own feelings, or with other people?", answerPatterns: [ ("feelings", .Question( question: "And do you suffer from love, or from ambition?", answerPatterns:[ ("love", .Statement(quotations:feelingsLoveQuotes)), ("ambition", .Statement(quotations:feelingsAmbitionQuotes)) ]) ), // ... ])

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

import Alexa !

Slide 67

Slide 67 text

Amazon Alexa Custom Skills • 1000s of them, some quite fun! • deployed on the Amazon Echo device • most easily hosted on Amazon Lambda ... • ... which only supports NodeJS, Python, Java

Slide 68

Slide 68 text

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

Conclusions: • Yes, Swift can go anywhere! ! • ... even into conversational UI • Real hardware/software advances in speech • Works best in constrained domains • An interesting world where design is writing • Don't believe the hype (except on neural nets)

Slide 74

Slide 74 text

Acknowledgements • Nick Jackson (@sheriffjackson), docker and terraform magic, in https://github.com/algal/SwiftOnLambda • Claus Höfele (@claushoefele), Alexa wrapper • Xavier Schott, Objective-C Eliza implementation • Swift@IBM, for Kitura and helpful docker images

Slide 75

Slide 75 text

end @alexisgallagher

Slide 76

Slide 76 text

Talking to Swift @alexisgallagher #swiftsummit

Slide 77

Slide 77 text

The Most Human Human by Brian Christian • Pop sci book on chat bots • How they work, where they fail • Turing Test winners pretend to be distractible

Slide 78

Slide 78 text

Swift Package Manager + Xcode: • isolate cross platform in modules • use SPM to manage its build • use SPM to generate xcodeproj ﬁles • contain iOS app in Xcode workspace • SPM-generated project is one imported module • iOS project imports and uses it

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

cgi-bin, all over again /** Reads all of stdin in a String buffer. Transforms the string. Prints result to stdout. */ func readTransformPrint(transform:(String)->String) { var input:String = "" for line in lineGenerator(file: stdin) { input += line } let result = transform(input) print(result, separator: "", terminator: "") }