Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Talking to Swift

Talking to Swift

Talk given at Swift Summit 2016, in San Francisco.


Alexis Gallagher

November 07, 2016

More Decks by Alexis Gallagher

Other Decks in Technology


  1. Conversational UI 1. What is it? Why the hype? Is

    it real? 2. How to design it? 3. How to build it?
  2. But now, we really use them! ! • Siri (iOS,

    macOS, watchOS, tvOS) • Google (Search, Maps, Google Home) • Amazon Alexa • XFinity X1 • Slackbot
  3. But now, we really use them! ! • Voice-based customer

    service phone tree • Text chat with automated customer service agent • Automated telephone survey • IM messaging spam • Email spam
  4. What is it? Conversational UI Reality Check(list): • Messaging? Evolving

    new interaction model. No magic AI. • Chatbots? Still quite limited. No magic AI. (afaict). • Voice? Yes! Significant breakthroughs thanks to neural nets.
  5. What If? by Anne Bernays & Pamela Painter • Exercises

    for characterization • What does the character want? • Remember most from childhood? • Usually feel two hours after lunch? • What are they in denial of?
  6. Voice UI is opposite of Message Thread UI Thread Voice

    history accessible no history identity visible no identity rich media text only guided input freeform text
  7. Rochefoucauld • François VI, Duc de la Rouchefoucauld, Prince de

    Marcillac • Born 1613, Died 1680 • Wealthy, noble, good-looking • Unhappily married, imprisoned, libeled, exiled, shot in the eye
  8. Rochefoucauld • Maxims and other works • 500+ aphorisms on

    love, ambition, self-delusion, ennui, envy, esteem, evils, the exchange of secrets, etc. • character: cynical, worldly wise, witty • domain: life's sorrows
  9. Speech Synthesis API class MyDelegate : AVSpeechSynthesisDelegate { public func

    speechSynthesizer(_ synthesizer: AVSpeechSynthesizer, didFinish utterance: AVSpeechUtterance) { print("Just finishe saying hello") } } let synth = AVSpeechSynthesizer() let delegate = MyDelegate() synth.delegate = delegate let utterance = AVSpeechUtterance(string:"Hello, world") synth.speak(utterance)
  10. Once, create and/or configure • AVAudioEngine, AVAudioSession, SFSpeechRecognizer Per utterance,

    request stream of live partial results • AVAudioNodeTapBlock, SFSpeechAudioBufferRecognitionRequest, SFSpeechAudioBufferRecognitionTask During utterances, handle callbacks with results • SFSpeechRecognitionResult • but SFSpeechRecognitionResult.isFinal is never
  11. /** Operation which recognizes speech and finishes after it has

    reached an "endpoint", an interval over which no speech has been recognized (e.g., a period of silence or of noise). */ public class SpeechRecognitionOperation: Operation, SFSpeechRecognitionTaskDelegate { open override var isAsynchronous: Bool { return true } public var output:String? = nil private var endpointTimer:Timer? private var request:SFSpeechAudioBufferRecognitionRequest? // ... public init(engine e:AVAudioEngine, recognizer r:SFSpeechRecognizer) { /* ... */ } // ... }
  12. // Represents a bot, e.g., Eliza, Rochefoucauld, etc.. protocol Interlocutor

    : class { init() func respondTo(saying:String) -> String } // wraps any Interlocutor in a speech rec/synth UI public class VoiceChatter : NSObject { public init(interlocutor:Interlocutor) { /* ... */ } } // receives a callback when a line of dialog is recognized/spoken. protocol VoiceChatterDelegate { func engineDidUpdateDialog(engine:VoiceChatter, dialogLines:[String]) }
  13. Tree, where every node is a Q&A or a set

    of quotations: let rocheTree:ConversationNode = .Question( question: "Is your problem with your own feelings, or with other people?", answerPatterns: [ ("feelings", .Question( question: "And do you suffer from love, or from ambition?", answerPatterns:[ ("love", .Statement(quotations:feelingsLoveQuotes)), ("ambition", .Statement(quotations:feelingsAmbitionQuotes)) ]) ), // ... ])
  14. Amazon Alexa Custom Skills • 1000s of them, some quite

    fun! • deployed on the Amazon Echo device • most easily hosted on Amazon Lambda ... • ... which only supports NodeJS, Python, Java
  15. !

  16. Conclusions: • Yes, Swift can go anywhere! ! • ...

    even into conversational UI • Real hardware/software advances in speech • Works best in constrained domains • An interesting world where design is writing • Don't believe the hype (except on neural nets)
  17. Acknowledgements • Nick Jackson (@sheriffjackson), docker and terraform magic, in

    https://github.com/algal/SwiftOnLambda • Claus Höfele (@claushoefele), Alexa wrapper • Xavier Schott, Objective-C Eliza implementation • Swift@IBM, for Kitura and helpful docker images
  18. The Most Human Human by Brian Christian • Pop sci

    book on chat bots • How they work, where they fail • Turing Test winners pretend to be distractible
  19. Swift Package Manager + Xcode: • isolate cross platform in

    modules • use SPM to manage its build • use SPM to generate xcodeproj files • contain iOS app in Xcode workspace • SPM-generated project is one imported module • iOS project imports and uses it
  20. What If? by Anne Bernays & Pamela Painter • Exercises

    for characterization • What does the character want? • Remember most from childhood? • Usually feel two hours after lunch? • What are they in denial of?
  21. cgi-bin, all over again /** Reads all of stdin in

    a String buffer. Transforms the string. Prints result to stdout. */ func readTransformPrint(transform:(String)->String) { var input:String = "" for line in lineGenerator(file: stdin) { input += line } let result = transform(input) print(result, separator: "", terminator: "") }