$30 off During Our Annual Pro Sale. View Details »

Talking to Swift

Talking to Swift

Talk given at Swift Summit 2016, in San Francisco.

https://www.skilled.io/u/swiftsummit/talking-to-swift

Alexis Gallagher

November 07, 2016
Tweet

More Decks by Alexis Gallagher

Other Decks in Technology

Transcript

  1. Talking to Swift
    @alexisgallagher

    View Slide

  2. View Slide

  3. View Slide

  4. "Just use the keyboard."
    "A keyboard, how quaint..."

    View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. Bots are here ?!

    View Slide

  9. Conversational UI
    1. What is it? Why the hype? Is it real?
    2. How to design it?
    3. How to build it?

    View Slide

  10. What is it?
    Trend 1: Messaging

    View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. Message Thread:
    an evolving interaction model

    View Slide

  16. What is it?
    Trend 2: Chatbots

    View Slide

  17. The
    DREAM
    is not new

    View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. The
    REALITY
    is not new either

    View Slide

  22. $ emacs -q -f doctor

    View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. But now, we really use them! !
    • Siri (iOS, macOS, watchOS, tvOS)
    • Google (Search, Maps, Google Home)
    • Amazon Alexa
    • XFinity X1
    • Slackbot

    View Slide

  27. But now, we really use them! !
    • Voice-based customer service phone tree
    • Text chat with automated customer service agent
    • Automated telephone survey
    • IM messaging spam
    • Email spam

    View Slide

  28. SiriKit?

    View Slide

  29. SiriKit.

    View Slide

  30. chatbots
    =
    restricted domain bots

    View Slide

  31. What is it?
    Trend 3: Voice

    View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. Progress in speech
    recognition
    (within the last month!)

    View Slide

  37. Progress in speech
    synthesis
    (within the last month!)

    View Slide

  38. View Slide

  39. What is it?
    Conversational UI Reality Check(list):
    • Messaging?
    Evolving new interaction model. No magic AI.
    • Chatbots?
    Still quite limited. No magic AI. (afaict).
    • Voice?
    Yes! Significant breakthroughs thanks to neural nets.

    View Slide

  40. How to Design it
    1. Develop a character

    View Slide

  41. View Slide

  42. What If?
    by Anne Bernays & Pamela Painter
    • Exercises for characterization
    • What does the character want?
    • Remember most from childhood?
    • Usually feel two hours after lunch?
    • What are they in denial of?

    View Slide

  43. How to Design it
    2. Restrict the domain

    View Slide

  44. Voice UI is opposite of
    Message Thread UI
    Thread Voice
    history
    accessible
    no history
    identity visible no identity
    rich media text only
    guided input freeform text

    View Slide

  45. Example
    character: Rochefoucauld
    domain: sorrows of life

    View Slide

  46. Rochefoucauld
    • François VI, Duc de la
    Rouchefoucauld, Prince de Marcillac
    • Born 1613, Died 1680
    • Wealthy, noble, good-looking
    • Unhappily married, imprisoned,
    libeled, exiled, shot in the eye

    View Slide

  47. View Slide

  48. Rochefoucauld
    • Maxims and other works
    • 500+ aphorisms on love, ambition,
    self-delusion, ennui, envy, esteem,
    evils, the exchange of secrets, etc.
    • character: cynical, worldly wise, witty
    • domain: life's sorrows

    View Slide

  49. We all have strength
    enough to bear the
    misfortunes of others

    View Slide

  50. To say that one never flirts
    is in itself a form of
    flirtation

    View Slide

  51. How to Build it

    View Slide

  52. View Slide

  53. import AVFoundation

    View Slide

  54. Speech Synthesis API
    class MyDelegate : AVSpeechSynthesisDelegate {
    public func speechSynthesizer(_ synthesizer: AVSpeechSynthesizer,
    didFinish utterance: AVSpeechUtterance) {
    print("Just finishe saying hello")
    }
    }
    let synth = AVSpeechSynthesizer()
    let delegate = MyDelegate()
    synth.delegate = delegate
    let utterance = AVSpeechUtterance(string:"Hello, world")
    synth.speak(utterance)

    View Slide

  55. import Speech

    View Slide

  56. Once, create and/or configure
    • AVAudioEngine, AVAudioSession, SFSpeechRecognizer
    Per utterance, request stream of live partial results
    • AVAudioNodeTapBlock,
    SFSpeechAudioBufferRecognitionRequest,
    SFSpeechAudioBufferRecognitionTask
    During utterances, handle callbacks with results
    • SFSpeechRecognitionResult
    • but SFSpeechRecognitionResult.isFinal is never

    View Slide

  57. Problem:
    "endpointing"

    View Slide

  58. Solution:
    Asynchronous Operation

    View Slide

  59. /**
    Operation which recognizes speech and finishes after it has reached
    an "endpoint", an interval over which no speech has been recognized
    (e.g., a period of silence or of noise).
    */
    public class SpeechRecognitionOperation:
    Operation, SFSpeechRecognitionTaskDelegate
    {
    open override var isAsynchronous: Bool { return true }
    public var output:String? = nil
    private var endpointTimer:Timer?
    private var request:SFSpeechAudioBufferRecognitionRequest?
    // ...
    public init(engine e:AVAudioEngine,
    recognizer r:SFSpeechRecognizer) { /* ... */ }
    // ...
    }

    View Slide

  60. View Slide

  61. // Represents a bot, e.g., Eliza, Rochefoucauld, etc..
    protocol Interlocutor : class {
    init()
    func respondTo(saying:String) -> String
    }
    // wraps any Interlocutor in a speech rec/synth UI
    public class VoiceChatter : NSObject {
    public init(interlocutor:Interlocutor) { /* ... */ }
    }
    // receives a callback when a line of dialog is recognized/spoken.
    protocol VoiceChatterDelegate {
    func engineDidUpdateDialog(engine:VoiceChatter, dialogLines:[String])
    }

    View Slide

  62. View Slide

  63. Tree, where every node is a Q&A or a set of quotations:
    let rocheTree:ConversationNode = .Question(
    question: "Is your problem with your own feelings, or with other people?",
    answerPatterns: [
    ("feelings",
    .Question(
    question: "And do you suffer from love, or from ambition?",
    answerPatterns:[
    ("love",
    .Statement(quotations:feelingsLoveQuotes)),
    ("ambition",
    .Statement(quotations:feelingsAmbitionQuotes))
    ])
    ),
    // ...
    ])

    View Slide

  64. View Slide

  65. View Slide

  66. import Alexa
    !

    View Slide

  67. Amazon Alexa Custom Skills
    • 1000s of them, some quite fun!
    • deployed on the Amazon Echo device
    • most easily hosted on Amazon Lambda ...
    • ... which only supports NodeJS, Python, Java

    View Slide

  68. !

    View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. View Slide

  73. Conclusions:
    • Yes, Swift can go anywhere! !
    • ... even into conversational UI
    • Real hardware/software advances in speech
    • Works best in constrained domains
    • An interesting world where design is writing
    • Don't believe the hype (except on neural nets)

    View Slide

  74. Acknowledgements
    • Nick Jackson (@sheriffjackson), docker and terraform magic,
    in https://github.com/algal/SwiftOnLambda
    • Claus Höfele (@claushoefele), Alexa wrapper
    • Xavier Schott, Objective-C Eliza implementation
    • Swift@IBM, for Kitura and helpful docker images

    View Slide

  75. end
    @alexisgallagher

    View Slide

  76. Talking to Swift
    @alexisgallagher
    #swiftsummit

    View Slide

  77. The Most Human Human
    by Brian Christian
    • Pop sci book on chat bots
    • How they work, where they fail
    • Turing Test winners pretend to be
    distractible

    View Slide

  78. Swift Package Manager + Xcode:
    • isolate cross platform in modules
    • use SPM to manage its build
    • use SPM to generate xcodeproj files
    • contain iOS app in Xcode workspace
    • SPM-generated project is one imported module
    • iOS project imports and uses it

    View Slide

  79. View Slide

  80. View Slide

  81. What If?
    by Anne Bernays & Pamela Painter
    • Exercises for characterization
    • What does the character want?
    • Remember most from childhood?
    • Usually feel two hours after lunch?
    • What are they in denial of?

    View Slide

  82. View Slide

  83. cgi-bin, all over again
    /**
    Reads all of stdin in a String buffer. Transforms the string. Prints result to stdout.
    */
    func readTransformPrint(transform:(String)->String)
    {
    var input:String = ""
    for line in lineGenerator(file: stdin) {
    input += line
    }
    let result = transform(input)
    print(result, separator: "", terminator: "")
    }

    View Slide

  84. View Slide

  85. View Slide

  86. View Slide

  87. View Slide

  88. View Slide

  89. View Slide