$30 off During Our Annual Pro Sale. View Details »

Hacking SiriKi - Mobilization 2016

Hacking SiriKi - Mobilization 2016

Esteban Torres

October 22, 2016
Tweet

More Decks by Esteban Torres

Other Decks in Technology

Transcript

  1. HACKING(?)
    SiriKit
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  2. I remember being at home sitting in the couch watching the keynote
    on my ! when they announced what we all have been hoping for
    ANNOUNCEMENT
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  3. !
    APPLE FINALLY RELEASES Siri
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  4. Right there and then I started thinking of all the possible
    applications for such a powerful framework.
    Can you imagine being able to use the power of Siri
    to command your app?
    !
    APPLE FINALLY RELEASES Siri
    maybe
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  5. Back when Siri was released to us I was working for a company called
    Brewbot; which basically made a machine that helped you brew beer
    and control the whole process from your device.
    Overall it was a really nice product and a super ambitious idea.
    If you didn't know making beer is a very hands-on task, and the whole
    premise of Brewbot's product at the time was to make it hands on so you
    felt like making beer instead of making a coffee out of a Keurig machine.
    A LITTLE BIT OF HISTORY
    !
    Brewbot
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  6. This is a very polarizing topic in the brew community but there's
    a point when had to stirr the mash, in our case, the app will send
    a PM telling the user to stirr and would wait until after the user
    confirms that they are done with the requested task.
    As you can imagine having your phone out there while stirring is
    quite risky, it could fall down and shatter the screen (which
    actually happened to us) or it could take a dive on the mash and
    turn into a brick, etc.
    So you could imagine how I was going crazy thinking about the
    possibility of telling Siri stuff like:
    ^> Siri: begin brewing False Bottomed Girls
    ^> Siri: continue the brew
    Which would potentially minimize the risk of damaging your
    STIRRING THE
    MASH
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  7. Keep in mind that all this possible scenarios were running in my head
    just in the few seconds between apple saying «we are releasing
    SiriKit» and apple saying «it will only work for a subset of applications»
    Needless to say I felt devastate
    !
    THEN TRAGEDY
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  8. I understand the resoning behind apple's decision; at least on my mind its because 
    is setting up the basicss and defining how Siri capable apps should behave and once
    everything is nice and smooth, just then then will release/unleash Siri's power for us.
    There were 2 possible paths I could take at that point. Resign myself from my crazy
    ideas (which I sort of did) or maybe…
    LIMITATIONS
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  9. In order to check if there's a way to circumvent the current limitations; or
    even «hacking» its behaviour we first need to understand how it works.
    !
    COULD IT BE «hacked»?
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  10. First the «boring» and easy to follow part; you configure your apps
    capabilities, entitlements, etc; which is no biggie, then comes the nice part.
    How do you enable or even integrate it in a nice way so the user can
    interact with it from anywhere on their phone; and even maybe say «Yo,
    Siri, do something»?
    How does it work??
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  11. The most interesting part about integrating Siri the way we want it to is via
    app extensions
    If you don't know what those are they let's think of them like mini-apps that
    get embedded inside of your binary/app and they installed on your device
    along with the «main» app (like today widgets or ᴡᴀᴛᴄʜ apps.
    They come bundled with your app but they work outside of the app's realm
    so in order to communicate to and from the main app you would have to
    use shared containers to «surpass» 's sandboxing.
    App Extensions
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  12. This are the app extensions that you would need to
    support SiriKit
    You don't need BOTH depending on what and how you
    would like to support SiriKit.
    INTENTS
    &
    UI INTENTS
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  13. An «intent» is the extension one which will get called
    when you execute a Siri command for your app; its an
    intent because its what Siri will try to call if there's a match.
    Basically is the way to represent the users's «intent» of
    requesting a command
    The way this works is because you'll say something like:
    INTENT
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  14. Here you can see how I've highlighted some of the important words.
    First we are obviously launching Siri, then we are telling it to Send something (this
    will tell Siri that the app we want to use needs to be registered as an app that can
    send messages; so it will narrow down the apps scope), then we are telling Siri
    what's the content of the message (which is not necessarily required to be
    provided at the time of launching the intent); we are then specifying a receiver by
    saying to whom we want to send the message (also not required) and then by
    saying using we are specifying the app we want to use to handle our intent
    With all this information Siri can now deconstruct everything
    HEY SIRI, Send Hello to Travis using
    Commander
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  15. Now siri can actually call our intent telling it exactly all of this, your user intents to send a message
    with this information, there you can actually look for the recipient in your custom list of contacts
    (or piggy back on the phone's contacts book) and decide if you can handle the message or not.
    ▸ Type: Message app
    ▸ Recipient: Travis
    ▸ Content: Hello
    ▸ Application: Commander
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  16. Back to the components required to support SiriKit we have the UI
    Intent which gives visual feedback to the user and its probably the most
    interesting one for my hack project.
    The UI Intent could be a complex view depending on your app or a
    pretty simple one and should look something like this:
    UI INTENT
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  17. Here you can see some HORRIBLE UI I used when I
    was playing around with Siri.
    There are a few things to notice from this screenshot.
    1. Travis; for the longest time i've wanted to launch
    Travis jobs with something other than a commit or the
    web ui, why? no idea; I just wanted to; so basically my
    hack app will hack Siri to communicate with Travis
    2. From what I told you before we can infer that the app
    is called Commander
    3. And the most important part; the UI…
    and of course i'm kidding, the Recipient on the bottom
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  18. I said send it to Travis but the UI reported a different recipient which
    sounds like an error…
    Or maybe it isn't; this pretty much tells us that the data can be
    «tweaked» or hacked or tinkered with, meaning that we can actually
    take advantage of SiriKit
    to Travis
    to Charlie's
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  19. We still have a long way ahead before being able to say
    something like that
    We still need to figure out a way to parse our actions and
    map our recipients
    perhaps…
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  20. But first, what is this app we are going to build?
    it has a couple of parts
    First one being a way to query Travis CI for a
    list of repos and their last build state
    ^ And also way for us to trigger a failed build so
    it can run again
    Commander
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  21. so; let's see some code and disect it so we can verify how
    «hackable» this is going to be.
    SEE SOME CODE
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  22. Maybe you can't see it but we'll disect it bit by bit so we can
    all understand what's going on and how to abuse it.
    func resolveRecipients(forSendMessage intent: INSendMessageIntent, with completion: @escaping ([INPersonResolutionResult]) -> Void) {
    var resolutionResults = [INPersonResolutionResult]()
    for recipient in recipients {

    switch matchingContacts.count {
    case 2 ... Int.max:
    // We need Siri's help to ask user to pick one from the matches.
    resolutionResults += [INPersonResolutionResult.disambiguation(with: matchingContacts)]
    case 1:
    // We have exactly one matching contact
    resolutionResults += [INPersonResolutionResult.success(with: recipient)]
    case 0:
    // We have no contacts matching the description provided
    resolutionResults += [INPersonResolutionResult.unsupported()]

    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  23. First, what happens if multiple contacts match our recipient?
    as you can see we can send back «feedback»/information to the
    user so they can actually choose which contact we want, which in
    this case can be totally abused by completely ignoring any kind of
    «contacts» concept and instead using our own entities.
    The other scenarios are self explanatory, when we have one and only
    one contact we return success setting that contact as the receiver
    We have NO contacts and thus we need to report it back to siri so
    the user can provide the user
    WHAT HAPPENS WHEN MULTIPLE CONTACTS «MATCH»?
    case 2 ... Int.max:
    // We need Siri's help to ask user to pick one from the matches.
    resolutionResults += [INPersonResolutionResult.disambiguation(with: matchingContacts)]
    FEEDBACK TO THE USER
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  24. For the first part I thought about maybe having a predefined
    contact called Travis and then send it a command like fetch
    CALLING Travis
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  25. Pretty simple and works pretty nice, why?
    We are defaulting to Travis as the recipient so in our code
    we can actually check if the recipient is travis and then
    proceed.
    Send a message to Travis
    using Commander
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  26. As you can see here when confirming the command we get to actually execute and hit
    the Travis api calls and even fail and give feedback to the user if the service failed; there's
    even a flag to indicate that our service is down (messaging service though).
    CONFIRMING THE COMMAND
    // Handle the completed intent (required).
    func handle(sendMessage intent: INSendMessageIntent, completion: @escaping (INSendMessageIntentResponse) -> Void) {
    let userActivity = NSUserActivity(activityType: NSStringFromClass(INSendMessageIntent.self))
    let response = INSendMessageIntentResponse(code: .success, userActivity: userActivity)
    // Here we should be hitting Travi's API
    completion(response)
    }
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  27. Basically we have most of the pseudo code ready, we are still lacking a way to actually
    parse the commands that the user is sending, we could do that on multiple parts of
    the app; but I think the best part to do so would be on resolveContent function
    VERIFYING THE
    COMMANDS
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  28. The code is pretty simple; at the top we are defining our actions
    as an enum with a string type so we can easily «parse» the
    commands sent to siri
    Inside the actual «resolve» we verify that the recipient is either
    Travis or a valid repo and that the user issued an expected
    command.
    If one of those requirements isn't met we can fail and tell the user:
    VERIFYING THE COMMANDS
    private enum Actions: String {
    case execute
    case status
    }
    func resolveContent(forSendMessage intent: INSendMessageIntent, with completion: @escaping (INStringResolutionResult) -> Void) {
    // Extract the 1st recipient (only expect 1 here)
    if let recipients = intent.recipients, let recipient = recipients.first {
    let key = (recipient.customIdentifier ?? recipient.displayName).lowercased()
    if (!self.repos.contains(key)) { // Repos should be read from a shared container DB
    completion(INStringResolutionResult.unsupported())
    }
    }
    // Extract the command
    if let text = intent.content, !text.isEmpty {
    // Extract the actual action
    if let _ = Actions(rawValue: text.lowercased()) {
    completion(INStringResolutionResult.success(with: text))
    } else {
    completion(INStringResolutionResult.unsupported())
    }
    } else {
    completion(INStringResolutionResult.needsValue())
    }
    }
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  29. Here the message is kind of misleading; but that's the whole point of
    hacking/piggy backing on another service.
    Simply we are failing because we are trying to send a message to
    Miguel which isn't Travis nor is a repo on the «predefined»
    account
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  30. In here we are sending a message to a valid recipient but we are trying to execute
    an invalid command, and thus we are failing with a more «valid» message.
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  31. Ok, we are almost there, we need to hit Travis api based on multiple
    different actionns and recipients.
    If the recipient is travis and the command is status we fetch the list of
    repos and their state
    Here I wanted to give a visual feedback to the user with the state of the
    repos via a custom UI Intent.
    The way it works is by specifying the type of Intent for which you want
    to support the UI on the info plist of the extensions; sadly for messaging
    apps the only valid value is for sending; which means there's no actual
    way to give the user immediate feedback of the status of the repos
    PUTTING ALL THE
    PARTS TOGETHER
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  32. Not too worry too much, we can keep on hacking over Siri to fit our needs.
    a couple of slides ago we said we were going to read the repos from a
    shared container db, basically when we execute the fetch we store the
    results in a shared container (if we want to also support querying and
    launching jobs from the actual app and not just the extensions)
    Then next time the user tries to send a message and doesn't specify a
    recipient we can «hijack» the recipient check and populate with the list of
    repos and a state flag/emoji
    SO WHAT THEN?
    !
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  33. Here is where we actually abuse Siri the most, we should map from our list of
    repos and then build our INPerson entities from that data and pass it back as a
    disambiguation to siri, this will generate this output:
    // If no recipients were provided we'll need to prompt for a value.
    if recipients.count == 0 {
    // TODO: Read from the DB/store the list of repos and statuses
    let person = INPerson(personHandle: INPersonHandle(value: "RxViewModel", type: .unknown),
    nameComponents: nil,
    displayName: "✅\tRxViewModel",
    image: nil,
    contactIdentifier: nil,
    customIdentifier: "RxViewModel")
    completion([INPersonResolutionResult.disambiguation(with: matchingContacts)])
    return
    }
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  34. Here we are giving the user enough visual feedback and
    information to make a conscious decision
    Once the user selects a repo siri will ask for a message or
    in this case a command
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  35. Here we can select which repo we want by taping on it and then saying execute which is a recognized task, then
    again, inside the function where we confirming sending the message we just need to extract the info we require
    and call the travis api with the «re run» command and that will actually launch a new job on the desired repo.
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  36. Well obviously we have proven that Siri is quite powerful due to the fact the it can
    interpret language really nice (caveat, depending on your language); english is ok-
    ish, same as spanish but i've heard that german needs to be quite robotic for siri to
    pick up context.
    The feedback and concepts are not too obvious or readable given that you need to
    abuse the vocabulary set for a completely different type of app
    SHORTCOMINGS
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  37. Basically SiriKit is quite powerful, it can be easily abused to fill our needs but is still
    a long way from being the native assistante we all want to integrate into our apps.
    Hopefully next year or maybe in a couple  will loosen up a little bit the restriction
    on apps that can use SiriKit and then maybe we can come up with really nice
    integreations
    The fitness apps ui intents are more dynamic and maybe could be abused in a
    really nice matter; just that for this specific scenario the vocabulary and
    foundations for fitness felt even weirder
    TAKEAWAYS
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  38. THANKS
    dzięki
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide

  39. YOU CAN FIND ME:
    ▸ Twitter: @esttorhe
    ▸ Github: esttorhe
    ▸ Blog: estebantorr.es
    ▸ Code sample1: github.com/esttorhe/travisboss
    #yatusabes
    1
    Code will be uploaded next week
    Roberto Esteban Torres - @esttorhe, Mobilization, 2016

    View Slide