Siri Shortcut Introduction

Transcript

1. Siri Shortcut DADA 2018/7/24 WWDC18 Study Group @honestbee
2. What is shortcut? • Shortcut let you expose the capabilities of your app to Siri. • It can accelerate the user to perform a key function of your app. • It should be of persistent interest to the user. • It should be executable at any time.
3. User Aspect
4. Where is shortcut?
5. Where is shortcut?
6. Shortcut App Manage shortcuts Settings ! "
7. Developer Aspect
8. Shortcut API
9. Create shortcut Creating Shortcuts Define Shortcut Donate Shortcut Handle Shortcut
10. User Engagement Siri Suggestion Voice Interaction Define Donate Handle Navigate to App Executes in background Create shortcut Input Output
12. Define NSUserActivityTypes
13. • Define NSUserActivity • Set -isEligibleForPrediction to true • Set -isEligibleForSearch to true first Donate • Donate to UIResponder to UIResponder
14. // How to donate a shortcut let userActivity = NSUserActivity(activityType: "com.myapp.name.my-activity-type") userActivity.isEligibleForSearch = true userActivity.isEligibleForPrediction = true userActivity.title = "Activity" userActivity.userInfo = ["key": "value"] userActivity.suggestedInvocationPhrase = "Let's do it" let attributes = CSSearchableItemAttributeSet(itemContentType: kUTTypeItem as String) let image = UIImage(named: "myImage")! attributes.thumbnailData = image.pngData() attributes.contentDescription = "Subtitle" userActivity.contentAttributeSet = attributes viewController.userActivity = userActivity
15. // How to donate a shortcut let userActivity = NSUserActivity(activityType: "com.myapp.name.my-activity-type") userActivity.isEligibleForSearch = true userActivity.isEligibleForPrediction = true userActivity.title = "Activity" userActivity.userInfo = ["key": "value"] userActivity.suggestedInvocationPhrase = "Let's do it" let attributes = CSSearchableItemAttributeSet(itemContentType: kUTTypeItem as String) let image = UIImage(named: "myImage")! attributes.thumbnailData = image.pngData() attributes.contentDescription = "Subtitle" userActivity.contentAttributeSet = attributes viewController.userActivity = userActivity Do not call becomeCurrent when donating to UIResponder
16. // How to handle a shortcut func application(_ application: UIApplication, continue userActivity: NSUserActivity, restorationHandler: @escaping ([Any]?) -> Void) -> Bool { if userActivity.activityType == "com.myapp.name.my-activity-type" { // Restore state for userActivity and userInfo } } Handle // How to donate a shortcut let userActivity = NSUserActivity(activityType: "com.myapp.name.my-activity-type") userActivity.isEligibleForSearch = true userActivity.isEligibleForPrediction = true userActivity.title = "Activity" userActivity.userInfo = ["key": "value"] userActivity.suggestedInvocationPhrase = "Let's do it" let attributes = CSSearchableItemAttributeSet(itemContentType: kUTTypeItem as String) let image = UIImage(named: "myImage")! attributes.thumbnailData = image.pngData() attributes.contentDescription = "Subtitle" userActivity.contentAttributeSet = attributes viewController.userActivity = userActivity - continueUserActivity
18. Define Siri Intent Go to File -> New File, and choose “SiriKit Intent Definition File.”
19. Define Siri Intent
20. Define Siri Intent
21. Define Siri Intent
22. Define Siri Intent Go to File -> New File, and choose “SiriKit Intent Definition File.”
23. public class OrderSoupIntent: INIntent { public var items: [INObject]? public var deliveryLocation: CLPlacemark? } public protocol OrderSoupIntentHandling: NSObjectProtocol { public func handle(intent: OrderSoupIntent, completion: @escaping (OrderSoupIntentResponse) -> Void) optional func confirm(intent: OrderSoupIntent, completion: @escaping (OrderSoupIntentResponse) -> Void) } // Donate your shortcut let intent = PlaceOrderIntent() intent.items = order.items.map({ INObject(identifier: $0.id.uuidString, display: $0.menuItem.displayName) }) intent.deliveryLocation = order.destinationLocation let interaction = INInteraction(intent: intent, response: nil) interaction.donate { error in // Handle error } Code Generation public class OrderSoupIntent: INIntent { public var items: [INObject]? public var deliveryLocation: CLPlacemark? } public protocol OrderSoupIntentHandling: NSObjectProtocol { public func handle(intent: OrderSoupIntent, completion: @escaping (OrderSoupIntentResponse) -> Void) optional func confirm(intent: OrderSoupIntent, completion: @escaping (OrderSoupIntentResponse) -> Void) } • Siri Intent • Handling Protocol Define
24. Donate // Donate your shortcut let intent = PlaceOrderIntent() intent.items = order.items.map({ INObject(identifier: $0.id.uuidString, display: $0.menuItem.displayName) }) intent.deliveryLocation = order.destinationLocation let interaction = INInteraction(intent: intent, response: nil) interaction.donate { error in // Handle error } // Donate your shortcut let intent = PlaceOrderIntent() intent.items = order.items.map({ INObject(identifier: $0.id.uuidString, display: $0.menuItem.displayName) }) intent.deliveryLocation = order.destinationLocation let interaction = INInteraction(intent: intent, response: nil) interaction.donate { error in // Handle error } to INInteraction
25. • - continueUserActivityActivity • support background execution • Goto File -> new Target then create Intents Extension. • Check the target membership. The Intent should be visible to Intents Extension. • Make the Intents Extension confirm the Handling Protocol. Handle
26. Handle // Handle your shortcut func application(_ application: UIApplication, continue userActivity: NSUserActivity, restorationHandler: @escaping ([Any]?) -> Void) -> Bool { if userActivity.activityType == "OrderSoupIntent", let intent = userActivity.interaction?.intent as? OrderSoupIntent { // Show ordering UI, pre-populated with the fields from the intent } } // Donate your shortcut let intent = PlaceOrderIntent() intent.items = order.items.map({ INObject(identifier: $0.id.uuidString, display: $0.menuItem.displayName) }) intent.deliveryLocation = order.destinationLocation let interaction = INInteraction(intent: intent, response: nil) interaction.donate { error in // Handle error } - continueUserActivity
27. // Handle your shortcut class IntentHandler: INExtension, OrderSoupIntentHandling { func confirm(intent: OrderSoupIntent, completion: (OrderSoupIntentResponse) -> Void) { completion(OrderSoupIntentResponse(code: .ready, userActivity: nil)) } func handle(intent: OrderSoupIntent, completion: (OrderSoupIntentResponse) -> Void) { // Order the soup completion(OrderSoupIntentResponse(code: .success, userActivity: nil)) } } Handle // Donate your shortcut let intent = PlaceOrderIntent() intent.items = order.items.map({ INObject(identifier: $0.id.uuidString, display: $0.menuItem.displayName) }) intent.deliveryLocation = order.destinationLocation let interaction = INInteraction(intent: intent, response: nil) interaction.donate { error in // Handle error } Intent Handling Protocol
 for background execution You should still handle -continueActivity, even though you have already confirmed Intent handling protocol
28. Which one to choose? • Use NSUserActivity if your shortcut: • Opens something in your app • Represents showing items that you index in Spotlight or offer for Handoff • For the best experience, adopt Intents. Intents-based shortcuts can: • Run inline, without launching your app • Include custom voice response or a custom UI Include granular predictions
29. The Intent Parameters
30. Custom Parameters • “Custom” results in an INObject which combines an ID with a display string.
31. Add Enum Parameters • Use enums when the values for a parameter are clearly bounded.
32. Voice Response
33. // Custom response object public class OrderSoupIntentResponse : INIntentResponse { public var waitTime: String? { get set } public var soup: INObject? { get set } public convenience init(code: OrderSoupIntentResponseCode, userActivity: NSUserActivity?) public static func success(soup: INObject, waitTime: String) -> OrderSoupIntentResponse public static func failureOutOfStock(soup: INObject) -> OrderSoupIntentResponse } Code Generation Voice Response
34. // Handle the intent class IntentHandler: INExtension, OrderSoupIntentHandling { func confirm(intent: OrderSoupIntent, completion: (OrderSoupIntentResponse) -> Void) { ... } func handle(intent: OrderSoupIntent, completion: (OrderSoupIntentResponse) -> Void) { guard let soup = intent.soup, let order = Order(from: intent) else { completion(OrderSoupIntentResponse(code: .failure, userActivity: nil)) return } let orderManager = SoupOrderDataManager() orderManager.placeOrder(order: order) completion(OrderSoupIntentResponse(code: .success, userActivity: nil)) } }
35. • You had better prepared these 4 types of response: • Confirmation • Success • Informational • Error Voice Response By Apple’s suggestions.
36. The Details…
37. Optimizing Suggestions • The suggestion’s goal: Find Routines / Patterns • NSUserActivity • requiredUserInfoKeys: Minimal amount of information necessary for restoration which used to find patters. • INIntent • Shortcut types: Find the most appropriate parameter combination.
38. Shortcut Testing Developer settings on device Shortcut Testing
39. Shortcut Testing Developer settings on device Set up a shortcut with Siri and edit the Xcode scheme to test Shortcut Testing
40. Localizations • Localize both your code and the strings in your intent definition files • It will create all contents of all patterns. • Handle pluralization in stringsdict
41. Localizations Intent Definition File Localizable content—strings NEW
42. Localizations Definition File content—strings NEW Intent Definition File Localizable content—strings NEW Handle Pluralization
43. Localizations Intent Definition File Localizable content—stringsdict
44. Deferred Localized Intents String Localization happens when the string is presented to the user Used when preparing the intent for donation intent.suggestedInvocationPhrase = NSString.deferredLocalizedIntentsString(with: “Go to Mars!")
45. Privacy • Delete Donations: New APIs Prevent suggesting content that’s no longer relevant to fit user expectation
46. Suggested Invocation Phrase • - suggestedInvocationPhrase
 For both NSUserActivity and INIntent
47. Default suggestion shortcut // How to suggest a shortcut for something a user hasn’t done yet import Intents let suggestions = [ INShortcut(intent: orderFavoriteSoupIntent), INShortcut(userActivity: orderFavoriteBeverageUserActivity), ] INVoiceShortcutCenter.shared.setShortcutSuggestions(suggestions) ! "
48. Bring shortcut into your App “Chowder time” “Tomato tomato” Thanks for your order! $8, office Soup Chef # Your clam chowder will be delivered to One Apple Park Way soon. Done Done Add to Siri INUIAddVoiceShortcutViewController INUIEditVoiceShortcutViewController Available shortcuts list API?! See nothing in documents…
49. Bring shortcut into your App // Adding a shortcut from an NSUserActivity import IntentsUI let userActivity = NSUserActivity(activityType: "com.unicorny.SoupChef.viewOrder") // Configure your NSUserActivity let shortcut = INShortcut(userActivity: userActivity) let viewController = INUIAddVoiceShortcutViewController(shortcut: shortcut) viewController.delegate = self present(viewController, animated: true)
50. INUIAddVoiceShortcutViewControllerDelegate • addVoiceShortcutViewController:didFinishWithVoiceShortcut:error: • addVoiceShortcutViewControllerDidCancel: INUIEditVoiceShortcutViewControllerDelegate • editVoiceShortcutViewController:didUpdateVoiceShortcut:error: • editVoiceShortcutViewController:didDeleteVoiceShortcutWithIdentifier: • editVoiceShortcutViewControllerDidCancel:
51. Media Shortcut Provide media playback shortcuts. • INPlayMediaIntent • Launches app in background • Suggested in playback controls on Lock Screen • Works on HomePod • INUpcomingMedia • Podcasts, TV shows
52. • INRelevantShortcut • INRelevanceProvider • Background Task • iOS Relevant Shortcuts on Siri Watch Face Siri Shortcuts on the Siri Watch Face
53. INRelevantShortcut // Relevant Shortcut open class INRelevantShortcut : NSObject, NSCopying, NSSecureCoding { @NSCopying open var shortcut: INShortcut { get } @NSCopying open var relevanceProviders: [INRelevanceProvider] @NSCopying open var watchTemplate: INDefaultCardTemplate? public init(shortcut: INShortcut) }
54. Define // Setting INRelevantShortcuts let userActivity = NSUserActivity(activityType: "com.myapp.LogMeal") userActivity.userInfo["meal"] = "Dinner" let shortcut = INShortcut(userActivity: userActivity)
 let relevantShortcut = INRelevantShortcut(shortcut: shortcut!) let cardTemplate = INDefaultCardTemplate(title: "Log Dinner") cardTemplate.subtitle = "Beat the veggie challenge!" cardTemplate.image = INImage(named: "MenacingBroccoli") relevantShortcut.watchTemplate = cardTemplate let relevanceProvider = INDailyRoutineRelevanceProvider(situation: .evening) relevantShortcut.relevanceProviders = [relevanceProvider] INRelevantShortcutStore.default.setRelevantShortcuts([relevantShortcut]) { error in NSUserActivity
55. Define INIntent // Setting INRelevantShortcuts let intent = LogFavoriteIntent() intent.favorite = "Cookies" intent.setImage(INImage(named: "Cookies"), forParameterNamed: "favorite") let shortcut = INShortcut(intent: intent)
 let relevantShortcut = INRelevantShortcut(shortcut: shortcut!) INRelevantShortcutStore.default.setRelevantShortcuts([relevantShortcut]) { error in if let error = error { /* */ } } // Setting INRelevantShortcuts let intent = LogFavoriteIntent() intent.favorite = "Cookies" intent.setImage(INImage(named: "Cookies"), forParameterNamed: "favorite") let shortcut = INShortcut(intent: intent)
 let relevantShortcut = INRelevantShortcut(shortcut: shortcut!) INRelevantShortcutStore.default.setRelevantShortcuts([relevantShortcut]) { error in if let error = error { /* */ } }
56. INRelevantShortcut // Relevant Shortcut Store open class INRelevantShortcutStore : NSObject { open class var `default`: INRelevantShortcutStore { get } open func setRelevantShortcuts(_ shortcuts: [INRelevantShortcut], completionHandler: ((Error?) -> Void)? = nil) }
57. Define setRelevantShortcuts // Setting INRelevantShortcuts let intent = LogFavoriteIntent() intent.favorite = "Cookies" intent.setImage(INImage(named: "Cookies"), forParameterNamed: "favorite") let shortcut = INShortcut(intent: intent)
 let relevantShortcut = INRelevantShortcut(shortcut: shortcut!) INRelevantShortcutStore.default.setRelevantShortcuts([relevantShortcut]) { error in if let error = error { /* */ } }
58. Donate NSUserActivity Donations Set eligibleForPrediction and eligibleForSearch Supported activity types in Info.plist Donate when visible let userActivity = NSUserActivity(activityType: "MyActivityType") userActivity.becomeCurrent() let userActivity = NSUserActivity(activityType: "MyActivityType") interfaceController.updateUserActivity(activityType) NEW NSUserActivity
59. Donate INIntent Donations Donate using INInteraction Indicate primary shortcut type let intent = /* Create intent and set parameters */ let interaction = INInteraction(intent: intent, response: nil) interaction.donate(completion: nil) INIntent
60. Donate Primary Shortcut Type Find the critical pattern
61. Handle // Handle restoring from NSUserActivity class ExtensionDelegate: NSObject, WKExtensionDelegate { func handle(_ userActivity: NSUserActivity) { if userActivity.activityType == "com.myapp.LogMeal" { WKExtension.shared().rootInterfaceController?.popToRootController() WKExtension.shared().rootInterfaceController?.pushController(withName: "LogMealInterfaceController", context: userActivity.userInfo!["meal"]) } else if userActivity.activityType == "LogFavoriteIntent" { let interaction = userActivity!.interaction let intent = interaction.intent // restore } } } NEW
62. Handle Background Task • WKRelevantShortcutRefreshBackgroundTask • Refresh data. • WKIntentDidRunRefreshBackgroundTask • It will be called after intent execution. • Update UI for snapshots or complication.
63. INRelevanceProvider • Describe content relevance • Evaluated again time, location and routine • INDateRelevanceProvider • INLocationRelevanceProvider • INDailyRoutineRelevanceProvider • Provide array to INRelevantShortcut
64. INDateRelevanceProvider //Relevance Providers open class INDateRelevanceProvider : INRelevanceProvider { open var startDate: Date { get } open var endDate: Date? { get } public init(start startDate: Date, end endDate: Date?) } let startEvent = /* event start time */ let endEvent = /* event end time */ let relevanceProvider = INDateRelevanceProvider(start: startEvent, end: endEvent)
65. INLocationRelevanceProvider //Relevance Providers open class INLocationRelevanceProvider : INRelevanceProvider { @NSCopying open var region: CLRegion { get } public init(region: CLRegion) } let center = CLLocationCoordinate2D(latitude: 37.334728, longitude: -122.008883) let region = CLCircularRegion(center: center, radius: 2.0, identifier: "Apple Park") let relevanceProvider = INLocationRelevanceProvider(region: region)
66. INDailyRoutineRelevanceProvider //Relevance Providers open class INDailyRoutineRelevanceProvider : INRelevanceProvider { public enum Situation : Int { case morning case evening case home case work case school case gym } open var situation: Situation { get } public init(situation: Situation) } let relevanceProvider = INDailyRoutineRelevanceProvider(situation: .gym)
67. INDailyRoutineRelevanceProvider //Relevance Providers open class INDailyRoutineRelevanceProvider : INRelevanceProvider { public enum Situation : Int { case morning case evening case home case work case school case gym } open var situation: Situation { get } public init(situation: Situation) } let relevanceProvider = INDailyRoutineRelevanceProvider(situation: .gym)
68. Short Strings Limited space NSStringVariableWidthRuleType in stringsdict Options for context • 38mm: 20 • 42mm: 24
69. Summary • Define, Donate, Handle • Consider the informations you want to donate. • Enable the shortcut capability for your NSUserActivity and INIntent.
70. Q & A