ANDROID AT LARGE: HOW TO BRING OPTIMISED EXPERIENCES TO THE BIG SCREEN

With Android making its way to new form factors, and OEMs pushing the boundaries with new technology, it’s time to take a look at what these environments bring to you and your users. It’s crucial that your developer workflow accounts for resizability, new display sizes and aspect ratios, multi display, and app continuity to provide your users the best experience no matter where they are. The days of taking the easy road with screenOrientation="portrait" are quickly coming to an end.

In this talk we discuss the challenges faced when looking to support Android on various platforms, and how to make sure that you’re providing a great experience on all of these form-factors. You will leave this talk with an actionable checklist that helps stabilise your app and user experience on any device.

Transcript

1. Android at Large Optimized Experiences for the Big Screen Pietro F. Maggi @pfmaggi Kenneth Ford @KennethFSWE
2. Android comes in all sizes
3. Phones Tablets Foldable Phones Desktop Environments
4. One size does not fit all! Handle different screen sizes & aspect ratios
5. Different contexts of use Quick actions vs longer sessions Gaming, Media, Social, Productivity Immersive Quick actions Multi-tasking Communication, Music Productivity, Communication
6. App Continuity Configuration changes are unavoidable
7. One APK for all of these devices? Android App Bundles to the rescue
8. UX & UI considerations
9. New Aspect Ratios Test & support non-standard aspect ratios If necessary, use new minAspectRatio or existing maxAspectRatio attributes to restrict the ratios your app supports.
10. Layouts You have more space, use it! Think about how you can surface more information or make your users more efficient Larger screens provide the opportunity for more immersive experiences No one size fits all solution Check out our Material Studies for inspiration! material.io/design/material-studies/
11. material.io/design/material-studies/reply.html
12. material.io/design/material-studies/rally.html
13. Input Don’t assume touch input. Also true on phones (accessibility needs, desktop modes). Mouse + Keyboard =
14. KeyboardFocus.kt // Standard view myView.setFocusable(true) // Override default arrow mapping myView.setNextFocusLeftId(R.id.view_to_left) myView.setNextFocusRightId(R.id.view_to_right) myView.setNextFocusTopId(R.id.view_to_top) myView.setNextFocusBottomId(R.id.view_to_bottom) // Override default tab mapping myView.setNextFocusForwardId(R.id.next_view)
15. EnterKey.kt // Listen for release of enter key myView.setOnKeyListener { v, keyCode, event -> if (event.action == KeyEvent.ACTION_UP && keyCode == KeyEvent.KEYCODE_ENTER) { // Insert behavior here true } else { false } }
16. EnterKey.kt // Listen for release of enter key myView.setOnKeyListener { v, keyCode, event -> if (event.action == KeyEvent.ACTION_UP && keyCode == KeyEvent.KEYCODE_ENTER) { // Insert behavior here true } else { false } }
17. EnterKey.kt // Listen for release of enter key myView.setOnKeyListener { v, keyCode, event -> if (event.action == KeyEvent.ACTION_UP && keyCode == KeyEvent.KEYCODE_ENTER) { // Insert behavior here true } else { false } }
18. Ctrl_Z.kt override fun dispatchKeyShortcutEvent(event: KeyEvent?): Boolean { if (event?.keyCode == KeyEvent.KEYCODE_Z && event.hasModifiers(KeyEvent.META_CTRL_ON)) { // Undo action return true } return super.dispatchKeyShortcutEvent(event) }
19. Ctrl_Shift_Z.kt override fun dispatchKeyShortcutEvent(event: KeyEvent?): Boolean { if (event?.keyCode == KeyEvent.KEYCODE_Z && event.hasModifiers(KeyEvent.META_CTRL_ON | KeyEvent.META_SHIFT_ON)) { // Redo action return true } return super.dispatchKeyShortcutEvent(event) }
20. Pointers Long press action == right click actions onContextClickListener Change colors/elevation when hovering onHoverListener Update mouse pointers on hover View.pointerIcon
21. yourView.setOnHoverListener{ view, motionEvent -> when (motionEvent.actionMasked) { MotionEvent.ACTION_HOVER_ENTER -> { // UI Change } MotionEvent.ACTION_HOVER_EXIT -> { // Revert } } } Highlight Elements on Hover
22. yourView.setOnHoverListener { view, motionEvent -> view.pointerIcon = PointerIcon.getSystemIcon(context, PointerIcon.TYPE_HAND) false } Update Mouse Pointer Icons github.com/chromeos/pointer-icon-sample
23. App continuity
24. App Continuity Seamlessly continue what you’re doing! The ability to fold and unfold devices brings more spotlight to configuration changes and how your app handles them. Your app should restore the same state and location the user was in before folding or unfolding.
25. App Continuity Beyond foldables, the ability to save and restore state through configuration changes is a priority on all platforms. Correctly handling rotation or free-form resizing of windows not only requires the ability to save and restore state, but to be able to re-draw and re-layout with minimal jank. Beyond foldables, the ability to save and restore state through configuration changes is a priority on all platforms.
26. android:resizeableActivity Indicates whether an activity supports multi-window and multi-display environment. However, setting this to false does not mean the activity never needs to resize because of orientation changes, free-form resizing or display folding.
27. Activity on external screen when folded Compat mode Activity running in compat mode when unfolded Android 10 has a new compatibility mode for activities that support neither multi-window nor orientation changes.
28. Handling configuration changes Unfolding triggers a configuration change for smallestScreenSize, screenLayout and screenSize. • Use onSaveInstanceState and ViewModel, or • Handle the configuration change without restarting via handleConfigChange=”..” For the best experience, declare resizeableActivity=”true”.
29. Multi-Window
30. Android 10 Resumed Resumed Resumed Multi-resume In multi-window, all top focusable activities in visible stacks are now in the RESUMED state. Activity can still end up in the PAUSED state if: • There is a transparent activity on top • It’s not currently focusable (PiP) In multi-window, all top focusable activities in visible stacks are now in the RESUMED state.
31. Resources While multiple apps are resumed, some will disconnect from available resources. Watch for camera availability callbacks to continue use. resizeableActivity=false does not guarantee camera access. While multiple apps are resumed, some will disconnect from available resources.
32. protected void onTopResumedActivityChanged(boolean topResumed) { if (topResumed) { // Top resumed activity // Can be a signal to re-acquire exclusive resources } else { // No longer the top resumed activity } } New lifecycle callbacks for top-resumed
33. Drag-n-drop With multi-window becoming more common, consider adding drag-n-drop support for text and images.
34. Multi-Display
35. Activities on secondary screens Keep in mind that the following things may occur: • Context update • Window resize • Resource Changes
36. // Get current display val activityDisplay = activity.windowManager.defaultDisplay // Get current window metrics val windowMetrics = DisplayMetrics() activityDisplay.getMetrics(windowMetrics) // … or val windowMetrics = activity.resources.displayMetrics Activity vs Application context Context contains information about display // Get current display val activityDisplay = activity.windowManager.defaultDisplay // Get current window metrics val windowMetrics = DisplayMetrics() activityDisplay.getMetrics(windowMetrics) // … or val windowMetrics = activity.resources.displayMetrics // Show toast on the current display Toast.makeText(activity, text, duration).show()
37. Configuration changes If activity handles configuration change, it will be notified onConfigurationChanged(newConfig: Configuration?) If it doesn’t, it will be relaunched. onCreate and onConfigurationChanged are good points to check what is the current display for an activity. android:configChanges="density|orientation|screenLayout |screenSize|smallestScreenSize |touchscreen"
38. Multiple instances FLAG_ACTIVITY_NEW_TASK|FLAG_ACTIVITY_MULTIPLE_TASK
39. Multiple instances FLAG_ACTIVITY_NEW_TASK|FLAG_ACTIVITY_MULTIPLE_TASK Resumed Resumed
40. // Get available displays val dm = getSystemService(Context.DISPLAY_SERVICE) as DisplayManager val displays = dm.displays // Check their characteristics Display.flags // To check if it’s a secure display, etc. Display.metrics // To get the size, resolution and density Display.state // To check if it’s ON/OFF Using secondary screens
41. // Check if launch is allowed activityManager.isActivityStartAllowedOnDisplay(context, displayId, intent) // Launch on specific display val options = ActivityOptions.makeBasic() options.launchDisplayId = targetDisplay.displayId startActivity(intent, options.toBundle()) Using secondary screens
42. Multiple display densities • •
43. android { bundle { language { enableSplit = true // true by default. } density { // The app bundle should not support configuration APKs for // display densities. These resources are instead packaged // with each base and dynamic feature APK. enableSplit = false } abi { enableSplit = true // true by default. } } } Android App Bundles and multiple-display
44. Launchers android.intent.category.SECONDARY_HOME
45. Jetpack Window Manager
46. Jetpack Window Manager Provide a single API surface for different types of foldable devices coming to the market. Target the entire category, not a single model.
47. Features @IntDef({ // A fold on the screen without a physical gap. TYPE_FOLD, // A physical separation with a hinge that // allows two display panels to fold. TYPE_HINGE, }) public @interface Type{} github.com/android/user-interface-samples/tree/master/WindowManager
48. Postures @IntDef({ POSTURE_UNKNOWN, POSTURE_CLOSED, POSTURE_HALF_OPENED, POSTURE_OPENED, POSTURE_FLIPPED }) public @interface Posture{} github.com/android/user-interface-samples/tree/master/WindowManager
49. Usage dependencies { implementation "androidx.window:window:1.0.0-alpha01" } github.com/android/user-interface-samples/tree/master/WindowManager var windowManager = WindowManager( this /* context */, null /* windowBackend */) val displayFeatures = windowManager.windowLayoutInfo.displayFeatures windowManager.registerDeviceStateChangeCallback( mainThreadExecutor /* Executor */, callback /* Consumer<DeviceState> */)
50. Mocking - WindowBackend class MidScreenFoldBackend : WindowBackend { override fun getDeviceState() = { /* */ } override fun getWindowLayoutInfo(context: Context): WindowLayoutInfo { /* */ } override fun registerDeviceStateChangeCallback( executor: Executor, callback: Consumer<DeviceState> ) { /* */ } override fun unregisterDeviceStateChangeCallback(callback: Consumer<DeviceState>) { /* */ } override fun registerLayoutChangeCallback( context: Context, executor: Executor, callback: Consumer<WindowLayoutInfo> ) { /* */ } override fun unregisterLayoutChangeCallback(callback: Consumer<WindowLayoutInfo>) { /* */ } } github.com/android/user-interface-samples/tree/master/WindowManager var windowManager = WindowManager( this /* context */, null /* windowBackend */)
51. Testing
52. Android Emulator Multi-Touch Multi-core support Quick Boot OpenGL ES 3.0 Multiple instances Display cutout Battery mode Headless build
53. Size Resolution Unfolded 8” 2200x2480 Folded 6.6” 1148x2480 Size Resolution Unfolded 7.3” 1536x2152 Folded 4.6” 840x1960 7.3” 7.3” and 8” Emulators 8” Android Emulator
54. App continuity Compat mode Multi-resume Emulator and Android 10
55. 6.7” Horizontal Fold-in Android Emulator
56. 7.4” Rollable Android Emulator androidstudio.googleblog.com/2020/09/emulator-3013-canary.html
57. 13.5” Freeform Android Emulator
58. Multiple Displays Android Emulator
59. ...and More Developing for Android 11 with the Android Emulator medium.com/androiddevelopers/developing-for-android-11-with-the-android-emulator-a9486af2d7ef
60. Thank You! Pietro F. Maggi @pfmaggi