ANDROID AT LARGE: HOW TO BRING OPTIMISED EXPERIENCES TO THE BIG SCREEN

With Android making its way to new form factors, and OEMs pushing the boundaries with new technology, it’s time to take a look at what these environments bring to you and your users. It’s crucial that your developer workflow accounts for resizability, new display sizes and aspect ratios, multi display, and app continuity to provide your users the best experience no matter where they are. The days of taking the easy road with screenOrientation="portrait" are quickly coming to an end.

In this talk we discuss the challenges faced when looking to support Android on various platforms, and how to make sure that you’re providing a great experience on all of these form-factors. You will leave this talk with an actionable checklist that helps stabilise your app and user experience on any device.

Transcript

1. Android at Large
   Optimized Experiences for the Big Screen
   Pietro F. Maggi
   @pfmaggi
   Kenneth Ford
   @KennethFSWE
   

   View Slide

2. Android comes in all sizes
   

   View Slide

3. Phones Tablets Foldable
   Phones
   Desktop
   Environments
   

   View Slide

4. One size does not fit all!
   Handle different screen sizes & aspect ratios
   

   View Slide

5. Different contexts of use
   Quick actions vs longer sessions
   Gaming, Media, Social,
   Productivity
   Immersive Quick actions Multi-tasking
   Communication, Music Productivity, Communication
   

   View Slide

6. App Continuity
   Configuration changes are unavoidable
   

   View Slide

7. One APK for all of these devices?
   Android App Bundles to the rescue
   

   View Slide

8. UX & UI
   considerations
   

   View Slide

9. New Aspect Ratios
   Test & support non-standard
   aspect ratios
   If necessary, use new minAspectRatio or existing
   maxAspectRatio attributes to restrict the ratios your
   app supports.
   

   View Slide

10. Layouts
    You have more space, use it!
    Think about how you can surface more information
    or make your users more efficient
    Larger screens provide the opportunity for more
    immersive experiences
    No one size fits all solution
    Check out our Material Studies for inspiration!
    material.io/design/material-studies/
    

    View Slide

11. material.io/design/material-studies/reply.html
    

    View Slide

12. material.io/design/material-studies/rally.html
    

    View Slide

13. Input
    Don’t assume touch input. Also true on phones
    (accessibility needs, desktop modes).
    Mouse + Keyboard =
    

    View Slide

14. KeyboardFocus.kt
    // Standard view
    myView.setFocusable(true)
    // Override default arrow mapping
    myView.setNextFocusLeftId(R.id.view_to_left)
    myView.setNextFocusRightId(R.id.view_to_right)
    myView.setNextFocusTopId(R.id.view_to_top)
    myView.setNextFocusBottomId(R.id.view_to_bottom)
    // Override default tab mapping
    myView.setNextFocusForwardId(R.id.next_view)
    

    View Slide

15. EnterKey.kt
    // Listen for release of enter key
    myView.setOnKeyListener { v, keyCode, event ->
    if (event.action == KeyEvent.ACTION_UP
    && keyCode == KeyEvent.KEYCODE_ENTER) {
    // Insert behavior here
    true
    } else {
    false
    }
    }
    

    View Slide

16. EnterKey.kt
    // Listen for release of enter key
    myView.setOnKeyListener { v, keyCode, event ->
    if (event.action == KeyEvent.ACTION_UP
    && keyCode == KeyEvent.KEYCODE_ENTER) {
    // Insert behavior here
    true
    } else {
    false
    }
    }
    

    View Slide

17. EnterKey.kt
    // Listen for release of enter key
    myView.setOnKeyListener { v, keyCode, event ->
    if (event.action == KeyEvent.ACTION_UP
    && keyCode == KeyEvent.KEYCODE_ENTER) {
    // Insert behavior here
    true
    } else {
    false
    }
    }
    

    View Slide

18. Ctrl_Z.kt
    override fun dispatchKeyShortcutEvent(event: KeyEvent?): Boolean {
    if (event?.keyCode == KeyEvent.KEYCODE_Z
    && event.hasModifiers(KeyEvent.META_CTRL_ON)) {
    // Undo action
    return true
    }
    return super.dispatchKeyShortcutEvent(event)
    }
    

    View Slide

19. Ctrl_Shift_Z.kt
    override fun dispatchKeyShortcutEvent(event: KeyEvent?): Boolean {
    if (event?.keyCode == KeyEvent.KEYCODE_Z
    && event.hasModifiers(KeyEvent.META_CTRL_ON |
    KeyEvent.META_SHIFT_ON)) {
    // Redo action
    return true
    }
    return super.dispatchKeyShortcutEvent(event)
    }
    

    View Slide

20. Pointers
    Long press action == right click actions
    onContextClickListener
    Change colors/elevation when hovering
    onHoverListener
    Update mouse pointers on hover
    View.pointerIcon
    

    View Slide

21. yourView.setOnHoverListener{ view, motionEvent ->
    when (motionEvent.actionMasked) {
    MotionEvent.ACTION_HOVER_ENTER -> {
    // UI Change
    }
    MotionEvent.ACTION_HOVER_EXIT -> {
    // Revert
    }
    }
    }
    Highlight Elements on Hover
    

    View Slide

22. yourView.setOnHoverListener { view, motionEvent ->
    view.pointerIcon =
    PointerIcon.getSystemIcon(context, PointerIcon.TYPE_HAND)
    false
    }
    Update Mouse Pointer Icons
    github.com/chromeos/pointer-icon-sample
    

    View Slide

23. App continuity
    

    View Slide

24. App Continuity
    Seamlessly continue what you’re doing!
    The ability to fold and unfold devices
    brings more spotlight to configuration
    changes and how your app handles them.
    Your app should restore the same state and
    location the user was in before folding or
    unfolding.
    

    View Slide

25. App Continuity
    Beyond foldables, the ability to save
    and restore state through configuration
    changes is a priority on all platforms.
    Correctly handling rotation or
    free-form resizing of windows not only
    requires the ability to save and restore
    state, but to be able to re-draw and
    re-layout with minimal jank.
    Beyond foldables, the ability to save
    and restore state through configuration
    changes is a priority on all platforms.
    

    View Slide

26. android:resizeableActivity
    Indicates whether an activity supports multi-window and
    multi-display environment.
    However, setting this to false does not mean the activity never
    needs to resize because of orientation changes, free-form resizing or
    display folding.
    

    View Slide

27. Activity on external
    screen when folded
    Compat mode
    Activity running in compat
    mode when unfolded
    Android 10 has a new
    compatibility mode for activities
    that support neither
    multi-window nor orientation
    changes.
    

    View Slide

28. Handling configuration changes
    Unfolding triggers a configuration change for smallestScreenSize, screenLayout
    and screenSize.
    ● Use onSaveInstanceState and ViewModel, or
    ● Handle the configuration change without restarting via handleConfigChange=”..”
    For the best experience, declare resizeableActivity=”true”.
    

    View Slide

29. Multi-Window
    

    View Slide

30. Android 10
    Resumed
    Resumed
    Resumed
    Multi-resume
    In multi-window, all top focusable
    activities in visible stacks are now in
    the RESUMED state.
    Activity can still end up in the
    PAUSED state if:
    ● There is a transparent activity on top
    ● It’s not currently focusable (PiP)
    In multi-window, all top focusable
    activities in visible stacks are now in
    the RESUMED state.
    

    View Slide

31. Resources
    While multiple apps are resumed,
    some will disconnect from available
    resources.
    Watch for camera availability callbacks
    to continue use.
    resizeableActivity=false does
    not guarantee camera access.
    While multiple apps are resumed,
    some will disconnect from available
    resources.
    

    View Slide

32. protected void onTopResumedActivityChanged(boolean topResumed) {
    if (topResumed) {
    // Top resumed activity
    // Can be a signal to re-acquire exclusive resources
    } else {
    // No longer the top resumed activity
    }
    }
    New lifecycle callbacks for top-resumed
    

    View Slide

33. Drag-n-drop
    With multi-window becoming
    more common, consider adding
    drag-n-drop support for text and
    images.
    

    View Slide

34. Multi-Display
    

    View Slide

35. Activities
    on secondary screens
    Keep in mind that the following
    things may occur:
    ● Context update
    ● Window resize
    ● Resource Changes
    

    View Slide

36. // Get current display
    val activityDisplay = activity.windowManager.defaultDisplay
    // Get current window metrics
    val windowMetrics = DisplayMetrics()
    activityDisplay.getMetrics(windowMetrics)
    // … or
    val windowMetrics = activity.resources.displayMetrics
    Activity vs Application context
    Context contains information about display
    // Get current display
    val activityDisplay = activity.windowManager.defaultDisplay
    // Get current window metrics
    val windowMetrics = DisplayMetrics()
    activityDisplay.getMetrics(windowMetrics)
    // … or
    val windowMetrics = activity.resources.displayMetrics
    // Show toast on the current display
    Toast.makeText(activity, text, duration).show()
    

    View Slide

37. Configuration changes
    If activity handles configuration change, it will be notified
    onConfigurationChanged(newConfig: Configuration?)
    If it doesn’t, it will be relaunched.
    onCreate and onConfigurationChanged are good points to check what is the
    current display for an activity.
    android:configChanges="density|orientation|screenLayout
    |screenSize|smallestScreenSize
    |touchscreen"
    

    View Slide

38. Multiple instances
    FLAG_ACTIVITY_NEW_TASK|FLAG_ACTIVITY_MULTIPLE_TASK
    

    View Slide

39. Multiple instances
    FLAG_ACTIVITY_NEW_TASK|FLAG_ACTIVITY_MULTIPLE_TASK
    Resumed
    Resumed
    

    View Slide

40. // Get available displays
    val dm = getSystemService(Context.DISPLAY_SERVICE) as DisplayManager
    val displays = dm.displays
    // Check their characteristics
    Display.flags // To check if it’s a secure display, etc.
    Display.metrics // To get the size, resolution and density
    Display.state // To check if it’s ON/OFF
    Using secondary screens
    

    View Slide

41. // Check if launch is allowed
    activityManager.isActivityStartAllowedOnDisplay(context, displayId, intent)
    // Launch on specific display
    val options = ActivityOptions.makeBasic()
    options.launchDisplayId = targetDisplay.displayId
    startActivity(intent, options.toBundle())
    Using secondary screens
    

    View Slide

42. Multiple display densities
    ●
    ●
    

    View Slide

43. android {
    bundle {
    language {
    enableSplit = true // true by default.
    }
    density {
    // The app bundle should not support configuration APKs for
    // display densities. These resources are instead packaged
    // with each base and dynamic feature APK.
    enableSplit = false
    }
    abi {
    enableSplit = true // true by default.
    }
    }
    }
    Android App Bundles and multiple-display
    

    View Slide

44. Launchers
    android.intent.category.SECONDARY_HOME
    

    View Slide

45. Jetpack
    Window Manager
    

    View Slide

46. Jetpack Window Manager
    Provide a single API surface for
    different types of foldable devices
    coming to the market.
    Target the entire category,
    not a single model.
    

    View Slide

47. Features
    @IntDef({
    // A fold on the screen without a physical gap.
    TYPE_FOLD,
    // A physical separation with a hinge that
    // allows two display panels to fold.
    TYPE_HINGE,
    })
    public @interface Type{}
    github.com/android/user-interface-samples/tree/master/WindowManager
    

    View Slide

48. Postures
    @IntDef({
    POSTURE_UNKNOWN,
    POSTURE_CLOSED,
    POSTURE_HALF_OPENED,
    POSTURE_OPENED,
    POSTURE_FLIPPED
    })
    public @interface Posture{}
    github.com/android/user-interface-samples/tree/master/WindowManager
    

    View Slide

49. Usage
    dependencies {
    implementation "androidx.window:window:1.0.0-alpha01"
    }
    github.com/android/user-interface-samples/tree/master/WindowManager
    var windowManager = WindowManager(
    this /* context */,
    null /* windowBackend */)
    val displayFeatures =
    windowManager.windowLayoutInfo.displayFeatures
    windowManager.registerDeviceStateChangeCallback(
    mainThreadExecutor /* Executor */,
    callback /* Consumer */)
    

    View Slide

50. Mocking - WindowBackend
    class MidScreenFoldBackend : WindowBackend {
    override fun getDeviceState() = { /* */ }
    override fun getWindowLayoutInfo(context: Context): WindowLayoutInfo { /* */ }
    override fun registerDeviceStateChangeCallback(
    executor: Executor,
    callback: Consumer
    ) { /* */ }
    override fun unregisterDeviceStateChangeCallback(callback: Consumer) { /* */ }
    override fun registerLayoutChangeCallback(
    context: Context,
    executor: Executor,
    callback: Consumer
    ) { /* */ }
    override fun unregisterLayoutChangeCallback(callback: Consumer) { /* */ }
    }
    github.com/android/user-interface-samples/tree/master/WindowManager
    var windowManager = WindowManager(
    this /* context */,
    null /* windowBackend */)
    

    View Slide

51. Testing
    

    View Slide

52. Android Emulator
    Multi-Touch
    Multi-core support
    Quick Boot
    OpenGL ES 3.0
    Multiple instances
    Display cutout
    Battery mode
    Headless build
    

    View Slide

53. Size Resolution
    Unfolded 8” 2200x2480
    Folded 6.6” 1148x2480
    Size Resolution
    Unfolded 7.3” 1536x2152
    Folded 4.6” 840x1960
    7.3”
    7.3” and 8” Emulators
    8”
    Android Emulator
    

    View Slide

54. App continuity Compat mode Multi-resume
    Emulator and Android 10
    

    View Slide

55. 6.7” Horizontal Fold-in
    Android Emulator
    

    View Slide

56. 7.4” Rollable
    Android Emulator
    androidstudio.googleblog.com/2020/09/emulator-3013-canary.html
    

    View Slide

57. 13.5” Freeform
    Android Emulator
    

    View Slide

58. Multiple Displays
    Android Emulator
    

    View Slide

59. ...and More
    Developing for Android 11 with the Android Emulator
    medium.com/androiddevelopers/developing-for-android-11-with-the-android-emulator-a9486af2d7ef
    

    View Slide

60. Thank You!
    Pietro F. Maggi
    @pfmaggi
    

    View Slide