Rethinking the Mobile App: Dual Screen and Multi-Posture

Mobile devices today come in different form factors - tablets, phones, watches, even televisions. We design our apps to adapt to these device affordances but we assume that each device has a fixed form factor and capability. What if you had a mobile device that could fold into a phone, open up as a book, fold out like a laptop or fold over like a tent? What new experiences can we design with dual screens and different device postures?

In this talk I'll introduce you to the features and functionality available in the Microsoft Surface Duo and talk about how you can adapt your existing apps to leverage the power of flexible layouts that can span screens or adapt to user postures. We'll look at how you can get started building apps for the device, and end by discussing ways in which we could rethink mobile user experiences with the emergence of new device modalities and affordances.

Event Site:
https://emamo.com/event/android-summit-2020/s/dual-screen-multi-posture-developing-new-mobile-experiences-aL4Kxa

Transcript

1. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   D U A L
   S C R E E N
   & M U L T I
   P O S T U R E
   R E T H I N K I N G T H E
   M O B I L E A P P
   Nitya Narasimhan, PhD
   Senior Cloud Advocate
   Microsoft
   #SurfaceDuoDev @nitya
   https://aka.ms/androidsummit-2020
   

   View Slide

2. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   A B O U T M E
   • Mobile Research
   PhD & Motorola Labs
   • Mobile Development
   Startups & Consulting
   • Mobile Teaching
   University & Community (8+ years)
   • Mobile Advocacy
   Microsoft Developer Relations
   #SurfaceDuoDev @nitya
   https://aka.ms/androidsummit-2020
   

   View Slide

3. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   Foldable
   Devices
   Dual Screen
   Devices
   Android For
   Foldables
   Android For
   Multi-Screen
   Design Considerations
   Microsoft
   Surface Duo
   API
   Dual Screen Libraries
   Emulator Design Patterns
   Code Samples
   Surface Duo SDK
   Jetpack
   Window Manager
   Development Options
   Microsoft Docs
   Surface Duo Development
   Microsoft Learn
   Surface Duo Dev Blog
   Surface Duo Forum
   Android Developers
   dev.to Community
   Tech Community
   A W A R E N E S S | A C T I O N | A N D R O I D
   

   View Slide

4. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   F O L DA B L E
   D E V I C E S
   & D U A L
   S C R E E N S
   W H A T A R E T H E Y ?
   W H Y D O T H E Y M A T T E R ?
   https://aka.ms/chicagoroboto
   https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   

   View Slide

5. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   F O L DA B L E = P O R TA B L E + F L E X I B L E !
   Ref: Android Developers
   Building Apps For Foldables
   Ref: Microsoft Developers
   Surface Duo Overview
   Portable device +
   rethink form factor!
   Multi-display option
   (single vs. dual)
   Multi-posture
   (flexible modes)
   Opportunity:
   new, enhanced UX
   Challenge:
   app continuity, state
   

   View Slide

6. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   W H AT I S " D U A L S C R E E N " ?
   Challenge: Orientation modes +
   postures drive many configuration
   changes. Handling visible "seams"
   Devices with two distinct screens
   (multi-display, ideally symmetric)
   Emphasis on user productivity
   (side-by-side, drag-and-drop)
   Opportunity: Flexibility in usage
   (single, spanned, paired apps)
   Ref: Microsoft Developers | Surface Duo Overview
   

   View Slide

7. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   A N D R O I D
   A P P S &
   F O L DA B L E S,
   M U L T I - S C R E E N
   D E S I G N C O N S I D E R A T I O N S
   M U L T I - S C R E E N S U P P O R T
   J E T P A C K W I N D O W M A N A G E R
   https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya
   https://aka.ms/androidsummit-2020
   

   View Slide

8. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   A N D R O I D & F O L DA B L E S / E V O L U T I O N
   Pervasive, Multi-Screen Apps
   Android Dev Summit, 2018
   "Screen Continuity UX"
   Build Apps for Foldable .. Devices
   Google I/O, 2019
   "Multi-Display Apps"
   Screens: Large, Small & Foldable
   11 Weeks Of Android, 2020
   "WindowManager API"
   

   View Slide

9. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
   A N D R O I D & M U L T I - S C R E E N / S U P P O R T
   Ref: Android Developers
   Building Apps For Foldables
   Multi-Display
   Multi-Resume, ResizeableActivity
   Ref: Android Developers / Medium
   Support new form factors.. | Foldables
   Multi-Configuration
   Device State, Display Features
   Ref: Android Developers
   Multi-Window Support
   Multi-Window
   Split Screen, PIP
   ANDROID 7
   API 24
   ANDROID 10
   API 29
   ANDROID 11?
   (2020 – alpha release)
   Closed
   Flipped
   Open
   Half-Opened
   

   View Slide

10. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    What was the user's INTENT
    in folding, flipping, or rotating the device?
    What was the user's EXPERIENCE,
    and did app behavior fulfill that intent?
    How can new device AFFORDANCES
    help up rethink mobile app UX?
    F O L DA B L E S & M U L T I - S C R E E N / E X P L O R E
    

    View Slide

11. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    APPLICATION
    CONTINUITY
    MULTI-WINDOW
    SUPPORT
    MULTI-DISPLAY
    SUPPORT
    TESTING
    SUPPORT
    DESIGN
    PATTERNS
    Folding or unfolding a device can cause apps to transition from one screen to another.
    Ensure consistent state. Adopt responsive layouts.
    Use resizeableActvity flag to indicate level of app support. Adopt drag-and-drop for better
    UX. Gracefully handle resource loss in multi-resume
    Activities can launch on second screen. Check your default display after creation or
    configuration change events. Adapt layout or behavior to suit
    Use foldable device emulators.Test app behavior under configuration changes (posture,
    orientation), multi-display (second screen) scenarios
    Look for general best practices and device-specific recommendations for handling postures
    (hinge), occlusion (seam), layouts (config changes)
    Ref: Android Developers
    Building Apps For Foldables
    F O L DA B L E S & A P P D E S I G N / G U I D A N C E
    

    View Slide

12. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    F O L DA B L E S & A P P T E S T I N G / E M U L A T O R S
    Ref: Android Developers
    Testing / Foldable Emulators
    Test default & developed responses
    to fold/unfold & rotation events
    

    View Slide

13. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    How do I know display
    features, device state and
    related change events?
    Ref: Android Developers
    Building Apps For Foldables
    

    View Slide

14. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    J E T PA C K W I N D O W M A N A G E R / A B S T R A C T
    provide a single API surface for different types of foldable
    devices coming to the market, so that application developers could
    target entire categories of devices instead of a single model.
    • Sample: WindowManager
    • Usage: Medium Post
    • With Surface Duo (DevBlog)
    V 1.0.0-alpha01
    Android Developers
    How do I know display
    features, device state and
    related change events?
    

    View Slide

15. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    S U R FAC E
    D U O &
    D E V E L O P M E N T
    G U I DA N C E
    I N S T A L L I N G T H E S D K
    E X P L O R I N G T H E E M U L A T O R
    U N P A C K I N G T H E A P I & L I B R A R I E S
    https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya
    https://aka.ms/androidsummit-2020
    

    View Slide

16. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    H E L L O, S U R FA C E D U O !
    Ref: Microsoft Website
    Surface Duo: Tech Specs
    Thin
    Lightweight
    Portable
    Foldable
    360 rotation
    Multiple postures
    Dual Screen
    Symmetric screens
    "Spanned Apps"
    Productivity
    Android + M365
    "App Groups"
    Seam
    

    View Slide

17. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    A P P D E V E L O P M E N T O P T I O N S
    Android
    Developers
    Kotlin
    Java
    Cross-Platform
    Developers
    React Native
    Xamarin
    Flutter
    Other
    Developers
    Unity (gaming)
    JavaScript (web)
    Ref: Microsoft GitHub
    Surface Duo: SDK & Samples
    

    View Slide

18. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    D E V I C E / C O N F I G U R A T I O N F A C T O R S
    Ref: Microsoft Docs
    Dual Screen: Introduction
    • Hinge: 0-360 degrees
    • Postures: 0-16 discrete levels
    ▪ Peek
    ▪ Book (Compose)
    ▪ Tent (Flip)
    ▪ Flat
    • Screens: Single or Dual
    • Orientation: Portrait or Landscape
    

    View Slide

19. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    A P P L I C AT I O N / B E H AV I O R S
    • Default: Apps open on 1 screen
    • Multi-Display: App can launch
    activity on 2nd screen (if available)
    • Move L/R: App can be moved from
    one screen to other
    • Multi-Instance: If supported, app
    instances can run side-by-side
    • Span: App can be displayed across both
    screens (on user request only)
    Ref: Microsoft Docs
    Dual Screen: Introduction
    

    View Slide

20. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    D E S I G N G U I DA N C E / A N Y F O L D A B L E
    1. Use Responsive Layouts: scale (to fit) or create custom layouts (to enhance)
    2. Support All Configurations: are users adopting new orientations, postures?
    3. Support Diverse Inputs: touch, pen, type, posture? (learn user preferences)
    4. Use Drag-And-Drop: particularly suited for dual-screen (give user feedback!)
    5. Support Multi-Instance: allow side-by-side app activation (free enhanced UX)
    6. Learn User Intent: when/why did user span app? change posture? orientation?
    Always leave users in control (avoid unpredictable effects or automation without consent)
    

    View Slide

21. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    D E S I G N G U I DA N C E / ' S E A M ' S T R A T E G I E S
    1. Do Nothing: test app UX – is behavior acceptable? (e.g., can slide maps for view)
    2. Avoid It: change dropdown directions, move to one screen, flip orientation etc.
    3. Snap To It: restructure grid-based views to have even columns (seam falls in middle)
    4. Mask It: if content, render under seam (human eye auto-magically connects the dots)
    5. Split It: if possible, break content into two and render each part separately (left, right)
    6. Embrace It: seams can provide a natural separation of concerns that can reduce
    user cognitive overload for multi-tasking. See design patterns when creating new apps!
    

    View Slide

22. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    D E V E L O P M E N T G U I DA N C E / I T E R A T I V E
    ❶ Download
    SDK/Emulator
    ❸ Adopt Android
    Best Practices
    For Foldables
    ❷ Test Default
    UX on Emulator
    ❹ Explore Device
    Design Patterns
    for Enhanced UX
    

    View Slide

23. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    G E T T I N G S TA R T E D / I N S T A L L S D K
    Ref: Microsoft.com
    Surface Duo SDK: Download
    Download Surface Duo SDK
    (Preview) Contains the Dual Screen
    libraries, emulator and relevant scripts
    Run the Surface Duo Emulator
    (Manual) Is now discoverable as a target
    device on Android Studio
    Get Surface Duo SDK Samples
    (Optional) Build and run samples in emulator
    to explore dual screen design patterns with code
    

    View Slide

24. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    S D K E M U L AT O R / U S I N G & T E S T I N G
    Simulate "spanning"
    your app for testing
    Simulate "hinge angle"
    change events using
    pressure sensors
    Simulate "posture"
    change events using
    humidity sensor
    0 - 360 0 - 16
    Test for orientation
    change effects by
    simulating rotation
    

    View Slide

25. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    S D K L I B R A R I E S / C O R E , L AY O U T, O T H E R
    CORE SDK
    ScreenHelper
    SurfaceDuoScreenManager
    BOTTOM NAVIGATION
    SurfaceDuoBottomNavigationView
    FRAGMENTS HANDLER
    FragmentManagerStateHandler
    LAYOUTS
    SurfaceDuoLayout
    SurfaceDuoFrameLayout
    Get screen (mode) info, register
    for mode change events **
    Bottom Nav helper, support to
    move view to left or right screen
    Handles fragments in different
    screen modes – restore correct
    fragments to match transition
    Containers to simplify layout
    and render with 1-screen
    and 2-screen child views
    

    View Slide

26. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    S U R FAC E
    D U O &
    D E S I G N
    PAT T E R N S
    L E A R N I N G F I V E D E S I G N P A T T E R N S
    E X P L O R I N G T H E C O D E S A M P L E S
    https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya
    https://aka.ms/androidsummit-2020
    

    View Slide

27. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    D E S I G N PAT T E R N S + C O D E S A M P L E S
    Ref: Microsoft Docs
    Dual Screen App Patterns
    

    View Slide

28. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    PAT T E R N / E X T E N D E D C A N VA S
    Ref: Microsoft Docs
    Sample: Extended Canvas
    Sample App - FrameLayout with a map view. Spanning 'extends canvas' to cover full area.
    Seam Strategy - Do nothing! Canvas can be scrolled to reveal occluded regions.
    

    View Slide

29. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    PAT T E R N / L I S T- D E T A I L
    Ref: Microsoft Docs
    Sample: List-Detail
    Sample App - List-Detail app. Shows usage of SurfaceDuoLayout & ScreenHelper (Core) libraries
    Seam Strategy - Split it! App lends itself naturally to rendering collection+content on dual screens
    

    View Slide

30. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    PAT T E R N / T W O PA G E
    Ref: Microsoft Docs
    Sample: Two Page
    Sample App - Book-like app. Usage of ViewPager & handling rotation events (vertical scroll)
    Seam Strategy - Embrace it! Seam enhances familiar "page-turning" UX, easy back/forth navigation
    

    View Slide

31. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    PAT T E R N / D U A L V I E W
    Ref: Microsoft Docs
    Sample: Dual View
    Sample App - Coordinated content experiences. (can also use multi-instance to emulate behavior)
    Seam Strategy - Embrace it! Create clear "compare & contrast" with two perspectives of same data
    

    View Slide

32. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    PAT T E R N / C O M PA N I O N P A N E
    Ref: Microsoft Docs
    Sample: Photo Editor App
    Sample App - Augmented content experiences. (Use added real estate to add 'extra' capability)
    Seam Strategy - Embrace it! Elevate second level content or controls to main display to enhance UX
    

    View Slide

33. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    H E R O A P P S / C O M B I N E PA T T E R N S I N U S E
    Ref: Microsoft Docs
    Surface Duo App Samples
    Source Editor Drag-and-Drop, Single Screen
    Mode, Dual Screen Mode, Mirrored Scrolling
    TwoNote Drag-and-Drop (Text & Images), Pen
    Events, List-Detail, Extended Canvas etc.
    

    View Slide

34. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    R E S O U R C E S / C O N T E N T , C O D E & C O M M U N I T Y
    1. Microsoft Docs: Introduction to Dual Screen
    2. Surface Duo: Developer Blog
    3. Surface Duo: SDK (Download)
    4. Surface Duo: SDK (Source)
    5. Surface Duo: Code Samples
    6. Surface Duo: Community Forum
    7. Android Developers: Foldables
    

    View Slide

35. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    R E C A P / A W A R E N E S S - A C T I O N - A N D R O I D
    Postures
    Displays
    Seams
    Focus on Productivity
    Surface Duo SDK
    Emulator
    Libraries
    Affordances
    App Layouts
    Device States
    Design Patterns
    Focus on UX Continuity
    Design Patterns
    Enhanced Canvas
    List-Detail
    Two Page
    Dual View
    Companion Pane
    Focus on Development
    Content Portals | Code Samples |Community Forum
    Focus on Learning
    

    View Slide

36. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya
    T H A N K Y O U !
    F E E D B A C K :
    aka.ms/androidsummit-
    2020*
    A W A R E N E S S
    A C T I O N
    A N D R O I D
    https://aka.ms/androidsummit-2020
    

    View Slide

37. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
    D U A L
    S C R E E N
    & M U L T I
    P O S T U R E
    R E T H I N K I N G T H E
    M O B I L E A P P
    Nitya Narasimhan, PhD
    Senior Cloud Advocate
    Microsoft
    #SurfaceDuoDev @nitya
    https://aka.ms/androidsummit-2020
    THANKS FOR
    LISTENING!
    

    View Slide