Rethinking the Mobile App: Dual Screen and Multi-Posture

Mobile devices today come in different form factors - tablets, phones, watches, even televisions. We design our apps to adapt to these device affordances but we assume that each device has a fixed form factor and capability. What if you had a mobile device that could fold into a phone, open up as a book, fold out like a laptop or fold over like a tent? What new experiences can we design with dual screens and different device postures?

In this talk I'll introduce you to the features and functionality available in the Microsoft Surface Duo and talk about how you can adapt your existing apps to leverage the power of flexible layouts that can span screens or adapt to user postures. We'll look at how you can get started building apps for the device, and end by discussing ways in which we could rethink mobile user experiences with the emergence of new device modalities and affordances.

Event Site:
https://emamo.com/event/android-summit-2020/s/dual-screen-multi-posture-developing-new-mobile-experiences-aL4Kxa

Transcript

1. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D U A L S C R E E N & M U L T I P O S T U R E R E T H I N K I N G T H E M O B I L E A P P Nitya Narasimhan, PhD Senior Cloud Advocate Microsoft #SurfaceDuoDev @nitya https://aka.ms/androidsummit-2020
2. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya A B O U T M E • Mobile Research PhD & Motorola Labs • Mobile Development Startups & Consulting • Mobile Teaching University & Community (8+ years) • Mobile Advocacy Microsoft Developer Relations #SurfaceDuoDev @nitya https://aka.ms/androidsummit-2020
3. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya Foldable Devices Dual Screen Devices Android For Foldables Android For Multi-Screen Design Considerations Microsoft Surface Duo API Dual Screen Libraries Emulator Design Patterns Code Samples Surface Duo SDK Jetpack Window Manager Development Options Microsoft Docs Surface Duo Development Microsoft Learn Surface Duo Dev Blog Surface Duo Forum Android Developers dev.to Community Tech Community A W A R E N E S S | A C T I O N | A N D R O I D
4. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya F O L DA B L E D E V I C E S & D U A L S C R E E N S W H A T A R E T H E Y ? W H Y D O T H E Y M A T T E R ? https://aka.ms/chicagoroboto https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya
5. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya F O L DA B L E = P O R TA B L E + F L E X I B L E ! Ref: Android Developers Building Apps For Foldables Ref: Microsoft Developers Surface Duo Overview Portable device + rethink form factor! Multi-display option (single vs. dual) Multi-posture (flexible modes) Opportunity: new, enhanced UX Challenge: app continuity, state
6. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya W H AT I S " D U A L S C R E E N " ? Challenge: Orientation modes + postures drive many configuration changes. Handling visible "seams" Devices with two distinct screens (multi-display, ideally symmetric) Emphasis on user productivity (side-by-side, drag-and-drop) Opportunity: Flexibility in usage (single, spanned, paired apps) Ref: Microsoft Developers | Surface Duo Overview
7. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya A N D R O I D A P P S & F O L DA B L E S, M U L T I - S C R E E N D E S I G N C O N S I D E R A T I O N S M U L T I - S C R E E N S U P P O R T J E T P A C K W I N D O W M A N A G E R https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya https://aka.ms/androidsummit-2020
8. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya A N D R O I D & F O L DA B L E S / E V O L U T I O N Pervasive, Multi-Screen Apps Android Dev Summit, 2018 "Screen Continuity UX" Build Apps for Foldable .. Devices Google I/O, 2019 "Multi-Display Apps" Screens: Large, Small & Foldable 11 Weeks Of Android, 2020 "WindowManager API"
9. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya A N D R O I D & M U L T I - S C R E E N / S U P P O R T Ref: Android Developers Building Apps For Foldables Multi-Display Multi-Resume, ResizeableActivity Ref: Android Developers / Medium Support new form factors.. | Foldables Multi-Configuration Device State, Display Features Ref: Android Developers Multi-Window Support Multi-Window Split Screen, PIP ANDROID 7 API 24 ANDROID 10 API 29 ANDROID 11? (2020 – alpha release) Closed Flipped Open Half-Opened
10. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya What was the user's INTENT in folding, flipping, or rotating the device? What was the user's EXPERIENCE, and did app behavior fulfill that intent? How can new device AFFORDANCES help up rethink mobile app UX? F O L DA B L E S & M U L T I - S C R E E N / E X P L O R E
11. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya APPLICATION CONTINUITY MULTI-WINDOW SUPPORT MULTI-DISPLAY SUPPORT TESTING SUPPORT DESIGN PATTERNS Folding or unfolding a device can cause apps to transition from one screen to another. Ensure consistent state. Adopt responsive layouts. Use resizeableActvity flag to indicate level of app support. Adopt drag-and-drop for better UX. Gracefully handle resource loss in multi-resume Activities can launch on second screen. Check your default display after creation or configuration change events. Adapt layout or behavior to suit Use foldable device emulators.Test app behavior under configuration changes (posture, orientation), multi-display (second screen) scenarios Look for general best practices and device-specific recommendations for handling postures (hinge), occlusion (seam), layouts (config changes) Ref: Android Developers Building Apps For Foldables F O L DA B L E S & A P P D E S I G N / G U I D A N C E
12. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya F O L DA B L E S & A P P T E S T I N G / E M U L A T O R S Ref: Android Developers Testing / Foldable Emulators Test default & developed responses to fold/unfold & rotation events
13. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya How do I know display features, device state and related change events? Ref: Android Developers Building Apps For Foldables
14. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya J E T PA C K W I N D O W M A N A G E R / A B S T R A C T provide a single API surface for different types of foldable devices coming to the market, so that application developers could target entire categories of devices instead of a single model. • Sample: WindowManager • Usage: Medium Post • With Surface Duo (DevBlog) V 1.0.0-alpha01 Android Developers How do I know display features, device state and related change events?
15. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya S U R FAC E D U O & D E V E L O P M E N T G U I DA N C E I N S T A L L I N G T H E S D K E X P L O R I N G T H E E M U L A T O R U N P A C K I N G T H E A P I & L I B R A R I E S https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya https://aka.ms/androidsummit-2020
16. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya H E L L O, S U R FA C E D U O ! Ref: Microsoft Website Surface Duo: Tech Specs Thin Lightweight Portable Foldable 360 rotation Multiple postures Dual Screen Symmetric screens "Spanned Apps" Productivity Android + M365 "App Groups" Seam
17. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya A P P D E V E L O P M E N T O P T I O N S Android Developers Kotlin Java Cross-Platform Developers React Native Xamarin Flutter Other Developers Unity (gaming) JavaScript (web) Ref: Microsoft GitHub Surface Duo: SDK & Samples
18. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D E V I C E / C O N F I G U R A T I O N F A C T O R S Ref: Microsoft Docs Dual Screen: Introduction • Hinge: 0-360 degrees • Postures: 0-16 discrete levels ▪ Peek ▪ Book (Compose) ▪ Tent (Flip) ▪ Flat • Screens: Single or Dual • Orientation: Portrait or Landscape
19. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya A P P L I C AT I O N / B E H AV I O R S • Default: Apps open on 1 screen • Multi-Display: App can launch activity on 2nd screen (if available) • Move L/R: App can be moved from one screen to other • Multi-Instance: If supported, app instances can run side-by-side • Span: App can be displayed across both screens (on user request only) Ref: Microsoft Docs Dual Screen: Introduction
20. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D E S I G N G U I DA N C E / A N Y F O L D A B L E 1. Use Responsive Layouts: scale (to fit) or create custom layouts (to enhance) 2. Support All Configurations: are users adopting new orientations, postures? 3. Support Diverse Inputs: touch, pen, type, posture? (learn user preferences) 4. Use Drag-And-Drop: particularly suited for dual-screen (give user feedback!) 5. Support Multi-Instance: allow side-by-side app activation (free enhanced UX) 6. Learn User Intent: when/why did user span app? change posture? orientation? Always leave users in control (avoid unpredictable effects or automation without consent)
21. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D E S I G N G U I DA N C E / ' S E A M ' S T R A T E G I E S 1. Do Nothing: test app UX – is behavior acceptable? (e.g., can slide maps for view) 2. Avoid It: change dropdown directions, move to one screen, flip orientation etc. 3. Snap To It: restructure grid-based views to have even columns (seam falls in middle) 4. Mask It: if content, render under seam (human eye auto-magically connects the dots) 5. Split It: if possible, break content into two and render each part separately (left, right) 6. Embrace It: seams can provide a natural separation of concerns that can reduce user cognitive overload for multi-tasking. See design patterns when creating new apps!
22. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D E V E L O P M E N T G U I DA N C E / I T E R A T I V E ❶ Download SDK/Emulator ❸ Adopt Android Best Practices For Foldables ❷ Test Default UX on Emulator ❹ Explore Device Design Patterns for Enhanced UX
23. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya G E T T I N G S TA R T E D / I N S T A L L S D K Ref: Microsoft.com Surface Duo SDK: Download Download Surface Duo SDK (Preview) Contains the Dual Screen libraries, emulator and relevant scripts Run the Surface Duo Emulator (Manual) Is now discoverable as a target device on Android Studio Get Surface Duo SDK Samples (Optional) Build and run samples in emulator to explore dual screen design patterns with code
24. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya S D K E M U L AT O R / U S I N G & T E S T I N G Simulate "spanning" your app for testing Simulate "hinge angle" change events using pressure sensors Simulate "posture" change events using humidity sensor 0 - 360 0 - 16 Test for orientation change effects by simulating rotation
25. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya S D K L I B R A R I E S / C O R E , L AY O U T, O T H E R CORE SDK ScreenHelper SurfaceDuoScreenManager BOTTOM NAVIGATION SurfaceDuoBottomNavigationView FRAGMENTS HANDLER FragmentManagerStateHandler LAYOUTS SurfaceDuoLayout SurfaceDuoFrameLayout Get screen (mode) info, register for mode change events ** Bottom Nav helper, support to move view to left or right screen Handles fragments in different screen modes – restore correct fragments to match transition Containers to simplify layout and render with 1-screen and 2-screen child views
26. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya S U R FAC E D U O & D E S I G N PAT T E R N S L E A R N I N G F I V E D E S I G N P A T T E R N S E X P L O R I N G T H E C O D E S A M P L E S https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya https://aka.ms/androidsummit-2020
27. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D E S I G N PAT T E R N S + C O D E S A M P L E S Ref: Microsoft Docs Dual Screen App Patterns
28. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya PAT T E R N / E X T E N D E D C A N VA S Ref: Microsoft Docs Sample: Extended Canvas Sample App - FrameLayout with a map view. Spanning 'extends canvas' to cover full area. Seam Strategy - Do nothing! Canvas can be scrolled to reveal occluded regions.
29. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya PAT T E R N / L I S T- D E T A I L Ref: Microsoft Docs Sample: List-Detail Sample App - List-Detail app. Shows usage of SurfaceDuoLayout & ScreenHelper (Core) libraries Seam Strategy - Split it! App lends itself naturally to rendering collection+content on dual screens
30. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya PAT T E R N / T W O PA G E Ref: Microsoft Docs Sample: Two Page Sample App - Book-like app. Usage of ViewPager & handling rotation events (vertical scroll) Seam Strategy - Embrace it! Seam enhances familiar "page-turning" UX, easy back/forth navigation
31. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya PAT T E R N / D U A L V I E W Ref: Microsoft Docs Sample: Dual View Sample App - Coordinated content experiences. (can also use multi-instance to emulate behavior) Seam Strategy - Embrace it! Create clear "compare & contrast" with two perspectives of same data
32. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya PAT T E R N / C O M PA N I O N P A N E Ref: Microsoft Docs Sample: Photo Editor App Sample App - Augmented content experiences. (Use added real estate to add 'extra' capability) Seam Strategy - Embrace it! Elevate second level content or controls to main display to enhance UX
33. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya H E R O A P P S / C O M B I N E PA T T E R N S I N U S E Ref: Microsoft Docs Surface Duo App Samples Source Editor Drag-and-Drop, Single Screen Mode, Dual Screen Mode, Mirrored Scrolling TwoNote Drag-and-Drop (Text & Images), Pen Events, List-Detail, Extended Canvas etc.
34. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya R E S O U R C E S / C O N T E N T , C O D E & C O M M U N I T Y 1. Microsoft Docs: Introduction to Dual Screen 2. Surface Duo: Developer Blog 3. Surface Duo: SDK (Download) 4. Surface Duo: SDK (Source) 5. Surface Duo: Code Samples 6. Surface Duo: Community Forum 7. Android Developers: Foldables
35. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya R E C A P / A W A R E N E S S - A C T I O N - A N D R O I D Postures Displays Seams Focus on Productivity Surface Duo SDK Emulator Libraries Affordances App Layouts Device States Design Patterns Focus on UX Continuity Design Patterns Enhanced Canvas List-Detail Two Page Dual View Companion Pane Focus on Development Content Portals | Code Samples |Community Forum Focus on Learning
36. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya https://aka.ms/chicagoroboto #SurfaceDuoDev @nitya T H A N K Y O U ! F E E D B A C K : aka.ms/androidsummit- 2020* A W A R E N E S S A C T I O N A N D R O I D https://aka.ms/androidsummit-2020
37. https://aka.ms/androidsummit-2020 #SurfaceDuoDev @nitya D U A L S C R E E N & M U L T I P O S T U R E R E T H I N K I N G T H E M O B I L E A P P Nitya Narasimhan, PhD Senior Cloud Advocate Microsoft #SurfaceDuoDev @nitya https://aka.ms/androidsummit-2020 THANKS FOR LISTENING!