Droidcon NYC 2024: Unpacking Compose Multiplatform Accessibility

In this talk, we will provide a comprehensive overview of the current state of accessibility in Compose Multiplatform. As an emerging UI framework in multiplatform development, understanding its accessibility features is critical for developers.

We will delve into the current accessibility capabilities of Compose Multiplatform, discuss its strengths and weaknesses, and provide practical strategies for leveraging its features to ensure the accessibility of your apps.

Transcript

1. Unpacking Compose Multiplatform Accessibility Colin Marsch - @colinmarsch
2. Hey, I’m Colin Marsch Android Engineer @ Cash App
3. Talk Overview • Compose Multiplatform overview • UI construction • Accessibility handling • Testing
4. What is Compose Multiplatform?
5. Kotlin Multiplatform Shared Logic Compose Multiplatform Shared UI
6. var showImage by remember { mutableStateOf(false) } Column { Button(onClick = { showImage = !showImage }) { Text("Toggle image") } AnimatedVisibility(showImage) { Image("Compose Multiplatform Logo") } }
8. How does Compose Multiplatform build UI?
9. On Android it’s just Compose UI
10. class MainActivity : ComponentActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContent { App() } } }
11. class MainActivity : ComponentActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContent { App() } } }
13. What about iOS?
14. fun MainViewController() : UIViewController = ComposeUIViewController { App() }
15. Skiko
16. Canvas Implementation Pros • Identical UI between platforms • UI won’t change during OS version updates • Less platform speci fi c code Cons • Possible performance overhead • Native UI interoperability complexity • Accessibility complexity
17. Native iOS UI Interoperability
18. Column { Text("How to use UIKitView inside Compose Multiplatform") UIKitView( factory = { MKMapView() }, modifier = Modifier.border(2.dp, Color.Blue).size(300.dp), ) }
19. Column { Text("How to use UIKitView inside Compose Multiplatform") UIKitView( factory = { MKMapView() }, modifier = Modifier.border(2.dp, Color.Blue).size(300.dp), ) }
20. You can also use SwiftUI within Compose Multiplatform
21. How is Compose Multiplatform related to accessibility?
22. Android Accessibility Tree
23. AccessibilityNodeInfo
24. AccessibilityNodeInfo.getContentDescription()
25. AccessibilityNodeInfo.getContentDescription() AccessibilityNodeInfo.isSelected()
26. AccessibilityNodeInfo.getContentDescription() AccessibilityNodeInfo.isSelected() AccessibilityNodeInfo.isImportantForAccessibility()
27. SemanticsNode
28. SemanticsNode.config.get(SemanticsProperties.ContentDescription)
29. SemanticsNode.config.get(SemanticsProperties.ContentDescription) SemanticsNode.config.get(SemanticsProperties.Selected)
30. SemanticsNode.config.get(SemanticsProperties.ContentDescription) SemanticsNode.config.get(SemanticsProperties.Selected) SemanticsNode.config.get(SemanticsProperties.Heading)
31. SemanticsNode -> AccessibilityNodeInfo
32. Compose Multiplatform == Compose UI
33. iOS Accessibility Tree
34. UIAccessibilityElement
35. UIAccessibilityElement.accessibilityLabel
36. UIAccessibilityElement.accessibilityLabel UIAccessibilityElement.accessibilityHint
37. UIAccessibilityElement.accessibilityLabel UIAccessibilityElement.accessibilityHint UIAccessibilityElement.accessibilityTraits.contains(.selected)
38. SemanticsNodes -> UIAccessibilityElement*
39. SemanticsNodes -> UIAccessibilityElement* * Lazily
40. Mapping iOS properties
41. override fun accessibilityLabel(): String? = getOrElse(CachedAccessibilityPropertyKeys.accessibilityLabel) { val contentDescription = config.getOrNull(SemanticsProperties.ContentDescription) if (contentDescription != null) { contentDescription } else { val editableText = config.getOrNull(SemanticsProperties.EditableText)?.text editableText ?: config.getOrNull(SemanticsProperties.Text) ?.joinToString("\n") { it.text } } }
42. override fun accessibilityLabel(): String? = getOrElse(CachedAccessibilityPropertyKeys.accessibilityLabel) { val contentDescription = config.getOrNull(SemanticsProperties.ContentDescription) if (contentDescription != null) { contentDescription } else { val editableText = config.getOrNull(SemanticsProperties.EditableText)?.text editableText ?: config.getOrNull(SemanticsProperties.Text) ?.joinToString("\n") { it.text } } }
43. override fun accessibilityLabel(): String? = getOrElse(CachedAccessibilityPropertyKeys.accessibilityLabel) { val contentDescription = config.getOrNull(SemanticsProperties.ContentDescription) if (contentDescription != null) { contentDescription } else { val editableText = config.getOrNull(SemanticsProperties.EditableText)?.text editableText ?: config.getOrNull(SemanticsProperties.Text) ?.joinToString("\n") { it.text } } }
44. override fun accessibilityLabel(): String? = getOrElse(CachedAccessibilityPropertyKeys.accessibilityLabel) { val contentDescription = config.getOrNull(SemanticsProperties.ContentDescription) if (contentDescription != null) { contentDescription } else { val editableText = config.getOrNull(SemanticsProperties.EditableText)?.text editableText ?: config.getOrNull(SemanticsProperties.Text) ?.joinToString("\n") { it.text } } }
45. override fun accessibilityHint(): String? = getOrElse(CachedAccessibilityPropertyKeys.accessibilityHint) { cachedConfig.getOrNull(SemanticsActions.OnClick)?.label }
46. override fun accessibilityHint(): String? = getOrElse(CachedAccessibilityPropertyKeys.accessibilityHint) { cachedConfig.getOrNull(SemanticsActions.OnClick)?.label }
47. override fun accessibilityTraits(): UIAccessibilityTraits = getOrElse(CachedAccessibilityPropertyKeys.accessibilityTraits) { config.getOrNull(SemanticsActions.OnClick)?.let { result = result or UIAccessibilityTraitButton } if (config.contains(SemanticsProperties.Heading)) { result = result or UIAccessibilityTraitHeader } }
48. override fun accessibilityTraits(): UIAccessibilityTraits = getOrElse(CachedAccessibilityPropertyKeys.accessibilityTraits) { config.getOrNull(SemanticsActions.OnClick)?.let { result = result or UIAccessibilityTraitButton } if (config.contains(SemanticsProperties.Heading)) { result = result or UIAccessibilityTraitHeader } }
49. override fun accessibilityTraits(): UIAccessibilityTraits = getOrElse(CachedAccessibilityPropertyKeys.accessibilityTraits) { config.getOrNull(SemanticsActions.OnClick)?.let { result = result or UIAccessibilityTraitButton } if (config.contains(SemanticsProperties.Heading)) { result = result or UIAccessibilityTraitHeader } }
50. Native iOS UI Accessibility Interoperability
51. UIKitView( factory = { MKMapView() }, modifier = Modifier.semantics { contentDescription = "Map of NYC” }, properties = UIKitInteropProperties( // false by default isNativeAccessibilityEnabled = false, ), )
52. UIKitView( factory = { MKMapView() }, properties = UIKitInteropProperties( isNativeAccessibilityEnabled = true, ), )
53. Testing Compose Multiplatform Accessibility
54. Android Accessibility Testing
55. App built by Google • Highlights accessibility issues via an overlay on top of the running app • Must install this app on your device or emulator • Performs pass/fail accessibility checks on the UI • Color contrast, touch target size, missing labels, etc. Accessibility Scanner
56. Manual screen reader testing • Android’s main screen reader technology • Requires interpreting the results • Can uncover deeper navigation and interaction issues • Should always be a part of your development work fl ow Talkback
57. • Espresso accessibility checks • Paparazzi’s AccessibilityRenderExtension • Can integrate into your CI setup Automated Testing
58. iOS Accessibility Testing
59. Manual screen reader testing • iOS’ main screen reader technology • Requires interpreting the results • Can uncover deeper navigation and interaction issues • Should always be a part of your development work fl ow VoiceOver
60. XCode developer tool • Examine accessibility properties of UI elements • Requires interpreting the results • Supports iOS simulators • Similar to screen readers (i.e. Talkback/VoiceOver) in a written format Accessibility Inspector
61. • performAccessibilityAudit on XCUIApplication in UI tests • Performs the same audit that is able to be manually run from the Accessibility Inspector tool • Performs pass/fail checks similar to Android’s Espresso checks Automated Testing
62. iOS Simulator Testing
63. iOSMain > MainViewController.kt fun MainViewController() = ComposeUIViewController() { App() } iOS Simulator Testing
64. iOSMain > MainViewController.kt fun MainViewController() = ComposeUIViewController( configure = { accessibilitySyncOptions = Always() } ) { App() } iOS Simulator Testing
65. Want to learn more? • github.com/JetBrains/compose-multiplatform: Examples and documentation • github.com/JetBrains/compose-multiplatform-core: All the internals • Compose Multiplatform on iOS: On the Road to Stable - Sebastian Aigner • Tomorrow at 3:25 PM
66. Thank you! Q&A?