Semantics in Jetpack Compose

In Jetpack Compose semantics are the key to unlocking accessibility and testing for your applications. We will explore what semantics are, how semantics work with the accessibility framework to make your apps accessible, and how you can use them to effectively test Composables.

Transcript

1. Jetpack Compose Semantics Bryan Herbst (@bryancherbst)
2. Jetpack Compose Semantics 1. Why do I need semantics? 2. How do I use semantics? 3. Deep Dives: • Accessibility • Testing
3. What do accessibility services and testing have in common?
4. Accessibility & Testing •Need to find UI elements •Need to act on UI elements •Need UI element metadata (e.g. content description, available interactions)
5. Composables emit UI (they don’t return anything!)
6. The Composition is a state- aware tree of Composables
7. How do we support these features in Compose?
8. Semantics to the rescue!
9. Semantics Tree •Parallel tree to the Composition •Describes the semantic meaning of Composables
10. Basic Semantics
11. Image requires a contentDescription
12. @Composable fun Image( painter: Painter, contentDescription: String? )
13. contentDescription •Text used by accessibility services to describe content •Same as View world
14. Not limited to images! Box( Modifier.semantics { contentDescription = ”I’m a box” } )
15. What else can semantics do?
16. SemanticsPropertyReceiver Modifier.semantics { } this
17. SemanticsPropertyReceiver •disabled() •heading() •selectableGroup() •collapse()/expand()
18. Text( modifier = Modifier.semantics { heading() }, text = ”I am a header” )
19. SemanticsPropertyReceiver •selected •stateDescription •contentDescription •role
20. Text( modifier = Modifier.semantics { selected = true }, text = ”I am selected” )
21. Shortcut Modifiers: clickable(), toggleable(), etc.
22. What is a Button?
23. Button Surface( Modifier.clickable {} )
24. Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { … }
25. Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { onClick(action = { onClick(); }) }
26. Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { onClick(action = { onClick(); }) if (!enabled) { disabled() } }
27. Modifier.toggleable()
28. Row { Checkbox() Text("Get milk") }
30. Row( Modifier.toggleable( value = checked, onValueChange = { checked = it } ) ) { … }
32. Row(…) { Checkbox( checked = checked, onCheckedChange = null ) }
34. Semantics and Accessibility
38. AndroidComposeView AccessibilityDelegateCompat
39. ACVADC maps semantics to AccessibilityNodeInfo
40. TalkBack has a View problem
42. Wait, we don’t have View classes!
43. ACVADC val className = when (it) { Role.Button -> "android.widget.Button” Role.Checkbox -> "android.widget.CheckBox” }
44. Roles to the rescue!
45. Modifier.clickable( role = Role.RadioButton )
46. Roles •Button •Checkbox •Switch •RadioButton •Tab •Image
47. Other semantics may imply other class names
48. More common accessibility features
49. What does clicking do?
50. Modifier.clickable( onClickLabel = "Create new folder" )
51. Merging Composables
52. Column { Text("100") Text("steps") }
53. Column( Modifier.semantics( mergeDescendants = true ) {} ) { … }
55. Heading
56. Text( modifier = Modifier.semantics { heading() }, text = "Title” )
57. Selectable Group
58. Selectable Group
59. Column( Modifier.selectableGroup() )
60. Column( Modifier.selectableGroup() )
61. State Description
62. State Description
63. Modifier.semantics { stateDescription = ”Subscribed” }
64. Modifier.semantics { stateDescription = ”Subscribed” }
65. Custom Actions
66. Modifier.semantics { customActions = listOf( CustomAccessibilityAction( "delete email” ), ) }
67. What’s Missing?
68. Announcements view.announceForAccessibility( "Something changed!” )
69. “This is not a recommend method in View system.” Source: https://issuetracker.google.com/issues/172590945
70. One option: Live Region
71. var text by remember { mutableStateOf("Hello") }
72. var text by remember { mutableStateOf("Hello") } LaunchedEffect(true) { delay(2000) text = "world!” }
74. Modifier.semantics { liveRegion = LiveRegionMode.Polite }
76. Semantics for Testing
77. Ye olde days
78. Ye olde days (probably today)
79. Espresso onView(withId(R.id.some_view)) .makeAnAssertion()
80. Espresso onView(withId(R.id.some_view)) .makeAnAssertion()
81. Espresso onView(withId(R.id.some_view)) .makeAnAssertion() onView(withContentDescription("text"))
82. Espresso onView(withId(R.id.some_view)) .makeAnAssertion() onView(withContentDescription("text")) onView(withText("text"))
83. Semantics power Compose finders & matchers
84. composeRule .onNodeWithContentDescription("…") .assertExists()
85. composeRule .onNodeWithText("Some Text") .assertExists()
86. composeRule .onNode(hasText("Some Text")) .assertExists()
87. fun hasText( text: String ): SemanticsMatcher { // … }
88. Semantics & Matchers •text -> hasText()
89. Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription()
90. Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription() •stateDescription -> hasStateDescription()
91. Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription() •stateDescription -> hasStateDescription() •selected -> isSelected()
92. Modifier of Last Resort: testTag()
93. Bonus: Testable generally implies accessible!
94. Merged/Unmerged Tree
95. Modifier.semantics( mergeDescendants = true )
96. Or clickable(), selectable(), etc.
97. Column( Modifier.clickable() ) { Text("Jane Doe") Text("555-555-5555") }
98. Column( Modifier.clickable() ) { Text("Jane Doe") Text("555-555-5555") }
99. composeTestRule .onNodeWithText("Jane Doe") .assertTextEquals("Jane Doe")
100. composeTestRule .onNodeWithText("Jane Doe") .assertTextEquals("Jane Doe") ❌ Test failed
101. composeTestRule .onRoot() .printToLog("whereIsJane")
102. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'
103. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'
104. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'
105. composeTestRule.onNodeWithText( text = "Jane Doe", useUnmergedTree = true ).assertTextEquals("Jane Doe")
106. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'true' Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Actions = [OnClick] MergeDescendants = 'true' |-Node #3 at (l=0.0, t=145.0, r=159.0, b=197.0)px | Text = '[Jane Doe]' | Actions = [GetTextLayoutResult] |-Node #5 at (l=0.0, t=197.0, r=232.0, b=249.0)px Text = '[555-555-5555]' Actions = [GetTextLayoutResult]
107. composeTestRule.onNodeWithText( text = "Jane Doe", useUnmergedTree = true ).assertTextEquals("Jane Doe") ✅ Test passed!
108. Custom Semantics
109. From Crane:
110. How can we assert on the state of each day?
111. enum class DayStatus { NoSelected, Selected, FirstDay, LastDay, }
112. val DayKey = SemanticsPropertyKey<DayStatus>("DayKey")
113. Modifier.semantics { }
114. SemanticsPropertyReceiver Modifier.semantics { } this
115. val DayKey = SemanticsPropertyKey<DayStatus>("DayKey") var SemanticsPropertyReceiver.dayStatus by DayKey
116. Modifier.semantics { dayStatus = DayStatus.FirstDay }
117. composeTestRule.onNode( SemanticsMatcher.expectValue( key = DayKey, expectedValue = DayStatus.FirstDay ) )
118. private fun ComposeTestRule.onDateNode( status: DaySelectedStatus ) = onNode( SemanticsMatcher.expectValue(…) )
119. Use lightly, avoid writing test- specific code!
120. Thanks!