Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Semantics in Jetpack Compose

Semantics in Jetpack Compose

In Jetpack Compose semantics are the key to unlocking accessibility and testing for your applications. We will explore what semantics are, how semantics work with the accessibility framework to make your apps accessible, and how you can use them to effectively test Composables.

108056ccba92f98fdbbabad534537573?s=128

Bryan Herbst

December 09, 2021
Tweet

More Decks by Bryan Herbst

Other Decks in Technology

Transcript

  1. Jetpack Compose Semantics Bryan Herbst (@bryancherbst)

  2. Jetpack Compose Semantics 1. Why do I need semantics? 2.

    How do I use semantics? 3. Deep Dives: • Accessibility • Testing
  3. What do accessibility services and testing have in common?

  4. Accessibility & Testing •Need to find UI elements •Need to

    act on UI elements •Need UI element metadata (e.g. content description, available interactions)
  5. Composables emit UI (they don’t return anything!)

  6. The Composition is a state- aware tree of Composables

  7. How do we support these features in Compose?

  8. Semantics to the rescue!

  9. Semantics Tree •Parallel tree to the Composition •Describes the semantic

    meaning of Composables
  10. Basic Semantics

  11. Image requires a contentDescription

  12. @Composable fun Image( painter: Painter, contentDescription: String? )

  13. contentDescription •Text used by accessibility services to describe content •Same

    as View world
  14. Not limited to images! Box( Modifier.semantics { contentDescription = ”I’m

    a box” } )
  15. What else can semantics do?

  16. SemanticsPropertyReceiver Modifier.semantics { } this

  17. SemanticsPropertyReceiver •disabled() •heading() •selectableGroup() •collapse()/expand()

  18. Text( modifier = Modifier.semantics { heading() }, text = ”I

    am a header” )
  19. SemanticsPropertyReceiver •selected •stateDescription •contentDescription •role

  20. Text( modifier = Modifier.semantics { selected = true }, text

    = ”I am selected” )
  21. Shortcut Modifiers: clickable(), toggleable(), etc.

  22. What is a Button?

  23. Button Surface( Modifier.clickable {} )

  24. Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { … }

  25. Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { onClick(action = {

    onClick(); }) }
  26. Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { onClick(action = {

    onClick(); }) if (!enabled) { disabled() } }
  27. Modifier.toggleable()

  28. Row { Checkbox() Text("Get milk") }

  29. None
  30. Row( Modifier.toggleable( value = checked, onValueChange = { checked =

    it } ) ) { … }
  31. None
  32. Row(…) { Checkbox( checked = checked, onCheckedChange = null )

    }
  33. None
  34. Semantics and Accessibility

  35. None
  36. None
  37. None
  38. AndroidComposeView AccessibilityDelegateCompat

  39. ACVADC maps semantics to AccessibilityNodeInfo

  40. TalkBack has a View problem

  41. None
  42. Wait, we don’t have View classes!

  43. ACVADC val className = when (it) { Role.Button -> "android.widget.Button”

    Role.Checkbox -> "android.widget.CheckBox” }
  44. Roles to the rescue!

  45. Modifier.clickable( role = Role.RadioButton )

  46. Roles •Button •Checkbox •Switch •RadioButton •Tab •Image

  47. Other semantics may imply other class names

  48. More common accessibility features

  49. What does clicking do?

  50. Modifier.clickable( onClickLabel = "Create new folder" )

  51. Merging Composables

  52. Column { Text("100") Text("steps") }

  53. Column( Modifier.semantics( mergeDescendants = true ) {} ) { …

    }
  54. None
  55. Heading

  56. Text( modifier = Modifier.semantics { heading() }, text = "Title”

    )
  57. Selectable Group

  58. Selectable Group

  59. Column( Modifier.selectableGroup() )

  60. Column( Modifier.selectableGroup() )

  61. State Description

  62. State Description

  63. Modifier.semantics { stateDescription = ”Subscribed” }

  64. Modifier.semantics { stateDescription = ”Subscribed” }

  65. Custom Actions

  66. Modifier.semantics { customActions = listOf( CustomAccessibilityAction( "delete email” ), )

    }
  67. What’s Missing?

  68. Announcements view.announceForAccessibility( "Something changed!” )

  69. “This is not a recommend method in View system.” Source:

    https://issuetracker.google.com/issues/172590945
  70. One option: Live Region

  71. var text by remember { mutableStateOf("Hello") }

  72. var text by remember { mutableStateOf("Hello") } LaunchedEffect(true) { delay(2000)

    text = "world!” }
  73. None
  74. Modifier.semantics { liveRegion = LiveRegionMode.Polite }

  75. None
  76. Semantics for Testing

  77. Ye olde days

  78. Ye olde days (probably today)

  79. Espresso onView(withId(R.id.some_view)) .makeAnAssertion()

  80. Espresso onView(withId(R.id.some_view)) .makeAnAssertion()

  81. Espresso onView(withId(R.id.some_view)) .makeAnAssertion() onView(withContentDescription("text"))

  82. Espresso onView(withId(R.id.some_view)) .makeAnAssertion() onView(withContentDescription("text")) onView(withText("text"))

  83. Semantics power Compose finders & matchers

  84. composeRule .onNodeWithContentDescription("…") .assertExists()

  85. composeRule .onNodeWithText("Some Text") .assertExists()

  86. composeRule .onNode(hasText("Some Text")) .assertExists()

  87. fun hasText( text: String ): SemanticsMatcher { // … }

  88. Semantics & Matchers •text -> hasText()

  89. Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription()

  90. Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription() •stateDescription

    -> hasStateDescription()
  91. Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription() •stateDescription

    -> hasStateDescription() •selected -> isSelected()
  92. Modifier of Last Resort: testTag()

  93. Bonus: Testable generally implies accessible!

  94. Merged/Unmerged Tree

  95. Modifier.semantics( mergeDescendants = true )

  96. Or clickable(), selectable(), etc.

  97. Column( Modifier.clickable() ) { Text("Jane Doe") Text("555-555-5555") }

  98. Column( Modifier.clickable() ) { Text("Jane Doe") Text("555-555-5555") }

  99. composeTestRule .onNodeWithText("Jane Doe") .assertTextEquals("Jane Doe")

  100. composeTestRule .onNodeWithText("Jane Doe") .assertTextEquals("Jane Doe") ❌ Test failed

  101. composeTestRule .onRoot() .printToLog("whereIsJane")

  102. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1

    at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'
  103. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1

    at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'
  104. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1

    at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'
  105. composeTestRule.onNodeWithText( text = "Jane Doe", useUnmergedTree = true ).assertTextEquals("Jane Doe")

  106. com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'true' Node #1

    at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Actions = [OnClick] MergeDescendants = 'true' |-Node #3 at (l=0.0, t=145.0, r=159.0, b=197.0)px | Text = '[Jane Doe]' | Actions = [GetTextLayoutResult] |-Node #5 at (l=0.0, t=197.0, r=232.0, b=249.0)px Text = '[555-555-5555]' Actions = [GetTextLayoutResult]
  107. composeTestRule.onNodeWithText( text = "Jane Doe", useUnmergedTree = true ).assertTextEquals("Jane Doe")

    ✅ Test passed!
  108. Custom Semantics

  109. From Crane:

  110. How can we assert on the state of each day?

  111. enum class DayStatus { NoSelected, Selected, FirstDay, LastDay, }

  112. val DayKey = SemanticsPropertyKey<DayStatus>("DayKey")

  113. Modifier.semantics { }

  114. SemanticsPropertyReceiver Modifier.semantics { } this

  115. val DayKey = SemanticsPropertyKey<DayStatus>("DayKey") var SemanticsPropertyReceiver.dayStatus by DayKey

  116. Modifier.semantics { dayStatus = DayStatus.FirstDay }

  117. composeTestRule.onNode( SemanticsMatcher.expectValue( key = DayKey, expectedValue = DayStatus.FirstDay ) )

  118. private fun ComposeTestRule.onDateNode( status: DaySelectedStatus ) = onNode( SemanticsMatcher.expectValue(…) )

  119. Use lightly, avoid writing test- specific code!

  120. Thanks!