Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Semantics in Jetpack Compose

Bryan Herbst
December 09, 2021

Semantics in Jetpack Compose

In Jetpack Compose semantics are the key to unlocking accessibility and testing for your applications. We will explore what semantics are, how semantics work with the accessibility framework to make your apps accessible, and how you can use them to effectively test Composables.

Bryan Herbst

December 09, 2021
Tweet

More Decks by Bryan Herbst

Other Decks in Technology

Transcript

  1. Jetpack Compose Semantics
    Bryan Herbst (@bryancherbst)

    View full-size slide

  2. Jetpack Compose Semantics
    1. Why do I need semantics?
    2. How do I use semantics?
    3. Deep Dives:
    • Accessibility
    • Testing

    View full-size slide

  3. What do accessibility services
    and testing have in common?

    View full-size slide

  4. Accessibility & Testing
    •Need to find UI elements
    •Need to act on UI elements
    •Need UI element metadata (e.g. content
    description, available interactions)

    View full-size slide

  5. Composables emit UI
    (they don’t return anything!)

    View full-size slide

  6. The Composition is a state-
    aware tree of Composables

    View full-size slide

  7. How do we support these
    features in Compose?

    View full-size slide

  8. Semantics to the rescue!

    View full-size slide

  9. Semantics Tree
    •Parallel tree to the Composition
    •Describes the semantic meaning of
    Composables

    View full-size slide

  10. Basic Semantics

    View full-size slide

  11. Image requires a
    contentDescription

    View full-size slide

  12. @Composable
    fun Image(
    painter: Painter,
    contentDescription: String?
    )

    View full-size slide

  13. contentDescription
    •Text used by accessibility services to describe
    content
    •Same as View world

    View full-size slide

  14. Not limited to images!
    Box(
    Modifier.semantics {
    contentDescription = ”I’m a box”
    }
    )

    View full-size slide

  15. What else can semantics do?

    View full-size slide

  16. SemanticsPropertyReceiver
    Modifier.semantics {
    }
    this

    View full-size slide

  17. SemanticsPropertyReceiver
    •disabled()
    •heading()
    •selectableGroup()
    •collapse()/expand()

    View full-size slide

  18. Text(
    modifier = Modifier.semantics {
    heading()
    },
    text = ”I am a header”
    )

    View full-size slide

  19. SemanticsPropertyReceiver
    •selected
    •stateDescription
    •contentDescription
    •role

    View full-size slide

  20. Text(
    modifier = Modifier.semantics {
    selected = true
    },
    text = ”I am selected”
    )

    View full-size slide

  21. Shortcut Modifiers:
    clickable(), toggleable(), etc.

    View full-size slide

  22. What is a Button?

    View full-size slide

  23. Button
    Surface(
    Modifier.clickable {}
    )

    View full-size slide

  24. Modifier.clickable()
    Modifier.semantics(
    mergeDescendants = true
    ) { … }

    View full-size slide

  25. Modifier.clickable()
    Modifier.semantics(
    mergeDescendants = true
    ) {
    onClick(action = { onClick(); })
    }

    View full-size slide

  26. Modifier.clickable()
    Modifier.semantics(
    mergeDescendants = true
    ) {
    onClick(action = { onClick(); })
    if (!enabled) {
    disabled()
    }
    }

    View full-size slide

  27. Modifier.toggleable()

    View full-size slide

  28. Row {
    Checkbox()
    Text("Get milk")
    }

    View full-size slide

  29. Row(
    Modifier.toggleable(
    value = checked,
    onValueChange = { checked = it }
    )
    ) { … }

    View full-size slide

  30. Row(…) {
    Checkbox(
    checked = checked,
    onCheckedChange = null
    )
    }

    View full-size slide

  31. Semantics and Accessibility

    View full-size slide

  32. AndroidComposeView
    AccessibilityDelegateCompat

    View full-size slide

  33. ACVADC maps semantics to
    AccessibilityNodeInfo

    View full-size slide

  34. TalkBack has a View problem

    View full-size slide

  35. Wait, we don’t have View
    classes!

    View full-size slide

  36. ACVADC
    val className = when (it) {
    Role.Button -> "android.widget.Button”
    Role.Checkbox -> "android.widget.CheckBox”
    }

    View full-size slide

  37. Roles to the rescue!

    View full-size slide

  38. Modifier.clickable(
    role = Role.RadioButton
    )

    View full-size slide

  39. Roles
    •Button
    •Checkbox
    •Switch
    •RadioButton
    •Tab
    •Image

    View full-size slide

  40. Other semantics may imply
    other class names

    View full-size slide

  41. More common accessibility
    features

    View full-size slide

  42. What does clicking do?

    View full-size slide

  43. Modifier.clickable(
    onClickLabel = "Create new folder"
    )

    View full-size slide

  44. Merging Composables

    View full-size slide

  45. Column {
    Text("100")
    Text("steps")
    }

    View full-size slide

  46. Column(
    Modifier.semantics(
    mergeDescendants = true
    ) {}
    ) { … }

    View full-size slide

  47. Text(
    modifier = Modifier.semantics {
    heading()
    },
    text = "Title”
    )

    View full-size slide

  48. Selectable Group

    View full-size slide

  49. Selectable Group

    View full-size slide

  50. Column(
    Modifier.selectableGroup()
    )

    View full-size slide

  51. Column(
    Modifier.selectableGroup()
    )

    View full-size slide

  52. State
    Description

    View full-size slide

  53. State
    Description

    View full-size slide

  54. Modifier.semantics {
    stateDescription = ”Subscribed”
    }

    View full-size slide

  55. Modifier.semantics {
    stateDescription = ”Subscribed”
    }

    View full-size slide

  56. Custom Actions

    View full-size slide

  57. Modifier.semantics {
    customActions = listOf(
    CustomAccessibilityAction(
    "delete email”
    ),
    )
    }

    View full-size slide

  58. What’s Missing?

    View full-size slide

  59. Announcements
    view.announceForAccessibility(
    "Something changed!”
    )

    View full-size slide

  60. “This is not a recommend
    method in View system.”
    Source: https://issuetracker.google.com/issues/172590945

    View full-size slide

  61. One option: Live Region

    View full-size slide

  62. var text by remember {
    mutableStateOf("Hello")
    }

    View full-size slide

  63. var text by remember {
    mutableStateOf("Hello")
    }
    LaunchedEffect(true) {
    delay(2000)
    text = "world!”
    }

    View full-size slide

  64. Modifier.semantics {
    liveRegion = LiveRegionMode.Polite
    }

    View full-size slide

  65. Semantics for Testing

    View full-size slide

  66. Ye olde days

    View full-size slide

  67. Ye olde days (probably today)

    View full-size slide

  68. Espresso
    onView(withId(R.id.some_view))
    .makeAnAssertion()

    View full-size slide

  69. Espresso
    onView(withId(R.id.some_view))
    .makeAnAssertion()

    View full-size slide

  70. Espresso
    onView(withId(R.id.some_view))
    .makeAnAssertion()
    onView(withContentDescription("text"))

    View full-size slide

  71. Espresso
    onView(withId(R.id.some_view))
    .makeAnAssertion()
    onView(withContentDescription("text"))
    onView(withText("text"))

    View full-size slide

  72. Semantics power Compose
    finders & matchers

    View full-size slide

  73. composeRule
    .onNodeWithContentDescription("…")
    .assertExists()

    View full-size slide

  74. composeRule
    .onNodeWithText("Some Text")
    .assertExists()

    View full-size slide

  75. composeRule
    .onNode(hasText("Some Text"))
    .assertExists()

    View full-size slide

  76. fun hasText(
    text: String
    ): SemanticsMatcher {
    // …
    }

    View full-size slide

  77. Semantics & Matchers
    •text -> hasText()

    View full-size slide

  78. Semantics & Matchers
    •text -> hasText()
    •contentDescription -> hasContentDescription()

    View full-size slide

  79. Semantics & Matchers
    •text -> hasText()
    •contentDescription -> hasContentDescription()
    •stateDescription -> hasStateDescription()

    View full-size slide

  80. Semantics & Matchers
    •text -> hasText()
    •contentDescription -> hasContentDescription()
    •stateDescription -> hasStateDescription()
    •selected -> isSelected()

    View full-size slide

  81. Modifier of Last Resort:
    testTag()

    View full-size slide

  82. Bonus: Testable generally
    implies accessible!

    View full-size slide

  83. Merged/Unmerged Tree

    View full-size slide

  84. Modifier.semantics(
    mergeDescendants = true
    )

    View full-size slide

  85. Or clickable(), selectable(),
    etc.

    View full-size slide

  86. Column(
    Modifier.clickable()
    ) {
    Text("Jane Doe")
    Text("555-555-5555")
    }

    View full-size slide

  87. Column(
    Modifier.clickable()
    ) {
    Text("Jane Doe")
    Text("555-555-5555")
    }

    View full-size slide

  88. composeTestRule
    .onNodeWithText("Jane Doe")
    .assertTextEquals("Jane Doe")

    View full-size slide

  89. composeTestRule
    .onNodeWithText("Jane Doe")
    .assertTextEquals("Jane Doe")
    ❌ Test failed

    View full-size slide

  90. composeTestRule
    .onRoot()
    .printToLog("whereIsJane")

    View full-size slide

  91. com.example D/whereIsJane: printToLog:
    Printing with useUnmergedTree = 'false'
    Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    Text = '[Jane Doe, 555-555-5555]'
    Actions = [OnClick, GetTextLayoutResult]
    MergeDescendants = 'true'

    View full-size slide

  92. com.example D/whereIsJane: printToLog:
    Printing with useUnmergedTree = 'false'
    Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    Text = '[Jane Doe, 555-555-5555]'
    Actions = [OnClick, GetTextLayoutResult]
    MergeDescendants = 'true'

    View full-size slide

  93. com.example D/whereIsJane: printToLog:
    Printing with useUnmergedTree = 'false'
    Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    Text = '[Jane Doe, 555-555-5555]'
    Actions = [OnClick, GetTextLayoutResult]
    MergeDescendants = 'true'

    View full-size slide

  94. composeTestRule.onNodeWithText(
    text = "Jane Doe",
    useUnmergedTree = true
    ).assertTextEquals("Jane Doe")

    View full-size slide

  95. com.example D/whereIsJane: printToLog:
    Printing with useUnmergedTree = 'true'
    Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px
    Actions = [OnClick]
    MergeDescendants = 'true'
    |-Node #3 at (l=0.0, t=145.0, r=159.0, b=197.0)px
    | Text = '[Jane Doe]'
    | Actions = [GetTextLayoutResult]
    |-Node #5 at (l=0.0, t=197.0, r=232.0, b=249.0)px
    Text = '[555-555-5555]'
    Actions = [GetTextLayoutResult]

    View full-size slide

  96. composeTestRule.onNodeWithText(
    text = "Jane Doe",
    useUnmergedTree = true
    ).assertTextEquals("Jane Doe")
    ✅ Test passed!

    View full-size slide

  97. Custom Semantics

    View full-size slide

  98. How can we assert on the
    state of each day?

    View full-size slide

  99. enum class DayStatus {
    NoSelected,
    Selected,
    FirstDay,
    LastDay,
    }

    View full-size slide

  100. val DayKey =
    SemanticsPropertyKey("DayKey")

    View full-size slide

  101. Modifier.semantics {
    }

    View full-size slide

  102. SemanticsPropertyReceiver
    Modifier.semantics {
    }
    this

    View full-size slide

  103. val DayKey =
    SemanticsPropertyKey("DayKey")
    var SemanticsPropertyReceiver.dayStatus
    by DayKey

    View full-size slide

  104. Modifier.semantics {
    dayStatus = DayStatus.FirstDay
    }

    View full-size slide

  105. composeTestRule.onNode(
    SemanticsMatcher.expectValue(
    key = DayKey,
    expectedValue = DayStatus.FirstDay
    )
    )

    View full-size slide

  106. private fun ComposeTestRule.onDateNode(
    status: DaySelectedStatus
    ) = onNode(
    SemanticsMatcher.expectValue(…)
    )

    View full-size slide

  107. Use lightly, avoid writing test-
    specific code!

    View full-size slide