Semantics in Jetpack Compose

Slide 1

Slide 1 text

Jetpack Compose Semantics Bryan Herbst (@bryancherbst)

Slide 2

Slide 2 text

Jetpack Compose Semantics 1. Why do I need semantics? 2. How do I use semantics? 3. Deep Dives: • Accessibility • Testing

Slide 3

Slide 3 text

What do accessibility services and testing have in common?

Slide 4

Slide 4 text

Accessibility & Testing •Need to find UI elements •Need to act on UI elements •Need UI element metadata (e.g. content description, available interactions)

Slide 5

Slide 5 text

Composables emit UI (they don’t return anything!)

Slide 6

Slide 6 text

The Composition is a state- aware tree of Composables

Slide 7

Slide 7 text

How do we support these features in Compose?

Slide 8

Slide 8 text

Semantics to the rescue!

Slide 9

Slide 9 text

Semantics Tree •Parallel tree to the Composition •Describes the semantic meaning of Composables

Slide 10

Slide 10 text

Basic Semantics

Slide 11

Slide 11 text

Image requires a contentDescription

Slide 12

Slide 12 text

@Composable fun Image( painter: Painter, contentDescription: String? )

Slide 13

Slide 13 text

contentDescription •Text used by accessibility services to describe content •Same as View world

Slide 14

Slide 14 text

Not limited to images! Box( Modifier.semantics { contentDescription = ”I’m a box” } )

Slide 15

Slide 15 text

What else can semantics do?

Slide 16

Slide 16 text

SemanticsPropertyReceiver Modifier.semantics { } this

Slide 17

Slide 17 text

SemanticsPropertyReceiver •disabled() •heading() •selectableGroup() •collapse()/expand()

Slide 18

Slide 18 text

Text( modifier = Modifier.semantics { heading() }, text = ”I am a header” )

Slide 19

Slide 19 text

SemanticsPropertyReceiver •selected •stateDescription •contentDescription •role

Slide 20

Slide 20 text

Text( modifier = Modifier.semantics { selected = true }, text = ”I am selected” )

Slide 21

Slide 21 text

Shortcut Modifiers: clickable(), toggleable(), etc.

Slide 22

Slide 22 text

What is a Button?

Slide 23

Slide 23 text

Button Surface( Modifier.clickable {} )

Slide 24

Slide 24 text

Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { … }

Slide 25

Slide 25 text

Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { onClick(action = { onClick(); }) }

Slide 26

Slide 26 text

Modifier.clickable() Modifier.semantics( mergeDescendants = true ) { onClick(action = { onClick(); }) if (!enabled) { disabled() } }

Slide 27

Slide 27 text

Modifier.toggleable()

Slide 28

Slide 28 text

Row { Checkbox() Text("Get milk") }

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Row( Modifier.toggleable( value = checked, onValueChange = { checked = it } ) ) { … }

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Row(…) { Checkbox( checked = checked, onCheckedChange = null ) }

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Semantics and Accessibility

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

AndroidComposeView AccessibilityDelegateCompat

Slide 39

Slide 39 text

ACVADC maps semantics to AccessibilityNodeInfo

Slide 40

Slide 40 text

TalkBack has a View problem

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Wait, we don’t have View classes!

Slide 43

Slide 43 text

ACVADC val className = when (it) { Role.Button -> "android.widget.Button” Role.Checkbox -> "android.widget.CheckBox” }

Slide 44

Slide 44 text

Roles to the rescue!

Slide 45

Slide 45 text

Modifier.clickable( role = Role.RadioButton )

Slide 46

Slide 46 text

Roles •Button •Checkbox •Switch •RadioButton •Tab •Image

Slide 47

Slide 47 text

Other semantics may imply other class names

Slide 48

Slide 48 text

More common accessibility features

Slide 49

Slide 49 text

What does clicking do?

Slide 50

Slide 50 text

Modifier.clickable( onClickLabel = "Create new folder" )

Slide 51

Slide 51 text

Merging Composables

Slide 52

Slide 52 text

Column { Text("100") Text("steps") }

Slide 53

Slide 53 text

Column( Modifier.semantics( mergeDescendants = true ) {} ) { … }

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Heading

Slide 56

Slide 56 text

Text( modifier = Modifier.semantics { heading() }, text = "Title” )

Slide 57

Slide 57 text

Selectable Group

Slide 58

Slide 58 text

Selectable Group

Slide 59

Slide 59 text

Column( Modifier.selectableGroup() )

Slide 60

Slide 60 text

Column( Modifier.selectableGroup() )

Slide 61

Slide 61 text

State Description

Slide 62

Slide 62 text

State Description

Slide 63

Slide 63 text

Modifier.semantics { stateDescription = ”Subscribed” }

Slide 64

Slide 64 text

Modifier.semantics { stateDescription = ”Subscribed” }

Slide 65

Slide 65 text

Custom Actions

Slide 66

Slide 66 text

Modifier.semantics { customActions = listOf( CustomAccessibilityAction( "delete email” ), ) }

Slide 67

Slide 67 text

What’s Missing?

Slide 68

Slide 68 text

Announcements view.announceForAccessibility( "Something changed!” )

Slide 69

Slide 69 text

“This is not a recommend method in View system.” Source: https://issuetracker.google.com/issues/172590945

Slide 70

Slide 70 text

One option: Live Region

Slide 71

Slide 71 text

var text by remember { mutableStateOf("Hello") }

Slide 72

Slide 72 text

var text by remember { mutableStateOf("Hello") } LaunchedEffect(true) { delay(2000) text = "world!” }

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Modifier.semantics { liveRegion = LiveRegionMode.Polite }

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

Semantics for Testing

Slide 77

Slide 77 text

Ye olde days

Slide 78

Slide 78 text

Ye olde days (probably today)

Slide 79

Slide 79 text

Espresso onView(withId(R.id.some_view)) .makeAnAssertion()

Slide 80

Slide 80 text

Espresso onView(withId(R.id.some_view)) .makeAnAssertion()

Slide 81

Slide 81 text

Espresso onView(withId(R.id.some_view)) .makeAnAssertion() onView(withContentDescription("text"))

Slide 82

Slide 82 text

Espresso onView(withId(R.id.some_view)) .makeAnAssertion() onView(withContentDescription("text")) onView(withText("text"))

Slide 83

Slide 83 text

Semantics power Compose finders & matchers

Slide 84

Slide 84 text

composeRule .onNodeWithContentDescription("…") .assertExists()

Slide 85

Slide 85 text

composeRule .onNodeWithText("Some Text") .assertExists()

Slide 86

Slide 86 text

composeRule .onNode(hasText("Some Text")) .assertExists()

Slide 87

Slide 87 text

fun hasText( text: String ): SemanticsMatcher { // … }

Slide 88

Slide 88 text

Semantics & Matchers •text -> hasText()

Slide 89

Slide 89 text

Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription()

Slide 90

Slide 90 text

Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription() •stateDescription -> hasStateDescription()

Slide 91

Slide 91 text

Semantics & Matchers •text -> hasText() •contentDescription -> hasContentDescription() •stateDescription -> hasStateDescription() •selected -> isSelected()

Slide 92

Slide 92 text

Modifier of Last Resort: testTag()

Slide 93

Slide 93 text

Bonus: Testable generally implies accessible!

Slide 94

Slide 94 text

Merged/Unmerged Tree

Slide 95

Slide 95 text

Modifier.semantics( mergeDescendants = true )

Slide 96

Slide 96 text

Or clickable(), selectable(), etc.

Slide 97

Slide 97 text

Column( Modifier.clickable() ) { Text("Jane Doe") Text("555-555-5555") }

Slide 98

Slide 98 text

Column( Modifier.clickable() ) { Text("Jane Doe") Text("555-555-5555") }

Slide 99

Slide 99 text

composeTestRule .onNodeWithText("Jane Doe") .assertTextEquals("Jane Doe")

Slide 100

Slide 100 text

composeTestRule .onNodeWithText("Jane Doe") .assertTextEquals("Jane Doe") ❌ Test failed

Slide 101

Slide 101 text

composeTestRule .onRoot() .printToLog("whereIsJane")

Slide 102

Slide 102 text

com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'false' Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Text = '[Jane Doe, 555-555-5555]' Actions = [OnClick, GetTextLayoutResult] MergeDescendants = 'true'

Slide 103

Slide 103 text

Slide 104

Slide 104 text

Slide 105

Slide 105 text

composeTestRule.onNodeWithText( text = "Jane Doe", useUnmergedTree = true ).assertTextEquals("Jane Doe")

Slide 106

Slide 106 text

com.example D/whereIsJane: printToLog: Printing with useUnmergedTree = 'true' Node #1 at (l=0.0, t=145.0, r=232.0, b=249.0)px |-Node #2 at (l=0.0, t=145.0, r=232.0, b=249.0)px Actions = [OnClick] MergeDescendants = 'true' |-Node #3 at (l=0.0, t=145.0, r=159.0, b=197.0)px | Text = '[Jane Doe]' | Actions = [GetTextLayoutResult] |-Node #5 at (l=0.0, t=197.0, r=232.0, b=249.0)px Text = '[555-555-5555]' Actions = [GetTextLayoutResult]

Slide 107

Slide 107 text

composeTestRule.onNodeWithText( text = "Jane Doe", useUnmergedTree = true ).assertTextEquals("Jane Doe") ✅ Test passed!

Slide 108

Slide 108 text

Custom Semantics

Slide 109

Slide 109 text

From Crane:

Slide 110

Slide 110 text

How can we assert on the state of each day?

Slide 111

Slide 111 text

enum class DayStatus { NoSelected, Selected, FirstDay, LastDay, }

Slide 112

Slide 112 text

val DayKey = SemanticsPropertyKey("DayKey")

Slide 113

Slide 113 text

Modifier.semantics { }

Slide 114

Slide 114 text

SemanticsPropertyReceiver Modifier.semantics { } this

Slide 115

Slide 115 text

val DayKey = SemanticsPropertyKey("DayKey") var SemanticsPropertyReceiver.dayStatus by DayKey

Slide 116

Slide 116 text

Modifier.semantics { dayStatus = DayStatus.FirstDay }

Slide 117

Slide 117 text

composeTestRule.onNode( SemanticsMatcher.expectValue( key = DayKey, expectedValue = DayStatus.FirstDay ) )

Slide 118

Slide 118 text

private fun ComposeTestRule.onDateNode( status: DaySelectedStatus ) = onNode( SemanticsMatcher.expectValue(…) )