Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Maestro: A paradigm shift in UI testing

Maestro: A paradigm shift in UI testing

Amanjeet Singh

November 30, 2022
Tweet

More Decks by Amanjeet Singh

Other Decks in Programming

Transcript

  1. Maestro: A paradigm shift in UI testing
    @droid_singh
    @amanjeetsingh150
    Amanjeet Singh

    View Slide

  2. Definition

    View Slide

  3. Definition
    Story📕

    View Slide

  4. Definition
    Story📕

    View Slide

  5. Definition
    Story📕

    View Slide

  6. Definition
    Story📕

    View Slide

  7. Definition
    Story📕

    View Slide

  8. Definition
    Story📕

    View Slide

  9. Definition
    Story📕
    • Develop more features

    View Slide

  10. Definition
    Story📕
    • Develop more features
    • More features = More Money

    View Slide

  11. Definition
    Story📕
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features

    View Slide

  12. Definition
    Story📕
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features

    View Slide

  13. Definition
    Story📕
    • Workload increases
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features

    View Slide

  14. Definition
    Story📕
    • Workload increases
    • Feature complexity increases

    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features

    View Slide

  15. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features

    View Slide

  16. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features

    View Slide

  17. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button

    View Slide

  18. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen

    View Slide

  19. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing

    View Slide

  20. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing
    • Realization:

    View Slide

  21. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing
    • Realization: We need to test before we ship

    View Slide

  22. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing
    • Realization: We need to test before we ship

    View Slide

  23. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing
    • Realization: We need to test before we ship
    • Hire QA engineers

    View Slide

  24. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing
    • Realization: We need to test before we ship
    • Hire QA engineers
    • Start writing unit tests for the features

    View Slide

  25. Definition
    Story📕
    • Workload increases
    • Feature complexity increases
    • Reports from users:

    • I am not able to login
    • Develop more features
    • More features = More Money
    • More subdivisions created, coming soon with more features
    • I am not able to click on the Add to Cart button
    • My app crashes when I go on Payments screen
    • Still Increasing
    • Realization: We need to test before we ship
    • Hire QA engineers
    • Start writing unit tests for the features

    View Slide

  26. Definition
    Story📕

    View Slide

  27. Definition
    Story📕
    • Started testing manually before releasing

    View Slide

  28. Definition
    Story📕
    • Started testing manually before releasing
    • Reports reduced and app got stabilised

    View Slide

  29. Definition
    Story📕
    • Started testing manually before releasing
    • Reports reduced and app got stabilised

    View Slide

  30. Definition
    Story📕
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    More Features

    View Slide

  31. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised

    View Slide

  32. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Onboarding cost to automation tools

    View Slide

  33. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Onboarding cost to automation tools
    • Setting up automation tools is painful

    View Slide

  34. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  35. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful
    • Time to get first flow automation working

    View Slide

  36. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  37. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • CI integration
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  38. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • Different API level devices
    • CI integration
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  39. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • Different API level devices
    • Managing traffic of builds
    • CI integration
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  40. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • Different API level devices
    • Managing traffic of builds
    • Different OEMs devices
    • CI integration
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  41. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • Different API level devices
    • Managing traffic of builds
    • Different OEMs devices
    • More
    • CI integration
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  42. Definition
    Story📕
    Automation
    More Features
    • Started testing manually before releasing
    • Reports reduced and app got stabilised
    • Time to get first flow automation working
    • Infrastructure to run tests
    • Different API level devices
    • Managing traffic of builds
    • Different OEMs devices
    • More
    • CI integration
    • Onboarding cost to automation tools
    • Automation tools are “non-declarative”
    • Setting up automation tools is painful

    View Slide

  43. Definition
    Story📕

    View Slide

  44. Definition
    Story📕
    • Gap between QAs and Developers

    View Slide

  45. Definition
    Story📕
    User Reports
    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  46. Definition
    Story📕
    User Reports
    Functional Bugs
    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  47. Definition
    Story📕
    User Reports
    Functional Bugs • App is launching slowly
    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  48. Definition
    Story📕
    User Reports
    Functional Bugs • App is launching slowly
    • Battery draining
    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  49. Definition
    Story📕
    User Reports
    Functional Bugs • App is launching slowly
    • Battery draining
    • Activity starts slowly
    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  50. Definition
    Story📕
    User Reports
    Functional Bugs • App is launching slowly
    • Battery draining
    • Activity starts slowly
    Non Functional Bugs
    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  51. Definition
    Story📕
    User Reports
    Functional Bugs • App is launching slowly
    • Battery draining
    • Activity starts slowly
    Non Functional Bugs
    • Appium support for solving Non Functional Bugs

    • Patterns in user reports
    • Gap between QAs and Developers

    View Slide

  52. Ideal setup for UI testing at scale

    View Slide

  53. • Setup should be very easy
    Ideal setup for UI testing at scale

    View Slide

  54. • Setup should be very easy
    • Starting and writing first flow should be easy
    Ideal setup for UI testing at scale

    View Slide

  55. • Setup should be very easy
    • Starting and writing first flow should be easy
    • Integration with CI should be very easy
    Ideal setup for UI testing at scale

    View Slide

  56. • Setup should be very easy
    • Starting and writing first flow should be easy
    • Integration with CI should be very easy
    • Managing infrastructure should be easy
    Ideal setup for UI testing at scale

    View Slide

  57. • Helpful for dealing with functional and non-functional issues
    • Setup should be very easy
    • Starting and writing first flow should be easy
    • Integration with CI should be very easy
    • Managing infrastructure should be easy
    Ideal setup for UI testing at scale

    View Slide

  58. Introducing Maestro

    View Slide

  59. Setup should be Easy
    brew tap mobile-dev-inc/tap


    brew install maestro


    View Slide

  60. or


    curl -Ls "https://get.maestro.mobile.dev" | bash


    Setup should be Easy
    brew tap mobile-dev-inc/tap


    brew install maestro


    View Slide

  61. or


    curl -Ls "https://get.maestro.mobile.dev" | bash


    Setup should be Easy
    brew tap mobile-dev-inc/tap


    brew install maestro


    View Slide

  62. Starting and writing first flow
    • Declarative, in YAML format ✅

    View Slide

  63. Starting and writing first flow
    • Declarative, in YAML format ✅
    • Hot reload or continues mode ✅

    View Slide

  64. Starting and writing first flow
    • Declarative, in YAML format ✅
    • Hot reload or continues mode ✅
    • Lets open twitter app and follow account

    View Slide

  65. Starting and writing first flow
    appId: com.twitter.android


    View Slide

  66. Starting and writing first flow
    appId: com.twitter.android


    ---


    - launchApp


    View Slide

  67. Starting and writing first flow
    appId: com.twitter.android


    ---


    - launchApp


    maestro test -c ./follow.yaml


    View Slide

  68. Starting and writing first flow
    appId: com.twitter.android


    ---


    - launchApp


    maestro test -c ./follow.yaml


    View Slide

  69. Starting and writing first flow
    appId: com.twitter.android


    ---


    - launchApp


    maestro test -c ./follow.yaml


    - tapOn: Search and Explore


    View Slide

  70. Starting and writing first flow
    appId: com.twitter.android


    ---


    - launchApp


    maestro test -c ./follow.yaml


    - tapOn: Search and Explore


    - tapOn: Search Twitter


    - inputText: "@mobile__dev"


    - tapOn: mobile.dev


    - tapOn: Follow


    - assertVisible: Following

    View Slide

  71. More commands

    View Slide

  72. More commands
    • Tap on commands

    View Slide

  73. More commands
    • Index
    • Tap on commands

    View Slide

  74. More commands
    • Index
    • Width, height
    • Tap on commands

    View Slide

  75. More commands
    • Index
    • Width, height
    • Relative positioning
    • Tap on commands

    View Slide

  76. More commands
    • Index
    • Width, height
    • Relative positioning
    • Tap on commands
    • id of view

    View Slide

  77. More commands
    • Index
    • Width, height
    • Relative positioning
    • Tap on commands
    • id of view
    • More

    View Slide

  78. More commands
    • Tap on commands
    • Long press
    - longPress: “text”

    or


    - longPress:

    id: “@id_some_view_id”

    View Slide

  79. More commands
    • Tap on commands
    • Long press
    • Swiping
    - swipe:

    direction: RIGHT

    duration: 300

    - swipe:

    start: x1, y1

    end: x2, y2

    duration: 400

    View Slide

  80. More commands
    • Tap on commands
    • Long press
    • Swiping
    • Erase Text
    - eraseText

    View Slide

  81. More commands
    • Tap on commands
    • Long press
    • Swiping
    • Erase Text
    • Much more: https://maestro.mobile.dev/reference

    View Slide

  82. Sit back!

    View Slide

  83. Sit back!
    • No need of idling resource

    View Slide

  84. Sit back!
    • No need of idling resource
    • No need pepper test case with Thread.sleep

    View Slide

  85. Sit back!
    • No need of idling resource
    • No need pepper test case with Thread.sleep
    appId: com.squareup


    name: Sign Up


    ---


    - launchApp


    - tapOn: Create account


    - tapOn:


    text: Accept all cookies


    optional: true


    - inputText: ${EMAIL}


    - tapOn: Continue


    - inputText: ${PASSWORD}


    - tapOn: Continue


    - tapOn:


    rightOf: I agree to.*


    retryTapIfNoChange: false


    - tapOn: Continue


    - tapOn: Confirm


    View Slide

  86. Sit back!
    • No need of idling resource
    • No need pepper test case with Thread.sleep
    • Not much jargon involved, making maestro tests sharable source

    View Slide

  87. Sit back!
    • No need of idling resource
    • No need pepper test case with Thread.sleep
    • Decouples with codebase, blazing fast iterations
    • Not much jargon involved, making maestro tests sharable source

    View Slide

  88. Sit back!
    • No need of idling resource
    • No need pepper test case with Thread.sleep
    • Decouples with codebase, blazing fast iterations
    • Not much jargon involved, making maestro tests sharable source
    • And, cross platform !

    View Slide

  89. Dealing with Functional Bugs

    View Slide

  90. Dealing with Functional Bugs
    • Maestro test command outputs

    View Slide

  91. Dealing with Functional Bugs
    • Maestro test command outputs
    • Locally

    View Slide

  92. Dealing with Functional Bugs
    • Maestro test command outputs
    • Locally
    • JUnit type format output available












    Element not found: Text matching regex: Follow











    View Slide

  93. Dealing with Non-Functional Bugs

    View Slide

  94. Dealing with Non-Functional Bugs
    • Frame drops
    😖

    View Slide

  95. Dealing with Non-Functional Bugs
    • Frame drops
    😖
    • Identify core flows of app which have high user traffic

    View Slide

  96. Dealing with Non-Functional Bugs
    • Frame drops
    😖
    • Identify core flows of app which have high user traffic
    • Following twitter page flow

    View Slide

  97. Dealing with Non-Functional Bugs
    • Frame drops
    😖
    • Identify core flows of app which have high user traffic
    • Following twitter page flow
    • Write Maestro Flow

    View Slide

  98. Dealing with Non-Functional Bugs
    • Frame drops
    😖
    • Identify core flows of app which have high user traffic
    • Following twitter page flow
    • Write Maestro Flow
    • Run: adb shell dumpsys gfxinfo com.twitter.android framestats

    View Slide

  99. Dealing with Non-Functional Bugs
    • Frame drops
    😖
    • Identify core flows of app which have high user traffic
    • Following twitter page flow
    • Write Maestro Flow
    • Run: adb shell dumpsys gfxinfo com.twitter.android framestats
    • Collect output and capture important metrics

    View Slide

  100. Dealing with Non-Functional Bugs
    • Frame drops
    😖
    • Identify core flows of app which have high user traffic
    • Following twitter page flow
    • Write Maestro Flow
    • Run: adb shell dumpsys gfxinfo com.twitter.android framestats
    • Collect output and capture important metrics
    • Repeat

    View Slide

  101. Dealing with Non-Functional Bugs
    • Root Causing
    Window: com.twitter.android/com.twitter.app.main.MainActivity


    Stats since: 100942401883393ns


    Total frames rendered: 387


    Janky frames: 110 (28.42%)


    50th percentile: 9ms


    90th percentile: 22ms


    95th percentile: 34ms


    99th percentile: 450ms

    View Slide

  102. Dealing with Non-Functional Bugs
    • Root Causing
    Window: com.twitter.android/com.twitter.app.main.MainActivity


    Stats since: 100942401883393ns


    Total frames rendered: 387


    Janky frames: 110 (28.42%)


    50th percentile: 9ms


    90th percentile: 22ms


    95th percentile: 34ms


    99th percentile: 450ms

    View Slide

  103. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage

    View Slide

  104. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage
    • Generate flame graphs to narrow down on heavy operations

    View Slide

  105. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage
    • Generate flame graphs to narrow down on heavy operations
    Debug API

    View Slide

  106. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage
    • Generate flame graphs to narrow down on heavy operations
    Debug API
    Debug.startMethodTracingSampling(


    context.cacheDir,


    maxBufferSize,


    samplingIntervalUs


    )


    View Slide

  107. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage
    • Generate flame graphs to narrow down on heavy operations
    Debug API
    Debug.startMethodTracingSampling(


    context.cacheDir,


    maxBufferSize,


    samplingIntervalUs


    )


    Debug.stopMethodTracing()
    ……


    ……


    ……

    View Slide

  108. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage
    • Generate flame graphs to narrow down on heavy operations

    View Slide

  109. Dealing with Non-Functional Bugs
    • Root Causing
    • Monitor janky frame percentage
    • Generate flame graphs to narrow down on heavy operations
    • Capture Strict mode violations

    View Slide

  110. Integrate with CI

    View Slide

  111. Integrate with CI
    maestro cloud

    View Slide

  112. Integrate with CI
    maestro cloud ./maestro —apiKey
    App file

    View Slide

  113. Integrate with CI
    maestro cloud ./maestro —apiKey
    Test directory



    |

    |___.maestro/

    | |

    | |____ Login.yaml


    | |

    | |____ Add to Cart.yaml


    |

    View Slide

  114. Integrate with CI
    maestro cloud ./maestro —apiKey

    View Slide

  115. Integrate with CI
    maestro cloud ./maestro —apiKey
    • Integrate with PRs ✅

    View Slide

  116. Integrate with CI
    • Device management and scaling
    maestro cloud ./maestro —apiKey
    • Integrate with PRs ✅

    View Slide

  117. Integrate with CI
    maestro cloud ./maestro —apiKey
    • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc.
    • Integrate with PRs ✅

    • Device management and scaling ✅

    View Slide

  118. Integrate with CI
    maestro cloud ./maestro —apiKey
    • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc.
    • Integrate with PRs ✅

    • Device management and scaling ✅
    • Trigger on PR stage

    View Slide

  119. Integrate with CI
    maestro cloud ./maestro —apiKey
    • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc.
    • Integrate with PRs ✅

    • Device management and scaling ✅
    • Trigger on PR stage

    • Different API level devices ✅

    View Slide

  120. Ideal setup of UI testing
    • Helpful for dealing with functional and non-functional issues
    • Setup should be very easy
    • Starting and writing first flow should be easy
    • Integration with CI should be very easy
    • Managing infrastructure should be easy





    View Slide

  121. Maestro is Open Source!

    View Slide

  122. Maestro is Open Source!
    • https://github.com/mobile-dev-inc/maestro/

    View Slide

  123. Maestro is Open Source!
    • https://github.com/mobile-dev-inc/maestro/

    View Slide

  124. Community Support is Great
    • Performance tool to get FPS, CPU usage for search and explore interaction
    by Bamlab:


    https://twitter.com/almouro/status/1595402239044198400

    View Slide

  125. Community Support is Great
    • Performance tool to get FPS, CPU usage for search and explore interaction
    by Craig Atkinson:


    https://www.atkinsondev.com/post/android-maestro-mock-rest-api/
    • Mock API servers for Maestro Android testing using MockServer and Gradle

    View Slide

  126. Community Support is Great
    • Performance tool to get FPS, CPU usage for search and explore interaction
    by Craig Atkinson:


    https://www.atkinsondev.com/post/android-maestro-testing/
    • Mock API servers for Maestro Android testing using MockServer and Gradle
    • High level UI testing with Maestro

    View Slide

  127. Community Support is Great
    by Manish Verma:


    https://twitter.com/mobile__dev/status/1580952966474846209
    • Facebook registration flow automation
    • Performance tool to get FPS, CPU usage for search and explore interaction
    • Mock API servers for Maestro Android testing using MockServer and Gradle
    • High level UI testing with Maestro

    View Slide

  128. Community Support is Great
    by Manish Verma:


    https://twitter.com/mobile__dev/status/1580952966474846209
    • Facebook registration flow automation
    • Performance tool to get FPS, CPU usage for search and explore interaction
    • Mock API servers for Maestro Android testing using MockServer and Gradle
    • High level UI testing with Maestro
    • Much more: Join Maestro slack community !!

    View Slide

  129. Fin! 🙋

    View Slide