Maestro: A paradigm shift in UI testing

Transcript

1. Maestro: A paradigm shift in UI testing @droid_singh @amanjeetsingh150 Amanjeet Singh
2. Definition
3. Definition Story📕
4. Definition Story📕
5. Definition Story📕
6. Definition Story📕
7. Definition Story📕
8. Definition Story📕
9. Definition Story📕 • Develop more features
10. Definition Story📕 • Develop more features • More features = More Money
11. Definition Story📕 • Develop more features • More features = More Money • More subdivisions created, coming soon with more features
12. Definition Story📕 • Develop more features • More features = More Money • More subdivisions created, coming soon with more features
13. Definition Story📕 • Workload increases • Develop more features • More features = More Money • More subdivisions created, coming soon with more features
14. Definition Story📕 • Workload increases • Feature complexity increases ⏫ • Develop more features • More features = More Money • More subdivisions created, coming soon with more features
15. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • Develop more features • More features = More Money • More subdivisions created, coming soon with more features
16. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features
17. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button
18. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen
19. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing
20. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization:
21. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship
22. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship
23. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship • Hire QA engineers
24. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship • Hire QA engineers • Start writing unit tests for the features
25. Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship • Hire QA engineers • Start writing unit tests for the features
26. Definition Story📕
27. Definition Story📕 • Started testing manually before releasing
28. Definition Story📕 • Started testing manually before releasing • Reports reduced and app got stabilised
29. Definition Story📕 • Started testing manually before releasing • Reports reduced and app got stabilised
30. Definition Story📕 • Started testing manually before releasing • Reports reduced and app got stabilised More Features
31. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised
32. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools
33. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools • Setting up automation tools is painful
34. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
35. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful • Time to get first flow automation working
36. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
37. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
38. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
39. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
40. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • Different OEMs devices • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
41. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • Different OEMs devices • More • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
42. Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • Different OEMs devices • More • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful
43. Definition Story📕
44. Definition Story📕 • Gap between QAs and Developers
45. Definition Story📕 User Reports • Patterns in user reports • Gap between QAs and Developers
46. Definition Story📕 User Reports Functional Bugs • Patterns in user reports • Gap between QAs and Developers
47. Definition Story📕 User Reports Functional Bugs • App is launching slowly • Patterns in user reports • Gap between QAs and Developers
48. Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Patterns in user reports • Gap between QAs and Developers
49. Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Activity starts slowly • Patterns in user reports • Gap between QAs and Developers
50. Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Activity starts slowly Non Functional Bugs • Patterns in user reports • Gap between QAs and Developers
51. Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Activity starts slowly Non Functional Bugs • Appium support for solving Non Functional Bugs ☹ • Patterns in user reports • Gap between QAs and Developers
52. Ideal setup for UI testing at scale
53. • Setup should be very easy Ideal setup for UI testing at scale
54. • Setup should be very easy • Starting and writing first flow should be easy Ideal setup for UI testing at scale
55. • Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy Ideal setup for UI testing at scale
56. • Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy • Managing infrastructure should be easy Ideal setup for UI testing at scale
57. • Helpful for dealing with functional and non-functional issues • Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy • Managing infrastructure should be easy Ideal setup for UI testing at scale
58. Introducing Maestro
59. Setup should be Easy brew tap mobile-dev-inc/tap brew install maestro
60. or curl -Ls "https://get.maestro.mobile.dev" | bash Setup should be Easy brew tap mobile-dev-inc/tap brew install maestro
61. or curl -Ls "https://get.maestro.mobile.dev" | bash Setup should be Easy brew tap mobile-dev-inc/tap brew install maestro
62. Starting and writing first flow • Declarative, in YAML format ✅
63. Starting and writing first flow • Declarative, in YAML format ✅ • Hot reload or continues mode ✅
64. Starting and writing first flow • Declarative, in YAML format ✅ • Hot reload or continues mode ✅ • Lets open twitter app and follow account
65. Starting and writing first flow appId: com.twitter.android
66. Starting and writing first flow appId: com.twitter.android --- - launchApp
67. Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml
68. Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml
69. Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml - tapOn: Search and Explore
70. Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml - tapOn: Search and Explore - tapOn: Search Twitter - inputText: "@mobile__dev" - tapOn: mobile.dev - tapOn: Follow - assertVisible: Following
71. More commands
72. More commands • Tap on commands
73. More commands • Index • Tap on commands
74. More commands • Index • Width, height • Tap on commands
75. More commands • Index • Width, height • Relative positioning • Tap on commands
76. More commands • Index • Width, height • Relative positioning • Tap on commands • id of view
77. More commands • Index • Width, height • Relative positioning • Tap on commands • id of view • More
78. More commands • Tap on commands • Long press - longPress: “text” 
 or - longPress: 
 id: “@id_some_view_id” 

79. More commands • Tap on commands • Long press • Swiping - swipe: 
 direction: RIGHT 
 duration: 300 
 - swipe: 
 start: x1, y1 
 end: x2, y2 
 duration: 400 

80. More commands • Tap on commands • Long press • Swiping • Erase Text - eraseText
81. More commands • Tap on commands • Long press • Swiping • Erase Text • Much more: https://maestro.mobile.dev/reference
82. Sit back!
83. Sit back! • No need of idling resource
84. Sit back! • No need of idling resource • No need pepper test case with Thread.sleep
85. Sit back! • No need of idling resource • No need pepper test case with Thread.sleep appId: com.squareup name: Sign Up --- - launchApp - tapOn: Create account - tapOn: text: Accept all cookies optional: true - inputText: ${EMAIL} - tapOn: Continue - inputText: ${PASSWORD} - tapOn: Continue - tapOn: rightOf: I agree to.* retryTapIfNoChange: false - tapOn: Continue - tapOn: Confirm
86. Sit back! • No need of idling resource • No need pepper test case with Thread.sleep • Not much jargon involved, making maestro tests sharable source
87. Sit back! • No need of idling resource • No need pepper test case with Thread.sleep • Decouples with codebase, blazing fast iterations • Not much jargon involved, making maestro tests sharable source
88. Sit back! • No need of idling resource • No need pepper test case with Thread.sleep • Decouples with codebase, blazing fast iterations • Not much jargon involved, making maestro tests sharable source • And, cross platform !
89. Dealing with Functional Bugs
90. Dealing with Functional Bugs • Maestro test command outputs
91. Dealing with Functional Bugs • Maestro test command outputs • Locally
92. Dealing with Functional Bugs • Maestro test command outputs • Locally • JUnit type format output available <?xml version='1.0' encoding='UTF-8'?> <testsuites> <testsuite name="Test Suite" device="emulator-5554" tests="1" failures="1"> <testcase id="follow" name="follow"> <failure>Element not found: Text matching regex: Follow</failure> </testcase> </testsuite> </testsuites>
93. Dealing with Non-Functional Bugs
94. Dealing with Non-Functional Bugs • Frame drops 😖
95. Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic
96. Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow
97. Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow
98. Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow • Run: adb shell dumpsys gfxinfo com.twitter.android framestats
99. Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow • Run: adb shell dumpsys gfxinfo com.twitter.android framestats • Collect output and capture important metrics
100. Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow • Run: adb shell dumpsys gfxinfo com.twitter.android framestats • Collect output and capture important metrics • Repeat
101. Dealing with Non-Functional Bugs • Root Causing Window: com.twitter.android/com.twitter.app.main.MainActivity Stats since: 100942401883393ns Total frames rendered: 387 Janky frames: 110 (28.42%) 50th percentile: 9ms 90th percentile: 22ms 95th percentile: 34ms 99th percentile: 450ms
102. Dealing with Non-Functional Bugs • Root Causing Window: com.twitter.android/com.twitter.app.main.MainActivity Stats since: 100942401883393ns Total frames rendered: 387 Janky frames: 110 (28.42%) 50th percentile: 9ms 90th percentile: 22ms 95th percentile: 34ms 99th percentile: 450ms
103. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage
104. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations
105. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations Debug API
106. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations Debug API Debug.startMethodTracingSampling( context.cacheDir, maxBufferSize, samplingIntervalUs )
107. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations Debug API Debug.startMethodTracingSampling( context.cacheDir, maxBufferSize, samplingIntervalUs ) Debug.stopMethodTracing() …… …… ……
108. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations
109. Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations • Capture Strict mode violations
110. Integrate with CI
111. Integrate with CI maestro cloud
112. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key> App file
113. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key> Test directory <root> | 
 |___.maestro/ 
 | | 
 | |____ Login.yaml | | 
 | |____ Add to Cart.yaml |
114. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key>
115. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key> • Integrate with PRs ✅
116. Integrate with CI • Device management and scaling maestro cloud <app-file> ./maestro —apiKey <api-key> • Integrate with PRs ✅ ✅
117. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key> • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc. • Integrate with PRs ✅ ✅ • Device management and scaling ✅
118. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key> • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc. • Integrate with PRs ✅ ✅ • Device management and scaling ✅ • Trigger on PR stage ✅
119. Integrate with CI maestro cloud <app-file> ./maestro —apiKey <api-key> • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc. • Integrate with PRs ✅ ✅ • Device management and scaling ✅ • Trigger on PR stage ✅ • Different API level devices ✅
120. Ideal setup of UI testing • Helpful for dealing with functional and non-functional issues • Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy • Managing infrastructure should be easy ✅ ✅ ✅ ✅ ✅
121. Maestro is Open Source!
122. Maestro is Open Source! • https://github.com/mobile-dev-inc/maestro/
123. Maestro is Open Source! • https://github.com/mobile-dev-inc/maestro/
124. Community Support is Great • Performance tool to get FPS, CPU usage for search and explore interaction by Bamlab: https://twitter.com/almouro/status/1595402239044198400
125. Community Support is Great • Performance tool to get FPS, CPU usage for search and explore interaction by Craig Atkinson: https://www.atkinsondev.com/post/android-maestro-mock-rest-api/ • Mock API servers for Maestro Android testing using MockServer and Gradle
126. Community Support is Great • Performance tool to get FPS, CPU usage for search and explore interaction by Craig Atkinson: https://www.atkinsondev.com/post/android-maestro-testing/ • Mock API servers for Maestro Android testing using MockServer and Gradle • High level UI testing with Maestro
127. Community Support is Great by Manish Verma: https://twitter.com/mobile__dev/status/1580952966474846209 • Facebook registration flow automation • Performance tool to get FPS, CPU usage for search and explore interaction • Mock API servers for Maestro Android testing using MockServer and Gradle • High level UI testing with Maestro
128. Community Support is Great by Manish Verma: https://twitter.com/mobile__dev/status/1580952966474846209 • Facebook registration flow automation • Performance tool to get FPS, CPU usage for search and explore interaction • Mock API servers for Maestro Android testing using MockServer and Gradle • High level UI testing with Maestro • Much more: Join Maestro slack community !!
129. Fin! 🙋