Tweet
Tweet

Slide 1

Slide 1 text

Maestro: A paradigm shift in UI testing @droid_singh @amanjeetsingh150 Amanjeet Singh

Slide 2

Slide 2 text

Definition

Slide 3

Slide 3 text

Definition Story📕

Slide 4

Slide 4 text

Definition Story📕

Slide 5

Slide 5 text

Definition Story📕

Slide 6

Slide 6 text

Definition Story📕

Slide 7

Slide 7 text

Definition Story📕

Slide 8

Slide 8 text

Definition Story📕

Slide 9

Slide 9 text

Definition Story📕 • Develop more features

Slide 10

Slide 10 text

Definition Story📕 • Develop more features • More features = More Money

Slide 11

Slide 11 text

Definition Story📕 • Develop more features • More features = More Money • More subdivisions created, coming soon with more features

Slide 12

Slide 12 text

Definition Story📕 • Develop more features • More features = More Money • More subdivisions created, coming soon with more features

Slide 13

Slide 13 text

Definition Story📕 • Workload increases • Develop more features • More features = More Money • More subdivisions created, coming soon with more features

Slide 14

Slide 14 text

Definition Story📕 • Workload increases • Feature complexity increases ⏫ • Develop more features • More features = More Money • More subdivisions created, coming soon with more features

Slide 15

Slide 15 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • Develop more features • More features = More Money • More subdivisions created, coming soon with more features

Slide 16

Slide 16 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features

Slide 17

Slide 17 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button

Slide 18

Slide 18 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen

Slide 19

Slide 19 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing

Slide 20

Slide 20 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization:

Slide 21

Slide 21 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship

Slide 22

Slide 22 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship

Slide 23

Slide 23 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship • Hire QA engineers

Slide 24

Slide 24 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship • Hire QA engineers • Start writing unit tests for the features

Slide 25

Slide 25 text

Definition Story📕 • Workload increases • Feature complexity increases • Reports from users: ⏫ • I am not able to login • Develop more features • More features = More Money • More subdivisions created, coming soon with more features • I am not able to click on the Add to Cart button • My app crashes when I go on Payments screen • Still Increasing • Realization: We need to test before we ship • Hire QA engineers • Start writing unit tests for the features

Slide 26

Slide 26 text

Definition Story📕

Slide 27

Slide 27 text

Definition Story📕 • Started testing manually before releasing

Slide 28

Slide 28 text

Definition Story📕 • Started testing manually before releasing • Reports reduced and app got stabilised

Slide 29

Slide 29 text

Definition Story📕 • Started testing manually before releasing • Reports reduced and app got stabilised

Slide 30

Slide 30 text

Definition Story📕 • Started testing manually before releasing • Reports reduced and app got stabilised More Features

Slide 31

Slide 31 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised

Slide 32

Slide 32 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools

Slide 33

Slide 33 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools • Setting up automation tools is painful

Slide 34

Slide 34 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 35

Slide 35 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful • Time to get first flow automation working

Slide 36

Slide 36 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 37

Slide 37 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 38

Slide 38 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 39

Slide 39 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 40

Slide 40 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • Different OEMs devices • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 41

Slide 41 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • Different OEMs devices • More • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 42

Slide 42 text

Definition Story📕 Automation More Features • Started testing manually before releasing • Reports reduced and app got stabilised • Time to get first flow automation working • Infrastructure to run tests • Different API level devices • Managing traffic of builds • Different OEMs devices • More • CI integration • Onboarding cost to automation tools • Automation tools are “non-declarative” • Setting up automation tools is painful

Slide 43

Slide 43 text

Definition Story📕

Slide 44

Slide 44 text

Definition Story📕 • Gap between QAs and Developers

Slide 45

Slide 45 text

Definition Story📕 User Reports • Patterns in user reports • Gap between QAs and Developers

Slide 46

Slide 46 text

Definition Story📕 User Reports Functional Bugs • Patterns in user reports • Gap between QAs and Developers

Slide 47

Slide 47 text

Definition Story📕 User Reports Functional Bugs • App is launching slowly • Patterns in user reports • Gap between QAs and Developers

Slide 48

Slide 48 text

Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Patterns in user reports • Gap between QAs and Developers

Slide 49

Slide 49 text

Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Activity starts slowly • Patterns in user reports • Gap between QAs and Developers

Slide 50

Slide 50 text

Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Activity starts slowly Non Functional Bugs • Patterns in user reports • Gap between QAs and Developers

Slide 51

Slide 51 text

Definition Story📕 User Reports Functional Bugs • App is launching slowly • Battery draining • Activity starts slowly Non Functional Bugs • Appium support for solving Non Functional Bugs ☹ • Patterns in user reports • Gap between QAs and Developers

Slide 52

Slide 52 text

Ideal setup for UI testing at scale

Slide 53

Slide 53 text

• Setup should be very easy Ideal setup for UI testing at scale

Slide 54

Slide 54 text

• Setup should be very easy • Starting and writing first flow should be easy Ideal setup for UI testing at scale

Slide 55

Slide 55 text

• Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy Ideal setup for UI testing at scale

Slide 56

Slide 56 text

• Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy • Managing infrastructure should be easy Ideal setup for UI testing at scale

Slide 57

Slide 57 text

• Helpful for dealing with functional and non-functional issues • Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy • Managing infrastructure should be easy Ideal setup for UI testing at scale

Slide 58

Slide 58 text

Introducing Maestro

Slide 59

Slide 59 text

Setup should be Easy brew tap mobile-dev-inc/tap brew install maestro

Slide 60

Slide 60 text

or curl -Ls "https://get.maestro.mobile.dev" | bash Setup should be Easy brew tap mobile-dev-inc/tap brew install maestro

Slide 61

Slide 61 text

or curl -Ls "https://get.maestro.mobile.dev" | bash Setup should be Easy brew tap mobile-dev-inc/tap brew install maestro

Slide 62

Slide 62 text

Starting and writing first flow • Declarative, in YAML format ✅

Slide 63

Slide 63 text

Starting and writing first flow • Declarative, in YAML format ✅ • Hot reload or continues mode ✅

Slide 64

Slide 64 text

Starting and writing first flow • Declarative, in YAML format ✅ • Hot reload or continues mode ✅ • Lets open twitter app and follow account

Slide 65

Slide 65 text

Starting and writing first flow appId: com.twitter.android

Slide 66

Slide 66 text

Starting and writing first flow appId: com.twitter.android --- - launchApp

Slide 67

Slide 67 text

Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml

Slide 68

Slide 68 text

Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml

Slide 69

Slide 69 text

Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml - tapOn: Search and Explore

Slide 70

Slide 70 text

Starting and writing first flow appId: com.twitter.android --- - launchApp maestro test -c ./follow.yaml - tapOn: Search and Explore - tapOn: Search Twitter - inputText: "@mobile__dev" - tapOn: mobile.dev - tapOn: Follow - assertVisible: Following

Slide 71

Slide 71 text

More commands

Slide 72

Slide 72 text

More commands • Tap on commands

Slide 73

Slide 73 text

More commands • Index • Tap on commands

Slide 74

Slide 74 text

More commands • Index • Width, height • Tap on commands

Slide 75

Slide 75 text

More commands • Index • Width, height • Relative positioning • Tap on commands

Slide 76

Slide 76 text

More commands • Index • Width, height • Relative positioning • Tap on commands • id of view

Slide 77

Slide 77 text

More commands • Index • Width, height • Relative positioning • Tap on commands • id of view • More

Slide 78

Slide 78 text

More commands • Tap on commands • Long press - longPress: “text” 
 or - longPress: 
 id: “@id_some_view_id” 


Slide 79

Slide 79 text

More commands • Tap on commands • Long press • Swiping - swipe: 
 direction: RIGHT 
 duration: 300 
 - swipe: 
 start: x1, y1 
 end: x2, y2 
 duration: 400 


Slide 80

Slide 80 text

More commands • Tap on commands • Long press • Swiping • Erase Text - eraseText

Slide 81

Slide 81 text

More commands • Tap on commands • Long press • Swiping • Erase Text • Much more: https://maestro.mobile.dev/reference

Slide 82

Slide 82 text

Sit back!

Slide 83

Slide 83 text

Sit back! • No need of idling resource

Slide 84

Slide 84 text

Sit back! • No need of idling resource • No need pepper test case with Thread.sleep

Slide 85

Slide 85 text

Sit back! • No need of idling resource • No need pepper test case with Thread.sleep appId: com.squareup name: Sign Up --- - launchApp - tapOn: Create account - tapOn: text: Accept all cookies optional: true - inputText: ${EMAIL} - tapOn: Continue - inputText: ${PASSWORD} - tapOn: Continue - tapOn: rightOf: I agree to.* retryTapIfNoChange: false - tapOn: Continue - tapOn: Confirm

Slide 86

Slide 86 text

Sit back! • No need of idling resource • No need pepper test case with Thread.sleep • Not much jargon involved, making maestro tests sharable source

Slide 87

Slide 87 text

Sit back! • No need of idling resource • No need pepper test case with Thread.sleep • Decouples with codebase, blazing fast iterations • Not much jargon involved, making maestro tests sharable source

Slide 88

Slide 88 text

Sit back! • No need of idling resource • No need pepper test case with Thread.sleep • Decouples with codebase, blazing fast iterations • Not much jargon involved, making maestro tests sharable source • And, cross platform !

Slide 89

Slide 89 text

Dealing with Functional Bugs

Slide 90

Slide 90 text

Dealing with Functional Bugs • Maestro test command outputs

Slide 91

Slide 91 text

Dealing with Functional Bugs • Maestro test command outputs • Locally

Slide 92

Slide 92 text

Dealing with Functional Bugs • Maestro test command outputs • Locally • JUnit type format output available Element not found: Text matching regex: Follow

Slide 93

Slide 93 text

Dealing with Non-Functional Bugs

Slide 94

Slide 94 text

Dealing with Non-Functional Bugs • Frame drops 😖

Slide 95

Slide 95 text

Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic

Slide 96

Slide 96 text

Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow

Slide 97

Slide 97 text

Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow

Slide 98

Slide 98 text

Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow • Run: adb shell dumpsys gfxinfo com.twitter.android framestats

Slide 99

Slide 99 text

Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow • Run: adb shell dumpsys gfxinfo com.twitter.android framestats • Collect output and capture important metrics

Slide 100

Slide 100 text

Dealing with Non-Functional Bugs • Frame drops 😖 • Identify core flows of app which have high user traffic • Following twitter page flow • Write Maestro Flow • Run: adb shell dumpsys gfxinfo com.twitter.android framestats • Collect output and capture important metrics • Repeat

Slide 101

Slide 101 text

Dealing with Non-Functional Bugs • Root Causing Window: com.twitter.android/com.twitter.app.main.MainActivity Stats since: 100942401883393ns Total frames rendered: 387 Janky frames: 110 (28.42%) 50th percentile: 9ms 90th percentile: 22ms 95th percentile: 34ms 99th percentile: 450ms

Slide 102

Slide 102 text

Dealing with Non-Functional Bugs • Root Causing Window: com.twitter.android/com.twitter.app.main.MainActivity Stats since: 100942401883393ns Total frames rendered: 387 Janky frames: 110 (28.42%) 50th percentile: 9ms 90th percentile: 22ms 95th percentile: 34ms 99th percentile: 450ms

Slide 103

Slide 103 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage

Slide 104

Slide 104 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations

Slide 105

Slide 105 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations Debug API

Slide 106

Slide 106 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations Debug API Debug.startMethodTracingSampling( context.cacheDir, maxBufferSize, samplingIntervalUs )

Slide 107

Slide 107 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations Debug API Debug.startMethodTracingSampling( context.cacheDir, maxBufferSize, samplingIntervalUs ) Debug.stopMethodTracing() …… …… ……

Slide 108

Slide 108 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations

Slide 109

Slide 109 text

Dealing with Non-Functional Bugs • Root Causing • Monitor janky frame percentage • Generate flame graphs to narrow down on heavy operations • Capture Strict mode violations

Slide 110

Slide 110 text

Integrate with CI

Slide 111

Slide 111 text

Integrate with CI maestro cloud

Slide 112

Slide 112 text

Integrate with CI maestro cloud ./maestro —apiKey App file

Slide 113

Slide 113 text

Integrate with CI maestro cloud ./maestro —apiKey Test directory | 
 |___.maestro/ 
 | | 
 | |____ Login.yaml | | 
 | |____ Add to Cart.yaml |

Slide 114

Slide 114 text

Integrate with CI maestro cloud ./maestro —apiKey

Slide 115

Slide 115 text

Integrate with CI maestro cloud ./maestro —apiKey • Integrate with PRs ✅

Slide 116

Slide 116 text

Integrate with CI • Device management and scaling maestro cloud ./maestro —apiKey • Integrate with PRs ✅ ✅

Slide 117

Slide 117 text

Integrate with CI maestro cloud ./maestro —apiKey • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc. • Integrate with PRs ✅ ✅ • Device management and scaling ✅

Slide 118

Slide 118 text

Integrate with CI maestro cloud ./maestro —apiKey • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc. • Integrate with PRs ✅ ✅ • Device management and scaling ✅ • Trigger on PR stage ✅

Slide 119

Slide 119 text

Integrate with CI maestro cloud ./maestro —apiKey • Natively integrating with your CI: Github, Gitlab, Bitbucket, etc. • Integrate with PRs ✅ ✅ • Device management and scaling ✅ • Trigger on PR stage ✅ • Different API level devices ✅

Slide 120

Slide 120 text

Ideal setup of UI testing • Helpful for dealing with functional and non-functional issues • Setup should be very easy • Starting and writing first flow should be easy • Integration with CI should be very easy • Managing infrastructure should be easy ✅ ✅ ✅ ✅ ✅

Slide 121

Slide 121 text

Maestro is Open Source!

Slide 122

Slide 122 text

Maestro is Open Source! • https://github.com/mobile-dev-inc/maestro/

Slide 123

Slide 123 text

Maestro is Open Source! • https://github.com/mobile-dev-inc/maestro/

Slide 124

Slide 124 text

Community Support is Great • Performance tool to get FPS, CPU usage for search and explore interaction by Bamlab: https://twitter.com/almouro/status/1595402239044198400

Slide 125

Slide 125 text

Community Support is Great • Performance tool to get FPS, CPU usage for search and explore interaction by Craig Atkinson: https://www.atkinsondev.com/post/android-maestro-mock-rest-api/ • Mock API servers for Maestro Android testing using MockServer and Gradle

Slide 126

Slide 126 text

Community Support is Great • Performance tool to get FPS, CPU usage for search and explore interaction by Craig Atkinson: https://www.atkinsondev.com/post/android-maestro-testing/ • Mock API servers for Maestro Android testing using MockServer and Gradle • High level UI testing with Maestro

Slide 127

Slide 127 text

Community Support is Great by Manish Verma: https://twitter.com/mobile__dev/status/1580952966474846209 • Facebook registration flow automation • Performance tool to get FPS, CPU usage for search and explore interaction • Mock API servers for Maestro Android testing using MockServer and Gradle • High level UI testing with Maestro

Slide 128

Slide 128 text

Community Support is Great by Manish Verma: https://twitter.com/mobile__dev/status/1580952966474846209 • Facebook registration flow automation • Performance tool to get FPS, CPU usage for search and explore interaction • Mock API servers for Maestro Android testing using MockServer and Gradle • High level UI testing with Maestro • Much more: Join Maestro slack community !!

Slide 129

Slide 129 text

Fin! 🙋