Tweet
Tweet

Slide 1

Slide 1 text

Mobile Acceptance Testing Workshop mdevcon 2013 did you do the setup steps? github.com/moredip/mdevcon-workshop Thursday, March 14, 13

Slide 2

Slide 2 text

Pete Hodgson @ph1 thepete.net [email protected] Thursday, March 14, 13

Slide 3

Slide 3 text

Thursday, March 14, 13

Slide 4

Slide 4 text

Thursday, March 14, 13

Slide 5

Slide 5 text

enough about me, what about you? Thursday, March 14, 13

Slide 6

Slide 6 text

Thursday, March 14, 13

Slide 7

Slide 7 text

The Plan Thursday, March 14, 13

Slide 8

Slide 8 text

The Plan some theory, but centered on practical know-how lots of opportunities for hands on Thursday, March 14, 13

Slide 9

Slide 9 text

The Plan focus will be iOS and Frank but almost all concepts are generally applicable Thursday, March 14, 13

Slide 10

Slide 10 text

Thursday, March 14, 13

Slide 11

Slide 11 text

Agenda the value of testing acceptance testing fundamentals selecting bits of UI interacting with bits of UI inspecting bits of UI “advanced stuff” Thursday, March 14, 13

Slide 12

Slide 12 text

Thursday, March 14, 13

Slide 13

Slide 13 text

The Goal leave here feeling empowered to write your first automated acceptance tests Thursday, March 14, 13

Slide 14

Slide 14 text

Bonus Goal learn some acceptance testing ProTipstm Thursday, March 14, 13

Slide 15

Slide 15 text

interrupt, raise your hand, ask questions Please Thursday, March 14, 13

Slide 16

Slide 16 text

Thursday, March 14, 13

Slide 17

Slide 17 text

Manual Testing Thursday, March 14, 13

Slide 18

Slide 18 text

launch the app > cd your_app_directory > frank launch LAUNCHING IN THE SIMULATOR... Thursday, March 14, 13

Slide 19

Slide 19 text

manual test script Scenario: adding a new counter - launch app - tap + to add a new counter - choose a CountUp counter - enter “My Group” as a group name - enter “My Counter” as a counter name - tap Save - check you are back on the main screen - check you see your group and counter name - check that tapping the counter makes it count Thursday, March 14, 13

Slide 20

Slide 20 text

manual test script Scenario: adding a new counter - launch app - tap + to add a new counter - choose a CountUp counter - enter “My Group” as a group name - enter “My Counter” as a counter name - tap Save - check you are back on the main screen - check you see your group and counter name - check that tapping the counter makes it count again Thursday, March 14, 13

Slide 21

Slide 21 text

Thursday, March 14, 13

Slide 22

Slide 22 text

what have we discovered? - testing is a way to explore the app - testing can be tedious - humans tend to optimize - humans tend to make mistakes Thursday, March 14, 13

Slide 23

Slide 23 text

what else? Thursday, March 14, 13

Slide 24

Slide 24 text

did we find any bugs? Thursday, March 14, 13

Slide 25

Slide 25 text

what else would you test? Thursday, March 14, 13

Slide 26

Slide 26 text

Thursday, March 14, 13

Slide 27

Slide 27 text

running our first automated test Thursday, March 14, 13

Slide 28

Slide 28 text

running our first automated test > cd your_app_directory > frank build > cucumber Frank/features Thursday, March 14, 13

Slide 29

Slide 29 text

what just happened? Thursday, March 14, 13

Slide 30

Slide 30 text

iOS app acceptance tests Thursday, March 14, 13

Slide 31

Slide 31 text

iOS app acceptance tests Frank server Frank driver HTTP test scripts Thursday, March 14, 13

Slide 32

Slide 32 text

iOS app acceptance tests Frank server Frank driver HTTP test scripts Thursday, March 14, 13

Slide 33

Slide 33 text

Thursday, March 14, 13

Slide 34

Slide 34 text

why do we test? Thursday, March 14, 13

Slide 35

Slide 35 text

why do we test? interacting with the app to see what it does Thursday, March 14, 13

Slide 36

Slide 36 text

“does it work?” Thursday, March 14, 13

Slide 37

Slide 37 text

“does it still work?” Thursday, March 14, 13

Slide 38

Slide 38 text

“what’s it supposed to do anyway?” Thursday, March 14, 13

Slide 39

Slide 39 text

manual testing is tedious ... Thursday, March 14, 13

Slide 40

Slide 40 text

and we’re not very good at it ... manual testing is tedious ... Thursday, March 14, 13

Slide 41

Slide 41 text

but computers are ... and we’re not very good at it ... manual testing is tedious ... Thursday, March 14, 13

Slide 42

Slide 42 text

“Computers are designed to do simple repetitive tasks. The second you have humans doing repetitive tasks, all the computers get together late at night and laugh at you…” - Neal Ford Thursday, March 14, 13

Slide 43

Slide 43 text

Thursday, March 14, 13

Slide 44

Slide 44 text

why automate your tests Thursday, March 14, 13

Slide 45

Slide 45 text

why automate your tests - more time for exploratory QA Thursday, March 14, 13

Slide 46

Slide 46 text

why automate your tests - more time for exploratory QA - consistency in our testing Thursday, March 14, 13

Slide 47

Slide 47 text

why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements Thursday, March 14, 13

Slide 48

Slide 48 text

why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements - protect against regressions Thursday, March 14, 13

Slide 49

Slide 49 text

why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements - protect against regressions - enable courageous change Thursday, March 14, 13

Slide 50

Slide 50 text

why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements - protect against regressions - enable courageous change - what else? Thursday, March 14, 13

Slide 51

Slide 51 text

Thursday, March 14, 13

Slide 52

Slide 52 text

types of automated testing Thursday, March 14, 13

Slide 53

Slide 53 text

Mobile App Thursday, March 14, 13

Slide 54

Slide 54 text

Mobile App End User Thursday, March 14, 13

Slide 55

Slide 55 text

Mobile App End User Backend Services Thursday, March 14, 13

Slide 56

Slide 56 text

Unit Test Mobile App End User Backend Services Thursday, March 14, 13

Slide 57

Slide 57 text

Integration Test Mobile App End User Backend Services Thursday, March 14, 13

Slide 58

Slide 58 text

Acceptance Test Mobile App End User Backend Services Thursday, March 14, 13

Slide 59

Slide 59 text

Acceptance Test Mobile App End User Backend Services The App Thursday, March 14, 13

Slide 60

Slide 60 text

Acceptance Unit Integration Thursday, March 14, 13

Slide 61

Slide 61 text

a few reliable tests are better than lots of flakey tests ProTiptm Thursday, March 14, 13

Slide 62

Slide 62 text

run your tests as frequently as you can ProTiptm Thursday, March 14, 13

Slide 63

Slide 63 text

Thursday, March 14, 13

Slide 64

Slide 64 text

the UI testing triad Thursday, March 14, 13

Slide 65

Slide 65 text

select a bit o UI Thursday, March 14, 13

Slide 66

Slide 66 text

select a bit o UI simulate interaction with it inspect state o it Thursday, March 14, 13

Slide 67

Slide 67 text

UI select interact inspect Thursday, March 14, 13

Slide 68

Slide 68 text

UI select interact inspect Thursday, March 14, 13

Slide 69

Slide 69 text

symbiote > cd your_app_directory > frank launch > frank inspect Thursday, March 14, 13

Slide 70

Slide 70 text

symbiote Thursday, March 14, 13

Slide 71

Slide 71 text

demo! Thursday, March 14, 13

Slide 72

Slide 72 text

Thursday, March 14, 13

Slide 73

Slide 73 text

view selectors Thursday, March 14, 13

Slide 74

Slide 74 text

view selectors - a little language to specify which views to inspect/ interact with - like CSS or XPath Thursday, March 14, 13

Slide 75

Slide 75 text

Thursday, March 14, 13

Slide 76

Slide 76 text

Thursday, March 14, 13

Slide 77

Slide 77 text

Thursday, March 14, 13

Slide 78

Slide 78 text

view marked:‘A Group’ by accessibility label Thursday, March 14, 13

Slide 79

Slide 79 text

accessibility labels are your friend ProTiptm Thursday, March 14, 13

Slide 80

Slide 80 text

view marked:‘A Group’ by accessibility label Thursday, March 14, 13

Slide 81

Slide 81 text

tableViewCell by view class Thursday, March 14, 13

Slide 82

Slide 82 text

tableViewCell label drill down Thursday, March 14, 13

Slide 83

Slide 83 text

tableViewCell label drill down Thursday, March 14, 13

Slide 84

Slide 84 text

tableViewCell label drill down Thursday, March 14, 13

Slide 85

Slide 85 text

putting it together Thursday, March 14, 13

Slide 86

Slide 86 text

tableViewCell label marked:’A Counter’ putting it together Thursday, March 14, 13

Slide 87

Slide 87 text

avoid brittle view selection ProTiptm Thursday, March 14, 13

Slide 88

Slide 88 text

record & playback: not a sustainable approach ProTiptm Thursday, March 14, 13

Slide 89

Slide 89 text

Thursday, March 14, 13

Slide 90

Slide 90 text

UI select interact inspect Thursday, March 14, 13

Slide 91

Slide 91 text

UI select interact inspect Thursday, March 14, 13

Slide 92

Slide 92 text

user simulation touch(...) type_into_keyboard(...) set_orientation(...) double_tap(...) tap_and_hold(...) drag_with_initial_delay(...) Thursday, March 14, 13

Slide 93

Slide 93 text

touch(...) Thursday, March 14, 13

Slide 94

Slide 94 text

touch(...) view selector Thursday, March 14, 13

Slide 95

Slide 95 text

frank console > cd your_app_directory > frank launch > frank console connecting to app... connected [1] pry(#)> Thursday, March 14, 13

Slide 96

Slide 96 text

frank console > touch “button marked:‘Add’” Thursday, March 14, 13

Slide 97

Slide 97 text

UI select interact inspect Thursday, March 14, 13

Slide 98

Slide 98 text

UI select interact inspect Thursday, March 14, 13

Slide 99

Slide 99 text

view introspection frankly_map(view_selector, method) runs an ObjectiveC method on all matching UIView instances Thursday, March 14, 13

Slide 100

Slide 100 text

back in the console Thursday, March 14, 13

Slide 101

Slide 101 text

back in the console > selector = “button marked:‘Edit’” > element_exists( selector ) => true > frankly_map( selector, ‘isEnabled’ ) => [true] > frankly_map( selector, ‘isHidden’ ) => [false] > frankly_map( selector, ‘accessibilityFrame’ ) => [{"size"=>{"width"=>50, "height"=>30}, "origin"=>{"x"=>5, "y"=>27}}] Thursday, March 14, 13

Slide 102

Slide 102 text

interact inspect select UI Thursday, March 14, 13

Slide 103

Slide 103 text

putting it all together Thursday, March 14, 13

Slide 104

Slide 104 text

back in the console > touch “view marked:‘Add’” > touch “label marked:'CountUp'” > touch “view:'UITextFieldLabel' marked:'Group Name'” > type_into_keyboard “My Group” > touch “view:'UITextFieldLabel' marked:'Counter Name'” > type_into_keyboard “My Counter” > touch “button marked:'Save'” > element_exists “view marked:’My Group’” > element_exists “view marked:’My Counter’” > touch “view marked:’My Counter’” Thursday, March 14, 13

Slide 105

Slide 105 text

Thursday, March 14, 13

Slide 106

Slide 106 text

other platforms Thursday, March 14, 13

Slide 107

Slide 107 text

Android Robotium Calabash Thursday, March 14, 13

Slide 108

Slide 108 text

Mobile Web WebDriver (aka Selenium 2) with Capybara (if you’re using ruby) Thursday, March 14, 13

Slide 109

Slide 109 text

which WebDriver? Thursday, March 14, 13

Slide 110

Slide 110 text

which WebDriver? iPhoneDriver ~ or ~ a desktop WebDriver (e.g. chromedriver) Thursday, March 14, 13

Slide 111

Slide 111 text

Headless capybara-webkit poltergeist ghostdriver htmlunitdriver Thursday, March 14, 13

Slide 112

Slide 112 text

Thursday, March 14, 13

Slide 113

Slide 113 text

cucumber Thursday, March 14, 13

Slide 114

Slide 114 text

cucumber write tests which describe behavior in terms of user value Thursday, March 14, 13

Slide 115

Slide 115 text

cucumber Given... When... Then... Thursday, March 14, 13

Slide 116

Slide 116 text

cucumber Given I have a simple counter And I have counted to 10 When I reset the counter Then the counter should be at 0 Thursday, March 14, 13

Slide 117

Slide 117 text

How does that work? Thursday, March 14, 13

Slide 118

Slide 118 text

How does that work? Regular Expressions (!!!) Thursday, March 14, 13

Slide 119

Slide 119 text

Cucumber steps Given I have a simple counter And I have counted to 10 When I reset the counter Then the counter should be at 0 Thursday, March 14, 13

Slide 120

Slide 120 text

Thursday, March 14, 13

Slide 121

Slide 121 text

Cucumber step definitions Thursday, March 14, 13

Slide 122

Slide 122 text

Cucumber step definitions When /^I reset the counter$/ do touch( “view marked:‘Zero Counts’” ) end Thursday, March 14, 13

Slide 123

Slide 123 text

Cucumber step definitions When /^I reset the counter$/ do touch( “view marked:‘Zero Counts’” ) end Thursday, March 14, 13

Slide 124

Slide 124 text

Cucumber step definitions When /^I reset the counter$/ do touch( “view marked:‘Zero Counts’” ) end Thursday, March 14, 13

Slide 125

Slide 125 text

so, what does Cucumber do for us? Thursday, March 14, 13

Slide 126

Slide 126 text

... ... touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard “my group” touch “view marked:'Counter Name'” type_into_keyboard “my counter” touch “button marked:‘Save’” ... ... Thursday, March 14, 13

Slide 127

Slide 127 text

When I create a “my counter” counter Thursday, March 14, 13

Slide 128

Slide 128 text

When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13

Slide 129

Slide 129 text

When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13

Slide 130

Slide 130 text

When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13

Slide 131

Slide 131 text

When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13

Slide 132

Slide 132 text

“foo” When I create a counter Thursday, March 14, 13

Slide 133

Slide 133 text

“bar” When I create a counter Thursday, March 14, 13

Slide 134

Slide 134 text

“baz” When I create a counter Thursday, March 14, 13

Slide 135

Slide 135 text

Don’t write test scripts in cucumber ProTiptm Thursday, March 14, 13

Slide 136

Slide 136 text

Cucumber should be high-level and user-focussed ProTiptm Thursday, March 14, 13

Slide 137

Slide 137 text

embrace patterns like Page Object ProTiptm Thursday, March 14, 13

Slide 138

Slide 138 text

Thursday, March 14, 13

Slide 139

Slide 139 text

race conditions Thursday, March 14, 13

Slide 140

Slide 140 text

Thursday, March 14, 13

Slide 141

Slide 141 text

Thursday, March 14, 13

Slide 142

Slide 142 text

Thursday, March 14, 13

Slide 143

Slide 143 text

did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 144

Slide 144 text

did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 145

Slide 145 text

did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 146

Slide 146 text

did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 147

Slide 147 text

what about the network? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 148

Slide 148 text

what about the network? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 149

Slide 149 text

what about the network? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 150

Slide 150 text

what about the network? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 151

Slide 151 text

what about the network? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” fail Thursday, March 14, 13

Slide 152

Slide 152 text

what about the network? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” fail Thursday, March 14, 13

Slide 153

Slide 153 text

a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 154

Slide 154 text

a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13

Slide 155

Slide 155 text

a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” PROBLEM slow tests Thursday, March 14, 13

Slide 156

Slide 156 text

a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” PROBLEM slow tests PROBLEM fragile tests Thursday, March 14, 13

Slide 157

Slide 157 text

wait_until SOLUTION: Thursday, March 14, 13

Slide 158

Slide 158 text

wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13

Slide 159

Slide 159 text

wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13

Slide 160

Slide 160 text

wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13

Slide 161

Slide 161 text

wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13

Slide 162

Slide 162 text

wait_until touch “button marked:‘Login’” wait_for_element_to_exist do “view marked:‘Welcome back!’” end Thursday, March 14, 13

Slide 163

Slide 163 text

wait_until touch “button marked:‘Login’” wait_for_element_to_exist do “view marked:‘Welcome back!’” end Thursday, March 14, 13

Slide 164

Slide 164 text

Spin Assert ProTiptm Thursday, March 14, 13

Slide 165

Slide 165 text

Thursday, March 14, 13

Slide 166

Slide 166 text

Thursday, March 14, 13

Slide 167

Slide 167 text

wrapping up the value of testing (feedback) definition of an acceptance test selecting/interacting/inspecting frank console and symbiote “advanced stuff” Thursday, March 14, 13

Slide 168

Slide 168 text

this slide deck will be at: http://slides.thepete.net Thursday, March 14, 13

Slide 169

Slide 169 text

further research - ‘Page Object’ pattern - Spin Asserts - Quarantine broken tests - acceptance testing Journeys - Continuous Integration - faking out external services Thursday, March 14, 13

Slide 170

Slide 170 text

helpful resources testingwithfrank.com the cucumber book the rspec book scripting w. Ruby book specification by example book rubymine (good IDE for ruby/cuke dev.) eradicating non- determinism in tests (good blog post) creating maintainable acceptance tests (presentation slides and video) the example tests in the Frank repo Thursday, March 14, 13

Slide 171

Slide 171 text

more questions? @ph1 http://blog.thepete.net [email protected] me, at the bar feedback plz! http://rate.thepete.net Thursday, March 14, 13