Automated Mobile Acceptance Testing Workshop - mdevcon 2013

Transcript

1. Mobile Acceptance Testing Workshop mdevcon 2013 did you do the setup steps? github.com/moredip/mdevcon-workshop Thursday, March 14, 13
2. Pete Hodgson @ph1 thepete.net phodgson@thoughtworks.com Thursday, March 14, 13
3. Thursday, March 14, 13
4. Thursday, March 14, 13
5. enough about me, what about you? Thursday, March 14, 13
6. Thursday, March 14, 13
7. The Plan Thursday, March 14, 13
8. The Plan some theory, but centered on practical know-how lots of opportunities for hands on Thursday, March 14, 13
9. The Plan focus will be iOS and Frank but almost all concepts are generally applicable Thursday, March 14, 13
10. Thursday, March 14, 13
11. Agenda the value of testing acceptance testing fundamentals selecting bits of UI interacting with bits of UI inspecting bits of UI “advanced stuff” Thursday, March 14, 13
12. Thursday, March 14, 13
13. The Goal leave here feeling empowered to write your first automated acceptance tests Thursday, March 14, 13
14. Bonus Goal learn some acceptance testing ProTipstm Thursday, March 14, 13
15. interrupt, raise your hand, ask questions Please Thursday, March 14, 13
16. Thursday, March 14, 13
17. Manual Testing Thursday, March 14, 13
18. launch the app > cd your_app_directory > frank launch LAUNCHING IN THE SIMULATOR... Thursday, March 14, 13
19. manual test script Scenario: adding a new counter - launch app - tap + to add a new counter - choose a CountUp counter - enter “My Group” as a group name - enter “My Counter” as a counter name - tap Save - check you are back on the main screen - check you see your group and counter name - check that tapping the counter makes it count Thursday, March 14, 13
20. manual test script Scenario: adding a new counter - launch app - tap + to add a new counter - choose a CountUp counter - enter “My Group” as a group name - enter “My Counter” as a counter name - tap Save - check you are back on the main screen - check you see your group and counter name - check that tapping the counter makes it count again Thursday, March 14, 13
21. Thursday, March 14, 13
22. what have we discovered? - testing is a way to explore the app - testing can be tedious - humans tend to optimize - humans tend to make mistakes Thursday, March 14, 13
23. what else? Thursday, March 14, 13
24. did we find any bugs? Thursday, March 14, 13
25. what else would you test? Thursday, March 14, 13
26. Thursday, March 14, 13
27. running our first automated test Thursday, March 14, 13
28. running our first automated test > cd your_app_directory > frank build > cucumber Frank/features Thursday, March 14, 13
29. what just happened? Thursday, March 14, 13
30. iOS app acceptance tests Thursday, March 14, 13
31. iOS app acceptance tests Frank server Frank driver HTTP test scripts Thursday, March 14, 13
32. iOS app acceptance tests Frank server Frank driver HTTP test scripts Thursday, March 14, 13
33. Thursday, March 14, 13
34. why do we test? Thursday, March 14, 13
35. why do we test? interacting with the app to see what it does Thursday, March 14, 13
36. “does it work?” Thursday, March 14, 13
37. “does it still work?” Thursday, March 14, 13
38. “what’s it supposed to do anyway?” Thursday, March 14, 13
39. manual testing is tedious ... Thursday, March 14, 13
40. and we’re not very good at it ... manual testing is tedious ... Thursday, March 14, 13
41. but computers are ... and we’re not very good at it ... manual testing is tedious ... Thursday, March 14, 13
42. “Computers are designed to do simple repetitive tasks. The second you have humans doing repetitive tasks, all the computers get together late at night and laugh at you…” - Neal Ford Thursday, March 14, 13
43. Thursday, March 14, 13
44. why automate your tests Thursday, March 14, 13
45. why automate your tests - more time for exploratory QA Thursday, March 14, 13
46. why automate your tests - more time for exploratory QA - consistency in our testing Thursday, March 14, 13
47. why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements Thursday, March 14, 13
48. why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements - protect against regressions Thursday, March 14, 13
49. why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements - protect against regressions - enable courageous change Thursday, March 14, 13
50. why automate your tests - more time for exploratory QA - consistency in our testing - drives focus on requirements - protect against regressions - enable courageous change - what else? Thursday, March 14, 13
51. Thursday, March 14, 13
52. types of automated testing Thursday, March 14, 13
53. Mobile App Thursday, March 14, 13
54. Mobile App End User Thursday, March 14, 13
55. Mobile App End User Backend Services Thursday, March 14, 13
56. Unit Test Mobile App End User Backend Services Thursday, March 14, 13
57. Integration Test Mobile App End User Backend Services Thursday, March 14, 13
58. Acceptance Test Mobile App End User Backend Services Thursday, March 14, 13
59. Acceptance Test Mobile App End User Backend Services The App Thursday, March 14, 13
60. Acceptance Unit Integration Thursday, March 14, 13
61. a few reliable tests are better than lots of flakey tests ProTiptm Thursday, March 14, 13
62. run your tests as frequently as you can ProTiptm Thursday, March 14, 13
63. Thursday, March 14, 13
64. the UI testing triad Thursday, March 14, 13
65. select a bit o UI Thursday, March 14, 13
66. select a bit o UI simulate interaction with it inspect state o it Thursday, March 14, 13
67. UI select interact inspect Thursday, March 14, 13
68. UI select interact inspect Thursday, March 14, 13
69. symbiote > cd your_app_directory > frank launch > frank inspect Thursday, March 14, 13
70. symbiote Thursday, March 14, 13
71. demo! Thursday, March 14, 13
72. Thursday, March 14, 13
73. view selectors Thursday, March 14, 13
74. view selectors - a little language to specify which views to inspect/ interact with - like CSS or XPath Thursday, March 14, 13
75. Thursday, March 14, 13
76. Thursday, March 14, 13
77. Thursday, March 14, 13
78. view marked:‘A Group’ by accessibility label Thursday, March 14, 13
79. accessibility labels are your friend ProTiptm Thursday, March 14, 13
80. view marked:‘A Group’ by accessibility label Thursday, March 14, 13
81. tableViewCell by view class Thursday, March 14, 13
82. tableViewCell label drill down Thursday, March 14, 13
83. tableViewCell label drill down Thursday, March 14, 13
84. tableViewCell label drill down Thursday, March 14, 13
85. putting it together Thursday, March 14, 13
86. tableViewCell label marked:’A Counter’ putting it together Thursday, March 14, 13
87. avoid brittle view selection ProTiptm Thursday, March 14, 13
88. record & playback: not a sustainable approach ProTiptm Thursday, March 14, 13
89. Thursday, March 14, 13
90. UI select interact inspect Thursday, March 14, 13
91. UI select interact inspect Thursday, March 14, 13
92. user simulation touch(...) type_into_keyboard(...) set_orientation(...) double_tap(...) tap_and_hold(...) drag_with_initial_delay(...) Thursday, March 14, 13
93. touch(...) Thursday, March 14, 13
94. touch(...) view selector Thursday, March 14, 13
95. frank console > cd your_app_directory > frank launch > frank console connecting to app... connected [1] pry(#<Frank::Console>)> Thursday, March 14, 13
96. frank console > touch “button marked:‘Add’” Thursday, March 14, 13
97. UI select interact inspect Thursday, March 14, 13
98. UI select interact inspect Thursday, March 14, 13
99. view introspection frankly_map(view_selector, method) runs an ObjectiveC method on all matching UIView instances Thursday, March 14, 13
100. back in the console Thursday, March 14, 13
101. back in the console > selector = “button marked:‘Edit’” > element_exists( selector ) => true > frankly_map( selector, ‘isEnabled’ ) => [true] > frankly_map( selector, ‘isHidden’ ) => [false] > frankly_map( selector, ‘accessibilityFrame’ ) => [{"size"=>{"width"=>50, "height"=>30}, "origin"=>{"x"=>5, "y"=>27}}] Thursday, March 14, 13
102. interact inspect select UI Thursday, March 14, 13
103. putting it all together Thursday, March 14, 13
104. back in the console > touch “view marked:‘Add’” > touch “label marked:'CountUp'” > touch “view:'UITextFieldLabel' marked:'Group Name'” > type_into_keyboard “My Group” > touch “view:'UITextFieldLabel' marked:'Counter Name'” > type_into_keyboard “My Counter” > touch “button marked:'Save'” > element_exists “view marked:’My Group’” > element_exists “view marked:’My Counter’” > touch “view marked:’My Counter’” Thursday, March 14, 13
105. Thursday, March 14, 13
106. other platforms Thursday, March 14, 13
107. Android Robotium Calabash Thursday, March 14, 13
108. Mobile Web WebDriver (aka Selenium 2) with Capybara (if you’re using ruby) Thursday, March 14, 13
109. which WebDriver? Thursday, March 14, 13
110. which WebDriver? iPhoneDriver ~ or ~ a desktop WebDriver (e.g. chromedriver) Thursday, March 14, 13
111. Headless capybara-webkit poltergeist ghostdriver htmlunitdriver Thursday, March 14, 13
112. Thursday, March 14, 13
113. cucumber Thursday, March 14, 13
114. cucumber write tests which describe behavior in terms of user value Thursday, March 14, 13
115. cucumber Given... When... Then... Thursday, March 14, 13
116. cucumber Given I have a simple counter And I have counted to 10 When I reset the counter Then the counter should be at 0 Thursday, March 14, 13
117. How does that work? Thursday, March 14, 13
118. How does that work? Regular Expressions (!!!) Thursday, March 14, 13
119. Cucumber steps Given I have a simple counter And I have counted to 10 When I reset the counter Then the counter should be at 0 Thursday, March 14, 13
120. Thursday, March 14, 13
121. Cucumber step definitions Thursday, March 14, 13
122. Cucumber step definitions When /^I reset the counter$/ do touch( “view marked:‘Zero Counts’” ) end Thursday, March 14, 13
123. Cucumber step definitions When /^I reset the counter$/ do touch( “view marked:‘Zero Counts’” ) end Thursday, March 14, 13
124. Cucumber step definitions When /^I reset the counter$/ do touch( “view marked:‘Zero Counts’” ) end Thursday, March 14, 13
125. so, what does Cucumber do for us? Thursday, March 14, 13
126. ... ... touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard “my group” touch “view marked:'Counter Name'” type_into_keyboard “my counter” touch “button marked:‘Save’” ... ... Thursday, March 14, 13
127. When I create a “my counter” counter Thursday, March 14, 13
128. When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13
129. When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13
130. When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13
131. When /^I create a "(.*?)" counter$/ do |counter| touch “button marked:'CountUp'” touch “view marked:'Group Name'” type_into_keyboard DEFAULT_GROUP_NAME touch “view marked:'Counter Name'” type_into_keyboard counter touch “button marked:‘Save’” end Thursday, March 14, 13
132. “foo” When I create a counter Thursday, March 14, 13
133. “bar” When I create a counter Thursday, March 14, 13
134. “baz” When I create a counter Thursday, March 14, 13
135. Don’t write test scripts in cucumber ProTiptm Thursday, March 14, 13
136. Cucumber should be high-level and user-focussed ProTiptm Thursday, March 14, 13
137. embrace patterns like Page Object ProTiptm Thursday, March 14, 13
138. Thursday, March 14, 13
139. race conditions Thursday, March 14, 13
140.      Thursday, March 14, 13
141.    Thursday, March 14, 13
142.  
       Thursday, March 14, 13
143. did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
144. did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
145. did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
146. did we log in? touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
147. what about the network?      touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
148. what about the network?      touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
149. what about the network?    touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
150. what about the network?    touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
151. what about the network?    touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” fail Thursday, March 14, 13
152. what about the network?  
       touch “button marked:‘Login’” check_element_exists “view marked:‘Welcome back!’” fail Thursday, March 14, 13
153. a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
154. a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” Thursday, March 14, 13
155. a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” PROBLEM slow tests Thursday, March 14, 13
156. a tempting solution touch “button marked:‘Login’” sleep 2 check_element_exists “view marked:‘Welcome back!’” PROBLEM slow tests PROBLEM fragile tests Thursday, March 14, 13
157. wait_until SOLUTION: Thursday, March 14, 13
158. wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13
159. wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13
160. wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13
161. wait_until touch “button marked:‘Login’” wait_until do element_exists “view marked:‘Welcome back!’” end Thursday, March 14, 13
162. wait_until touch “button marked:‘Login’” wait_for_element_to_exist do “view marked:‘Welcome back!’” end Thursday, March 14, 13
163. wait_until touch “button marked:‘Login’” wait_for_element_to_exist do “view marked:‘Welcome back!’” end Thursday, March 14, 13
164. Spin Assert ProTiptm Thursday, March 14, 13
165. Thursday, March 14, 13
166. Thursday, March 14, 13
167. wrapping up the value of testing (feedback) definition of an acceptance test selecting/interacting/inspecting frank console and symbiote “advanced stuff” Thursday, March 14, 13
168. this slide deck will be at: http://slides.thepete.net Thursday, March 14, 13
169. further research - ‘Page Object’ pattern - Spin Asserts - Quarantine broken tests - acceptance testing Journeys - Continuous Integration - faking out external services Thursday, March 14, 13
170. helpful resources testingwithfrank.com the cucumber book the rspec book scripting w. Ruby book specification by example book rubymine (good IDE for ruby/cuke dev.) eradicating non- determinism in tests (good blog post) creating maintainable acceptance tests (presentation slides and video) the example tests in the Frank repo Thursday, March 14, 13
171. more questions? @ph1 http://blog.thepete.net phodgson@thoughtworks.com me, at the bar feedback plz! http://rate.thepete.net Thursday, March 14, 13