Building a Device Lab

by Lara Hogan and Destiny Montague
http://larahogan.me/devicelab/

Transcript

1. Building a Device Lab Destiny Montague and Lara Hogan co-LAB-orators at Etsy Slides and links: larahogan.me/devicelab/
2. Reminder:
   have your visitor’s
   device data ready!
3. What is Etsy?
   (and who are we?)
4. Engineering Teams at Etsy • Feature Teams • Listing page • Seller tools • Checkout
5. Engineering Teams at Etsy • Feature Teams • Listing page • Seller tools • Checkout • Infrastructure Teams • Performance • Dev tools • Security Destiny
6. Why’d we build
   a device lab?
7. traffic to Etsy from
   mobile web+apps: > 50%
8. How many times have we built a device lab?
9. v1: Gray Cabinet v2: Bookshelves v3: Leaning Tower Our device lab’s history
10. Our device lab by the numbers • 36 devices • 8 operating systems • 45 average users/month • 4 median checkouts/device • 8 devices retired this year, 
 12 new purchased v3: Leaning Tower
11. a note about privilege
12. what works for us
    may not work for you
13. What we’ll cover today 1. Choosing Devices (10 min) 2. Power (20 min) 3. Device Setup (10 min) 4. Setting Up Testing (5 min) 5. Networking, RFID Readers (15 min) 6. User Experience & Organization (15 min)
14. Choosing Devices
15. find your visitors’ devices
17. Device OS Version % Total Sessions Size Apple iPhone 7.1 40% handset Apple iPad 7.1 7.9% small/large tablet Apple iPhone 7.0.6 4.9% handset LG Nexus 5 4.4.2 4.5% handset Google Nexus 4 4.4.2 4% handset Apple iPhone 7.0.4 3.2% handset Apple iPhone 7.1.1 3.2% handset Google Nexus 4 4.3.1 2.5% handset Google Nexus 7 4.4.2 2.1% small tablet Apple iPhone 6.1.3 1.5% handset
18. Get the most bang
    for your buck
19. focus on the 20% of devices that produce 
 80% of your traffic
20. Device OS Version % Total Sessions Size Apple iPhone 7.1 40% handset Apple iPad 7.1 7.9% small/large tablet Apple iPhone 7.0.6 4.9% handset LG Nexus 5 4.4.2 4.5% handset Google Nexus 4 4.4.2 4% handset Apple iPhone 7.0.4 3.2% handset Apple iPhone 7.1.1 3.2% handset Google Nexus 4 4.3.1 2.5% handset Google Nexus 7 4.4.2 2.1% small tablet Apple iPhone 6.1.3 1.5% handset
21. cover your visitors’ screen sizes and operating systems
22. Device OS Version % Total Sessions Size Apple iPhone 7.1 40% handset Apple iPad 7.1 7.9% small/large tablet Apple iPhone 7.0.6 4.9% handset LG Nexus 5 4.4.2 4.5% handset Google Nexus 4 4.4.2 4% handset Apple iPhone 7.0.4 3.2% handset Apple iPhone 7.1.1 3.2% handset Google Nexus 4 4.3.1 2.5% handset Google Nexus 7 4.4.2 2.1% small tablet Apple iPhone 6.1.3 1.5% handset
23. buggiest:
    Kindle, older Android
24. Device OS Version % Total Sessions Size Apple iPhone 7.1 40% handset Apple iPad 7.1 7.9% small/large tablet Apple iPhone 7.0.6 4.9% handset LG Nexus 5 4.4.2 4.5% handset Google Nexus 4 4.4.2 4% handset Apple iPhone 7.0.4 3.2% handset Apple iPhone 7.1.1 3.2% handset Google Nexus 4 4.3.1 2.5% handset Google Nexus 7 4.4.2 2.1% small tablet Apple iPhone 6.1.3 1.5% handset
25. Does usage differ
    if you segment by audience region
    or target market?
26. combination of
    touch and cursor
27. retina/DPI
28. Device OS Version % Total Sessions Size Apple iPhone 7.1 40% handset Apple iPad 7.1 7.9% small/large tablet Apple iPhone 7.0.6 4.9% handset LG Nexus 5 4.4.2 4.5% handset Google Nexus 4 4.4.2 4% handset Apple iPhone 7.0.4 3.2% handset Apple iPhone 7.1.1 3.2% handset Google Nexus 4 4.3.1 2.5% handset Google Nexus 7 4.4.2 2.1% small tablet Apple iPhone 6.1.3 1.5% handset
29. Device OS Version % Total Sessions Size Apple iPhone 7.1 40% handset Apple iPad 7.1 7.9% small/large tablet Apple iPhone 7.0.6 4.9% handset LG Nexus 5 4.4.2 4.5% handset Google Nexus 4 4.4.2 4% handset Apple iPhone 7.0.4 3.2% handset Apple iPhone 7.1.1 3.2% handset Google Nexus 4 4.3.1 2.5% handset Google Nexus 7 4.4.2 2.1% small tablet Apple iPhone 6.1.3 1.5% handset
30. purchasing devices
31. used devices better- represent battery life
32. wifi-capable vs. data plan
33. eBay, Amazon,
    local cell phone dealers
34. our devices
35. In our lab: • 12 Android phones (a range of operating system versions and major manufacturers) • 3 Android tablets (a range of screen resolutions) • 1 Blackberry phone • 1 Firefox phone (developer preview) • 5 iPads (including 1 mini, different generations)
36. In our lab (cont’): • 5 iPhones/iPods • 1 Kindle Fires (range of screen resolutions) • 1 Windows Phone, 1 Windows Surface, 1 Dell laptop • 1 Google Glass, 1 Chromebook Pixel • 1 Raspberry Pi, and some Kubis
37. More reading • Strategies for choosing test devices: 
 http://stephanierieger.com/strategies-for-choosing-test- devices/ • How to build a device lab, with a thorough first section on choosing and acquiring devices:
 http://dmolsen.com/2012/06/26/how-to-build-a- device-lab-part-1/
38. questions?
39. Power
41. electricity 101
42. https://www.flickr.com/photos/marktee/9223003535 electricity = water
43. https://www.flickr.com/photos/marktee/9223003535 usb/power cables = garden hose
44. amps vs. volts
45. amperage
46. voltage
47. volts * amps = watts!
49. power considerations
50. how much power does each device draw, 
 according to the manufacturer's specs?
51. how much power can the 
 surge protector
    support?
52. how much power can each usb hub support?
54. build to support
    your most
    power-hungry devices
55. power problems
56. power problems won’t accept a charge cycle on/off forever won’t turn on blinking battery can’t hold a charge someone stole the battery
57. http://www.microsoft.com/surface/en-us/support/warranty-service-and-recovery/
58. troubleshooting
59. start simple
60. "When you see hoof prints,
    look for horses,
    not zebras"
61. isolate the problem
62. DEVICE HARDWARE SOFTWARE
63. DEVICE HARDWARE cables charging port peripherals SOFTWARE battery display
64. DEVICE HARDWARE SOFTWARE apps operating system your base image cables charging port peripherals battery display
65. why?
66. •devices drew too much power
    ! •devices weren’t getting enough power
    ! •bad USB cables / peripherals
    ! •bad software
    ! •bad device
67. iterating on power
68. v1 - power strips
69. v2 - stupid USB hubs
70. v3 - better USB hubs
71. Cambrionix hub
72. Cambrionix hub • 32 ports • smart power delegation • fault indicator • disables problem devices
73. v4 - better cabling/monitoring
74. energy consumption
76. modlet timer
78. pro power tips
79. http://iheartorganizing.blogspot.com/2012/03/you-asked-organizing-cords.html
80. have spares/duplicates
    as back-ups
81. have extra cables handy for people to steal
82. questions?
83. Device Setup
84. configuration
85. factory (re)format
86. disable
    auto-updates
87. create an
    app store account
88. install apps
89. restrict devices from syncing to computers
90. set browser
    default homepage
91. make use of wallpaper
92. set up a passcode
93. have a way to identify devices
94. have a backup image
95. enroll devices in a
    mobile device management console (MDM)
96. choosing your MDM
97. scalability
98. app management
99. admin console
100. security
101. http://www.enterpriseios.com/wiki/Comparison_MDM_Providers MDM providers
102. questions?
103. Testing
106. 0. set up an Adobe account with the device lab
     email address
     ! (save this password somewhere;
     you’re gonna need it later.)
107. 1. install the Adobe Edge Inspect app via Creative Cloud desktop app
     ! (yeah, you need them both. and it has to be running for everything else to work.)
108. 2. install the
     browser
     extension
     ! (it’ll remind you to turn on the
     application if you haven’t yet.)
109. 3. install the app on
     iOS, Android and
     Kindle devices
     ! (right, it’s not on every operating system yet,
     and this is a great case for getting an MDM.)
110. 4. enter the pin for
     each device in the
     browser extension
     ! (you’ll have to upgrade to test on
     multiple devices at once.) http://www.adobe.com/inspire/2012/12/web-designs-edge-inspect.html
111. if you upgrade your account, you’ll want to reuse it on every laptop used for testing...
112. which is why it’s great to keep one laptop in the lab for this express purpose.
114. save all the
     screenshots at once! http://www.adobe.com/inspire/2012/12/web-designs-edge-inspect.html
115. development machines
     & staging environments
116. ensure it’s easy to
     test development on your devices
117. ensure it’s easy to
     test all environments on
     your devices
119. questions?
120. Networking
121. get connected
122. use wifi
123. bypass activation
124. http://www.droidxforums.com/forum/droid-x-faqs/506-bypass-android-activation-screen-motorola-droid-x.html B A
125. designate your
     own network
126. private SSID
127. Guest Network Internal Network Device Lab Network Access - basic web - no internal sites - VPN is blocked ! - basic web - access to internal sites - some sites require VPN access to prod network - basic web - only pre-production/ testing & VM stuff that is essential for testing our internal wifi networks
128. networking problems
129. decrease in
     wireless range
130. decrease in data throughput
131. intermittent or dropped wifi connection
132. difficulty pairing
     with bluetooth
133. signal-to-noise ratio
134. signal-to-noise ratio
136. RFID <3
138. what is RFID?
139. “Wireless non-contact use of radio-frequency electromagnetic fields to transfer data.”
141. high or low
     frequency
142. employee badges
143. usefulness
144. links to employee
145. saves time
146. can provide stats
     and trends
147. less expensive
148. components
149. office h4x0rz
150. RFID tags
151. RFID reader
152. raspberry pi
153. LCD screen, keyboard, misc cables
154. the stump by Trish Czech of “Timid Monsters”
158. problems
159. interference
160. identifying input
161. questions?
162. User Experience
163. surprises
164. surprises changed passcode left unplugged broken screen stolen cables lost devices connected personal facebook account changed wallpaper changed network connected personal email address upgraded operating system stolen stands stolen battery
165. make it easy to
     plug devices back in
166. cable drops, hooks or ties
167. mark cables and hubs
168. make stealing cables hard
169. Access to power In v2, power strips were very accessible so that we could watch the timed power strip for greenness.
170. Access to power In v3 we hid the power strips so users can’t easily steal cables and the lab looks more organized. We still made it easy for our team to go in and 
 re-cable devices.
171. make it easy to
     stand devices up
     for testing & returning
172. purchase a range of sizes
173. not one-size-fits-all
174. or, build-in stands
175. give people
     other things to steal
176. Group your devices There are many ways to organize your devices. Be consistent, such as grouping by: • device size (handsets, small tablets, large tablets, etc.) • operating system • unique devices on the top shelf (Google Glass, Firefox OS, Windows phone)
177. Labels on devices and shelves Make it easy to match devices to their home base. • Label the front of the shelf and the back of the device with the same wording. • Color-code by operating system so it’s easy to scan.
178. Device information For each device, label nearby: • device name and ID number • screen resolution, pixel density • operating system version
180. Keep major passwords a secret The device unlock passcodes should all be the same and well-known. But don’t tell anybody: • app store passwords • email passwords • WiFi network password (if you’re using a subnet specifically for the device lab)
181. check the lab regularly
182. plugged in & charging
183. missing devices
184. unlock to find surprises
185. usability tips
186. Document a how-to • Passcodes (the same for every device) • Instructions for using Adobe Edge Inspect • Instructions for testing on VMs/staging • Instructions for taking screenshots and checking email on various operating systems
187. Email on all devices • Set up a common email address (devicelab@) for testing email sends • Install a default email client and any other major email apps for testing • Configure it to receive all devicelab@ emails
188. Put reminders everywhere You’ll need to constantly remind people to not upgrade apps or OSes, and to check out devices. • as background images across devices • on the wall next to the lab • on all wiki docs
189. Make it easy to check out devices • Library card underneath 
 each device (color-coded, 
 of course) • RFID tags on each device and reader that works with employee badges
190. Make it easy to test • Set up a shared laptop to connect with Adobe Edge Inspect • you only have to enter all the pins once • use a laptop (like Windows!) that can stay in your lab • Make sure VMs and staging environments can be accessed
191. Make it easy to collaborate • Set up comfortable seating nearby • Nearby long tables can be handy, too • Make sure other outlets are available for people to plug in their own laptops • Make it so that people don’t have to walk back to their desks to test their work
192. Make it attractive A good-looking lab is inviting. • New hires see it on their tour and guests want to check it out • No one wants to be near clutter • The more organized, the more likely they’ll put stuff back properly and want to use it
193. examples
194. Physical shelving: v2 1. Unique devices 2. iOS devices 3. Kindles and Android tablets 4. Android handsets 5. Crates and power
195. Physical shelving: v2 • Cable drops for organization • Library cards and washi tape for labels • Different-height shelves
196. Physical shelving: v3 1. Unique devices 2. Kindles | iOS tablets 3. Android | iOS handsets 4. Android | iOS handsets 5. Crates and misc.
197. Physical shelving: v3 • Cables & power hidden behind shelving • Internal dashboard for device info and checkouts • Different-height shelves
198. Thank you!
     larahogan.me/devicelab/
     @lara_hogan @thisiscarlsagan