Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Adapting to Input

Adapting to Input

Responsive Web Design has forced us to accept that we don't know the size of our canvas, and we've learned to embrace the squishiness of the web. Input, it turns out, is every bit as challenging as screen size. We have tablets with keyboards, laptops that become tablets, laptops with touch screens, phones with physical keyboards, and even phones that become desktop computers. In this session, Jason will guide you through the input landscape, showing you new forms of input like sensors and voice control, as well as new lessons about old input standbys. You'll learn the design principles necessary to build web sites that respond and adapt to whatever input people use.


Jason Grigsby

June 14, 2016


  1. Adapting to Input Jason Grigsby • @grigs • cloudfour.com

  2. Web was created formless. https://www.flickr.com/photos/chiaralily/3068047592

  3. http://www.wired.co.uk/magazine/archive/ 2014/03/web-at-25/tim-berners-lee

  4. https://www.flickr.com/photos/inpivic/5206515374/

  5. https://www.flickr.com/photos/tomokyo-blue/5812115264

  6. http://www.flickr.com/photos/60415054@N00/14301113/ So we told ourselves that the web was…

  7. http://www.flickr.com/photos/60415054@N00/14301113/ 640 px 480 px

  8. 640 px 480 px

  9. 800 px 600 px

  10. 1024 px 768 px

  11. http://www.flickr.com/photos/adactio/6153481666/ Then mobile came and made us realize…

  12. that it was a consensual hallucination all along. http://www.flickr.com/photos/garibaldi/303085857/

  13. The web never had a fixed canvas. http://www.flickr.com/photos/paulocarrillo/124755065/

  14. None
  15. https://www.flickr.com/photos/giuliatanini/7125944117/

  16. http://www.flickr.com/photos/69797234@N06/7203485148/ We’ve made tremendous prog

  17. But there is another consensual hallucination. http://www.flickr.com/photos/garibaldi/303085857/

  18. = =


  20. None
  21. None
  22. None
  23. It’s not that we’re technically incapable, but adapting a phone

    UI to a tablet UI is not so dissimilar from trying to automatically adapt desktop UI to a phone. They are fundamentally different platforms with different usability considerations… —Todd Anglin, Kendo UI http://www.kendoui.com/blogs/teamblog/posts/12-09-11/universal_mobile_apps_with_html5_and_kendo_ui.aspx
  24. It’s fairly certain that the highest-value use will stay predominantly

    on desktop… Most complex tasks have vastly better user experience on the desktop and thus will be performed there. —Jacob Nielsen https://www.nngroup.com/articles/transmedia-design-for-3-screens/
  25. http://www.flickr.com/photos/royalsapien/2387707860

  26. We envision desktop users as different from mobile. http://demos.kendoui.com/web/grid/editing.html

  27. http://www.flickr.com/photos/shantellmartin/4543010568

  28. https://www.flickr.com/photos/keh-s/6020885164/ It’s time to break free of our assumptions about

  29. Four truths about input

  30. #1 Input is exploding

  31. An incomplete history of computer input

  32. Focus on broad adoption Using Apple as a lens

  33. Keyboard 1874 First typewriters in 1860s Remington Typewriter 1874 Popularizes

    Qwerty keyboard Qwerty patent granted 1878
  34. Mouse 1984 Mouse prototypes 1960s by Engelbart and Mallebrein. Xerox

    uses mouse in 70s, early 80s. Apple Macintosh popularizes mouse and GUI in 1984.
  35. Many alternatives to the mouse are created—trackballs, trackpads, and joysticks

    to name a few—but all replace the mouse and control the cursor. No change in behavior.
  36. Scroll Wheel 1996 First scrolling mouse in 1985. Scroll wheel

    on side. Microsoft Intellimouse popularizes scroll wheel in 1996. Scroll wheel controls windows, not cursor.
  37. Camera 2005 / 2006 Cameras are now included in phones

    and computers. Cameraphones sold in 2005 alone greater than all digital cameras sold to date. All Mac computers include cameras starting 2006.
  38. iPhone 2007 Multi-touch screen Camera Accelerometer Proximity Sensor Ambient Light

  39. Macbook Air January 2008 Multi-touch trackpad Gestures in Mac OS

    Controls more than cursor
  40. iPhone 3G 2008 GPS (Location services)

  41. iPhone 3GS 2009 Voice control Magnetometer (Compass)

  42. iPhone 4 2010 Gyroscope Front facing camera

  43. Magic Trackpad 2010 Multi-touch trackpad Gestures in Mac OS No

    longer limited to laptops
  44. iPhone 4S 2011 Siri Bluetooth LE

  45. iPhone 5 2012 Nothing?

  46. iPhone 5S 2013 Fingerprint Sensor

  47. iPhone 6 2014 NFC (Apple Pay) Barometer

  48. iPhone 6S 2015 3D Touch

  49. Same story could be told using Android or Microsoft.

  50. 1984 1874 142 years 1996 12 years

  51. 1984 1874 1996

  52. 1984 1996 2005 9 years 2 years 2007

  53. 1984 1996 2005 2007

  54. 2007

  55. 2007 Touch Camera Accelerometer Proximity Sensor Ambient Light Sensor GPS

    Trackpad 2008 Voice control Magnetometer 2009 2010 Gyroscope Front-facing Camera Magic Trackpad 2011 Siri Bluetooth LE 2012 2013 Fingerprint Sensor 2014 NFC (Apple Pay) 2015 3D Touch 2016 ?
  56. Touch 3D Touch Fingerprint sensor Camera (video and image) GPS

    (location) Compass Bluetooth LE Audio (Siri) Gyroscope Accelerometer Barometer Proximity sensor Ambient light sensor NFC (Apple Pay) iBeacon
  57. Things aren’t slowing down. https://www.flickr.com/photos/andreastsonis/8675283555

  58. Siri gets all of the hype… http://www.98ps.com/viewnews-15222.html

  59. but both Microsoft and Google have compelling voice input in

    their products.
  60. None
  61. None
  62. None
  63. How should web pages change to support voice control?

  64. None
  65. None
  66. Google voice search

  67. https://www.youtube.com/watch?v=kwn9Lh0E_vU

  68. Remember Leap Motion?

  69. https://www.youtube.com/watch?v=LJPxyWM9Ujg

  70. https://www.youtube.com/watch?v=fiZkEYLXctE

  71. #2 Input is a continuum

  72. = =

  73. Supports hover and pointer events.

  74. Keyboard and touch.

  75. Even the iPhone can have a keyboard.

  76. Are these laptops or tablets?

  77. Desktop computer with 23” touch screen

  78. None
  79. Luke Wroblewski nailed it. http://static.lukew.com/unified_device_design.png

  80. We can no longer make assumptions about input based on

    screen size or form factor.
  81. 3. Input is undetectable

  82. Higher precision with mouse means smaller targets possible Hover state

    Less precise than mouse and requires larger touch targets Typing easier for many No hover state Typing often more difficult Right clicking and “power” tools Single and multi-touch gestures
  83. None
  84. I got this. Detect touch. http://www.flickr.com/photos/28096801@N05/5012309802

  85. Whatever you may think, it currently isn't possible to reliably

    detect whether or not the current device has a touchscreen, from within the browser. —Stu Cox http://www.stucox.com/blog/you-cant-detect-a-touchscreen/
  86. Chrome has entertained idea of enabling touch by default. https://code.google.com/p/chromium/issues/detail?id=159527

  87. Detect a mouse? Not reliably.

  88. Surely we can detect a keyboard? NOPE

  89. https://www.flickr.com/photos/mediaflex/4628277817/

  90. https://www.flickr.com/photos/ dentyone/2410917872/

  91. Be thankful. This saves us from ourselves.

  92. Interactive touch laptop experiments by Boris Smus

  93. None
  94. We can only detect input when it is used and

    that’s too late for our user interfaces.
  95. 4. Input is transient

  96. None
  97. Knowing what input is used one moment tells you nothing

    about what will be used next.
  98. Input is exploding continuum undetectable transient

  99. Overwhelmed? https://www.flickr.com/photos/cheo70/2702682262/

  100. https://www.flickr.com/photos/evilerin/3723714381/

  101. None
  102. We just need to rewire our brains again.

  103. Adapting to input

  104. #1 Design for the largest target by default

  105. None
  106. Fitts’s Law The time to acquire a target is a

    function of the distance to and size of the target.
  107. After poking at this problem for a few weeks, my

    conclusion is: every desktop UI should be designed for touch now. When any desktop machine could have a touch interface, we have to proceed as if they all do. —Josh Clark http://globalmoxie.com/blog/desktop-touch-design.shtml
  108. #2 Design for modes of interaction instead of inputs

  109. Display density settings

  110. Th Dream Experience - … Uploaded 2 years ago More

    Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode
  111. Couch Mode + See all Centric TV’s videos / Recently

    viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Couch Mode + See all Centric TV’s videos / Recently viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode
  112. Vimeo Couch Mode

  113. You’re designing for user need—not for a specific form factor

    or input.
  114. #3 Make things accessible

  115. D-pad remote controls

  116. http://www.flickr.com/photos/stewc/6669743035/ TVs browsers that support d-pad, send arrow key events.

  117. If then http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/

  118. is undetectable.

  119. If then http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/

  120. Support http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/

  121. Building accessibly increases the likelihood of support for future, unknown

  122. None
  123. None
  124. #4 Design for multiple concurrent inputs

  125. = =

  126. Interactive touch laptop experiments by Boris Smus

  127. At minimum, don’t assume if one input is present that

    the person doesn’t have access to other types of input.
  128. Look for opportunities to combine input to create more powerful

  129. #5 Abstract baseline input

  130. Mobile Tablet Desktop

  131. Small Medium Large

  132. TAP https://www.flickr.com/photos/openexhibits/5884451755

  133. CLICK


  135. Mouse events and touch events are fundamentally different beasts in

    browsers today, and that makes it hard to write cross-platform apps. These incompatibilities lead to applications having to listen to 2 sets of events, mouse on desktop and touch for mobile. https://github.com/jquery/PEP/
  136. http://ie.microsoft.com/testdrive/ieblog/2011/Sep/20_TouchInputforIE10andMetrostyleApps_1.png http://www.w3.org/TR/pointerevents/ http://blog.webplatform.org/2013/02/pointing-toward-the-future/ New Pointer Events spec normalizes touch, mouse

    and stylus Pointer Events builds on the DOM event model to offer a new way to handle input on the web, enabling developers to build touch-first experiences that work with mouse, pen, and other pointing devices as well…They are also designed from the ground up to allow modern browsers to accelerate the touch-surface performance, leading to a smoother user experience.
  137. Shipped Opposed In development

  138. jQuery Pointer Events Polyfill https://github.com/jquery/PEP/

  139. #6 Progressively enhance input

  140. Opportunity is knocking!

  141. None
  142. None
  143. We’re pretty good at using Geolocation.

  144. Compass.js http://ai.github.io/compass.js/

  145. Warby Parker Gyroscope https://www.warbyparker.com/eyeglasses/women/simone/tea-rose-fade

  146. Or hover

  147. None
  148. https://www.youtube.com/watch?v=7Yp0BpiDzXE

  149. None
  150. None
  151. https://www.youtube.com/watch?v=Ow8gA0qe9SI

  152. <input type="email" autocomplete="email" name="email"> <input type="text" autocomplete="cc-name" name="card-name"> <input type="text"

    autocomplete="cc-number" name="card-num"> <input type="text" autocomplete="cc-exp-month" name="card-exp-month"> <input type="text" autocomplete="cc-exp-year" name="card-exp-year"> <input type="text" autocomplete="cc-csc" name="card-csc"> Remember to support browser auto-fill and new Payment Request API http://www.html5rocks.com/en/tutorials/forms/requestautocomplete/
  153. None
  154. None
  155. You can use speech recognition too. http://www.google.com/intl/en/chrome/demos/speech.html http://www.moreawesomeweb.com/demos/speech_translate.html

  156. Special thanks to Eric Bidelman http://moreawesomeweb.com Speech Recognition API Support

  157. https://www.youtube.com/watch?v=_BUwOBdLjzQ Web Bluetooth

  158. https://www.youtube.com/watch?v=FYxQ5TeVFdE Physical Web

  159. https://www.youtube.com/watch?v=6z9ED4fmi1w Physical Web &
 Web Bluetooth

  160. #7 Make input part of test plans

  161. Need to add input to our device labs http://blog.adtile.me/2014/01/08/adtile-device-lab/

  162. https://www.flickr.com/photos/jeepersmedia/14692542501

  163. https://www.flickr.com/photos/yuckymuffintech/6988362001

  164. None
  165. 1. Design for the largest target by default 2. Design

    for modes of interaction instead of inputs 3. Make things accessible 4. Design for multiple concurrent inputs 5. Abstract baseline input 6. Progressively enhance input 7. Make input part of test plans
  166. Input is exploding continuum undetectable transient

  167. www.flickr.com/photos/brunauto/5062644167/ Learn from our mobile context mistakes.

  168. Who are we to judge which input is better? http://www.flickr.com/photos/fensterbme/4783366926

  169. We need to learn to adapt. http://www.flickr.com/photos/cdm/147947664/

  170. Learn how to let go of illusions that comfort us.

  171. Thank You!