Adaptive Input — WebVisions Portland 2014

Transcript

1. Adaptive Input Jason Grigsby • @grigs • cloudfour.com
2. Follow along at @grigs_talks http://bit.ly/grigs-wvpdx14
3. http://www.flickr.com/photos/cdm/51747860/
4. http://www.flickr.com/photos/rheaney/4397863376 It started with TVs.
5. Designing for a 10-foot UI is very different. http://www.flickr.com/photos/chrisbartow/5835428673
6. Larger text and fewer words.
7. Make up, down, left, right directions clear. http://images.dailytech.com/nimage/29122_large_amazon_prime_screen_5.jpg
8. How do we know what is a TV?
9. This is HDTV.
10. This is HDTV. 1980 px 1080 px
11. Resolution does not define the optimal experience.
12. Next came responsive web apps. https://twitter.com/freediverx/status/354698695041744896
13. Responsive design for apps is inevitable. http://blog.cloudfour.com/responsive-design-for-apps-part-1/
14. Lines between device classes are blurring Model Type Size Size Display Resolution Resolution Viewport Viewport W H W H W H Samsung Galaxy Note 2 Phone 3.17” 5.95” 5.5” 720 1280 360 640 Motorola RAZR HD Phone 2.67” 5.19” 4.7” 720 1280 360 519 Motorola Atrix HD Phone 2.75” 5.26” 4.5” 720 1280 540 812 HTC Droid DNA Phone 2.78” 5.5” 5” 1080 1920 360 640 Nexus 7 Tablet 4.72” 7.81” 7” 800 1280 600 793 Kindle Fire Tablet 4.72” 7.44” 7” 600 1024 600 819 Kindle Fire HD Tablet 5.4” 7.6” 7” 800 1280 533 731
15. 640 px 600 px 519 px 640 px 622 px 533 px 812 px Which are phones and which are tablets?
16. http://www.flickr.com/photos/geatchy/8489505999
17. How do I make this responsive? How do I make this responsive?
19. mobile desktop THE ART OF WEB DEVELOPMENT THE ART OF WEB DEVELOPMENT Web widgets THE ART OF WEB DEVELOPMENT THE ART OF WEB DEVELOPMENT Mobile widgets
26. It’s not that we’re technically incapable, but adapting a phone UI to a tablet UI is not so dissimilar from trying to automatically adapt desktop UI to a phone. They are fundamentally different platforms with different usability considerations, and something that makes sense on phones may or may not belong on tablets. —Todd Anglin, Kendo UI http://www.kendoui.com/blogs/teamblog/posts/12-09-11/universal_mobile_apps_with_html5_and_kendo_ui.aspx
27. Sometimes it’s hard to envision a responsive version. http://demos.kendoui.com/web/grid/editing.html
28. http://www.flickr.com/photos/jesuspresley/384080245/ We want people to be productive…
29. and stay in the zone. http://www.flickr.com/photos/raccatography/8038855203
30. http://www.flickr.com/photos/shantellmartin/4543010568 Which seems very different from playing on an iPad.
31. For both the TV…
32. and the desktop web app…
33. input matters much more than screen size.
34. The grid is important to support d-pad interaction. http://images.dailytech.com/nimage/29122_large_amazon_prime_screen_5.jpg
35. http://www.flickr.com/photos/royalsapien/2387707860
36. And keyboard and mouse are what we envision work is. http://www.flickr.com/photos/royalsapien/2387707860
37. http://www.flickr.com/photos/hellogeri/6154034099/ A few years ago, Jeremy talked about how…
38. http://www.flickr.com/photos/60415054@N00/14301113/ we told ourselves that the web was…
39. http://www.flickr.com/photos/60415054@N00/14301113/
40. http://www.flickr.com/photos/60415054@N00/14301113/ 640 px 480 px
41. 640 px 480 px
46. 1024 px 768 px
47. http://www.flickr.com/photos/adactio/6153481666/
48. http://www.flickr.com/photos/adactio/6153481666/ Then mobile came and made us realize…
49. that it was a consensual hallucination all along. http://www.flickr.com/photos/garibaldi/303085857/
50. The web never had a fixed canvas. http://www.flickr.com/photos/paulocarrillo/124755065/
51. Even our tools perpetuate the lie.
52. http://www.flickr.com/photos/69797234@N06/7203485148/ We’ve made tremendous progress.
53. But there is another consensual hallucination. http://www.flickr.com/photos/garibaldi/ 303085857/
55. = =
57. Supports hover and pointer events.
60. Keyboard and touch.
63. Even the iPhone can have a keyboard.
66. Are these laptops or tablets?
69. Desktop computer with 23” touch screen
71. Luke nailed it. http://static.lukew.com/unified_device_design.png
73. We can no longer make assumptions about input based on screen size or form factor. And we probably never should have.
74. http://www.flickr.com/photos/cblue98/7254221968
75. Input represents a bigger challenge than screen size. http://www.flickr.com/photos/cblue98/7254221968
76. http://www.flickr.com/photos/taedc/9278192929
77. And it is changing more rapidly than ever before. http://www.flickr.com/photos/taedc/9278192929
78. So let’s take a closer look…
79. Let’s start with futuristic input. http://www.flickr.com/photos/jdhancock/3714748769/
80. http://uncyclopedia.wikia.com/wiki/ File:Man_yelling_at_computer.JPG VOICE
81. http://uncyclopedia.wikia.com/wiki/ File:Man_yelling_at_computer.JPG VOICE
82. http://www.98ps.com/viewnews-15222.html
83. Siri gets all of the hype… http://www.98ps.com/viewnews-15222.html
84. but both Microsoft and Google have compelling voice input in their products.
88. How should web pages change to support voice control?
91. Google voice search
92. You can use speech recognition too. http://www.google.com/intl/en/chrome/demos/speech.html http://www.moreawesomeweb.com/demos/speech_translate.html
93. Web Speech API Specification 19 October 2012 Editors: Glen Shires, Google Inc. Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups.
94. Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult
95. Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult
96. Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult
97. Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events
98. Hans Wennborg, Google Inc. Please refer to the errata for this document, which may include some normative corrections. Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events 5.1.4 SpeechRecognitionError 5.1.5 SpeechRecognitionAlternative 5.1.6 SpeechRecognitionResult Copyright © 2012 the Contributors to the Web Speech API Specification, published by the Speech API Community Group under the W3C Community Final Specification Agreement (FSA). A human-readable summary is available. Abstract This specification defines a JavaScript API to enable web developers to incorporate speech recognition and synthesis into their web pages. It enables developers to use scripting to generate text-to-speech output and to use speech recognition as an input for forms, continuous dictation and control. The JavaScript API allows web pages to control activation and timing and to handle results and alternatives. Status of This Document This specification was published by the Speech API Community Group. It is not a W3C Standard nor is it on the W3C Standards Track. Please note that under the W3C Community Final Specification Agreement (FSA) other conditions apply. Learn more about W3C Community and Business Groups. All feedback is welcome. Table of Contents 1 Conformance requirements 2 Introduction 3 Use Cases 4 Security and privacy considerations 5 API Description 5.1 The SpeechRecognition Interface 5.1.1 SpeechRecognition Attributes 5.1.2 SpeechRecognition Methods 5.1.3 SpeechRecognition Events
99. Special thanks to Eric Bidelman http://moreawesomeweb.com Speech Recognition API Support ?
101. Gestures?
102. http://leapmotion.com
103. https://www.kickstarter.com/projects/1761670738/ring-shortcut-everything
104. https://www.kickstarter.com/projects/1761670738/ring-shortcut-everything
106. Amazing, but too new to know what, if anything, this technology will mean for the web.
107. Let’s come back from the future and look at something much Dumber. Dumber
108. Dumber
109. Dumber
110. -pad remote controls D
111. -pad remote controls D
112. -pad remote controls D
113. http://www.flickr.com/photos/stewc/6669743035/
114. http://www.flickr.com/photos/stewc/6669743035/ TVs browsers that support d-pad, send arrow key events.
115. http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/
116. If then http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/
118. is undetectable. This is a recurring theme for input.
119. Sensors and camera http://www.flickr.com/photos/retrocactus/2170677056
120. Sensors and camera Camera http://www.flickr.com/photos/retrocactus/2170677056
121. GPS http://www.flickr.com/photos/3dking/149450434
122. GPS GeoLocation http://www.flickr.com/photos/3dking/149450434
124. Gyroscope & Accelerometer
125. http://www.flickr.com/photos/chrisjagers/4694134078
126. Back to today’s problems. http://www.flickr.com/photos/chrisjagers/4694134078
128. Hover state No hover state
129. Hover state Typing easier for many No hover state Typing often more difficult
130. Higher precision with mouse means smaller targets possible Hover state Less precise than mouse and requires larger touch targets Typing easier for many No hover state Typing often more difficult
131. Higher precision with mouse means smaller targets possible Hover state Less precise than mouse and requires larger touch targets Typing easier for many No hover state Typing often more difficult Right clicking and “power” tools Single and multi-touch gestures
134. http://www.flickr.com/photos/28096801@N05/5012309802
135. I got this. Detect touch. http://www.flickr.com/photos/28096801@N05/5012309802
137. Whatever you may think, it currently isn't possible to reliably detect whether or not the current device has a touchscreen, from within the browser. —Stu Cox, You Can’t Reliably Detect a Touch Screen http://www.stucox.com/blog/you-cant-detect-a-touchscreen/
138. Chrome has entertained idea of enabling touch by default. https://code.google.com/p/chromium/issues/detail?id=159527 https://docs.google.com/a/cloudfour.com/presentation/d/1-n1qyzewpagREbzW2zm0wOalq33UhbtbSkWf9mEdly8/ edit#slide=id.gc2d80e5b_171
140. Detect a mouse? Not reliably.
143. Surely we can detect a keyboard?
144. Surely we can detect a keyboard? NOPE
146. Input is dynamic.
147. Input is dynamic.
148. Boris Smus’s experiments responding to input. http://smus.com/touch-laptop-experiments/
149. Boris Smus’s experiments responding to input. http://smus.com/touch-laptop-experiments/
150. http://www.flickr.com/photos/lyza/7382235106 Maybe we need to be more zen about input.
152. After poking at this problem for a few weeks, my conclusion is: every desktop UI should be designed for touch now. When any desktop machine could have a touch interface, we have to proceed as if they all do. —Josh Clark http://globalmoxie.com/blog/desktop-touch-design.shtml
154. http://ie.microsoft.com/testdrive/ieblog/2011/Sep/20_TouchInputforIE10andMetrostyleApps_1.png http://www.w3.org/TR/pointerevents/ http://blog.webplatform.org/2013/02/pointing-toward-the-future/ New Pointer Events spec normalizes touch and mouse Pointer Events builds on the DOM event model to offer a new way to handle input on the web, enabling developers to build touch-first experiences that work with mouse, pen, and other pointing devices as well…They are also designed from the ground up to allow modern browsers to accelerate the touch-surface performance, leading to a smoother user experience.
155. What about those who won’t let go of their “power” interfaces? http://www.flickr.com/photos/ecos/4092571213/
156. http://www.flickr.com/photos/scarygami/5689980135/ One option: give them a choice.
158. Gmail display density settings
159. Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode
160. Couch Mode + See all Centric TV’s videos / Recently viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode
161. Couch Mode + See all Centric TV’s videos / Recently viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Couch Mode + See all Centric TV’s videos / Recently viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode
162. Vimeo Couch Mode
164. The key benefit of this approach: You’re designing for user need not for a specific form factor or input.
165. http://www.flickr.com/photos/raver_mikey/504815463
167. Progressive Input?
168. Graph from Chapter 1 of Adaptive Web Design by Aaron Gustafson http://easy-readers.net/books/ adaptive-web-design/
169. Graph from Chapter 1 of Adaptive Web Design by Aaron Gustafson http://easy-readers.net/books/ adaptive-web-design/ Progressive enhancement contains a value judgment
170. Who are we to judge what input is better? http://www.flickr.com/photos/fensterbme/4783366926
171. We need to adapt and respond. http://www.flickr.com/photos/cdm/147947664/
172. Learn how to let go of the illusions that comfort us. http://www.flickr.com/photos/garibaldi/ 303085857/
176. This is the web as it should be. As it wants to be. The web in its natural state. http://www.flickr.com/photos/25062265@N06/6069101123
177. It is what our customers expect. http://www.flickr.com/photos/johanl/6798184016
179. Thank You! SLIDES: http://bit.ly/grigs-wvpdx14 Special thanks to Flickr users for generously sharing their photos under creative commons license.