Adapting to Input - Speaker Deck

Slide 1

Slide 1 text

Adapting to Input Jason Grigsby • @grigs • cloudfour.com

Slide 2

Slide 2 text

Web was created formless. https://www.ﬂickr.com/photos/chiaralily/3068047592

Slide 3

Slide 3 text

http://www.wired.co.uk/magazine/archive/ 2014/03/web-at-25/tim-berners-lee

Slide 4

Slide 4 text

https://www.ﬂickr.com/photos/inpivic/5206515374/

Slide 5

Slide 5 text

https://www.ﬂickr.com/photos/tomokyo-blue/5812115264

Slide 6

Slide 6 text

http://www.flickr.com/photos/60415054@N00/14301113/ So we told ourselves that the web was…

Slide 7

Slide 7 text

http://www.flickr.com/photos/60415054@N00/14301113/ 640 px 480 px

Slide 8

Slide 8 text

640 px 480 px

Slide 9

Slide 9 text

800 px 600 px

Slide 10

Slide 10 text

1024 px 768 px

Slide 11

Slide 11 text

http://www.flickr.com/photos/adactio/6153481666/ Then mobile came and made us realize…

Slide 12

Slide 12 text

that it was a consensual hallucination all along. http://www.flickr.com/photos/garibaldi/303085857/

Slide 13

Slide 13 text

The web never had a ﬁxed canvas. http://www.flickr.com/photos/paulocarrillo/124755065/

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

https://www.ﬂickr.com/photos/giuliatanini/7125944117/

Slide 16

Slide 16 text

http://www.flickr.com/photos/69797234@N06/7203485148/ We’ve made tremendous prog

Slide 17

Slide 17 text

But there is another consensual hallucination. http://www.flickr.com/photos/garibaldi/303085857/

Slide 18

Slide 18 text

= =

Slide 19

Slide 19 text

mobile desktop THE ART OF WEB DEVELOPMENT THE ART OF WEB DEVELOPMENT Web widgets THE ART OF WEB DEVELOPMENT THE ART OF WEB DEVELOPMENT Mobile widgets

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

It’s not that we’re technically incapable, but adapting a phone UI to a tablet UI is not so dissimilar from trying to automatically adapt desktop UI to a phone. They are fundamentally diﬀerent platforms with diﬀerent usability considerations… —Todd Anglin, Kendo UI http://www.kendoui.com/blogs/teamblog/posts/12-09-11/universal_mobile_apps_with_html5_and_kendo_ui.aspx

Slide 24

Slide 24 text

It’s fairly certain that the highest-value use will stay predominantly on desktop… Most complex tasks have vastly better user experience on the desktop and thus will be performed there. —Jacob Nielsen https://www.nngroup.com/articles/transmedia-design-for-3-screens/

Slide 25

Slide 25 text

http://www.flickr.com/photos/royalsapien/2387707860

Slide 26

Slide 26 text

We envision desktop users as diﬀerent from mobile. http://demos.kendoui.com/web/grid/editing.html

Slide 27

Slide 27 text

http://www.flickr.com/photos/shantellmartin/4543010568

Slide 28

Slide 28 text

https://www.ﬂickr.com/photos/keh-s/6020885164/ It’s time to break free of our assumptions about input.

Slide 29

Slide 29 text

Four truths about input

Slide 30

Slide 30 text

#1 Input is exploding

Slide 31

Slide 31 text

An incomplete history of computer input

Slide 32

Slide 32 text

Focus on broad adoption Using Apple as a lens

Slide 33

Slide 33 text

Keyboard 1874 First typewriters in 1860s Remington Typewriter 1874 Popularizes Qwerty keyboard Qwerty patent granted 1878

Slide 34

Slide 34 text

Mouse 1984 Mouse prototypes 1960s by Engelbart and Mallebrein. Xerox uses mouse in 70s, early 80s. Apple Macintosh popularizes mouse and GUI in 1984.

Slide 35

Slide 35 text

Many alternatives to the mouse are created—trackballs, trackpads, and joysticks to name a few—but all replace the mouse and control the cursor. No change in behavior.

Slide 36

Slide 36 text

Scroll Wheel 1996 First scrolling mouse in 1985. Scroll wheel on side. Microsoft Intellimouse popularizes scroll wheel in 1996. Scroll wheel controls windows, not cursor.

Slide 37

Slide 37 text

Camera 2005 / 2006 Cameras are now included in phones and computers. Cameraphones sold in 2005 alone greater than all digital cameras sold to date. All Mac computers include cameras starting 2006.

Slide 38

Slide 38 text

iPhone 2007 Multi-touch screen Camera Accelerometer Proximity Sensor Ambient Light Sensor

Slide 39

Slide 39 text

Macbook Air January 2008 Multi-touch trackpad Gestures in Mac OS Controls more than cursor

Slide 40

Slide 40 text

iPhone 3G 2008 GPS (Location services)

Slide 41

Slide 41 text

iPhone 3GS 2009 Voice control Magnetometer (Compass)

Slide 42

Slide 42 text

iPhone 4 2010 Gyroscope Front facing camera

Slide 43

Slide 43 text

Magic Trackpad 2010 Multi-touch trackpad Gestures in Mac OS No longer limited to laptops

Slide 44

Slide 44 text

iPhone 4S 2011 Siri Bluetooth LE

Slide 45

Slide 45 text

iPhone 5 2012 Nothing?

Slide 46

Slide 46 text

iPhone 5S 2013 Fingerprint Sensor

Slide 47

Slide 47 text

iPhone 6 2014 NFC (Apple Pay) Barometer

Slide 48

Slide 48 text

iPhone 6S 2015 3D Touch

Slide 49

Slide 49 text

Same story could be told using Android or Microsoft.

Slide 50

Slide 50 text

1984 1874 142 years 1996 12 years

Slide 51

Slide 51 text

1984 1874 1996

Slide 52

Slide 52 text

1984 1996 2005 9 years 2 years 2007

Slide 53

Slide 53 text

1984 1996 2005 2007

Slide 54

Slide 54 text

2007

Slide 55

Slide 55 text

2007 Touch Camera Accelerometer Proximity Sensor Ambient Light Sensor GPS Trackpad 2008 Voice control Magnetometer 2009 2010 Gyroscope Front-facing Camera Magic Trackpad 2011 Siri Bluetooth LE 2012 2013 Fingerprint Sensor 2014 NFC (Apple Pay) 2015 3D Touch 2016 ?

Slide 56

Slide 56 text

Touch 3D Touch Fingerprint sensor Camera (video and image) GPS (location) Compass Bluetooth LE Audio (Siri) Gyroscope Accelerometer Barometer Proximity sensor Ambient light sensor NFC (Apple Pay) iBeacon

Slide 57

Slide 57 text

Things aren’t slowing down. https://www.ﬂickr.com/photos/andreastsonis/8675283555

Slide 58

Slide 58 text

Siri gets all of the hype… http://www.98ps.com/viewnews-15222.html

Slide 59

Slide 59 text

but both Microsoft and Google have compelling voice input in their products.

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

How should web pages change to support voice control?

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Google voice search

Slide 67

Slide 67 text

https://www.youtube.com/watch?v=kwn9Lh0E_vU

Slide 68

Slide 68 text

Remember Leap Motion?

Slide 69

Slide 69 text

https://www.youtube.com/watch?v=LJPxyWM9Ujg

Slide 70

Slide 70 text

https://www.youtube.com/watch?v=ﬁZkEYLXctE

Slide 71

Slide 71 text

#2 Input is a continuum

Slide 72

Slide 72 text

= =

Slide 73

Slide 73 text

Supports hover and pointer events.

Slide 74

Slide 74 text

Keyboard and touch.

Slide 75

Slide 75 text

Even the iPhone can have a keyboard.

Slide 76

Slide 76 text

Are these laptops or tablets?

Slide 77

Slide 77 text

Desktop computer with 23” touch screen

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

Luke Wroblewski nailed it. http://static.lukew.com/unified_device_design.png

Slide 80

Slide 80 text

We can no longer make assumptions about input based on screen size or form factor.

Slide 81

Slide 81 text

3. Input is undetectable

Slide 82

Slide 82 text

Higher precision with mouse means smaller targets possible Hover state Less precise than mouse and requires larger touch targets Typing easier for many No hover state Typing often more diﬃcult Right clicking and “power” tools Single and multi-touch gestures

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

I got this. Detect touch. http://www.flickr.com/photos/28096801@N05/5012309802

Slide 85

Slide 85 text

Whatever you may think, it currently isn't possible to reliably detect whether or not the current device has a touchscreen, from within the browser. —Stu Cox http://www.stucox.com/blog/you-cant-detect-a-touchscreen/

Slide 86

Slide 86 text

Chrome has entertained idea of enabling touch by default. https://code.google.com/p/chromium/issues/detail?id=159527 https://docs.google.com/a/cloudfour.com/presentation/d/1-n1qyzewpagREbzW2zm0wOalq33UhbtbSkWf9mEdly8/edit#slide=id.gc2d80e5b_171

Slide 87

Slide 87 text

Detect a mouse? Not reliably.

Slide 88

Slide 88 text

Surely we can detect a keyboard? NOPE

Slide 89

Slide 89 text

https://www.ﬂickr.com/photos/mediaﬂex/4628277817/

Slide 90

Slide 90 text

https://www.ﬂickr.com/photos/ dentyone/2410917872/

Slide 91

Slide 91 text

Be thankful. This saves us from ourselves.

Slide 92

Slide 92 text

Interactive touch laptop experiments by Boris Smus

Slide 93

Slide 93 text

No content

Slide 94

Slide 94 text

We can only detect input when it is used and that’s too late for our user interfaces.

Slide 95

Slide 95 text

4. Input is transient

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

Knowing what input is used one moment tells you nothing about what will be used next.

Slide 98

Slide 98 text

Input is exploding continuum undetectable transient

Slide 99

Slide 99 text

Overwhelmed? https://www.ﬂickr.com/photos/cheo70/2702682262/

Slide 100

Slide 100 text

https://www.ﬂickr.com/photos/evilerin/3723714381/

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

We just need to rewire our brains again.

Slide 103

Slide 103 text

Adapting to input

Slide 104

Slide 104 text

#1 Design for the largest target by default

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

Fitts’s Law The time to acquire a target is a function of the distance to and size of the target.

Slide 107

Slide 107 text

After poking at this problem for a few weeks, my conclusion is: every desktop UI should be designed for touch now. When any desktop machine could have a touch interface, we have to proceed as if they all do. —Josh Clark http://globalmoxie.com/blog/desktop-touch-design.shtml

Slide 108

Slide 108 text

#2 Design for modes of interaction instead of inputs

Slide 109

Slide 109 text

Display density settings

Slide 110

Slide 110 text

Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode

Slide 111

Slide 111 text

Couch Mode + See all Centric TV’s videos / Recently viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Couch Mode + See all Centric TV’s videos / Recently viewed / Related videos Th Dream Experience - … Uploaded 2 years ago More Of The Dream Exp… Uploaded 2 years ago The Dream Experience -… Uploaded 2 years ago The Dream Experience … Uploaded 2 years ago The Love King Breaks It… Uploaded 2 years ago PROMOTED War Paint for Trees From Lincoln Motor Company Join Log In Create Watch Upload Search s [ ] – VIDEOS Vimeo Couch Mode

Slide 112

Slide 112 text

Vimeo Couch Mode

Slide 113

Slide 113 text

You’re designing for user need—not for a speciﬁc form factor or input.

Slide 114

Slide 114 text

#3 Make things accessible

Slide 115

Slide 115 text

D-pad remote controls

Slide 116

Slide 116 text

http://www.flickr.com/photos/stewc/6669743035/ TVs browsers that support d-pad, send arrow key events.

Slide 117

Slide 117 text

If then http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/

Slide 118

Slide 118 text

is undetectable.

Slide 119

Slide 119 text

If then http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/

Slide 120

Slide 120 text

Support http://www.wasdkeyboards.com/index.php/catalog/product/gallery/id/7164/image/343/

Slide 121

Slide 121 text

Building accessibly increases the likelihood of support for future, unknown inputs.

Slide 122

Slide 122 text

No content

Slide 123

Slide 123 text

No content

Slide 124

Slide 124 text

#4 Design for multiple concurrent inputs

Slide 125

Slide 125 text

= =

Slide 126

Slide 126 text

Interactive touch laptop experiments by Boris Smus

Slide 127

Slide 127 text

At minimum, don’t assume if one input is present that the person doesn’t have access to other types of input.

Slide 128

Slide 128 text

Look for opportunities to combine input to create more powerful experiences.

Slide 129

Slide 129 text

#5 Abstract baseline input

Slide 130

Slide 130 text

Mobile Tablet Desktop

Slide 131

Slide 131 text

Small Medium Large

Slide 132

Slide 132 text

TAP https://www.ﬂickr.com/photos/openexhibits/5884451755

Slide 133

Slide 133 text

CLICK

Slide 134

Slide 134 text

POINT SELECT

Slide 135

Slide 135 text

Mouse events and touch events are fundamentally diﬀerent beasts in browsers today, and that makes it hard to write cross-platform apps. These incompatibilities lead to applications having to listen to 2 sets of events, mouse on desktop and touch for mobile. https://github.com/jquery/PEP/

Slide 136

Slide 136 text

http://ie.microsoft.com/testdrive/ieblog/2011/Sep/20_TouchInputforIE10andMetrostyleApps_1.png http://www.w3.org/TR/pointerevents/ http://blog.webplatform.org/2013/02/pointing-toward-the-future/ New Pointer Events spec normalizes touch, mouse and stylus Pointer Events builds on the DOM event model to oﬀer a new way to handle input on the web, enabling developers to build touch-first experiences that work with mouse, pen, and other pointing devices as well…They are also designed from the ground up to allow modern browsers to accelerate the touch-surface performance, leading to a smoother user experience.

Slide 137

Slide 137 text

Shipped Opposed In development

Slide 138

Slide 138 text

jQuery Pointer Events Polyﬁll https://github.com/jquery/PEP/

Slide 139

Slide 139 text

#6 Progressively enhance input

Slide 140

Slide 140 text

Opportunity is knocking!

Slide 141

Slide 141 text

No content

Slide 142

Slide 142 text

No content

Slide 143

Slide 143 text

We’re pretty good at using Geolocation.

Slide 144

Slide 144 text

Compass.js http://ai.github.io/compass.js/

Slide 145

Slide 145 text

Warby Parker Gyroscope https://www.warbyparker.com/eyeglasses/women/simone/tea-rose-fade

Slide 146

Slide 146 text

Or hover

Slide 147

Slide 147 text

No content

Slide 148

Slide 148 text

https://www.youtube.com/watch?v=7Yp0BpiDzXE

Slide 149

Slide 149 text

No content

Slide 150

Slide 150 text

No content

Slide 151

Slide 151 text

https://www.youtube.com/watch?v=Ow8gA0qe9SI

Slide 152

Slide 152 text

Remember to support browser auto-ﬁll and new Payment Request API http://www.html5rocks.com/en/tutorials/forms/requestautocomplete/

Slide 153

Slide 153 text

No content

Slide 154

Slide 154 text

No content

Slide 155

Slide 155 text

You can use speech recognition too. http://www.google.com/intl/en/chrome/demos/speech.html http://www.moreawesomeweb.com/demos/speech_translate.html

Slide 156

Slide 156 text

Special thanks to Eric Bidelman http://moreawesomeweb.com Speech Recognition API Support

Slide 157

Slide 157 text

https://www.youtube.com/watch?v=_BUwOBdLjzQ Web Bluetooth

Slide 158

Slide 158 text

https://www.youtube.com/watch?v=FYxQ5TeVFdE Physical Web

Slide 159

Slide 159 text

https://www.youtube.com/watch?v=6z9ED4fmi1w Physical Web &  Web Bluetooth

Slide 160

Slide 160 text

#7 Make input part of test plans

Slide 161

Slide 161 text

Need to add input to our device labs http://blog.adtile.me/2014/01/08/adtile-device-lab/

Slide 162

Slide 162 text

https://www.ﬂickr.com/photos/jeepersmedia/14692542501

Slide 163

Slide 163 text

https://www.ﬂickr.com/photos/yuckymuﬃntech/6988362001

Slide 164

Slide 164 text

No content

Slide 165

Slide 165 text

1. Design for the largest target by default 2. Design for modes of interaction instead of inputs 3. Make things accessible 4. Design for multiple concurrent inputs 5. Abstract baseline input 6. Progressively enhance input 7. Make input part of test plans

Slide 166

Slide 166 text

Input is exploding continuum undetectable transient

Slide 167

Slide 167 text

www.ﬂickr.com/photos/brunauto/5062644167/ Learn from our mobile context mistakes.

Slide 168

Slide 168 text

Who are we to judge which input is better? http://www.flickr.com/photos/fensterbme/4783366926

Slide 169

Slide 169 text

We need to learn to adapt. http://www.flickr.com/photos/cdm/147947664/

Slide 170

Slide 170 text

Learn how to let go of illusions that comfort us. http://www.flickr.com/photos/garibaldi/303085857/

Slide 171

Slide 171 text

Thank You!