THE WORLD IS THE SCREEN Convey UX 2014 | Seattle WA | Andrew Hinton | @inkblurt Understanding Information Environments (Note for PDF: this is a shorter version of a talk I’ve done over the last year; some slides and content are newer, but some have been removed. For other versions, see slideshare.net/ andrewhinton ) 

Andrew Hinton | @inkblurt INFORMATION ENVIRONMENT http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg This is an environment. It’s a natural setting, full of physical, naturally occurring surfaces and objects. It’s the sort of environment that humans as a species spent most of its evolutionary history in. Notice there’s a stone wall here. It’s something made by humans that adds structure to the environment. It’s an object that impedes motion, but it’s also an object with cultural meaning -- it’s telling us something about this place. Maybe it’s marking off owned property, or it says “this is where my sheep are contained.” >> What if we think of this as not only an environment, but an information environment? In what ways is that true? That’s largely what this talk is about. Because networked technology is becoming so pervasive, so much a part of the fabric of our surroundings, that we need to understand the fundamentals of how humans comprehend their environment, regardless of the specific technology. And understanding that means understanding the sorts of information that are in the world, and how they work together to create places, contexts, and meaning. http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg 

Andrew Hinton | @inkblurt INFORMATION ENVIRONMENT 3 INFORMATION ENVIRONMENT These are two information environments that involve office supplies. What is the difference, really, between shopping for them in a brick and mortar retail store, and shopping for them through an online retailer like Amazon? If we think of the physical store as an information interface, how much information is conveyed, and of what kind, through one interface versus another? How useful is it for the activity of deciding between one printer or stapler and another? There are things the digital environment affords us that the physical one does not: we can compare things more easily, sort them, search them; we can see more of a selection; we can see what others have said about the products. 

Andrew Hinton | @inkblurt Subway station + Food store Here we have a mashup where a place is simultaneously a subway station and a virtual store, with pictures of products on the walls where shoppers can scan them for delivery or pickup using the store’s mobile app. It stretches what we assume about how an environment works, and even what a place is. 

Andrew Hinton | @inkblurt Google Glass Leap Motion Microsoft Kinect We’re being surrounded by sensors and digital agents that are trying to read our actions, or layers of digital information added to what we see around us. left glass image wired.com / others from google Leap motion from their site Kinect from blogs.msdn.com 

Andrew Hinton | @inkblurt Now, even your refrigerator can spam you. http://www.bbc.co.uk/news/technology-25780908 bestbuy.com Our environment is full of even mundane things that are networked and active in ways we used to only joke about. Like the smart refrigerator that was recently caught being used as a spam bot. 

Andrew Hinton | @inkblurt Understanding the nature of our materials. We’ve gone for a long time in our work being comfortable with knowing that what we make is confined to a specific screen -- a bounded interactive surface. But now *all* the surfaces around us are potentially digital and networked. Walls and windows, coffee pots and shoes. So we need to re-assess, what is the nature of our materials? Potters have a deep understanding of clay; masons have a deep understanding of stone. We need to deepen our understanding of information. 

Andrew Hinton | @inkblurt INFORMATION Digital Semantic Physical Information is a word we use a lot, but I don’t think that in our general practice we have a common framework for what it is, and the role it plays in the environments we inhabit. I tried finding a single, official definition, but there really isn’t one -- instead, there are many different ways of looking at various things in our world that people call information. The problem is, all of them are pretty useful and valuable. So rather than just pick one, I decided to pick several and see how they work together and inform each other, so to speak. These modes or ways we experience information are physical, semantic, and digital. 

Andrew Hinton | @inkblurt Physical Information 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital People communicating with people. Semantic Animals (including people) perceiving the environment. Physical Physical information has to do with the ecological relationship between an animal and its physical environment. The green field we looked at earlier was mostly made up of physical information that creatures use to know where they can go and what they can do in those surroundings. The way I’m framing this comes from something called ecological psychology, which has been adapted for a branch of study called embodied cognition. 

Andrew Hinton | @inkblurt Ecological Information "Pick-up" Agent Perception Action Ambient, structured energy arrays Embodied cognition: Body & Environment, not just Brain. Embodied cognition sees the environment and the body as part of a larger dynamic, in which the brain is a participant, but not necessarily the center-point. It may even just be a supporting player. A big part of this is *action*: we don’t perceive the world by sitting still and taking in sensory data; we perceive by moving and interacting with the environment. Much of our daily activity is unconsciously nudged and controlled by our environment. 

Andrew Hinton | @inkblurt “... there’s no such thing as disembodied cognition [for] living creatures. All of our cognition has to be embodied ... Brains evolved in the service of action.” - Louise Barrett http://podcasts.ox.ac.uk/embodiment-taking-sociality-seriously-audio Louise Barrett, who wrote a great book on embodied cognition called Beyond the Brain, puts it nicely -- -- there’s no such thing as disembodied cognition for living creatures, all of our cognition has to be embodied because brains evolved in the service of action. We didn’t arrive on the planet as brains that grew bodies to supplement their disembodied brain activity. Brains evolved to supplement the abilities of bodies in environments. They’re the product of the structures of their surroundings. 

Andrew Hinton | @inkblurt James J Gibson - Ecological Psychology of Perception Now seen as pioneer of embodied cognition. Some embodied cognition theorists have heavily adopted the work of James J Gibson, who was a scientist of something called “ecological psychology” in the mid 20th century -- a field he was part of inventing. 

Andrew Hinton | @inkblurt Ecological Information "Pick-up" Agent Perception Action Ambient, structured energy arrays Gibson used the word “information” to mean the elements we perceive in the environment that tell us what it “affords.” “INFORMATION” Gibson purposefully used the word “information” for this body-environment relationship. He argued that our bodies do something he called “information pickup” -- where the body’s perceptual system detects structural cues in the environment that specify what actions the body can take. And he argued that this happens directly, with little or no need for brain processing. 

Andrew Hinton | @inkblurt The perceived functional properties of objects, places and events in relation to an individual perceiver. AFFORDANCE Perception exists only insofar as we perceive affordances. Gibson also invented the concept of affordance. For Gibson, affordance isn’t a thing you add to something. It’s the result of a property in the environment that presents the opportunity for action to a perceiving organism. This is true in the built environment as well as the natural one; the structure of these stairs interacts with energy in the environment that my perception picks up on, and my body detects that the specific structure of the steps will take my body upward if climb them. Affordance was and still is a radical idea, in the sense that Gibson meant it. Affordance is the organizing principle behind *ALL* perception. We don’t perceive anything unless it affords meaningful action for a given context. Gibson even went on to discuss how higher-order combinations of affordances can be found in the complexities of human culture. A mailbox, for example, has the affordance for sending mail -- but only because we’ve learned a lot of stuff about the world around that box that makes it a special sort of container connected to a broader cultural system. 

Andrew Hinton | @inkblurt We perceive environment as “nested,” not in logical hierarchy. We perceive elements in the environment as invariant (persistent) or variant (in flux). Invariance grounds our experience (literally). Invariance Nestedness Two of Gibson’s principles of environmental perception. Gibson created a marvelous system of principles and components that work as building blocks for how we perceive our environment and its affordances. I can only barely touch on them here, but I’ll mention a few important ones. >> Invariance is the property of environmental structure that makes some of it persistent over time, so that we can make sense of the whole. Terrestrial animals like us evolved on stable ground, under a persistent sky. Substances and surfaces that persist allow us to learn what properties they have, and what they afford us. This can be a useful idea in digital design because in software, even though we can make everything fluid and variant, we need to design invariant structures that ground the experience. >> Nestedness: we perceive the environment as structures that relate to one another as nested -- a stream is in a valley, which is between mountains, which are part of a range, which are between earth and sky. There aren’t always definite boundaries, though, and this isn’t a purely logical relationship -- there are overlaps and redundancies, and shifts in meaning. A stick on the ground can be kindling or a club, or it can be part of a roof over our heads in a shelter. It’s not just one defined thing in how it relates to the rest of the environment. This is a powerful idea when we look at something like faceted navigation, where the meaning of a piece of content or an object shifts depending on what perspective a user is bringing to it. 

Andrew Hinton | @inkblurt Surfaces Objects Layout Place Some of the components that make up environments. There are many components Gibson outlines, like surface, object, layout and place -- that work together to form the environment. There isn’t time to cover it all here -- but I think it’s a useful framework we can borrow from, because all the environments we design should meet the expectations of people’s embodied experience. Our bodies want and need environments to work in these ways. 

Andrew Hinton | @inkblurt Semantic Information 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital Animals (including people) perceiving the environment. Physical People communicating with people. Semantic Of course, humans don’t just exist in raw physical surfaces and objects -- we add to the environment. In addition to buildings and roads and such, we add language to the world. The semantic mode, in short, is language. But I mean language in the broad sense of things we put into the environment to communicate with people. This can be all sorts of stuff: speech, gestures, text, iconography, even buildings have semantic qualities. 

Andrew Hinton | @inkblurt LANGUAGE IS ENVIRONMENT STRUCTURE THAT ENABLES US TO MAKE MORE STRUCTURE Language is “a form of mind- transforming cognitive scaffolding: a ... symbolic edi ce [playing a] critical role in promoting thought and reason.” - Andy Clark Supersizing the Mind http://en.wikipedia.org/wiki/File:CantileverScaffold.jpg Language is environment. In another excellent book on embodied cognition, Andy Clark frames language as a form of cognitive scaffolding; it persists in the environment, as new structure that allows us to make other structures. We wouldn’t have most of what we think of as civilization without the scaffolding of language. 

Andrew Hinton | @inkblurt Language is the primary human interface to complex systems. Language is both the infrastructure we use for creating complexity, as well as the primary interface we have for understanding and inhabiting complex systems. 

Andrew Hinton | @inkblurt WE LIVE IN LANGUAGE STRUCTURES Language structures are an architecture that we live within together, whether we know it or not. Whether these structures are defined explicitly like in this early IBM management diagram, or defined tacitly through the collective assumptions within a shared culture, the way we talk and write about our shared environment is also a structural feature of that environment. 

Andrew Hinton | @inkblurt Physical Affordance + Semantic (or “Linguistic”) Affordance Physical Semantic Humans aren’t just perceivers of physical structure. We depend as much or more on the symbolic workings of language – semantic information – as a part of our environment for working together and figuring out our world, and making stuff – as well as just getting around. In this picture of a staircase at City Lights Bookstore in SF, we see stairs that have intrinsic structural information that our bodies perceive as affording the ability to walk upward. But there’s this other thing – a label – which signifies what where we will be going when we go up the stairs. We might call this semantic affordance. It’s an abstracted, symbolic version of how we perceive and comprehend physical structure. In linguistics we also hear of signification -- which I won’t go into in detail here -- but there are signifiers at work in both the stairs and the label. The important point I want to get across here is that affordance is at work even at the high-order level of meaning we have with language, not just in physical structures. 

Andrew Hinton | @inkblurt Physical Physical + Semantic Simulated- Physical + Semantic Mostly Semantic Affordance across a spectrum of physical to semantic. Affordance is at the root of how we use language as well as the simulated physicality of the stuff we put into interfaces. >> The stairs intrinsically contain all the information needed for your body to know what will happen when you climb them. >> But look at this button. When people started making machines and systems that were more complex, we had physical affordances that did more than we could tell just by touching or looking at them. Because I’ve pushed similar buttons before, I know that the physical information here means I should be able to push in the red part and it will probably spring back out -- that’s all that the physical information is telling me. In order to know what the button is going to cause elsewhere, I need a label. >> When Microsoft updated Windows to add the Start button, they were basically adding an invariant structure to the operating system so that users would know that, no matter what, they could always “start” a new action from here. And like lots of other software, they used a simulated button object, borrowing from what we were used to with buttons like the one on the left. The button isn’t physical information, really ... it’s using semantic information to simulate a physical object. That’s what graphical user interfaces do. >> In other parts of interfaces, and especially on the web with hypertext, labels may not have these simulated object qualities; so we have to rely a lot on the context of a label to know if it just affords reading or if it’s something we can activate. This is the issue at the heart of flat versus skeuomorphic design -- it’s really not about aesthetics, it’s about solving this issue of affordance. Additionally, the web introduced broad usage of labels as paths -- tapping a hyperlink doesn’t just activate some mechanism, it takes you on a journey to what feels to your perception like a new place. 

Andrew Hinton | @inkblurt Which red x???? Looks like a “con rm” action. Simulated-Physical & Semantic Information In Con ict Here are some examples of where things perform both semantically and ecologically, in confusing ways, in a UI. In this email application, logically speaking, the red X’s are all very different -- but physically they’re barely different at all; they require a lot of thought to disambiguate their semantic meaning. When I was in a hurry, I would reach for the closest red X to do whatever I’m trying to do - close the message, decline an invite, or delete it entirely. About half the time, I would click the wrong one. >>In an unsubscribe interface for fab.com, my wife discovered that she was apparently re- subscribing without realizing it, because that big red button -- like a big berry you can’t help but pick -- contextually feels like it’s a confirmation, not a cancellation/re-subscription action. 

Andrew Hinton | @inkblurt Animals (including people) perceiving the environment. Physical People communicating with people. Semantic Digital Information 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital So the examples we just looked at weren’t just any sort of semantic information, they were semantic information driven by digital technology. And digital technology relies on digital information. Digital information is how the black boxes talk to the other black boxes. It’s the lifeblood of information technology. The whole point of digital information is to strip human meaning out of it, which makes it more efficient for machines to transmit, store, and retrieve. This isn’t stuff we see face to face very much. Mainly we encounter its *effects* in the environment. 

Andrew Hinton | @inkblurt Digital Information Mode Leaking into Semantic Environment We see machines around us trying to get us to perceive what they are saying, or what they want to hear from us. We see them murmuring to each other in weird, noisy machine-only semantics that we do not comprehend either ecologically or semantically. The gas pump has to have a sticker added to it that explains what “Enter Data” means. >>The Twitter profile with the iPhone coordinates expresses my location not in a semantic way (the name of a city, for instance) but in a Cartesian grid that I have no contextual orientation for, either semantically or ecologically. >>The Delta app has information that I, as a human, can read, but it gives priority to the machines that I encounter in the workflow of the airport. 

Andrew Hinton | @inkblurt Digital information enables pervasive semantic place-making. I don’t mean to paint digital information as a villain. It isn’t. The ability to transmit, store and retrieve information in this way is a miracle. An platform I like a lot is Avocado - it lets a couple keep in touch and share a place together, pervasively. It has nice touches that key into embodied experience of semantic information, like sending a hug by touching the screen to your heart. Another nice touch: the couple shares the same password - making a word into a very real link of co-ownership of the place, like having the same keys to your home. This sort of pervasively available place would be impossible without digital information in the background. But it also requires a lot of discipline with semantic information structure to make the place coherent. 

Andrew Hinton | @inkblurt Google Plus on Desktop Google Plus on Mobile Easily post content in the circle I’m in. Confusingly requires re-establishing the circle where I’m posting. Broken “invariants” across contexts. Google Plus has invariance in how it establishes place between mobile and desktop versions. When I’m on desktop, and I’m in my “Family” Circle reading posts, when I post something new it automatically goes into that circle, just like I’d expect if I were having a conversation in a room. But on mobile, if I’m in my Family circle, a new post defaults to viewable by everyone; I have to work through a half dozen interactions to make it post just to the circle I perceived I was already “in.” 

Andrew Hinton | @inkblurt The ontology of “news” is confused here. • News Feed inside News Feed? • If All Friends, is it no longer News? or Recent? • Isn’t News “recent” by de nition? • What, then, is News Feed as a place? How am I changing the kind of place it is here? Ontology is the basis that we use for creating any sort of taxonomy - or relationship between semantic invariants. In the mobile version of Facebook’s “news feed” there are several options for how you want to see the feed. Their meanings don’t make much sense together, though -- it’s not clear how these are nested. Is news feed another news feed inside news feed? What does all friends mean? If I choose that, am I not seeing news anymore, or the most recent? Isn’t news recent by definition? What is news-feed as a place? How am I changing the mode of the place when I select something else? 

Andrew Hinton | @inkblurt What is “card” in this environment? Lowes launched a service called MyLowes -- that requires the registration of a card. But they also have a “Lowe’s Card” that’s a consumer credit card. Conversations at checkout can end up like a “who’s on first” routine -- “do you have your Lowe’s card?” “My Lowe’s card? That’s what I’m paying with.” “No I mean your ‘my lowe’s’ card.” “This IS my lowe’s card!” 

Andrew Hinton | @inkblurt Shopping Simultaneously in a Store & Online http://stevenglassman.de/2013/11/25/america-vs-deutschland/ In these omni-channel scenarios, the semantic information has to align with how it’s presented both online and physically in the store’s structure. Knowing if a huge store has a specific peanut butter or not is great; knowing exactly which aisle to get it from is even better. 

Andrew Hinton | @inkblurt The Thing Ontology helps answer: “What is a thing across multiple contexts?” Presently our various fields are preoccupied with how to have content and functionality make sense in various contexts. Ontology is at the heart of this problem. In many organizations and project teams, there’s an over-obsession with things like layout in each of the instantiations of a thing, but not enough discussion about how to define the nature of the thing in abstract. That requires an ontological perspective. And, done properly, it forms the main structures of an information environment - the invariant pillars, so to speak - that allow language to stitch together coherence across channels. 

Andrew Hinton | @inkblurt LANGUAGE IS INFRASTRUCTURE. LABEL LABEL LABEL LABEL LABEL RULES Labels Connections Rules All this stuff is essentially made out of language... it’s language working as infrastructure. We essentially make things out of labels, connections and rules. Too often, we assume the labels are something to add later - but in reality they’re the thing we have to figure out first. This is why issues like ontology and taxonomy are so important - they establish the “invariant” features of the environments we make. 

Andrew Hinton | @inkblurt • What are the invariants for perception & action? What grounds the experience? • How is it structurally nested in relation to a perceiver? Is it resilient for different perspectives & contexts? • Is it clear what is a place and what is an object? • How are semantic and physical information working together, or not? • Are the rules behind the environment clearly expressed in its information? Some of the questions we can ask ... So in sum, this is just scratching the surface, but there are some questions we can ask. >>What are the invariants for perception & action? What grounds the experience? >>How is it structurally nested in relation to a perceiver? Is it resilient for different perspectives & contexts? >>Is it clear what is a place and what is an object? >>How are semantic and physical information working together, or not? >>Are the rules behind the environment clearly expressed in its information? We have a long way to go to build heuristics and frameworks for responsibly creating the broader environment that our users inhabit. Hopefully this is a helpful start. 

Andrew Hinton | @inkblurt THANKS! Andrew Hinton | @inkblurt | andrewhinton.com [email protected]