THE WORLD IS THE SCREEN Blend Conference 2013 | Charlotte NC | Andrew Hinton | @inkblurt Understanding Information Environments This is a version of the talk that I gave for Blend Conference in Charlotte, NC, September 2013

Andrew Hinton | @inkblurt INFORMATION ENVIRONMENT This is an environment. It’s a natural setting, but notice there’s a stone wall here. It’s an example of an environment where someone changed the kind of place it is by creating a structure. >> But what if we think of this as not only an environment, but an information environment? In what ways is that true? That’s largely what this talk is about. Because networked technology is becoming so pervasive, so much a part of the fabric of our surroundings, that we need to understand the fundamentals of how humans comprehend their environment, regardless of the specific technology. And understanding that means understanding the sorts of information that are in the world, and how they work together to create places, contexts, and meaning. http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg

Andrew Hinton | @inkblurt INFORMATION ENVIRONMENT 3 INFORMATION ENVIRONMENT What is the difference, really, between shopping for office supplies in a brick and mortar retail store, and shopping for them through an online retailer like Amazon? If we frame both of these as “information environments” -- does that help us understand why one might be eating the other’s lunch? If we think of the physical store as an information interface, how much information is conveyed, and of what kind, through one interface versus another?

Andrew Hinton | @inkblurt A CURATED, ENCLOSED, BLENDED INFORMATION ENVIRONMENT What about when information environments use all sorts of methods for communication, all at once? We’ve had blended information environments for a while. Here’s a marvelous exhibit at the American Museum of Natural History in New York. It has a whole taxonomy -- the old-school meaning of taxonomy, meaning an organized hierarchy of creatures. But this one is instantiated on the wall. It also has it represented in written form, in a printed document, and in digitally presented form, in a kiosk. (These were taken when I took my daughter there some years back.) This is a curated, complex information environment. Physical objects, digital interfaces, lots of language and labels around. All connected together to form a whole experience. It’s is a highly controlled version of the world we now live in -- which is more emergent, messier, but even more pervasively connected & digitally enabled. photos by andrew hinton

Andrew Hinton | @inkblurt AN OPEN-ENDED, CUMULATIVE, BLENDED INFORMATION ENVIRONMENT Here’s an everyday intersection in Dublin. This is an environment that also has many different layers and modalities, but it’s not controlled and curated in the same way as the museum. It’s been added to, streets have been widened, signage added, infrastructure installed. And on top of that, lots of other information is pouring through it in the form of newspapers, or advertisements on buses and vans. The digital signs are something relatively new for our environments. It used to be that signs said one thing, and you learned what they said, and then you could forget about them until somebody put up a new one. These days, we can’t depend on surfaces being stable, persistent homes for written information. The stuff is embedded in all sorts of places. This street intersection in Dublin has digital signage mixed in with everything else. Pervasive computing technology means that the world is only getting weirder and more complex. We’re not talking about just consumer devices, but whole infrastructures, urban networks, and wired economies. photo by andrew hinton

Andrew Hinton | @inkblurt Google Glass Leap Motion Microsoft Kinect Another way the world is the screen is because of technology like Google Glass, which essentially lays a screen over the world around us, mediating between our perception and the stuff we’re perceiving. As well as things like Leap Motion and Kinect - these are just primitive first steps toward turning our entire surroundings into an interactive, digitally enabled environment. It’s also the case that these things -- all sorts of device screens and augmented displays - are getting integrated into our environmental experience. left glass image wired.com / others from google Leap motion from their site Kinect from blogs.msdn.com

Andrew Hinton | @inkblurt Understanding the nature of our materials. Much of what I’ll cover today may seem like “just theory” but I don’t think there’s anything more practical than understanding the nature of our materials. So far, we’ve tended to just use words like “information” or “environment” or “context” as if we all understand what they mean and what they’re made of. But we don’t really. So let’s see if we can dig down and find out.

Andrew Hinton | @inkblurt Start Here PACE LAYERS OF INFORMATION ENVIRONMENTS I’m working out a sort of pace layer stack about information environments - based on “pace layers” from Stewart Brand. Lower layers are older and change more slowly; the higher we go, the more things change quickly and are in flux. At the root is our perception and cognition of environment -- these are things that don’t, at core, change much at all over millennia. Then there’s spoken language, something we’ve had with us possibly for over a million years -- to the point that it’s probably a shaping factor in our evolution as a species. Writing and graphical symbolic language come later than speech. They’re technologies, in a sense, for encoding, recording and sharing spoken language. And only later do we get into information organization and design, or what we call “information technology” -- digital computing, networks, & devices. We tend to start our work through the lens of the upper two layers - but they’re the ones that change and fluctuate the fastest. >> I think we should start with perception/cognition as the lens for understanding the rest.

Andrew Hinton | @inkblurt Information in Three Modes 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital Animals (including people) perceiving the environment. Ecological People communicating with people. Semantic There’s a long history of people trying to define information. I’m not into defining things so much these days -- I’m more interested in describing them. And that frees us up to understand a thing in more than one mode or dimension -- to be OK with grasping something in all its facets. Rather than defining information, I’d like to describe how it operates. I think information affects perception and understanding in three major modes. Let me mention them all, then we’ll look at each in more detail. >> First is “ecological” information. It’s about how animals perceive their environment. >> The second is “semantic” information: it’s the mode people use to communicate with one another. >> Third is digital information: digital information is information used by digital systems to transmit to and receive from other digital systems. It’s what happens between the black boxes of our digital infrastructure. Like I said, we’ll look at each of these more closely. Let’s start with ecological information. (12 min)

Andrew Hinton | @inkblurt Ecological Information 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital People communicating with people. Semantic Animals (including people) perceiving the environment. Ecological So, starting with ecological information. The word ecological means having to do with the relationship between an animal and its natural environment. I’m using the term this way because many of the ideas I’m using are based on ecological psychology and embodied cognition, which is different from mainstream cognitive science.

Andrew Hinton | @inkblurt Mainstream Cognitive Science • Sensory perception provides “inputs” ... • to a computer-like brain that “processes” information ... • using disembodied representations (rules, models, frameworks) ... • to codify knowledge and “instruct” the body on how to act. Mainstream cognitive science sees cognition this way SENSORY PERCEPTION DELIBERATED ACTION COGNITIVE PRODUCTION CODIFIED KNOWLEDGE RULES, MODELS, FRAMEWORKS EMBODIED DISEMBODIED Based on a diagram in Ambient Commons: Attention in the Age of Embodied Information Malcolm McCullough 2013 MIT Press. Mainstream cognitive science, which still forms the foundation for most HCI theory and practice, assumes that the brain works like a computer as a sort of information processor. The brain takes sensory inputs from a sort of dumb, robotic body, processes those inputs as “information” -- representational images and symbols of the world, along with images and symbols stored in memory -- and once it has figured out what to do, it tells the body how to respond. This is still the predominant way of seeing how the brain works. It’s part of the assumptions built into many of our methods and training.

Andrew Hinton | @inkblurt Embodied Cognition (not yet mainstream) First we need to look at embodied cognition theory. Embodied cognition argues that cognition is not brain-exclusive, but actually uses the body and even the environment around the body for cognitive activity. There are many flavors and schools of thought even within the embodiment movement; but one in particular is what some call “radical embodied cognition” that says we should not try to marry embodiment with the traditional cognitive science perspective, but replace it entirely. Full disclosure: the ‘radical’ or ‘replacement’ camp is the one I find myself aligning with.

Andrew Hinton | @inkblurt “... there’s no such thing as disembodied cognition, really, if you’re talking about living creatures. All of our cognition has to be embodied ... brains evolved in the service of action.” - Louise Barrett http://podcasts.ox.ac.uk/embodiment-taking-sociality-seriously-audio First we need to look at embodied cognition theory. Embodied cognition argues that cognition is not brain-exclusive, but actually uses the body and even the environment around the body for cognitive activity. There are many flavors and schools of thought even within the embodiment movement; but one in particular is what some call “radical embodied cognition” that says we should not try to marry embodiment with the traditional cognitive science perspective, but replace it entirely. Full disclosure: the ‘radical’ or ‘replacement’ camp is the one I find myself aligning with.

Andrew Hinton | @inkblurt James J Gibson - Ecological Psychology of Perception Long sidelined, now hailed as pioneer of embodied cognition. The so-called ‘radical embodiment’ movement has adopted the work of James J Gibson, who was a scientist of something called “ecological psychology” in the mid 20th century.. He started out studying WWII pilots - and found that centuries-old assumptions about how people comprehend their environment were simply wrong. His ideas have been acknowledged and quasi-appropriated here and there, but now many are starting to see his whole corpus of thought more clearly -- he was really writing about embodied cognition (but calling it ecological psychology).

Andrew Hinton | @inkblurt Ecological Information "Pick-up" Agent Perception Action Ambient, structured energy arrays ORGANISM'S SENSORIMOTOR ACTIVITY ENVIRONMENTAL STRUCTURES (Affordances) THE "PERCEPTUAL SYSTEM" BRAIN ACTIVITY Information pickup theory considers the whole “perceptual system” -- including the environment. We perceive affordance through something called “information pick-up.” A perceiver, takes *action* in an environment in order to discover its affordances. The action part is very important. Gibson rails against traditional cognitive science laboratories that strap people into chairs to keep their heads still -- cognition doesn’t function from stationary positions. We evolved as active, moving, interacting creatures that perceive through action. And when we act in the environment, we perceive, which then affects our action, which then affects what we perceive, in a continuous loop of cognitive activity. This is a very different way of thinking about “information” - but it’s valid, and forms the basis for all the other sorts of information in our lives.

Andrew Hinton | @inkblurt “The perceived functional properties of objects, places and events in relation to an individual perceiver.” - Gibson AFFORDANCE Perception exists only insofar as we perceive affordances. JJ Gibson invented the concept of affordance. Others have since popularized it, but gotten it somewhat wrong -- mainly because they’re coming at it from a traditional cognitive-science perspective, not an embodied perspective. For Gibson, affordance isn’t a thing you add to something. Affordance is the organizing principle behind *ALL* perception. We don’t perceive anything unless it affords meaningful action for a given context.

Andrew Hinton | @inkblurt A few key ideas from Gibson’s theories We perceive the environment in human-scale terms, not scientific abstractions. We perceive environment as “nested,” not in logical hierarchy. We perceive elements in the environment as invariant (persistent) or variant (in flux). Invariance grounds our experience (literally). Invariance Human-scale Nestedness There’s no way to cover all the important stuff from Gibson in this talk, but here are a few key ideas. >> We perceive elements in the environment as invariant or variant. Invariant elements are necessary for orientation of everything else. They include at the widest scale, the earth and the sky. Or perhaps a mountain range. Or even the occluding edge of one’s nose. Variant elements are things that are in flux that we don’t rely on for their persistent structures. >> We perceive the environment in human-scale terms, not scientific abstractions. Perception doesn’t grasp the abstraction of space or time. Our bodies don’t perceive a fallen tree limb in terms of centimeters, but in terms of whether it will fit in the hand, or if it’s too heavy to pick up. >> We perceive the environment as nested. A stream is nested between banks, which are nested between hills, which are nested within a range of larger hills, all of which is nested within the canopy of sky. This is importantly different from strict hierarchy, though. It overlaps and shifts depending on the activity of the perceiver. A cave might feel like “inside” but then feel like “outside” when rain starts leaking in. A stone may just be clutter to me when I walk by it the first time, but when I need a stone to pound something, it becomes an object I can pick up. Then when I pick it up, it becomes an extension of my body. All of these are important ideas for the structures we make for digital and other systems, because our cognition expects the world to accommodate these ways of perceiving.

Andrew Hinton | @inkblurt habitats are made up of places; perceived as nested in the environment; identified/exist because of their affordances. (There is no “space” ecologically speaking.) substance partially or wholly surrounded by the medium; can be attached or detached. the persisting arrangement of surfaces relative to one another and to the ground; layout only exists insofar as objects & surfaces afford meaningful action; everything else is “Clutter.” any change of a substance, place or object; this is how we ecologically perceive “Time.” Place Object Layout Event A few key ideas from Gibson’s theories See explanations in slide.

Andrew Hinton | @inkblurt The structure of all information is rooted in the ecological structure of our environment.

Andrew Hinton | @inkblurt Semantic Information 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital Animals (including people) perceiving the environment. Ecological People communicating with people. Semantic The semantic mode, in short, is language. But I mean language in the broad sense of things we put into the environment to communicate with people. This can be all sorts of stuff: speech, gestures, text, iconography, even buildings have semantic qualities.

Andrew Hinton | @inkblurt SEMANTIC INFORMATION CHANGES HOW WE EXPERIENCE ENVIRONMENT flickr - uicdigital Information (in the sense we tend to mean it colloquially) is what creates and changes much of what we consider to be contextual reality. Look at this photo -- there’s information everywhere in this scene. >>The lines on the road tell us where to drive; the traffic light is a virtual barrier that affects our behavior; the road signs give us a layer of instruction that adds meaning to the city around us. without the information here, it would quite literally be a different place. Really, you could have civilization without cars, lightposts and buildings, but you couldn’t have it without language. Language is our reality in many ways. And a city is as much a construct made of language - speech as well as labels, signs, other semantic artifacts - as one made of atoms. http://www.flickr.com/photos/uicdigital/5410417461/

Andrew Hinton | @inkblurt WE LIVE IN LANGUAGE STRUCTURES In a presentation last year, I heard Peter Merholz talk about how a cube farm in an office building is like the org chart “made manifest.” That’s due to the fact that language structures are an architecture that we live within together, whether we know it or not. Whether these structures are defined explicitly like in this early IBM management diagram, or defined tacitly through the collective assumptions within a shared culture, the way we talk and write about our shared environment is also a structural feature of that environment.

Andrew Hinton | @inkblurt LANGUAGE IS ENVIRONMENT “a form of cognitive scaffolding...” - Andy Clark - Supersizing the Mind Language is not information. Language is environment. When I am speaking I’m vibrating the air - affecting the environment, putting structures into it that weren’t there before. The same goes for writing - it’s environmental structure we’re adding to the world. We then pick up information about what those environmental features mean; you hear the vibrations and, because you’ve learned what those words mean, they have affordance for you.

Andrew Hinton | @inkblurt Semantic + Ecological Affordance Without going up the stairs, I wouldn’t know where this staircase at City Lights Bookstore goes, unless it told me.

Andrew Hinton | @inkblurt Semantic information can be both textual and graphical. We use diagrams to simulate/mimic the structures of the world, in order to explain things. Like this map of Napoleon’s campaign, often cited by Edward Tufte, the graphic diagrams we use make use of our embodied-cognitive comprehension of shape and structure to say things about ideas.

Andrew Hinton | @inkblurt Graphic User Interfaces use graphical semantic information to simulate ecologial information. Physical button Simulated button We use the physical-simulation capabilities of graphics all the time in user-interface work. Graphical user interfaces are essentially simulated ecological information. Objects with affordances, simulated on screens. This is really what is at the heart of “skeuomorphism” vs “flat” design - at its root it isn’t really just about aesthetics or style, but about simulated affordance information and how we use it, pixel by pixel. The same goes for gestures and so-called “natural” or “no UI” ... there’s no such thing as “no UI” because all affordances must be learned one way or another. engine button from ecomodder.com

Andrew Hinton | @inkblurt Which red x???? Looks like a “con rm” action. Digital-Ecological & Semantic Information In Con ict Here are some examples of where things perform both semantically and ecologically, in confusing ways, in a UI. Logically speaking, the red X’s in the first example are all very different -- but ecologically, they require too much thought to disambiguate. In this app I found myself always deleting rather than declining, closing rather than deleting, etc. When I’m in a hurry, I just reach for the closest red X to do whatever I’m trying to do - close the message, decline an invite, or delete it entirely. About half the time, I end up clicking the wrong one if I don’t stop and think about it explicitly. >>In an unsubscribe interface for fab.com, my wife discovered that she was apparently re- subscribing without realizing it, because that big red button -- like a big berry you can’t help but pick -- contextually feels like it’s a confirmation, not a cancellation/re-subscription action.

Andrew Hinton | @inkblurt Which of these will accidentally tweet publicly? (Simulated) Ecological Information = Affordance for action. Very little semantic or ecological information about what context I’m in The infamous Twitter “DM Fail” problem is largely caused by users responding to DMs via SMS. In this case, it’s hard to tell: which of these is a Twitter app that will safely allow me to DM someone, and which is my SMS app that will tweet to everyone who follows me? The physicality of the interface can easily override my perception of the semantic information’s differentiating cues.

Andrew Hinton | @inkblurt Animals (including people) perceiving the environment. Ecological People communicating with people. Semantic Digital Information 10100010 01001000 01110011 10100010 01001000 01110011 Digital systems transmitting to & receiving from other digital systems. Digital So the examples we just looked at weren’t just any sort of semantic information, they were semantic information driven by digital technology. And digital technology relies on digital information. Digital information is how the black boxes talk to the other black boxes. It’s the lifeblood of information technology. The whole point of digital information is to strip human meaning out of it to make it efficient for transmitting and storing encoded information. This isn’t stuff we see face to face very much. Mainly we encounter its *effects* in the environment.

Andrew Hinton | @inkblurt Digital Information Mode Leaking into Semantic Environment We see machines around us trying to get us to perceive what they are saying, or what they want to hear from us. We see them murmuring to each other in weird, noisy machine-only semantics that we do not comprehend either ecologically or semantically. •The gas pump above has to have a sticker added to it that explains what “Enter Data” means. >>The Twitter profile with the iPhone coordinates expresses my location not in a semantic way (the name of a city, for instance) but in a Cartesian grid that I have no contextual orientation for, either semantically or ecologically. >>The Delta app has information that I, as a human, can read, but it gives priority to the machines that I encounter in the workflow of the airport.

Andrew Hinton | @inkblurt Digital information enables pervasive semantic place-making. I don’t mean to paint digital information as a villain. It isn’t. The ability to transmit, store and retrieve information in this way is a miracle. An platform I like a lot is Avocado - it lets a couple keep in touch and share a place together, pervasively. It has nice touches that key into embodied experience of semantic information, like sending a hug by touching the screen to your heart. Another nice touch: the couple shares the same password - making a word into a very real link of co-ownership of the place, like having the same keys to your home. This sort of pervasively available place would be impossible without digital information in the background. But it also requires a lot of discipline with semantic information structure to make the place coherent.

Andrew Hinton | @inkblurt IF YOU COULD SMOKE THE PIPE. INFORMATION MAKES PLACES, KIND OF LIKE THIS PICTURE MAKES A PIPE. This is the famous Magritte painting -- it says “this is not a pipe” The picture definitely shows a pipe but it’s not a real pipe you can smoke. >>Information is kind of like this in the way it makes places. >>Except for a key difference that, with Information, you can smoke the pipe.

Andrew Hinton | @inkblurt LANGUAGE IS INFRASTRUCTURE LABEL LABEL LABEL LABEL LABEL RULES And Language is Infrastructure. We essentially make things out of labels, connections and rules. Too often, we assume the labels are something to add later - but in reality they’re the thing we have to figure out first. This is why issues like ontology and taxonomy are so important - they establish the “invariant” features of the environments we make.

Andrew Hinton | @inkblurt What am I? What is my world? How do I exist in it? Please describe a formal, explicit specification of a shared conceptualization for purposes of structuring semantic data. 00101011100100101110100101 ONTOLOGY Behind the scenes of all this is Ontology. Ontology can be the philosophical sort -- about the nature of one’s being and the relationship of the self to the environment. Or it can be the information-technology sort - developed for digital information work, to define the formal specification for data purposes. A big part of what IA should be doing is bridging these two planes of existence.

Andrew Hinton | @inkblurt The Thing ONTOLOGY Presently our various fields are preoccupied with how to have content and functionality make sense in various contexts. Ontology is at the heart of this problem. In many organizations and project teams, there’s an over-obsession with things like layout in each of the instantiations of a thing, but not enough discussion about how to define the nature of the thing in abstract. That requires an ontological perspective. And, done properly, it forms the main structures of an information environment - the invariant pillars, so to speak - that allow language to stitch together coherence across channels.

Andrew Hinton | @inkblurt What is “card” in this environment? We are disambiguating relationships. Here’s an ontology example. Lowes launched a service called MyLowes -- that requires the registration of a card. But they also have a “Lowe’s Card” that’s a consumer credit card. Conversations at checkout can end up like a “who’s on first” routine -- “do you have your Lowe’s card?” “My Lowe’s card? That’s what I’m paying with.” “No I mean your ‘my lowe’s’ card.” “This IS my lowe’s card!”

Andrew Hinton | @inkblurt Semantic-information “place” signi ed by “account” Digital architecture determining ecological & semantic context. If I walked into a bank and asked to access an account, it’d be clear what I meant. But online, it can mean different things (my profile-account represents me in the digital context -- and needs a label, which happens to also be “account”). The digital systems behind the scenes at Kohls require that these two things we call “account” be separate - requiring disambiguation. The ontology of ‘account’ is in question here. It’s one of the many sorts of things we have to sort out with language, when we’re working in an environment that’s made of almost nothing *but* language.

Andrew Hinton | @inkblurt Shopping Simultaneously in a Store & the Cloud Now that retailers are trying to be in the cloud and on the ground at the same time, context is especially confounding. It requires a great deal of work to situate the user’s perception of place. For many retailers, product price and availability are driven by location - yet shoppers online tend to come to the experience as if it’s a cloud-based store, not thinking about geography yet. It puts the user in a strange environmental position of being in a local store and in an amorphous web-shop experience simultaneously. The ontology of place is dissonant.

Andrew Hinton | @inkblurt Subway station + Food store And here we have a situation where a subway station is also filled with pictures of products that you can actually buy -- not unlike Magritte pipes that you can smoke. With the QR code sprinkled throughout -- digital information wrapped in massive simulacra of ecological information -- plus the semantic information of labels/brands. This could have just been a list of words with QR codes next to them, but perhaps wisely, the retailer decided to create the place in our image, to help bring the “reality” of shopping for groceries into what would otherwise override perception as a subway station.

Andrew Hinton | @inkblurt INFORMATION MODES & PACE LAYERS Digital Ecological Semantic The examples we’ve looked at are going to seem primitive in a matter of just a few years. So we need ways of breaking down whole environments into their essential elements - and those elements are bound up in human perception & cognition. This has been a very cursory overview of what I hope are a useful beginning for principles and frameworks for doing this work into the future.

Andrew Hinton | @inkblurt THANKS! Digital Ecological Semantic Andrew Hinton | @inkblurt | andrewhinton.com The examples we’ve looked at are going to seem primitive in a matter of just a few years. So we need ways of breaking down whole environments into their essential elements - and those elements are bound up in human perception & cognition. This has been a very cursory overview of what I hope are a useful beginning for principles and frameworks for doing this work into the future.