Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ConveyUX - The World is the Screen - Feb 2014 version

Andrew Hinton
February 07, 2014

ConveyUX - The World is the Screen - Feb 2014 version

This is a shorter, but more recent and recently refined, version of the talk I gave throughout 2013.

Andrew Hinton

February 07, 2014
Tweet

More Decks by Andrew Hinton

Other Decks in Design

Transcript

  1. THE WORLD IS
    THE SCREEN
    Convey UX 2014 | Seattle WA | Andrew Hinton | @inkblurt
    Understanding Information Environments
    (Note for PDF: this is a shorter version of a talk I’ve done over the last year; some slides and
    content are newer, but some have been removed. For other versions, see slideshare.net/
    andrewhinton )

    View full-size slide

  2. Andrew Hinton | @inkblurt
    INFORMATION ENVIRONMENT
    http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg
    This is an environment. It’s a natural setting, full of physical, naturally occurring surfaces and
    objects. It’s the sort of environment that humans as a species spent most of its evolutionary
    history in.
    Notice there’s a stone wall here. It’s something made by humans that adds structure to the
    environment. It’s an object that impedes motion, but it’s also an object with cultural meaning
    -- it’s telling us something about this place. Maybe it’s marking off owned property, or it
    says “this is where my sheep are contained.”
    >>
    What if we think of this as not only an environment, but an information environment? In what
    ways is that true?
    That’s largely what this talk is about.
    Because networked technology is becoming so pervasive, so much a part of the fabric of our
    surroundings, that we need to understand the fundamentals of how humans comprehend
    their environment, regardless of the specific technology.
    And understanding that means understanding the sorts of information that are in the world,
    and how they work together to create places, contexts, and meaning.
    http://commons.wikimedia.org/wiki/File:Derbyshire_Landscape.jpg

    View full-size slide

  3. Andrew Hinton | @inkblurt
    INFORMATION
    ENVIRONMENT
    3
    INFORMATION
    ENVIRONMENT
    These are two information environments that involve office supplies.
    What is the difference, really, between shopping for them in a brick and mortar retail store,
    and shopping for them through an online retailer like Amazon?
    If we think of the physical store as an information interface, how much information is
    conveyed, and of what kind, through one interface versus another? How useful is it for the
    activity of deciding between one printer or stapler and another? There are things the digital
    environment affords us that the physical one does not: we can compare things more easily,
    sort them, search them; we can see more of a selection; we can see what others have said
    about the products.

    View full-size slide

  4. Andrew Hinton | @inkblurt
    Subway station + Food store
    Here we have a mashup where a place is simultaneously a subway station and a virtual store,
    with pictures of products on the walls where shoppers can scan them for delivery or pickup
    using the store’s mobile app. It stretches what we assume about how an environment works,
    and even what a place is.

    View full-size slide

  5. Andrew Hinton | @inkblurt
    Google Glass
    Leap Motion Microsoft Kinect
    We’re being surrounded by sensors and digital agents that are trying to read our actions, or
    layers of digital information added to what we see around us.
    left glass image wired.com / others from google
    Leap motion from their site
    Kinect from blogs.msdn.com

    View full-size slide

  6. Andrew Hinton | @inkblurt
    Now, even your refrigerator can spam you.
    http://www.bbc.co.uk/news/technology-25780908 bestbuy.com
    Our environment is full of even mundane things that are networked and active in ways we
    used to only joke about. Like the smart refrigerator that was recently caught being used as a
    spam bot.

    View full-size slide

  7. Andrew Hinton | @inkblurt
    Understanding the nature
    of our materials.
    We’ve gone for a long time in our work being comfortable with knowing that what we make is
    confined to a specific screen -- a bounded interactive surface. But now *all* the surfaces
    around us are potentially digital and networked. Walls and windows, coffee pots and shoes.
    So we need to re-assess, what is the nature of our materials?
    Potters have a deep understanding of clay; masons have a deep understanding of stone.
    We need to deepen our understanding of information.

    View full-size slide

  8. Andrew Hinton | @inkblurt
    INFORMATION
    Digital
    Semantic
    Physical
    Information is a word we use a lot, but I don’t think that in our general practice we have a
    common framework for what it is, and the role it plays in the environments we inhabit.
    I tried finding a single, official definition, but there really isn’t one -- instead, there are many
    different ways of looking at various things in our world that people call information.
    The problem is, all of them are pretty useful and valuable. So rather than just pick one, I
    decided to pick several and see how they work together and inform each other, so to speak.
    These modes or ways we experience information are physical, semantic, and digital.

    View full-size slide

  9. Andrew Hinton | @inkblurt
    Physical Information
    10100010
    01001000
    01110011
    10100010
    01001000
    01110011
    Digital systems transmitting
    to & receiving from other
    digital systems.
    Digital
    People communicating
    with people.
    Semantic
    Animals (including
    people) perceiving the
    environment.
    Physical
    Physical information has to do with the ecological relationship between an animal and its
    physical environment.
    The green field we looked at earlier was mostly made up of physical information that
    creatures use to know where they can go and what they can do in those surroundings.
    The way I’m framing this comes from something called ecological psychology, which has been
    adapted for a branch of study called embodied cognition.

    View full-size slide

  10. Andrew Hinton | @inkblurt
    Ecological Information
    "Pick-up"
    Agent
    Perception
    Action
    Ambient,
    structured
    energy arrays
    Embodied cognition:
    Body & Environment, not just Brain.
    Embodied cognition sees the environment and the body as part of a larger dynamic, in which
    the brain is a participant, but not necessarily the center-point. It may even just be a
    supporting player.
    A big part of this is *action*: we don’t perceive the world by sitting still and taking in sensory
    data; we perceive by moving and interacting with the environment.
    Much of our daily activity is unconsciously nudged and controlled by our environment.

    View full-size slide

  11. Andrew Hinton | @inkblurt
    “... there’s no such thing as
    disembodied cognition [for]
    living creatures.
    All of our cognition has to
    be embodied ...
    Brains evolved in the
    service of action.”
    - Louise Barrett
    http://podcasts.ox.ac.uk/embodiment-taking-sociality-seriously-audio
    Louise Barrett, who wrote a great book on embodied cognition called Beyond the Brain, puts
    it nicely -- -- there’s no such thing as disembodied cognition for living creatures, all of our
    cognition has to be embodied because brains evolved in the service of action.
    We didn’t arrive on the planet as brains that grew bodies to supplement their disembodied
    brain activity. Brains evolved to supplement the abilities of bodies in environments. They’re
    the product of the structures of their surroundings.

    View full-size slide

  12. Andrew Hinton | @inkblurt
    James J Gibson - Ecological Psychology of Perception
    Now seen as pioneer of embodied cognition.
    Some embodied cognition theorists have heavily adopted the work of James J Gibson, who
    was a scientist of something called “ecological psychology” in the mid 20th century -- a field
    he was part of inventing.

    View full-size slide

  13. Andrew Hinton | @inkblurt
    Ecological Information
    "Pick-up"
    Agent
    Perception
    Action
    Ambient,
    structured
    energy arrays
    Gibson used the word “information” to mean the
    elements we perceive in the environment that tell us
    what it “affords.”
    “INFORMATION”
    Gibson purposefully used the word “information” for this body-environment relationship. He
    argued that our bodies do something he called “information pickup” -- where the body’s
    perceptual system detects structural cues in the environment that specify what actions the
    body can take. And he argued that this happens directly, with little or no need for brain
    processing.

    View full-size slide

  14. Andrew Hinton | @inkblurt
    The perceived functional
    properties of objects, places
    and events in relation to an
    individual perceiver.
    AFFORDANCE
    Perception exists only insofar
    as we perceive affordances.
    Gibson also invented the concept of affordance.
    For Gibson, affordance isn’t a thing you add to something.
    It’s the result of a property in the environment that presents the opportunity for action to a
    perceiving organism.
    This is true in the built environment as well as the natural one; the structure of these stairs
    interacts with energy in the environment that my perception picks up on, and my body
    detects that the specific structure of the steps will take my body upward if climb them.
    Affordance was and still is a radical idea, in the sense that Gibson meant it.
    Affordance is the organizing principle behind *ALL* perception. We don’t perceive anything
    unless it affords meaningful action for a given context.
    Gibson even went on to discuss how higher-order combinations of affordances can be found
    in the complexities of human culture. A mailbox, for example, has the affordance for sending
    mail -- but only because we’ve learned a lot of stuff about the world around that box that
    makes it a special sort of container connected to a broader cultural system.

    View full-size slide

  15. Andrew Hinton | @inkblurt
    We perceive environment as “nested,”
    not in logical hierarchy.
    We perceive elements in the
    environment as invariant (persistent) or
    variant (in flux). Invariance grounds our
    experience (literally).
    Invariance
    Nestedness
    Two of Gibson’s principles of
    environmental perception.
    Gibson created a marvelous system of principles and components that work as building
    blocks for how we perceive our environment and its affordances. I can only barely touch on
    them here, but I’ll mention a few important ones.
    >> Invariance is the property of environmental structure that makes some of it persistent
    over time, so that we can make sense of the whole. Terrestrial animals like us evolved on
    stable ground, under a persistent sky. Substances and surfaces that persist allow us to learn
    what properties they have, and what they afford us. This can be a useful idea in digital design
    because in software, even though we can make everything fluid and variant, we need to
    design invariant structures that ground the experience.
    >> Nestedness: we perceive the environment as structures that relate to one another as
    nested -- a stream is in a valley, which is between mountains, which are part of a range,
    which are between earth and sky. There aren’t always definite boundaries, though, and this
    isn’t a purely logical relationship -- there are overlaps and redundancies, and shifts in
    meaning. A stick on the ground can be kindling or a club, or it can be part of a roof over our
    heads in a shelter. It’s not just one defined thing in how it relates to the rest of the
    environment.
    This is a powerful idea when we look at something like faceted navigation, where the
    meaning of a piece of content or an object shifts depending on what perspective a user is
    bringing to it.

    View full-size slide

  16. Andrew Hinton | @inkblurt
    Surfaces
    Objects
    Layout
    Place
    Some of the components that
    make up environments.
    There are many components Gibson outlines, like surface, object, layout and place -- that
    work together to form the environment.
    There isn’t time to cover it all here -- but I think it’s a useful framework we can borrow from,
    because all the environments we design should meet the expectations of people’s embodied
    experience. Our bodies want and need environments to work in these ways.

    View full-size slide

  17. Andrew Hinton | @inkblurt
    Semantic Information
    10100010
    01001000
    01110011
    10100010
    01001000
    01110011
    Digital systems transmitting
    to & receiving from other
    digital systems.
    Digital
    Animals (including
    people) perceiving the
    environment.
    Physical
    People communicating
    with people.
    Semantic
    Of course, humans don’t just exist in raw physical surfaces and objects -- we add to the
    environment. In addition to buildings and roads and such, we add language to the world.
    The semantic mode, in short, is language. But I mean language in the broad sense of things
    we put into the environment to communicate with people. This can be all sorts of stuff:
    speech, gestures, text, iconography, even buildings have semantic qualities.

    View full-size slide

  18. Andrew Hinton | @inkblurt
    LANGUAGE IS ENVIRONMENT
    STRUCTURE THAT ENABLES US TO MAKE MORE STRUCTURE
    Language is “a form of mind-
    transforming cognitive
    scaffolding: a ... symbolic
    edi ce [playing a] critical role
    in promoting thought and
    reason.”
    - Andy Clark Supersizing the Mind
    http://en.wikipedia.org/wiki/File:CantileverScaffold.jpg
    Language is environment. In another excellent book on embodied cognition, Andy Clark
    frames language as a form of cognitive scaffolding; it persists in the environment, as new
    structure that allows us to make other structures. We wouldn’t have most of what we think of
    as civilization without the scaffolding of language.

    View full-size slide

  19. Andrew Hinton | @inkblurt
    Language is the primary human interface to
    complex systems.
    Language is both the infrastructure we use for creating complexity, as well as the primary
    interface we have for understanding and inhabiting complex systems.

    View full-size slide

  20. Andrew Hinton | @inkblurt
    WE LIVE IN LANGUAGE STRUCTURES
    Language structures are an architecture that we live within together, whether we know it or
    not. Whether these structures are defined explicitly like in this early IBM management
    diagram, or defined tacitly through the collective assumptions within a shared culture, the
    way we talk and write about our shared environment is also a structural feature of that
    environment.

    View full-size slide

  21. Andrew Hinton | @inkblurt
    Physical Affordance +
    Semantic (or “Linguistic”) Affordance
    Physical
    Semantic
    Humans aren’t just perceivers of physical structure. We depend as much or more on the
    symbolic workings of language – semantic information – as a part of our environment for
    working together and figuring out our world, and making stuff – as well as just getting
    around.
    In this picture of a staircase at City Lights Bookstore in SF, we see stairs that have intrinsic
    structural information that our bodies perceive as affording the ability to walk upward.
    But there’s this other thing – a label – which signifies what where we will be going when we go
    up the stairs.
    We might call this semantic affordance. It’s an abstracted, symbolic version of how we
    perceive and comprehend physical structure.
    In linguistics we also hear of signification -- which I won’t go into in detail here -- but there
    are signifiers at work in both the stairs and the label. The important point I want to get across
    here is that affordance is at work even at the high-order level of meaning we have with
    language, not just in physical structures.

    View full-size slide

  22. Andrew Hinton | @inkblurt
    Physical
    Physical
    + Semantic
    Simulated-
    Physical
    + Semantic
    Mostly
    Semantic
    Affordance across a spectrum
    of physical to semantic.
    Affordance is at the root of how we use language as well as the simulated physicality of the
    stuff we put into interfaces.
    >> The stairs intrinsically contain all the information needed for your body to know what will
    happen when you climb them.
    >> But look at this button. When people started making machines and systems that were
    more complex, we had physical affordances that did more than we could tell just by touching
    or looking at them. Because I’ve pushed similar buttons before, I know that the physical
    information here means I should be able to push in the red part and it will probably spring
    back out -- that’s all that the physical information is telling me. In order to know what the
    button is going to cause elsewhere, I need a label.
    >> When Microsoft updated Windows to add the Start button, they were basically adding an
    invariant structure to the operating system so that users would know that, no matter what,
    they could always “start” a new action from here. And like lots of other software, they used a
    simulated button object, borrowing from what we were used to with buttons like the one on
    the left. The button isn’t physical information, really ... it’s using semantic information to
    simulate a physical object. That’s what graphical user interfaces do.
    >> In other parts of interfaces, and especially on the web with hypertext, labels may not have
    these simulated object qualities; so we have to rely a lot on the context of a label to know if it
    just affords reading or if it’s something we can activate.
    This is the issue at the heart of flat versus skeuomorphic design -- it’s really not about
    aesthetics, it’s about solving this issue of affordance.
    Additionally, the web introduced broad usage of labels as paths -- tapping a hyperlink
    doesn’t just activate some mechanism, it takes you on a journey to what feels to your
    perception like a new place.

    View full-size slide

  23. Andrew Hinton | @inkblurt
    Which red x????
    Looks like a
    “con rm” action.
    Simulated-Physical & Semantic Information
    In Con ict
    Here are some examples of where things perform both semantically and ecologically, in
    confusing ways, in a UI.
    In this email application, logically speaking, the red X’s are all very different -- but physically
    they’re barely different at all; they require a lot of thought to disambiguate their semantic
    meaning. When I was in a hurry, I would reach for the closest red X to do whatever I’m trying
    to do - close the message, decline an invite, or delete it entirely. About half the time, I would
    click the wrong one.
    >>In an unsubscribe interface for fab.com, my wife discovered that she was apparently re-
    subscribing without realizing it, because that big red button -- like a big berry you can’t help
    but pick -- contextually feels like it’s a confirmation, not a cancellation/re-subscription
    action.

    View full-size slide

  24. Andrew Hinton | @inkblurt
    Animals (including
    people) perceiving the
    environment.
    Physical
    People communicating
    with people.
    Semantic
    Digital Information
    10100010
    01001000
    01110011
    10100010
    01001000
    01110011
    Digital systems transmitting
    to & receiving from other
    digital systems.
    Digital
    So the examples we just looked at weren’t just any sort of semantic information, they were
    semantic information driven by digital technology. And digital technology relies on digital
    information.
    Digital information is how the black boxes talk to the other black boxes. It’s the lifeblood of
    information technology. The whole point of digital information is to strip human meaning out
    of it, which makes it more efficient for machines to transmit, store, and retrieve.
    This isn’t stuff we see face to face very much. Mainly we encounter its *effects* in the
    environment.

    View full-size slide

  25. Andrew Hinton | @inkblurt
    Digital Information Mode Leaking into Semantic Environment
    We see machines around us trying to get us to perceive what they are saying, or what they
    want to hear from us. We see them murmuring to each other in weird, noisy machine-only
    semantics that we do not comprehend either ecologically or semantically. The gas pump has
    to have a sticker added to it that explains what “Enter Data” means.
    >>The Twitter profile with the iPhone coordinates expresses my location not in a semantic
    way (the name of a city, for instance) but in a Cartesian grid that I have no contextual
    orientation for, either semantically or ecologically.
    >>The Delta app has information that I, as a human, can read, but it gives priority to the
    machines that I encounter in the workflow of the airport.

    View full-size slide

  26. Andrew Hinton | @inkblurt
    Digital information enables
    pervasive semantic place-making.
    I don’t mean to paint digital information as a villain. It isn’t.
    The ability to transmit, store and retrieve information in this way is a miracle.
    An platform I like a lot is Avocado - it lets a couple keep in touch and share a place together,
    pervasively.
    It has nice touches that key into embodied experience of semantic information, like sending a
    hug by touching the screen to your heart.
    Another nice touch: the couple shares the same password - making a word into a very real
    link of co-ownership of the place, like having the same keys to your home.
    This sort of pervasively available place would be impossible without digital information in the
    background. But it also requires a lot of discipline with semantic information structure to
    make the place coherent.

    View full-size slide

  27. Andrew Hinton | @inkblurt
    Google Plus on Desktop Google Plus on Mobile
    Easily post content in the circle I’m in.
    Confusingly requires re-establishing the
    circle where I’m posting.
    Broken “invariants” across contexts.
    Google Plus has invariance in how it establishes place between mobile and desktop versions.
    When I’m on desktop, and I’m in my “Family” Circle reading posts, when I post something
    new it automatically goes into that circle, just like I’d expect if I were having a conversation in
    a room. But on mobile, if I’m in my Family circle, a new post defaults to viewable by everyone;
    I have to work through a half dozen interactions to make it post just to the circle I perceived I
    was already “in.”

    View full-size slide

  28. Andrew Hinton | @inkblurt
    The ontology of “news” is confused here.
    • News Feed inside News
    Feed?
    • If All Friends, is it no longer
    News? or Recent?
    • Isn’t News “recent” by
    de nition?
    • What, then, is News Feed as
    a place? How am I changing
    the kind of place it is here?
    Ontology is the basis that we use for creating any sort of taxonomy - or relationship between
    semantic invariants.
    In the mobile version of Facebook’s “news feed” there are several options for how you want to
    see the feed. Their meanings don’t make much sense together, though -- it’s not clear how
    these are nested.
    Is news feed another news feed inside news feed?
    What does all friends mean? If I choose that, am I not seeing news anymore, or the most
    recent?
    Isn’t news recent by definition?
    What is news-feed as a place? How am I changing the mode of the place when I select
    something else?

    View full-size slide

  29. Andrew Hinton | @inkblurt
    What is “card” in this environment?
    Lowes launched a service called MyLowes -- that requires the registration of a card. But they
    also have a “Lowe’s Card” that’s a consumer credit card.
    Conversations at checkout can end up like a “who’s on first” routine -- “do you have your
    Lowe’s card?” “My Lowe’s card? That’s what I’m paying with.” “No I mean your ‘my lowe’s’
    card.” “This IS my lowe’s card!”

    View full-size slide

  30. Andrew Hinton | @inkblurt
    Shopping Simultaneously in a Store & Online
    http://stevenglassman.de/2013/11/25/america-vs-deutschland/
    In these omni-channel scenarios, the semantic information has to align with how it’s
    presented both online and physically in the store’s structure. Knowing if a huge store has a
    specific peanut butter or not is great; knowing exactly which aisle to get it from is even
    better.

    View full-size slide

  31. Andrew Hinton | @inkblurt
    The
    Thing
    Ontology helps answer:
    “What is a thing across multiple contexts?”
    Presently our various fields are preoccupied with how to have content and functionality make
    sense in various contexts.
    Ontology is at the heart of this problem.
    In many organizations and project teams, there’s an over-obsession with things like layout in
    each of the instantiations of a thing, but not enough discussion about how to define the
    nature of the thing in abstract. That requires an ontological perspective.
    And, done properly, it forms the main structures of an information environment - the
    invariant pillars, so to speak - that allow language to stitch together coherence across
    channels.

    View full-size slide

  32. Andrew Hinton | @inkblurt
    LANGUAGE IS
    INFRASTRUCTURE.
    LABEL
    LABEL
    LABEL
    LABEL
    LABEL
    RULES
    Labels
    Connections
    Rules
    All this stuff is essentially made out of language... it’s language working as infrastructure.
    We essentially make things out of labels, connections and rules.
    Too often, we assume the labels are something to add later - but in reality they’re the thing
    we have to figure out first.
    This is why issues like ontology and taxonomy are so important - they establish the
    “invariant” features of the environments we make.

    View full-size slide

  33. Andrew Hinton | @inkblurt
    • What are the invariants for perception
    & action? What grounds the
    experience?
    • How is it structurally nested in relation
    to a perceiver? Is it resilient for
    different perspectives & contexts?
    • Is it clear what is a place and what is
    an object?
    • How are semantic and physical
    information working together, or not?
    • Are the rules behind the environment
    clearly expressed in its information?
    Some of the questions we can ask ...
    So in sum, this is just scratching the surface, but there are some questions we can ask.
    >>What are the invariants for perception & action? What grounds the experience?
    >>How is it structurally nested in relation to a perceiver? Is it resilient for different
    perspectives & contexts?
    >>Is it clear what is a place and what is an object?
    >>How are semantic and physical information working together, or not?
    >>Are the rules behind the environment clearly expressed in its information?
    We have a long way to go to build heuristics and frameworks for responsibly creating the
    broader environment that our users inhabit. Hopefully this is a helpful start.

    View full-size slide

  34. Andrew Hinton | @inkblurt
    THANKS!
    Andrew Hinton | @inkblurt | andrewhinton.com
    [email protected]

    View full-size slide