2 December 2005 Human-Computer Interaction Human Perception and Cognition Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com

Beat Signer - Department of Computer Science - [email protected] 2 October 14, 2024 Human Senses and Modalities ▪ Input as well as output modalities Sensory Perception Sense Organ Modality sight eyes visual hearing ears auditory smell nose olfactory taste tongue gustatory touch skin tactile (balance) vestibular system vestibular Modality “refers to the type of communication channel used to convey or acquire information. It also covers the way an idea is expressed or perceived, or the manner an action is performed.” Nigay and Coutaz, 1993

Beat Signer - Department of Computer Science - [email protected] 3 October 14, 2024 Model of Visual Perceptual Processing "What" system

Beat Signer - Department of Computer Science - [email protected] 4 October 14, 2024 Model of Perceptual Processing ... ▪ Stage 1: Parallel processing to extract low-level features ▪ billions of neurons working in parallel (in eye and visual cortex) ▪ orientation, colour, texture and motion ▪ results in a set of feature maps ▪ Stage 2: Pattern perception ▪ slower serial processing ▪ continuous contours, regions of same colour or texture ▪ two-visual-system theory: "Action" system and "What" system ▪ Stage 3: Visual working memory ▪ only a few objects in visual working memory ▪ may provide answers to visual query (cognitive processing) ▪ Attention affects all three stages!

Beat Signer - Department of Computer Science - [email protected] 5 October 14, 2024 Anatomy of the Human Eye https://commons.wikimedia.org/wiki/File:Blausen_0388_EyeAnatomy_01.png Fovea

Beat Signer - Department of Computer Science - [email protected] 6 October 14, 2024 Anatomy of the Human Eye ... ▪ Variable focus lens ▪ Pupil (aperture) ▪ Retina (sensor array) ▪ what we see ≠ image on the retina ▪ two types of cells - rods (~100 million), highly sensitive at low light levels - cones (~6 million), three distinct colour receptors (S-cones, M-cones and L-cones) (trichromacy) ▪ Brain forms our sight ▪ focal length ▪ maximum sharpness ▪ …

Beat Signer - Department of Computer Science - [email protected] 7 October 14, 2024 Cone Cell Sensitivity

Beat Signer - Department of Computer Science - [email protected] 8 October 14, 2024 Brightness ▪ Colour space can be divided into luminance (grey scale) and two chromatic (hue and saturation) dimensions ▪ Light receptors in the eye ▪ do not measure the amount of light on the retina ▪ measure relative light changes over time and over adjacent spots on the retina ▪ Eye is a change meter rather than a light meter ▪ Luminance refers to the measured amount of light coming from some region of space

Beat Signer - Department of Computer Science - [email protected] 9 October 14, 2024 Brightness Illusions ▪ Retinal ganglion cell receives input from multiple receptors (receptive field) ▪ Processing tries to enhance edges ▪ lateral inhibition from neighbouring neurons Herman grid illusion [Information Visualization, Colin Ware, 2013]

Beat Signer - Department of Computer Science - [email protected] 10 October 14, 2024 Relative Luminance Perception ▪ Perception of luminance is contextual based on the contrast with surrounding colours

Beat Signer - Department of Computer Science - [email protected] 11 October 14, 2024 Simultaneous Brightness Contrast Simultaneous brightness contrast [Information Visualization, Colin Ware, 2013]

Beat Signer - Department of Computer Science - [email protected] 12 October 14, 2024 Highlighting via Contrast Consider using adjustments in luminance contrast as a highlighting method. It can be applied by reducing the contrast of unimportant items or by locally adjusting the background to increase the luminance contrast of critical areas. Highlighting [Information Visualization, Colin Ware, 2013]

Beat Signer - Department of Computer Science - [email protected] 13 October 14, 2024 Colour (Hue) Perception ▪ Our visual systems evolved to provide colour constancy ▪ same surface identifiable across illumination conditions ▪ visual system might work against simple colour encodings [Visualization Analysis & Design, Tamara Munzner, 2014]

Beat Signer - Department of Computer Science - [email protected] 14 October 14, 2024 Saturation of Small/Large Areas Use more saturated colours when colour coding small symbols, thin lines, or other small areas. Use less saturated colours for coding large areas.

Beat Signer - Department of Computer Science - [email protected] 15 October 14, 2024 Colour Blindness ▪ About 10% of the male population and about 1% of the female population have some form of colour vision deficiency ▪ Most commonly a lack of either the L-cones (protanopia) or the M-cones (deuteranopia) ▪ both of these result in an inability to distinguish red and green

Beat Signer - Department of Computer Science - [email protected] 16 October 14, 2024 Visual Field Visual field [Information Visualization, Colin Ware, 2013]

Beat Signer - Department of Computer Science - [email protected] 17 October 14, 2024 Peripheral Vision ▪ Pixel density much higher in the centre of our view (fovea) ▪ fovea has around 160’000 cone cells/mm2 and the rest only 9000 cone cells/mm2 ▪ fovea cone cells have 1:1 connection to ganglion neuron cells while multiple cone cells outside of the fovea a grouped to a single ganglion neuron cell (compression) ▪ Brain’s visual cortex devotes about 50% of its processing to input from the fovea (1% of the retina) ▪ We have very poor peripheral vision ▪ but eyes move rapidly (about 3 times/s) to focus fovea on different parts of the environment ▪ brain “fills in” the gaps

Beat Signer - Department of Computer Science - [email protected] 18 October 14, 2024 Ninio's Extinction Illusion

Beat Signer - Department of Computer Science - [email protected] 19 October 14, 2024 Example of Peripheral Vision

Beat Signer - Department of Computer Science - [email protected] 20 October 14, 2024 Peripheral Vision ▪ Guides fovea towards objects matching our goals ▪ low-resolution cues to guide eye movements ▪ Good in detecting motion ▪ Better view in the dark ▪ rods-only vision ▪ given that there are no rods in the fovea, better to not directly focus on object in low-light conditions ▪ Might for example influence where we should show messages in a user interface ▪ place important information where users will be looking ▪ only neural networks that start in the fovea are involved in reading - cannot read in the periphery

Beat Signer - Department of Computer Science - [email protected] 21 October 14, 2024 Perception ▪ Perception of the world around us is not an exact depiction of what is there ▪ Perception is heavily biased by ▪ the past (our experience) ▪ the present (current context) ▪ the future (our goals)

Beat Signer - Department of Computer Science - [email protected] 22 October 14, 2024 Perception Biased by Past (Experience) ▪ Perceptual priming ▪ Familiar perceptual patterns/frames ▪ e.g.do not suddenly switch the order of buttons in an interface ▪ Habituation ▪ e.g.same warning appearing again and again → users will not bother to read them anymore ▪ Attentional blink ▪ after spotting something important, we are nearly blind to other visual stimuli for 0.15 to 0.45 seconds

Beat Signer - Department of Computer Science - [email protected] 23 October 14, 2024 Perception Biased by Present Context ▪ Visual perception is not simply a bottom-up process ▪ e.g.reading Fold napkins. Polish silverware. Wash dishes. French napkins. Polish silverware. German dishes.

Beat Signer - Department of Computer Science - [email protected] 24 October 14, 2024 McGurk Effect ▪ Modalities can affect each other ▪ e.g.what we see can override what we hear

Beat Signer - Department of Computer Science - [email protected] 25 October 14, 2024 Perception Biased by Future (Goals) ▪ Things unrelated to our goals seem to be filtered out preconsciously ▪ e.g.“cocktail party” effect ▪ Based on our current goal we might not really perceive certain elements in a user interface ▪ perception is active, not passive → we actively perceive what we need (e.g.for a task)

Beat Signer - Department of Computer Science - [email protected] 26 October 14, 2024 Designing for Biased Perception ▪ Avoid ambiguity ▪ test design to ensure that all users have the same interpretation ▪ Be consistent ▪ e.g.controls with the same function always at the same place ▪ Understand the goals ▪ information needed to achieve goals should be available and prominent

Beat Signer - Department of Computer Science - [email protected] 27 October 14, 2024 Gestalt Principles (Video)

Beat Signer - Department of Computer Science - [email protected] 28 October 14, 2024 Gestalt Principles ▪ Law of Proximity ▪ items that are closer together appear to be grouped ▪ Law of Similarity ▪ similar-looking items appear grouped ▪ Law of Continuity ▪ visually-aligned items are perceived as continuous forms ▪ Law of Closure ▪ visual system automatically tries to close open figures ▪ Law of Symmetry (Simplicity) ▪ parse complex scenes in a way that reduces complexity

Beat Signer - Department of Computer Science - [email protected] 29 October 14, 2024 Gestalt Principles … ▪ Law of Figure/Ground ▪ our mind separates the visual field into figure (foreground) and ground (background) ▪ Law of Common Fate ▪ items that are moved together are perceived as grouped or related ▪ Often multiple (all) Gestalt principles used together ▪ check design whether it suggests any unintended relationships between items Liagala, reddit

Beat Signer - Department of Computer Science - [email protected] 30 October 14, 2024 Long-Term Memory ▪ Permanent memory store intended for the long-term storage of information (everything that we know including experiential knowledge, procedural skills, etc.) ▪ huge (if not unlimited) capacity ▪ relatively slow access time ▪ forgetting (when it occurs) happens much slower than with short-term memory ▪ Three long-term memory subsystems ▪ episodic memory: memory of events and experiences in a serial form (chronology) ▪ semantic memory: structured record of facts and concepts that we have acquired ▪ procedural skills: "know-how" memory (skills, procedures)

Beat Signer - Department of Computer Science - [email protected] 31 October 14, 2024 Immediate Memory ▪ Short-Term Memory ▪ simple “storage” of individual items ▪ normally up to 7±2 items (Miller, 1956) - study participants were only college students!? → number might be too high ▪ Working Memory ▪ sensory-specific working memory ▪ long-term memories reactivated through recall ▪ everything that we are conscious of at a given time (current focus of our attention) ▪ Memory formation consists of changes in the neurons involved in a neural activity pattern ▪ single memory system for immediate and long-term memory

Beat Signer - Department of Computer Science - [email protected] 32 October 14, 2024 Design Implications by Working Memory ▪ Some interface types (e.g.voice interface) put higher load on immediate memory ▪ Support working memory for specific tasks ▪ e.g.when using a search function do not only show the result but also the original search terms ▪ broad and shallow navigation hierarchies are easier - provide “bread crumb” paths

Beat Signer - Department of Computer Science - [email protected] 33 October 14, 2024 Design Guidelines ▪ Design guidelines are based on human psychology: how people perceive, learn, reason, remember and convert intentions into action

Beat Signer - Department of Computer Science - [email protected] 34 October 14, 2024 Exercise 3 ▪ Low-Fidelity Prototyping

Beat Signer - Department of Computer Science - [email protected] 35 October 14, 2024 Further Reading ▪ Major parts of this lecture are based on the book Designing with the Mind in Mind

Beat Signer - Department of Computer Science - [email protected] 36 October 14, 2024 References ▪ Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines, Jeff Johnson, Morgan Kaufmann (3rd edition), November 2020, 978-0128182024 ▪ Information Visualization: Perception for Design, Colin Ware, Morgan Kaufmann (3rd edition) May 2012, ISBN-13: 978-0123814647 ▪ McGurk Effect ▪ https://www.youtube.com/watch?v=2k8fHR9jKVM ▪ Peripheral Vision Distorting Faces ▪ https://www.youtube.com/watch?v=VT9i99D_9gI

2 December 2005 Next Lecture Design Guidelines and Models