2 December 2005 Human-Computer Interaction Use Cases and Course Review Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com

Beat Signer - Department of Computer Science - [email protected] 2 December 9, 2024 HCI Research Contributions Revisited ▪ Empirical contributions ▪ Artefact contributions (systems research) ▪ Methodological contributions ▪ Theoretical contributions ▪ Dataset contributions ▪ Literature survey contributions ▪ Opinion contributions

Beat Signer - Department of Computer Science - [email protected] 3 December 9, 2024

Beat Signer - Department of Computer Science - [email protected] 4 December 9, 2024 Prof. Dr. Beat Signer Cross-Media Technology, Interac- tive Paper, Data Physicalisation Dr. Audrey Sanctorum User-defined XDI and IoT Inter- action, Human-AI Interaction CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING Maxim Van de Wynckel Interoperable Positioning Systems, Implicit HCI A person with long hair Description automatically generated with low confidence Yoshi Malaise Technology-enhanced Learning, Content-driven Presentations CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA)

Beat Signer - Department of Computer Science - [email protected] 5 December 9, 2024 logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Ekene Attoh IoT Middleware, Context-aware Computing, Implicit HCI Isaac Valadez Knowledge Physicalisation and Augmentation, Tangible UIs Xuyao Zhang Extensible Platform for Dynamic Data Physicalisation A close-up of a person smiling Description automatically generated Ingela Rossing Dynamic Data Physicalisation Framework and Guidelines

Beat Signer - Department of Computer Science - [email protected] 6 December 9, 2024 logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Piet Van Der Paelt Julia-based Framework for Simulation and Optimisation Evan Cole Technology-enhanced Learning, Study Lenses A person wearing glasses Description automatically generated with medium confidence Qi Xu Next Generation Scholarly Workflows, Cross-Media PIM A person with glasses and a white background Description automatically generated Bingyan Lee Next Generation User Interfaces for Human-Information Interaction

Beat Signer - Department of Computer Science - [email protected] 7 December 9, 2024 logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Arun Sojan Physicalisation for Digital Twins

Beat Signer - Department of Computer Science - [email protected] 8 December 9, 2024 Interactive Paper edfestComponents Global Information Systems Group ETH Zurich

Beat Signer - Department of Computer Science - [email protected] 9 December 9, 2024 Visual and Interaction Design ▪ Visualisation of interactive areas and functionality ▪ design patterns ▪ Interaction design ▪ online versus offline input processing ▪ lack of modal dialogues ▪ multimodal interaction ▪ Findings might be appli- cable to traditional GUIs ▪ e.g.cross-application widget interaction booklet 2004 booklet 2005 Global Information Systems Group, ETH Zurich

Beat Signer - Department of Computer Science - [email protected] 10 December 9, 2024 iGesture Framework ▪ iGesture Workbench ▪ create/test gesture sets and algorithms ▪ Different modalities ▪ digital pen, tablet PC, mouse, Wii remote, … ▪ multimodal gestures ▪ Open source ▪ http://www.igesture.org gestureSets.jpg captureFormNew.jpg

Beat Signer - Department of Computer Science - [email protected] 11 December 9, 2024 edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Fluid Cross-Media Information Spaces

Beat Signer - Department of Computer Science - [email protected] 12 December 9, 2024 RSL Hypermedia Metamodel iwebfirefox movieplugin ipaper Global Information Systems Group, ETH Zurich

Beat Signer - Department of Computer Science - [email protected] 13 December 9, 2024 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues ▪ unlimited canvas with zoomable user interface ▪ non-linear navigation and associative linking ▪ content-based approach with automatic visualisation

Beat Signer - Department of Computer Science - [email protected] 14 December 9, 2024 Presentation Solutions ▪ PaperPoint ▪ interactive PowerPoint paper handouts ▪ non-linear navigation and annotations ▪ MindXpres ▪ extensible presentation platform ▪ content-driven approach ▪ non-linear navigation via zoomable user interface ▪ associative linking ▪ rich media plug-ins

Beat Signer - Department of Computer Science - [email protected] 15 December 9, 2024 Interactive Source Code Plug-in Paul Mestereaga

Beat Signer - Department of Computer Science - [email protected] 16 December 9, 2024 MindXpres Video

Beat Signer - Department of Computer Science - [email protected] 17 December 9, 2024 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ OC2 PIM model ▪ based on RSL hypermedia metamodel ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities

Beat Signer - Department of Computer Science - [email protected] 18 December 9, 2024 PimVis Document View Audrey Sanctorum

Beat Signer - Department of Computer Science - [email protected] 19 December 9, 2024 PimVis Setup

Beat Signer - Department of Computer Science - [email protected] 20 December 9, 2024 EUD of Cross-Device and IoT Applications ▪ Rapid prototyping platform for cross-device and IoT applications (eSPACE) ▪ End-user authoring ▪ customised distribution of user interface components ▪ mashup tool for digital and physical (IoT) components

Beat Signer - Department of Computer Science - [email protected] 21 December 9, 2024 Tangible Holograms (TangHo) Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 22 December 9, 2024 Collaborative TangHo Interaction

Beat Signer - Department of Computer Science - [email protected] 23 December 9, 2024 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ replaceable end effectors for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system is body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin

Beat Signer - Department of Computer Science - [email protected] 24 December 9, 2024 TangHo Interactions

Beat Signer - Department of Computer Science - [email protected] 25 December 9, 2024 Real-Time Cross-Media Data Exploration

Beat Signer - Department of Computer Science - [email protected] 26 December 9, 2024 Dynamic Data Physicalisation ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Data physicalisation design guidelines ▪ what are the just-perceptual differences that a physical variable can convey? ▪ Dynamic data physicalisation framework ▪ data physicalisation grammar ▪ data processing with device-independent output ▪ software drivers for different physicalisations (e.g.TangHo) ▪ New application domains Jacques Bertin, 1967

Beat Signer - Department of Computer Science - [email protected] 27 December 9, 2024 Dynamic Data Physicalisation Framework ...

Beat Signer - Department of Computer Science - [email protected] 28 December 9, 2024 Hypertext principles can play a major role in realising a general infrastructure for human- information interaction in mixed reality environments

Beat Signer - Department of Computer Science - [email protected] 29 December 9, 2024 General Infrastructure for Mixed Reality ▪ W3C WebXR Device API ▪ cross-platform compatibility ▪ web-based applications running in a browser ▪ W3C Web of Things(WoT) Thing Description (TD) ▪ metadata and interfaces of IoT devices & passive things ▪ Solid Project ▪ decentralised Solid Pods in addition to the Web ▪ collaborative authoring and sharing of information Solid

Beat Signer - Department of Computer Science - [email protected] 30 December 9, 2024 Course Summary 1. Introduction ▪ HCI and Interaction Design ▪ history of human-computer interaction - analogue computers, oNLine System (NLS) and desktop computers ▪ “evolution” of interfaces ▪ interface types - command-based, WIMP, pen, touch, speech, gesture, VR, AR, multimodal, … 2. HCI and Interaction Design ▪ what to design - who (user), how (task) and where (context) ▪ usability goals - effectiveness, efficiency, safety, utility, learnability, memorability ▪ user experience (UX) goals

Beat Signer - Department of Computer Science - [email protected] 31 December 9, 2024 Course Summary … HCI and Interaction Design … ▪ Double Diamond Design Process ▪ user-centred design principles ▪ Interaction Design Lifecycle Model - establishing requirements, designing alternatives, prototyping and evaluating 3. Requirements Analysis and Prototyping ▪ types of requirements - functional, data, environmental, user characteristics, usability, user experience ▪ requirements gathering techniques - interviews, focus groups, surveys, direct and indirect observation, … ▪ task description - scenarios and use cases ▪ prototyping - horizontal vs.vertical, throwaway vs.evolutionary

Beat Signer - Department of Computer Science - [email protected] 32 December 9, 2024 Course Summary … 4. Human Perception and Cognition ▪ human senses and modalities ▪ model of visual perception processing ▪ anatomy of the human eye - lens, pupil, retina (rods and cones), … ▪ brightness perception and illusions ▪ visual field and peripheral vision ▪ perception - biased by past, present and future - designing for biased perception ▪ Gestalt principles - law of proximity, law of similarity, law of continuity, law of closure, … ▪ long-term memory and immediate memory

Beat Signer - Department of Computer Science - [email protected] 33 December 9, 2024 Course Summary … 5. Design Guidelines and Models ▪ Shneiderman’s eight golden rules of interfaces design ▪ Nielsen’s ten usability heuristics for UI design ▪ visibility, simplicity and mapping ▪ guidelines for using colours ▪ inattentional/change blindness ▪ affordances ▪ Fitts’s law

Beat Signer - Department of Computer Science - [email protected] 34 December 9, 2024 Course Summary … 6. Evaluation Methods ▪ why, what, where and when? ▪ DECIDE evaluation framework - goals, questions, methods, practical issues, ethical issues, evaluate/present ▪ usability testing ▪ experimental design - between subjects, within-subjects and pair-wise design ▪ UEQ+ user experience questionnaire ▪ inspection - heuristic evaluation and walkthroughs ▪ analytics - Google analytics, A/B testing, … ▪ predictive models - GOMS model, keystroke level model and Fitts’s law

Beat Signer - Department of Computer Science - [email protected] 35 December 9, 2024 Course Summary … 7. HCI Research Methods ▪ seven types of HCI research contributions - empirical contributions, artefact contributions (systems research), methodological contributions, theoretical contributions, dataset contributions, literature survey contributions, opinion contributions ▪ descriptive, relational and experimental research - statistical significance (e.g. t tests) - correlation does not imply a causal relationship ▪ surveys, diaries and interviews ▪ grounded theory (inductive research) ▪ ethnography ▪ automatic data collection - measuring the human and online data ▪ human computation

Beat Signer - Department of Computer Science - [email protected] 36 December 9, 2024 Course Summary … 8. Use Cases and Course Review ▪ use cases for different HCI research contributions

Beat Signer - Department of Computer Science - [email protected] 37 December 9, 2024 Exam ▪ Written closed book exam ▪ covers content of lectures and exercises ▪ Date ▪ January 7, 13:00–16:00 ▪ Overall grade = written exam (50%) + assignment (50%) ▪ for the assignment students have some flexibility in distributing the grades (±2 points) - send us an email by January 5, 2025 with all group members in CC if you want to distribute some points ▪ note that the grade for the written exam as well as for the assign- ment have to be 8/20 or higher in order to pass the exam! ▪ Submission of the assignment via Canvas ▪ deadline: December 23, 24:00 (Brussels timezone)

Beat Signer - Department of Computer Science - [email protected] 38 December 9, 2024 Exam … ▪ The exam will cover all the content presented in the lectures as well as any additional information from the exercise sessions ▪ includes the videos shown in some of the lectures ▪ Make sure that you understand the basic concepts first ▪ however, we might ask questions at any level of detail to evaluate your knowledge ▪ Applying the theoretical knowledge to concrete use cases ▪ e.g.given a description of a specific problem in a given domain - decide about requirements gathering techniques - define different types of requirements

Beat Signer - Department of Computer Science - [email protected] 39 December 9, 2024 Exam … ▪ Applying the theoretical knowledge to concrete use cases … ▪ e.g. given a description/pictures of a specific UI - identify potential issues based on heuristics, Gestalt principles, Fitts’s law and other guidelines ▪ Describing (and potentially applying) certain concepts ▪ Interaction Design Lifecyle Model ▪ Double Diamond Design Process ▪ steps of the DECIDE evaluation framework ▪ usability goals ▪ …

Beat Signer - Department of Computer Science - [email protected] 40 December 9, 2024 Other Related Courses ▪ Specialisation: Data Management and Analytics (DAMA) ▪ Information Visualisation (MA) ▪ representation of data (encoding of value/relation) ▪ presentation of data (visualisation techniques) ▪ interaction ▪ dashboards and case studies ▪ Next Generation User Interfaces (MA) ▪ multimodal interaction - theoretical concepts, fusion and fission, ... ▪ interactive tabletops and surfaces ▪ pen-based interaction ▪ gesture-based interaction ▪ tangible interaction and mixed reality

Beat Signer - Department of Computer Science - [email protected] 41 December 9, 2024 Other Related Courses ... ▪ Advanced Topics in Big Data (MA) ▪ seminar about recent developments in Big Data ▪ scalable data management and big data analytics ▪ graph processing and distributed query processing ▪ human-data interaction and human-in-the loop data processing ▪ information visualisation and data physicalisation ▪ Bachelor's and a Master's thesis

Beat Signer - Department of Computer Science - [email protected] 42 December 9, 2024 References ▪ B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Second Edition, ISBN 978-3-8370-2713-6, August 2017 cover2.jpg B. Signer, Pen-based Interaction, Handbook of Human Computer Interaction, Major Reference Work, Springer Nature, 2024  https://beatsigner.com/publications/pen-based-interaction.pdf ▪ J.O. Wobbrock and J.A. Kientz, Research Contributions in Human-Computer Interaction, Interactions 23(3), April 2016 ▪ https://doi.org/10.1145/2907069 A blue and white sign with text Description automatically generated

Beat Signer - Department of Computer Science - [email protected] 43 December 9, 2024 References ... ▪ B. Signer and T.J. Curtin, Tangible Holograms: Towards Mobile Physical Augmentation of Virtual Objects, Technical Report WISE Lab, WISE-2017-01, March 2017 ▪ https://beatsigner.com/publications/signer_arXiv2017.pdf ▪ B. Signer, P. Ebrahimi, T.J. Curtin and Ahmed K.A. Abdullah, Towards a Framework for Dynamic Data Physicalisation, International Workshop Toward a Design Language for Data Physicalisation, Berlin, Germany, October 2018 ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf

Beat Signer - Department of Computer Science - [email protected] 44 December 9, 2024 References ... ▪ E.I. Reuss, B. Signer and M.C. Norrie, PowerPoint Multimedia Presentations in Computer Science Education: What do Users Need?, Proceedings of USAB 2008, 4th Symposium on Usability & HCI for Education and Work, Graz, Austria, November 2008 ▪ https://beatsigner.com/publications/reuss_USAB2008.pdf ▪ A. Sanctorum and B. Signer, eSPACE: Leveraging Theoretical Foundations for the End-User Development of Cross-Device and IoT Applications, ACM Transactions on Computer-Human Interaction (TOCHI), 2025 ▪ https://wise.vub.ac.be/index.php/publication/espace-leveraging- theoretical-foundations-end-user-development-cross-device-and-iot

Beat Signer - Department of Computer Science - [email protected] 45 December 9, 2024 References ... ▪ R. Roels and B. Signer, MindXpres: An Extensible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th International Conference on Web Information System Engineering, Thessaloniki, Greece, October, 2014 ▪ https://beatsigner.com/publications/roels_WISE2014.pdf ▪ R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016 ▪ https://beatsigner.com/publications/roels_CCIS2016.pdf

Beat Signer - Department of Computer Science - [email protected] 46 December 9, 2024 References ... ▪ M.C. Norrie, B. Signer, M. Grossniklaus, R. Belotti, C. Decurtins and N. Weibel, Context-aware Platform for Mobile Data Management, Wireless Networks (WINET), 13(6), Springer, December 2007 ▪ https://beatsigner.com/publications/norrie_WINET2007.pdf ▪ B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/signer_RCIS2019.pdf

Beat Signer - Department of Computer Science - [email protected] 47 December 9, 2024 References ... ▪ B. Signer, U. Kurmann and M.C. Norrie, iGesture: A General Gesture Recognition Framework, Proceedings of ICDAR 2007, 9th International Conference on Document Analysis and Recognition, Curitiba, Brazil, September 2007 ▪ https://beatsigner.com/publications/signer_ICDAR2007.pdf ▪ S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross-Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf

Beat Signer - Department of Computer Science - [email protected] 48 December 9, 2024 References ... ▪ B. Signer, As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction, Proceedings of HUMAN 2024, Poznań, Poland, September 2024 ▪ https://beatsigner.com/publications/signer_HUMAN2024.pdf ▪ MindXpres Project ▪ https://mindxpres.com ▪ Dynamic Data Physicalisation and Tangible Holograms ▪ https://wise.vub.ac.be/topic/dynamic-data-physicalisation ▪ https://wise.vub.ac.be/topic/tangible-holograms-tangho ▪ iGesture ▪ http://igesture.org

2 December 2005 The End Good Luck with the Exam!