2 December 2005 Next Generation User Interfaces Introduction Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2 February 10, 2025 Course Organisation ▪ Prof. Beat Signer Vrije Universiteit Brussel PL9.3.60 (Pleinlaan 9) +32 2 629 1239 bsigner@vub.be wise.vub.ac.be/beat-signer ▪ Maxim Van de Wynckel Vrije Universiteit Brussel PL9.3.58 (Pleinlaan 9) +32 2 629 3487 mvdewync@vub.be wise.vub.ac.be/maxim-van-de-wynckel

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3 February 10, 2025 Prerequisites ▪ Note that this is an advanced Master's level course and the following previous knowledge is required ▪ good programming skills ▪ We further expect that you have some basic knowledge about Human-Computer Interaction ▪ otherwise, it is best to follow the 'Human Computer Interaction' (1023841ANR) Bachelor course as a "repair course" ▪ It is not impossible to follow the course without these prerequisites, but in this case, you should not complain about the potential additional workload!

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4 February 10, 2025 Course Goals ▪ After attending the course on Next Generation User Interfaces, the student has an understanding of the interaction principles introduced by new devices such as smartphones, multi-touch tables or gesture-based inter- faces as well as the theoretical background behind these interaction principles. The student is able to reflect on the qualities and shortcomings of different interaction styles, while placing the user at the core of the interface design process. ▪ The student can apply the theoretical background and interaction principles discussed in the course and create interfaces that go beyond the classical WIMP metaphor.

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5 February 10, 2025 Course Goals … ▪ The course should give students the ability to under- stand the possibilities and limitations of next generation user interfaces and to recognise variations of such inter- faces. They should further be able to understand and evaluate new developments and technologies in fields related to the course and have the skills to independently study and master these new technologies.

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6 February 10, 2025 Exercises ▪ The course content is further investigated in a number of exercise sessions ▪ exercise sessions might also be helpful for the assignment ▪ assistant: Maxim Van de Wynckel - Monday 16:00–18:00 (exact slots and locations on the lecture schedule) ▪ Lab Session ▪ work on the assignment and get feedback from the assistant - Monday, 12.5.2024, 16:00–18:00 in I.1.04 - you can also meet and get feedback (appointment via email) ▪ Additional content might be covered in exercise sessions ▪ strongly recommended to attend all exercise sessions! ▪ exam covers content of lectures and exercises

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7 February 10, 2025 Course Material ▪ All material will be available on Canvas ▪ lecture slides, exercises, research papers, tutorials, ... ▪ Make sure that you are subscribed to the Next Generation User Interfaces course on Canvas ▪ https://canvas.vub.be/courses/37403 ▪ Handouts normally available on Canvas at least the day before the lecture ▪ Similar information is also available on the WISE website (including links to last year's lecture slides) ▪ https://wise.vub.ac.be/course/next-generation-user-interfaces

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8 February 10, 2025 Lecture Schedule Exercise 1: Phidgets, Arduino and ESP32 23 24 25 26 Lecture 2: Information Architectures Lecture 3: Multimodal Interaction Initial Project Presentations Lecture 4: Pen-based Interaction Short Progress Presentations I.1.04 I.1.04 I.1.04 I.1.04 E.1.04 I.1.04 I.1.04 27 28 Lecture 1: Introduction 22 Information about Assignment and Group Formation I.1.04 Exercise 2: 3D Printing I.1.04 I.1.04 Lecture 5: Interactive Tabletops and Surfaces Exercise 3: Evaluation I.1.04 I.1.04 Lecture 6: Gesture-based Interaction Lecture 7: Tangible, Embedded and Embodied Interaction Exercise 4: Positioning Systems and Location Awareness I.1.04 E.1.04

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9 February 10, 2025 Lecture Schedule … No Lecture No Exercise 30 31 32 33 34 35 No Lecture Lecture 10: Implicit and Cross-Device Interaction Exercise 6: Discussion of Papers Lecture 11: Human-AI Interaction No Exercise 29 Lab Session Lecture 8: Virtual and Augmented Reality I.1.04 I.1.04 I.1.04 I.1.04 I.1.04 Exercise 5: WebXR E.1.04 ⋮ 36 TBD Final Project Presentations Lecture 12: Course Review I.1.04 Lecture 9: Data Physicalisation I.1.04 Interim Project Presentations (Working Prototype) I.1.04

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10 February 10, 2025 Assignment ▪ Next generation user interface ▪ realisation of a user interface for the domain of your choice (e.g.information management, analysis or gaming) - various presentations and reports - evaluated based on creativity, exploration of technologies, documentation, presentations, requirements analysis and evaluation, source code, … ▪ Assignment handed out later this week ▪ group project with 4 students per group - send an email with the 4 group members and your team’s name to Maxim Van de Wynckel by Friday, February 14 (mvdewync@vub.be) - final presentation (week of May 19), final report and code (May 25) ▪ assignment counts for 60% of the final grade

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11 February 10, 2025 Available Hardware Phidgets Raspberry Pi Smartphones and Tablets Pico Projectors Kinect M5 Stack

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12 February 10, 2025 Available Hardware … Web Cams Leap Motion Hand Tracking Digital Pens Multi-Touch Tabletop Myo Gesture Control Armband

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13 February 10, 2025 Available Hardware … HTC VIVE Pro 2 Ultimaker S5 3D Printer Microsoft HoloLens Meta Quest 3

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14 February 10, 2025 Available Hardware … ▪ You might further make use of the VUB FabLab ▪ laser cutter, CNC machines, 3D printers, …

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15 February 10, 2025 Exam ▪ Oral exam in English ▪ covers content of lectures and exercises ▪ counts 40% for the overall grade ▪ 5 mins questions about the assignment ▪ 15 mins questions about the course content (no preparation time) ▪ Overall grade = oral exam (40%) + assignment (60%) ▪ assignment is composed out of two grades - overall grade for project where students have some flexibility in distributing the grades (±2 points) (70%) - your contribution/knowledge to the project as checked in oral exam (30%) ▪ note that the grade for the oral exam as well as for the assign- ment have to be 8/20 or higher in order to pass the exam!

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16 February 10, 2025 Course Outline 1. Introduction ▪ interface types ▪ interaction design ▪ requirements analysis, prototyping and evaluation 2. Information Architectures ▪ personal information management (PIM) ▪ cross-media information systems 3. Multimodal Interaction ▪ human senses ▪ multimodal fusion and fission

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17 February 10, 2025 Course Outline … 4. Pen-based Interaction ▪ affordances of pen and paper ▪ digital pen and paper solutions ▪ pen and touch 5. Interactive Tabletops and Surfaces ▪ frameworks and technologies ▪ applications 6. Gesture-based Interaction ▪ single and multi-touch gestures ▪ offline vs. online gestures ▪ mid-air gestures

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18 February 10, 2025 Course Outline … 7. Tangible, Embedded and Embodied Interaction ▪ from tangible bits to radical atoms ▪ characteristics of tangible interfaces 8. Virtual and Augmented Reality ▪ technologies ▪ virtual and augmented reality applications 9. Data Physicalisation ▪ physical variables ▪ data physicalisation frameworks ▪ tangible holograms (TangHo)

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19 February 10, 2025 Course Outline … 10.Implicit and Cross-Device Interaction ▪ context-aware implicit human-computer interaction (HCII) ▪ cross-device and distributed user interfaces 11.Human-AI Interaction ▪ issues and guidelines ▪ use cases 12.Course Review & Final Project Presentations

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20 February 10, 2025 Video: Microsoft Office Labs Vision 2019

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21 February 10, 2025 "Evolution" of Interfaces date January 1984 January 2024 +40 years price $2500 $4568 × 1.83 CPU 68000 Motorola 8 MHz 0.7 MIPS 12(30)-core M2 Max 3.7 GHz 3.6 TFLOPS × 463 “× 5142000” memory 128 kB 32 GB × 264000 storage 400 kB floppy drive 1 TB SSD drive × 2684000 monitor 9" black and white 512 x 342 68 dpi 27" colour 5120 x 2880 218 dpi × 3 × 84 × 3.2 devices mouse keyboard mouse keyboard same same GUI desktop WIMP desktop WIMP same [partly based on Beaudouin-Lafon 2004] original Macintosh Mac Studio comparison

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22 February 10, 2025 Recent Changes and Opportunities in HCI ▪ Advances in graphical interfaces, speech, gesture and handwriting recognition ▪ Emergence of the Internet, cell phones, wireless networks, sensor technologies as well as large and small screens ▪ innovative interaction with digital information and services ▪ combining the physical and digital in new ways - mixed reality, cross-media spaces, tangible interfaces, wearable computing, ... ▪ collaborative interfaces with social interaction

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23 February 10, 2025 Interface Types ▪ Command-based ▪ WIMP and GUI ▪ Multimedia ▪ Virtual reality ▪ Web ▪ Consumer electronics ▪ Mobile ▪ Speech (Voice) ▪ Pen ▪ Touch ▪ Gesture ▪ Haptic ▪ Multimodal ▪ Shareable ▪ Tangible ▪ Augmented and mixed reality ▪ Wearable ▪ Robots ▪ Brain-computer ▪ Smart interfaces

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24 February 10, 2025 Towards Natural User Interfaces? ▪ Natural User Interfaces (NUI) enable a user to interact with a computer in the same way as they interact with the physical world ▪ use of speech, touch, mid-air gestures, face recognition etc. ▪ How natural are natural user interfaces? ▪ is it more natural to say “open” rather than to flick a switch to open door? ▪ is it more natural to raise both arms rather than to press a button on a remote control to change a TV channel? ▪ NUIs are effective for certain domains but might not replace existing user interfaces

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25 February 10, 2025 Which Interface Should We Use? ▪ In the last few years there is a significant increase in the number of user interface types ▪ How to decide which interface is preferable for a given task or activity? ▪ multimedia vs.tangible interface for learning ▪ speech vs.command-based interface ▪ multimodal vs.monomodal interface ▪ wearable vs.mobile interface ▪ virtual reality vs.augmented reality ▪ Many of these questions are currently being researched ▪ this course will provide you some more insights about the design process and different types of interfaces

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26 February 10, 2025 What to Design ▪ Who (user) is going to use an interactive product, how (task) is it going to be used and where (context)? ▪ Have to understand the activities people are doing while using the products ▪ How to optimise a user’s interactions with a system, environment or product to support and extend their activities in effective, useful and usable ways? ▪ take into account what people are good and bad at (human technology teamwork) ▪ what might help people with the way they currently do things ▪ listen to what people want and get them involved in the design ▪ use established user-based techniques during the design process

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27 February 10, 2025 Interaction Design Lifecycle Model Establishing requirements Designing alternatives Prototyping Evaluating Final product

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28 February 10, 2025 Interaction Design Process The interaction design process involves four basic activities 1. Establishing requirements ▪ know target users and the required support 2. Designing alternatives that meet the requirements ▪ conceptual design ▪ physical design 3. Prototyping the alternative designs in order that they can be communicated and assessed ▪ paper-based prototypes ▪ software prototypes

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29 February 10, 2025 Interaction Design Process 4. Evaluating ▪ determining the usability and acceptability ▪ observing or talking to users ▪ interviews or questionnaires

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30 February 10, 2025 Designing Alternatives ▪ Should not just stick to a solution that is “good enough” but also consider alternative solutions ▪ Creativity of the designer plays an important role ▪ discussion with other designers ▪ studying other designs - pay attention to copyrights and patent laws ▪ solve new problems based on knowledge gained from solving previous similar problems ▪ creativity workshops and brainstorming sessions “The best way to get a good idea, is to get lots of ideas.” Linus Pauling

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31 February 10, 2025 Types of Requirements ▪ Functional requirements ▪ what should the product do (fundamental) ▪ Data requirements ▪ type of data ▪ amount of data ▪ data accuracy ▪ … ▪ Environmental requirements (context of use) ▪ physical environment - lighting, noise, movement, dust, … ▪ social environment - synchronous or asynchronous sharing of data, co-located or distributed, …

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32 February 10, 2025 Types of Requirements … ▪ Environmental requirements (context of use) … ▪ organisational environment - user support, resources for training, how hierarchical is the management, … ▪ technical environment - technologies the product will run on, compatibility, technological limitations, … ▪ User characteristics ▪ key attributes of intended user group - abilities and skills - nationality and educational background - preferences - physical or mental disabilities - level of expertise (novice, expert, casual user, frequent user, …) ▪ user profile consists of a collection of attributes for a typical user

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33 February 10, 2025 Types of Requirements … ▪ Usability goals ▪ define measures for agreed usability goals - objective measure of a user’s performance - effectiveness, efficiency, safety, utility, learnability, memorability ▪ User experience goals ▪ Different forms of data gathering for requirements ▪ interviews, focus groups, questionnaires, direct observation, indirect observation, studying documentation or researching similar products

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34 February 10, 2025 User-Centred Design 1. Early focus on users and tasks ▪ who will be the users ▪ study the characteristics of users 2. Empirical measurement ▪ identify specific usability and user experience goals - helps to choose between alternative designs - can be used to check the progress ▪ sketches, description in natural language and prototypes help to observe and analyse the performance and reactions of users 3. Iterative design ▪ problems identified in user testing are fixed and evaluated in a next iteration - iteration is particularly important when trying to innovate

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35 February 10, 2025 Prototyping ▪ “It is often said that users can’t tell you what they want, but when they see something and get to use it, they soon know what they don’t want.” ▪ A prototype is a manifestation of design that allows stakeholders to interact with it and explore its suitability ▪ paper-based storyboard or outline of a screen ▪ electronic picture or video simulation of a task ▪ 3D cardboard mock-up or object printed with a 3D printer ▪ piece of software ▪ … ▪ Prototypes help to choose between design alternatives ▪ Building a prototype encourages reflection in design

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36 February 10, 2025 Low-Fidelity Prototyping ▪ Sketching ▪ hand-drawn sketches containing computer components, icons, dialogue boxes, … ▪ Prototyping with index cards ▪ each card represents one screen or one element of a task ▪ Wizard of Oz experiment ▪ assumes that one has a software-based prototype ▪ human operator (wizard) simulates the software’s response to the user ▪ PowerPoint ▪ balances the provisionality of paper with the polished appearance of software prototypes - characteristics of low and high fidelity

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37 February 10, 2025 Evaluation ▪ Evaluation is an integral part of the design process ▪ usability of the system ▪ user experience ▪ Observe participants and measure their perfor- mance ▪ usability testing ▪ experiments ▪ field studies

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38 February 10, 2025 Why, What, Where and When to Evaluate ▪ Why evaluate ▪ do we fulfill user requirements? ▪ ensure that users can use the product and they like it ▪ What to evaluate ▪ conceptual models ▪ early low-fidelity prototypes or high-fidelity prototypes ▪ individual function, complete workflow, aesthetic design, safety, … “User experience encompasses all aspects of the end-user’s interaction … the first requirement for an exemplary user experience is to meet the exact needs of the customer, without fuss or bother. Next come simplicity and elegance, which produces products that are a joy to own, a joy to use.” Nielsen Norman Group

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39 February 10, 2025 Why, What, Where and When to Evaluate … ▪ Where to evaluate ▪ laboratory ▪ natural setting (in-the-wild studies) - better for user experience ▪ living labs ▪ When to evaluate ▪ formative evaluations - throughout the design process - what and how to redesign? ▪ summative evaluations - assess the final product - how well did we do? Aware Home, Georgia Tech

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40 February 10, 2025 UEQ+ User Experience Questionnaire ▪ Tool to build customised UX questionnaire ▪ modular extension of UEQ ▪ customised selection of UX scales ▪ Word templates with questions - more than 20 languages - answers on a 7-point Likert scale ▪ UEQ+ Data Analysis Tool ▪ Excel document creating graphics etc.

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41 February 10, 2025 UEQ+ Available Scales

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42 February 10, 2025 Online Survey Tools Qualtrics

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43 February 10, 2025 Assignment ▪ Focus first on problem and user needs ▪ different interface types and hardware technologies should only be considered when designing alternatives ▪ Consider the use of user-centred design ▪ requirements, prototyping as well as evaluation ▪ Iterative design process with rapid low-fidelity prototyping ▪ Do not forget to evaluate your solution!

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44 February 10, 2025 References ▪ Interaction Design: Beyond Human-Computer Interaction, Yvonne Rogers, Helen Sharp and Jenny Preece, Wiley (6th edition), April 2023 ISBN-13: 978-1119901099 ▪ Designing with the Mind in Mind: Simple Guide to Understanding User Interface Design Guidelines, Jeff Johnson, Morgan Kaufmann (3rd edition), November 2020, 978-0128182024 ▪ Human-Computer Interaction, Alan Dix, Janet E. Finlay, Gregory D. Abowd and Russell Beale, Prentice Hall (3rd edition), December 2003 ISBN-13: 978-0130461094

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45 February 10, 2025 References ... ▪ Research Methods in Human-Computer Inter- action, Jonathan Lazar, Jinjuan Heidi Feng and Harry Hochheiser, Morgan Kaufmann (2nd edition), May 2019, ISBN-13: 978-0128053904 ▪ Designing for Interaction: Creating Innovative Applications and Devices, Dan Saffer, New Riders (2nd edition), August 2009, ISBN-13: 978-0321643391 ▪ The Design of Everyday Things, Don Norman, Basic Books (revised and expanded edition), November 2013, ISBN-13: 978-0465050659

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46 February 10, 2025 References ... ▪ Human Computer Interaction (1023841ANR) course ▪ https://wise.vub.ac.be/index.php/course/human-computer- interaction ▪ Microsoft Office Labs Vision 2019 ▪ https://www.youtube.com/watch?v=Zp-_oUwdSeY ▪ MAXQDA ▪ https://www.maxqda.com ▪ https://softweb.vub.be ▪ Qualtrics XM ▪ https://www.qualtrics.com ▪ https://softweb.vub.be

Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47 February 10, 2025 References ... ▪ UEQ+: A Modular Extension of the User Experience Questionnaire ▪ https://ueqplus.ueq-research.org

2 December 2005 Next Lecture Information Architectures