2 December 2005 Human-Computer Interaction HCI Research Methods Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com

Beat Signer - Department of Computer Science - [email protected] 2 November 4, 2024 Human-Computer Interaction ▪ Human-Computer Interaction is a multidisciplinary field ▪ Computer Science ▪ Design ▪ Cognitive Science ▪ Psychology ▪ … Human-computer interaction is a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them. ACM SIGCHI Curricula for Human-Computer Interaction

Beat Signer - Department of Computer Science - [email protected] 3 November 4, 2024 HCI Research Contributions ▪ Empirical contributions ▪ collected quantitative or qualitative data ▪ Artefact contributions (systems research) ▪ development and evaluation of new artefacts ▪ interfaces, toolkits, mock-ups, … ▪ Methodological contributions ▪ new method or modification of existing method, new metrics ▪ Theoretical contributions ▪ concepts and models as vehicles for thought ▪ frameworks, design spaces, conceptual models ▪ Dataset contributions ▪ corpus for the benefit of the research community

Beat Signer - Department of Computer Science - [email protected] 4 November 4, 2024 HCI Research Contributions … ▪ Literature survey contributions ▪ review and synthesis of work done in a specific area ▪ Opinion contributions ▪ trying to persuade the readers to change their minds ▪ HCI research as well as HCI research methods have changed over time ▪ web interfaces, user-generated content, touch screens, collaboration, … ▪ usability engineering, eye tracking, crowdsourcing, …

Beat Signer - Department of Computer Science - [email protected] 5 November 4, 2024 HCI Research ▪ Most HCI researchers must collect their own data ▪ in other domains (e.g.sociology or economics) data often collected by large entities or government agencies ▪ Studies with many participants (big data) can help us determine correlations ▪ Smaller studies might provide us with a deeper understanding of the meaning of data ▪ Longitudinal studies in HCI are rare ▪ technology and tools change rapidly → often does not make sense to compare over multiple years or decades ▪ often seen as a shortcoming

Beat Signer - Department of Computer Science - [email protected] 6 November 4, 2024 Measurement ▪ What to measure? ▪ In early days of HCI (early 1980s) mainly human perfor- mance for individual tasks (micro-HCI) measured in labs ▪ task correctness ▪ time performance ▪ error rate ▪ time to learn ▪ user satisfaction ▪ Broader level (macro-HCI) such as motivation, collabo- ration or trust not easy to measure with existing metrics ▪ use of multimethod approaches - case studies, observations, interviews, data logging, …

Beat Signer - Department of Computer Science - [email protected] 7 November 4, 2024 Different Types of Research Type of Research Focus General Claims Typical Methods Descriptive Describe a situation or a set of events X is happening Observations, field studies, focus groups, interviews Relational Identify relations between multiple variables (correlation) X is related to Y Observations, field studies surveys Experimental Identify causes of a situation or a set of events (causality) X is responsible for Y Controlled experiments

Beat Signer - Department of Computer Science - [email protected] 8 November 4, 2024 Experimental Research ▪ Null hypothesis and alternative hypothesis (see lecture 6) ▪ Independent variables ▪ technology, e.g.typing vs.voice input, mouse vs.joystick, … ▪ different types of design, e.g.pull-down vs.pop-up menu, font size, colours, … ▪ user related, e.g.age, gender, culture, computer experience, … ▪ context, e.g.noise, lighting, temperature, … Data Hypothesis Study

Beat Signer - Department of Computer Science - [email protected] 9 November 4, 2024 Experimental Research … ▪ Dependent variables ▪ efficiency - time to complete task, speed (e.g. words per minute) ▪ accuracy - error rate ▪ subjective satisfaction - normally collected via Likert scale ratings (e.g. via questionnaires) ▪ ease of learning or memorability and retention rate - important for adoption of information technology ▪ physical or cognitive demand - how long can we interact without significant fatigue

Beat Signer - Department of Computer Science - [email protected] 10 November 4, 2024 Experimental Procedure 1. Identify a research hypothesis ▪ how many independent variables? ▪ how many values for each independent variable? 2. Specify experimental design (see lecture 6) ▪ between-subjects design, within-subjects design and pair-wise design 3. Run a pilot study ▪ test design and study instruments 4. Recruit participants 5. Run the data collection sessions 6. Analyse the data (statistical analysis) 7. Report the results

Beat Signer - Department of Computer Science - [email protected] 11 November 4, 2024 Statistical Analysis ▪ Preprocessing of data ▪ cleaning up data and fixing errors if possible - in anonymous studies we might not be able to contact participants for clarification - if error cannot be fixed, we have to remove the data item and treat it as missing value in the analysis ▪ coding data - e.g. gender: “male” → 1 and “female” → 0 - e.g. degree: “high school” → 1, “university college” → 2 and “university” → 3 ▪ organising data - bring data in layout /format for specific data processing software (e.g.SPSS (Statistical Package for the Social Sciences, by IBM), available via the VUB software shop)

Beat Signer - Department of Computer Science - [email protected] 12 November 4, 2024 Descriptive Statistical Tests ▪ Distribution of data points, ▪ means, medians, variances, standard deviations and ranges ▪ Comparing means ▪ cannot just compare the means but also have to compute some statistical significance tests (e.g.t tests or ANOVA) ▪ Example ▪ Null hypothesis: “There is no significant difference in the task completion time between individuals who use the word-prediction software and those who do not use the software” ▪ Collect some data in an experiment where users use two different versions of the software (with and without word-prediction)

Beat Signer - Department of Computer Science - [email protected] 13 November 4, 2024 Descriptive Statistical Tests … ▪ Example … ▪ independent-samples t test for data on the next slide results in a t value of 2.169 - higher than the t value (2.131) for the specific degree of freedom (df =15) at the 95% confidence interval - “An independent-samples t test suggests that there is significant difference in the task completion time between the group who used the standard word- processing software and the group who used word-processing software with word prediction functions (t(15) = 2.169, p < 0.05).”

Beat Signer - Department of Computer Science - [email protected] 14 November 4, 2024 Data for Independent-Samples t Test

Beat Signer - Department of Computer Science - [email protected] 15 November 4, 2024 Data for Paired-Samples t Test

Beat Signer - Department of Computer Science - [email protected] 16 November 4, 2024 Identifying Relationships ▪ Identify whether there is a relationship (correlation) between various variables ▪ compute Pearson’s product moment correlation coefficient - SPSS will compute a correlation matrix between all variables ▪ Pearson’s r value ranges from -1 to 1 - r=-1.0 means that there is a perfect negative linear relationship between two variables → specific increase in one variable perfectly predicts decrease in the other variable - r=1.0 means that there is a perfect positive linear relationship between two variable → specific increase in one variable perfectly predicts increase in the other variable - r=0 means that there is no linear relationship between the two variables ▪ Note that correlation does not imply a causal relationship ▪ might also be based on hidden (“intervening”) variable

Beat Signer - Department of Computer Science - [email protected] 17 November 4, 2024 Limitations of Experimental Research ▪ Requires well-defined testable hypotheses with a limited number of independent and dependant variables ▪ many problems not clearly defined or involve a large number of potential variables ▪ Need strict control of factors that might influence the dependent variables ▪ not always possible to control these factors ▪ Lab experiments might not be a good representation of a user’s typical interaction behaviour ▪ study participants might behave differently in lab-based experiments - stress of being observed, different environment, …

Beat Signer - Department of Computer Science - [email protected] 18 November 4, 2024 Questionnaires/Surveys ▪ Closed as well as open questions (see lecture 3) ▪ Pilot test to ensure validity and reliability ▪ ensure that questions are clear, unambiguous and unbiased ▪ Need higher response rates than surveys in interaction design to get statistically relevant results ▪ Data analysis ▪ clean the data ▪ statistical analysis of closed questions - often just descriptive statistics (percentages etc.) - inferential statistics by understanding the relationships between variables ▪ Might be combined with other research methods ▪ e.g. interviews, focus groups or diaries

Beat Signer - Department of Computer Science - [email protected] 19 November 4, 2024 Diaries ▪ Individual maintaining regular recordings ▪ fills the gap between observations in naturalistic settings and lab studies ▪ Advantages ▪ good for understanding the “why” of user interaction with technology ▪ useful for technology that is used on the go - difficult to do in lab setting or via observation ▪ Disadvantages ▪ participants might not record a sufficient number of entries ▪ data analysis (mix of qualitative and quantitative) might take a long time

Beat Signer - Department of Computer Science - [email protected] 20 November 4, 2024 Interviews ▪ Open-ended (unstructured), semi-structured or structured interviews (see lecture 3) ▪ Content analysis ▪ usually relies on qualitative methods for coding data - try to find common structures and themes from qualitative data - frequency of terms etc. (e.g. use MAXQDA) ▪ if validity is a particular concern, then multiple researchers should independently analyse the interviews - individual analysts might have some bias ▪ Might best be conducted as complements to other data collection approaches

Beat Signer - Department of Computer Science - [email protected] 21 November 4, 2024 Grounded Theory ▪ For new topics with limited literature to build on ▪ no established theories to develop coding categories ▪ Use emergent coding approach based on the notion of grounded theory ▪ qualitative research method that seeks to develop theory that is “grounded in data systematically gathered and analysed” ▪ might have multiple rounds of data collection and analysis to allow the underlying theory to fully emerge Data Theory Study

Beat Signer - Department of Computer Science - [email protected] 22 November 4, 2024 Ethnography ▪ Combination of observation, interviews and participation ▪ has its roots in anthropological studies ▪ Examples ▪ home settings - country, culture and religion have a great impact on how technology is used in homes ▪ work settings - e.g. London underground control centre (video ethnography) - non-office-based settings (e.g. vineyards and use of sensors) ▪ educational settings - use of technologies in learning activities ▪ Virtual ethnography ▪ use of web cams or videos

Beat Signer - Department of Computer Science - [email protected] 23 November 4, 2024 Ethnography … ▪ Very useful in understanding the context of technology usage ▪ Often used as a first step to understand a group of users, their problems, challenges, norms and processes ▪ eventual goal of building some type of technology for them or with them ▪ More recently also ethnographic investigations of ubiquitous computing environments ▪ e.g.navigation needs of firefighters to find their way out of hazardous, smoke-filled environments

Beat Signer - Department of Computer Science - [email protected] 24 November 4, 2024 Automated Data Collection Methods ▪ Website access log analysers ▪ Activity logging software ▪ Custom software with reporting features

Beat Signer - Department of Computer Science - [email protected] 25 November 4, 2024 Measuring the Human ▪ Eye tracking ▪ Muscular and skeletal position sensing ▪ Wii remote ▪ smartwatches ▪ Microsoft Kinect ▪ Motion tracking ▪ e.g.for interaction with large wall-sized displays ▪ Physiological data ▪ heart rate and blood volume/pressure ▪ galvanic skin response ▪ respiration ▪ brain activity (EEG)

Beat Signer - Department of Computer Science - [email protected] 26 November 4, 2024 Online HCI Research ▪ Observational online studies ▪ usability and think-aloud studies ▪ remote screen sharing ▪ web cam live feed (audio/video) ▪ remote keyboard/mouse control ▪ potential recording of different streams ▪ A/B testing for Internet business

Beat Signer - Department of Computer Science - [email protected] 27 November 4, 2024 Human Computation ▪ Humans are often better in tasks requiring detailed interpretation of complex input ▪ Computers can be used to augment humans (see NLS project in lecture 1), but humans can also be used to augment computers → human computation ▪ Task that is hard for a computer but easy for humans ▪ ask multiple humans to complete small pieces of the task ▪ e.g.reCAPTCHA

Beat Signer - Department of Computer Science - [email protected] 28 November 4, 2024 Conducting Human Computation Studies ▪ Use crowdsourcing ser- vices for large pool of inexpensive study partici- pants ▪ e.g.Amazon’s Mechanical Turk with specific APIs ▪ potentially less bias and increased validity - participants do not directly interact with researchers

Beat Signer - Department of Computer Science - [email protected] 29 November 4, 2024 Main HCI Conferences ▪ CHI: ACM Conference on Human Factors in Computing Systems ▪ https://dl.acm.org/conference/chi/ ▪ UIST: ACM Symposium on User Interface Software and Technology ▪ https://dl.acm.org/conference/uist/ ▪ CSCW: ACM Conference on Computer-supported Cooperative Work and Social Computing ▪ https://dl.acm.org/conference/cscw/ ▪ IUI: Annual Conference on Intelligent User Interfaces ▪ https://dl.acm.org/conference/iui/

Beat Signer - Department of Computer Science - [email protected] 30 November 4, 2024 Main HCI Conferences … ▪ TEI: International Conference on Tangible, Embedded, and Embodied Interaction ▪ https://dl.acm.org/conference/tei/ ▪ EICS: International Conference on Engineering Interactive Computing Systems ▪ https://dl.acm.org/conference/eics/ ▪ DIS: ACM Conference on Designing Interactive Systems ▪ https://dl.acm.org/conference/dis/ ▪ ICMI: International Conference on Multimodal Interaction ▪ https://dl.acm.org/conference/icmi/

Beat Signer - Department of Computer Science - [email protected] 31 November 4, 2024 Exercise 6 ▪ Heuristic Evaluation

Beat Signer - Department of Computer Science - [email protected] 32 November 4, 2024 Further Reading ▪ Major parts of this lecture are based on the book Research Methods in Human-Computer Interaction ▪ chapter 1–14

Beat Signer - Department of Computer Science - [email protected] 33 November 4, 2024 References ▪ Research Methods in Human-Computer Interaction, Jonathan Lazar, Jinjuan Heidi Feng and Harry Hochheiser, Morgan Kaufmann (2nd edition), May 2019, ISBN-13: 978-0128053904 ▪ Technology in Action (Learning in Doing: Social, Cognitive and Computational Perspectives), Christian Heath and Paul Luff, Cambridge Univer- sity Press, June 2000, ISBN-13: 978-0521568692 ▪ ▪ SPSS (Statistical Package for the Social Sciences) ▪ https://www.ibm.com/spss ▪ available via VUB Software Webshop

Beat Signer - Department of Computer Science - [email protected] 34 November 4, 2024 References … Statistics for HCI: Making Sense of Quantitative Data, A. Dix, Morgan & Claypool, April 2020, ISB-13: 978-1681737430 ▪ J. O. Wobbrock and J. A. Kientz, Research Contributions in Human-Computer Interaction, Interactions 23(3), April 2016 ▪ https://doi.org/10.1145/2907069

2 December 2005 Next Lecture Use Cases and Course Review