User-centred design of a student facing learning analytics dashboard

Presentation at the Keele Annual Learning & Teaching Conference 2018
https://www.keele.ac.uk/lpdc/learningteaching/keelelearningandteachingconference/

Current Learning Analytics (LA) systems are primarily designed with University staff members as the target audience; very few are aimed at students, with almost none being developed with direct student involvement (de Quincey et al., 2016). Involving the user in the development process however has been shown to have a positive impact on the success of a system (Bano and Zowghi, 2013). Keele’s HEFCE funded “Learner Centred Design for Learning Analytics” project has therefore employed a variety of methods to engage students in the design/development of a LA dashboard which has then been implemented and piloted with 2 undergraduate modules. The design of the dashboard has been influenced by student feedback, using a novel approach of trying to understand the reasons why students want to study at university (e.g. career, self-development, attainment) and mapping their engagement and predicted outcomes to these motivations. Machine learning algorithms have been used to model the behaviour of student activity for last year’s cohort using KLE interactions, attendance and their final module grades. We then use this model to analyse the activity of current students and present their activity data back to them as scores, mapped to their chosen motivations in the LA dashboard. The end result is a dashboard personalised to each student allowing them to understand how their academic behaviour links to their motivations to study.

Transcript

1. User-centred design of a student facing learning analytics dashboard Dr Ed de Quincey & Chris Briggs School of Computing and Mathematics, Keele University
2. Dr Ed de Quincey @eddequincey Senior Lecturer in Computer Science, UG and PG Course Director School of Computing and Mathematics, Keele University Senior Fellow of the HEA instagram.com/eddequincey
3. Chris Briggs @confusedmatrix Research Software Engineer in Learning Analytics School of Computing and Mathematics, Keele University instagram.com/confusedmatrix/
4. Learning Analytics has been defined as a method for “deciphering trends and patterns from educational big data … to further the advancement of a personalized, supportive system of higher education.” (Johnson et al., 2013) Co-authorship network map of physicians publishing on hepatitis C (detail) Source: http://www.flickr.com/photos/speedoflife/8274993170/
5. 
   
6. WP1. KLE Data Review
    • Inconsistent formats • Reports take a long time to run • No way to bulk download all “raw” data • Student Overview for a Single Course is the most useful BUT have to run for each student Around 11 “reports” available 9 Course Reports, a Performance Dashboard and a Retention Centre
7. Blackboard Analytics
8. Apple Lisa PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS, USING USER CENTRIC QUERIES, FORMATS and METAPHORS
9. User-Centered Design Process Map http://www.usability.gov/how-to-and-tools/resources/ucd-map.html
10. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in matched funding from Keele) Title: Learner Centred Design for Learning Analytics This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the user experience, and therefore the ability to interpret and use the information. By sharing the data directly with students, using student-centred representations of their learning activity, this project aims to facilitate a common understanding of the learning experience between lecturers and students. Expanding on a successful teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and personalise the learning experience of cohorts with differing levels of technical experience and levels of digital literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele University and incorporate them into the delivery of relevant modules with the key aims of increasing engagement, making the VLE a more active space for learning and teaching and bridging the current gap between physical and digital spaces.
11. Outcomes for Students Students in lecture hall ©Jirka Matousek via Flickr • Access to personalised notifications and support e.g. highlighting/suggesting resources that have not been viewed. • Increased levels of engagement, in particular VLE usage. • Personalisation of cohort module delivery by the lecturer • Real-time feedback for students enabling them to judge their progress during a module using a different metric than current models of formative and summative feedback. • Direct involvement with the development of tools that support their learning.
12. 1 Being the best you can be/Effort (ability to maintain effort) 2 Build self-confidence 3 Career/Vocation/Job prospects 4 Industry 5 Giving yourself options 6 Grades/Marks/Qualifications 7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually 8 Mentoring/Family 9 Money 10 Part of a Professional community 11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy) 12 Sense of connectedness with others with similar goals/ Success as a group of peers 13 Social Prestige/Recognition Motivators for Studying in HE(Literature) Student Ambassadors
13. Laddering Sessions run by Student Ambassadors 10
14. Looks how they feel Shows how hard they are trying No point in spending £9,000 if you’re not going to try hard and do well A B ↑ ↑ “When thinking about what motivates you to study your degree, which of these do you prefer and why?”. “Why..?” “Why..?” Laddering
15. 14 Supporting family/Home life 15 Negative imagery 16 Rising above circumstances 17 Financial security 18 Controlling own destiny 19 Individuality 20 Opportunity to travel 21 Fear of wasting University opportunity Additional Motivators for Studying in HE Laddering Sessions run by Student Ambassadors 10
16. Revised Motivators (21 grouped into 9)
19. Timeline Formed basis of 6 Focus Groups with 20 students organised and run by Student Ambassadors
24. Professional Theme
25. Personal/Personified Theme
26. Personal/Personified Theme
27. Personal/Personified Theme
28. Personal/Personified Theme
29. Personal/Personified Theme
30. Both Themes
32. Mobile Optimised
33. Weekly Emails
34. 14 Features used to create model Total number of content accesses Total duration of content accesses Average duration of content accesses Number of days per week content is accessed Average length of time between content accesses Average group size of simultaneous content accesses Number of times specific content is accessed Number of clicks on content of a certain type Number of clicks grouped by parent folder Seen/unseen content Total number of lecture capture views Number of times specific lectures are viewed Grades Absences From KLE click data From lecture capture data From SCIMS data
35. Model to predict “Grade” using Decision Trees 5 = 1st 4 = 2:1 3 = 2:2 2 = 3rd 1 = Fail
36. Interventions Text based on decision trees
37. Lecturer Dashboard
38. Learning Analytics in the classroom? Students in lecture hall ©Jirka Matousek via Flickr
39. What motivates students to study at University? ESC-20030 Regional Landscapes
40. 2:2 38% of students 2:1 62% of students Predicted attainment ESC-20030 Regional Landscapes (as of week 10)
41. 0 5 10 15 20 25 30 35 40 45 1 2 3 4 5 6 7 8 9 10 # students week Content type clicks by week ESC-20030 Regional Landscapes Assessments Lectures Playback Seminars
42. Most popular content ESC-20030 Regional Landscapes week Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38 Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33 Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18 Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38 Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27 Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24 Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33
43. Least popular content ESC-20030 Regional Landscapes week Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments- ACL/Week 11 - Thermokarst Lakes 0 0 0 0 0 0 0 0 0 0 0 0 0 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos 0 0 0 0 0 1 0 0 0 0 0.1 1 1 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts 0 0 0 1 0 1 0 0 0 1 0.3 3 1 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers 0 0 0 2 0 2 0 0 0 0 0.4 4 2 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature 0 0 0 0 4 1 0 1 0 0 0.6 6 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River 0 0 0 4 2 0 0 0 0 0 0.6 6 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments- ACL/Week 11 - Proglacial lakes 2 0 0 0 0 0 0 0 0 0 7 0.7 7 7 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-9-Active-Layer-Azonal-Processes/Supplementary reading 0 0 0 3 0 2 0 3 0 0 0.8 8 3 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-6-7-Desert-Environments-RIW/Lecture 14 - Desert landsystems & climate change 0 0 0 0 0 0 0 4 3 1 0.8 8 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos 0 0 0 8 0 1 0 0 0 0 0.9 9 8
44. Think Aloud Sessions run by Student Ambassadors and Chris 10
45. V2
46. V2
47. V2
48. V2
49. V2
50. Comparison with peers Next Stage
51. 2:2 38% of students 2:1 62% of students Did it work? 94 students in Stage 1 Next Stage
52. Dr Ed de Quincey e.de quincey@keele.ac.uk Chris Briggs c.briggs@keele.ac.uk KEELE UNIVERSITY SCHOOL OF COMPUTING MOTIVATION METRICS V6 ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK CEO