Upgrade to Pro — share decks privately, control downloads, hide ads and more …

User-centred design of a student facing learning analytics dashboard

User-centred design of a student facing learning analytics dashboard

Presentation at the Keele Annual Learning & Teaching Conference 2018
https://www.keele.ac.uk/lpdc/learningteaching/keelelearningandteachingconference/

Current Learning Analytics (LA) systems are primarily designed with University staff members as the target audience; very few are aimed at students, with almost none being developed with direct student involvement (de Quincey et al., 2016). Involving the user in the development process however has been shown to have a positive impact on the success of a system (Bano and Zowghi, 2013). Keele’s HEFCE funded “Learner Centred Design for Learning Analytics” project has therefore employed a variety of methods to engage students in the design/development of a LA dashboard which has then been implemented and piloted with 2 undergraduate modules. The design of the dashboard has been influenced by student feedback, using a novel approach of trying to understand the reasons why students want to study at university (e.g. career, self-development, attainment) and mapping their engagement and predicted outcomes to these motivations. Machine learning algorithms have been used to model the behaviour of student activity for last year’s cohort using KLE interactions, attendance and their final module grades. We then use this model to analyse the activity of current students and present their activity data back to them as scores, mapped to their chosen motivations in the LA dashboard. The end result is a dashboard personalised to each student allowing them to understand how their academic behaviour links to their motivations to study.

Dafb8db748341892bda61117ecd16a43?s=128

Ed de Quincey

January 16, 2018
Tweet

More Decks by Ed de Quincey

Other Decks in Technology

Transcript

  1. User-centred design of a student facing learning analytics dashboard Dr

    Ed de Quincey & Chris Briggs School of Computing and Mathematics, Keele University
  2. Dr Ed de Quincey @eddequincey Senior Lecturer in Computer Science,

    UG and PG Course Director School of Computing and Mathematics, Keele University Senior Fellow of the HEA instagram.com/eddequincey
  3. Chris Briggs @confusedmatrix Research Software Engineer in Learning Analytics School

    of Computing and Mathematics, Keele University instagram.com/confusedmatrix/
  4. Learning Analytics has been defined as a method for “deciphering

    trends and patterns from educational big data … to further the advancement of a personalized, supportive system of higher education.” (Johnson et al., 2013) Co-authorship network map of physicians publishing on hepatitis C (detail) Source: http://www.flickr.com/photos/speedoflife/8274993170/
  5.  

  6. WP1. KLE Data Review   • Inconsistent formats •

    Reports take a long time to run • No way to bulk download all “raw” data • Student Overview for a Single Course is the most useful BUT have to run for each student Around 11 “reports” available 9 Course Reports, a Performance Dashboard and a Retention Centre
  7. Blackboard Analytics

  8. Apple Lisa PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,

    USING USER CENTRIC QUERIES, FORMATS and METAPHORS
  9. User-Centered Design Process Map http://www.usability.gov/how-to-and-tools/resources/ucd-map.html

  10. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in

    matched funding from Keele) Title: Learner Centred Design for Learning Analytics This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the user experience, and therefore the ability to interpret and use the information. By sharing the data directly with students, using student-centred representations of their learning activity, this project aims to facilitate a common understanding of the learning experience between lecturers and students. Expanding on a successful teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and personalise the learning experience of cohorts with differing levels of technical experience and levels of digital literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele University and incorporate them into the delivery of relevant modules with the key aims of increasing engagement, making the VLE a more active space for learning and teaching and bridging the current gap between physical and digital spaces.
  11. Outcomes for Students Students in lecture hall ©Jirka Matousek via

    Flickr • Access to personalised notifications and support e.g. highlighting/suggesting resources that have not been viewed. • Increased levels of engagement, in particular VLE usage. • Personalisation of cohort module delivery by the lecturer • Real-time feedback for students enabling them to judge their progress during a module using a different metric than current models of formative and summative feedback. • Direct involvement with the development of tools that support their learning.
  12. 1 Being the best you can be/Effort (ability to maintain

    effort) 2 Build self-confidence 3 Career/Vocation/Job prospects 4 Industry 5 Giving yourself options 6 Grades/Marks/Qualifications 7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually 8 Mentoring/Family 9 Money 10 Part of a Professional community 11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy) 12 Sense of connectedness with others with similar goals/ Success as a group of peers 13 Social Prestige/Recognition Motivators for Studying in HE(Literature) Student Ambassadors
  13. Laddering Sessions run by Student Ambassadors 10

  14. Looks how they feel Shows how hard they are trying

    No point in spending £9,000 if you’re not going to try hard and do well A B ↑ ↑ “When thinking about what motivates you to study your degree, which of these do you prefer and why?”. “Why..?” “Why..?” Laddering
  15. 14 Supporting family/Home life 15 Negative imagery 16 Rising above

    circumstances 17 Financial security 18 Controlling own destiny 19 Individuality 20 Opportunity to travel 21 Fear of wasting University opportunity Additional Motivators for Studying in HE Laddering Sessions run by Student Ambassadors 10
  16. Revised Motivators (21 grouped into 9)

  17. None
  18. None
  19. Timeline Formed basis of 6 Focus Groups with 20 students

    organised and run by Student Ambassadors
  20. None
  21. None
  22. None
  23. None
  24. Professional Theme

  25. Personal/Personified Theme

  26. Personal/Personified Theme

  27. Personal/Personified Theme

  28. Personal/Personified Theme

  29. Personal/Personified Theme

  30. Both Themes

  31. None
  32. Mobile Optimised

  33. Weekly Emails

  34. 14 Features used to create model Total number of content

    accesses Total duration of content accesses Average duration of content accesses Number of days per week content is accessed Average length of time between content accesses Average group size of simultaneous content accesses Number of times specific content is accessed Number of clicks on content of a certain type Number of clicks grouped by parent folder Seen/unseen content Total number of lecture capture views Number of times specific lectures are viewed Grades Absences From KLE click data From lecture capture data From SCIMS data
  35. Model to predict “Grade” using Decision Trees 5 = 1st

    4 = 2:1 3 = 2:2 2 = 3rd 1 = Fail
  36. Interventions Text based on decision trees

  37. Lecturer Dashboard

  38. Learning Analytics in the classroom? Students in lecture hall ©Jirka

    Matousek via Flickr
  39. What motivates students to study at University? ESC-20030 Regional Landscapes

  40. 2:2 38% of students 2:1 62% of students Predicted attainment

    ESC-20030 Regional Landscapes (as of week 10)
  41. 0 5 10 15 20 25 30 35 40 45

    1 2 3 4 5 6 7 8 9 10 # students week Content type clicks by week ESC-20030 Regional Landscapes Assessments Lectures Playback Seminars
  42. Most popular content ESC-20030 Regional Landscapes week Content 1 2

    3 4 5 6 7 8 9 10 Mean Total Max Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38 Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33 Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18 Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38 Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27 Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24 Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33
  43. Least popular content ESC-20030 Regional Landscapes week Content 1 2

    3 4 5 6 7 8 9 10 Mean Total Max Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments- ACL/Week 11 - Thermokarst Lakes 0 0 0 0 0 0 0 0 0 0 0 0 0 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos 0 0 0 0 0 1 0 0 0 0 0.1 1 1 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts 0 0 0 1 0 1 0 0 0 1 0.3 3 1 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers 0 0 0 2 0 2 0 0 0 0 0.4 4 2 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature 0 0 0 0 4 1 0 1 0 0 0.6 6 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River 0 0 0 4 2 0 0 0 0 0 0.6 6 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments- ACL/Week 11 - Proglacial lakes 2 0 0 0 0 0 0 0 0 0 7 0.7 7 7 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-9-Active-Layer-Azonal-Processes/Supplementary reading 0 0 0 3 0 2 0 3 0 0 0.8 8 3 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-6-7-Desert-Environments-RIW/Lecture 14 - Desert landsystems & climate change 0 0 0 0 0 0 0 4 3 1 0.8 8 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments- RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos 0 0 0 8 0 1 0 0 0 0 0.9 9 8
  44. Think Aloud Sessions run by Student Ambassadors and Chris 10

  45. V2

  46. V2

  47. V2

  48. V2

  49. V2

  50. Comparison with peers Next Stage

  51. 2:2 38% of students 2:1 62% of students Did it

    work? 94 students in Stage 1 Next Stage
  52. Dr Ed de Quincey e.de quincey@keele.ac.uk Chris Briggs c.briggs@keele.ac.uk KEELE

    UNIVERSITY SCHOOL OF COMPUTING MOTIVATION METRICS V6 ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK CEO