User-centred design of a student facing learning analytics dashboard

Presentation at the Keele Annual Learning & Teaching Conference 2018
https://www.keele.ac.uk/lpdc/learningteaching/keelelearningandteachingconference/

Current Learning Analytics (LA) systems are primarily designed with University staff members as the target audience; very few are aimed at students, with almost none being developed with direct student involvement (de Quincey et al., 2016). Involving the user in the development process however has been shown to have a positive impact on the success of a system (Bano and Zowghi, 2013). Keele’s HEFCE funded “Learner Centred Design for Learning Analytics” project has therefore employed a variety of methods to engage students in the design/development of a LA dashboard which has then been implemented and piloted with 2 undergraduate modules. The design of the dashboard has been influenced by student feedback, using a novel approach of trying to understand the reasons why students want to study at university (e.g. career, self-development, attainment) and mapping their engagement and predicted outcomes to these motivations. Machine learning algorithms have been used to model the behaviour of student activity for last year’s cohort using KLE interactions, attendance and their final module grades. We then use this model to analyse the activity of current students and present their activity data back to them as scores, mapped to their chosen motivations in the LA dashboard. The end result is a dashboard personalised to each student allowing them to understand how their academic behaviour links to their motivations to study.

Transcript

1. User-centred design of a student facing
   learning analytics dashboard
   Dr Ed de Quincey & Chris Briggs School of Computing and Mathematics, Keele University
   

   View Slide

2. Dr Ed de Quincey @eddequincey
   Senior Lecturer in Computer Science, UG and PG Course Director
   School of Computing and Mathematics, Keele University
   Senior Fellow of the HEA
   instagram.com/eddequincey
   

   View Slide

3. Chris Briggs @confusedmatrix
   Research Software Engineer in Learning Analytics
   School of Computing and Mathematics, Keele University
   instagram.com/confusedmatrix/
   

   View Slide

4. Learning Analytics
   has been defined as a method for
   “deciphering trends
   and patterns from
   educational big data
   … to further the
   advancement of a
   personalized,
   supportive system
   of higher education.”
   (Johnson et al., 2013)
   Co-authorship network map of physicians publishing on hepatitis C (detail)
   Source: http://www.flickr.com/photos/speedoflife/8274993170/
   

   View Slide

5. 
   
   

   View Slide

6. WP1. KLE Data Review
   
   
   • Inconsistent formats
   • Reports take a long time to run
   • No way to bulk download all “raw” data
   • Student Overview for a Single Course is the most useful BUT
   have to run for each student
   Around 11 “reports” available
   9 Course Reports, a Performance Dashboard and a Retention Centre
   

   View Slide

7. Blackboard Analytics
   

   View Slide

8. Apple Lisa
   PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,
   USING USER CENTRIC QUERIES, FORMATS and METAPHORS
   

   View Slide

9. User-Centered Design Process Map
   http://www.usability.gov/how-to-and-tools/resources/ucd-map.html
   

   View Slide

10. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in matched funding from Keele)
    Title: Learner Centred Design for Learning Analytics
    This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the
    user experience, and therefore the ability to interpret and use the information. By sharing the data directly
    with students, using student-centred representations of their learning activity, this project aims to facilitate a
    common understanding of the learning experience between lecturers and students. Expanding on a successful
    teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and
    personalise the learning experience of cohorts with differing levels of technical experience and levels of digital
    literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele
    University and incorporate them into the delivery of relevant modules with the key aims of
    increasing engagement, making the VLE a more active space for learning and teaching and bridging the current
    gap between physical and digital spaces.
    

    View Slide

11. Outcomes for Students
    Students in lecture hall ©Jirka Matousek via Flickr
    • Access to personalised notifications and support
    e.g. highlighting/suggesting resources that have not been viewed.
    • Increased levels of engagement, in particular VLE usage.
    • Personalisation of cohort module delivery by the lecturer
    • Real-time feedback for students enabling them to judge
    their progress during a module using a different metric
    than current models of formative and summative feedback.
    • Direct involvement with the development of tools that
    support their learning.
    

    View Slide

12. 1 Being the best you can be/Effort (ability to maintain effort)
    2 Build self-confidence
    3 Career/Vocation/Job prospects
    4 Industry
    5 Giving yourself options
    6 Grades/Marks/Qualifications
    7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually
    8 Mentoring/Family
    9 Money
    10 Part of a Professional community
    11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy)
    12
    Sense of connectedness with others with similar goals/ Success as a group of
    peers
    13 Social Prestige/Recognition
    Motivators for Studying in HE(Literature)
    Student Ambassadors
    

    View Slide

13. Laddering Sessions
    run by Student Ambassadors
    10
    

    View Slide

14. Looks how they feel
    Shows how hard they are
    trying
    No point in spending
    £9,000 if you’re not going
    to try hard and do well
    A B
    ↑
    ↑
    “When thinking about what motivates you to study your
    degree, which of these do you prefer and why?”.
    “Why..?”
    “Why..?”
    Laddering
    

    View Slide

15. 14 Supporting family/Home life
    15 Negative imagery
    16 Rising above circumstances
    17 Financial security
    18 Controlling own destiny
    19 Individuality
    20 Opportunity to travel
    21 Fear of wasting University opportunity
    Additional Motivators for Studying in HE
    Laddering Sessions
    run by Student Ambassadors
    10
    

    View Slide

16. Revised Motivators
    (21 grouped into 9)
    

    View Slide

17. View Slide

18. View Slide

19. Timeline
    Formed basis of 6 Focus Groups with 20 students
    organised and run by Student Ambassadors
    

    View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. Professional Theme
    

    View Slide

25. Personal/Personified Theme
    

    View Slide

26. Personal/Personified Theme
    

    View Slide

27. Personal/Personified Theme
    

    View Slide

28. Personal/Personified Theme
    

    View Slide

29. Personal/Personified Theme
    

    View Slide

30. Both Themes
    

    View Slide

31. View Slide

32. Mobile Optimised
    

    View Slide

33. Weekly Emails
    

    View Slide

34. 14 Features used to create model
    Total number of content accesses
    Total duration of content accesses
    Average duration of content accesses
    Number of days per week content is accessed
    Average length of time between content accesses
    Average group size of simultaneous content accesses
    Number of times specific content is accessed
    Number of clicks on content of a certain type
    Number of clicks grouped by parent folder
    Seen/unseen content
    Total number of lecture capture views
    Number of times specific lectures are viewed
    Grades
    Absences
    From KLE click data
    From lecture capture data
    From SCIMS data
    

    View Slide

35. Model to predict “Grade” using Decision Trees
    5 = 1st
    4 = 2:1
    3 = 2:2
    2 = 3rd
    1 = Fail
    

    View Slide

36. Interventions
    Text based on decision trees
    

    View Slide

37. Lecturer Dashboard
    

    View Slide

38. Learning Analytics in the classroom?
    Students in lecture hall ©Jirka Matousek via Flickr
    

    View Slide

39. What motivates students to study at University?
    ESC-20030 Regional Landscapes
    

    View Slide

40. 2:2
    38% of students
    2:1
    62% of students
    Predicted attainment
    ESC-20030 Regional Landscapes
    (as of week 10)
    

    View Slide

41. 0
    5
    10
    15
    20
    25
    30
    35
    40
    45
    1 2 3 4 5 6 7 8 9 10
    # students
    week
    Content type clicks by week
    ESC-20030 Regional Landscapes
    Assessments Lectures Playback Seminars
    

    View Slide

42. Most popular content
    ESC-20030 Regional Landscapes
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38
    Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33
    Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18
    Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38
    Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33
    

    View Slide

43. Least popular content
    ESC-20030 Regional Landscapes
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-
    ACL/Week 11 - Thermokarst Lakes
    0 0 0 0 0 0 0 0 0 0 0 0 0
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos
    0 0 0 0 0 1 0 0 0 0 0.1 1 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts
    0 0 0 1 0 1 0 0 0 1 0.3 3 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers
    0 0 0 2 0 2 0 0 0 0 0.4 4 2
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature
    0 0 0 0 4 1 0 1 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River
    0 0 0 4 2 0 0 0 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-
    ACL/Week 11 - Proglacial lakes 2
    0 0 0 0 0 0 0 0 0 7 0.7 7 7
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-9-Active-Layer-Azonal-Processes/Supplementary reading
    0 0 0 3 0 2 0 3 0 0 0.8 8 3
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-6-7-Desert-Environments-RIW/Lecture
    14 - Desert landsystems & climate change
    0 0 0 0 0 0 0 4 3 1 0.8 8 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos
    0 0 0 8 0 1 0 0 0 0 0.9 9 8
    

    View Slide

44. Think Aloud Sessions
    run by Student Ambassadors and Chris
    10
    

    View Slide

45. V2
    

    View Slide

46. V2
    

    View Slide

47. V2
    

    View Slide

48. V2
    

    View Slide

49. V2
    

    View Slide

50. Comparison with peers
    Next Stage
    

    View Slide

51. 2:2
    38% of students
    2:1
    62% of students
    Did it work?
    94 students in Stage 1
    Next Stage
    

    View Slide

52. Dr Ed de Quincey e.de quincey@keele.ac.uk
    Chris Briggs c.briggs@keele.ac.uk
    KEELE UNIVERSITY SCHOOL OF COMPUTING
    MOTIVATION METRICS V6
    ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK
    CEO
    

    View Slide