Upgrade to Pro — share decks privately, control downloads, hide ads and more …

User-centred design of a student facing learning analytics dashboard

User-centred design of a student facing learning analytics dashboard

Presentation at the Keele Annual Learning & Teaching Conference 2018
https://www.keele.ac.uk/lpdc/learningteaching/keelelearningandteachingconference/

Current Learning Analytics (LA) systems are primarily designed with University staff members as the target audience; very few are aimed at students, with almost none being developed with direct student involvement (de Quincey et al., 2016). Involving the user in the development process however has been shown to have a positive impact on the success of a system (Bano and Zowghi, 2013). Keele’s HEFCE funded “Learner Centred Design for Learning Analytics” project has therefore employed a variety of methods to engage students in the design/development of a LA dashboard which has then been implemented and piloted with 2 undergraduate modules. The design of the dashboard has been influenced by student feedback, using a novel approach of trying to understand the reasons why students want to study at university (e.g. career, self-development, attainment) and mapping their engagement and predicted outcomes to these motivations. Machine learning algorithms have been used to model the behaviour of student activity for last year’s cohort using KLE interactions, attendance and their final module grades. We then use this model to analyse the activity of current students and present their activity data back to them as scores, mapped to their chosen motivations in the LA dashboard. The end result is a dashboard personalised to each student allowing them to understand how their academic behaviour links to their motivations to study.

Ed de Quincey

January 16, 2018
Tweet

More Decks by Ed de Quincey

Other Decks in Technology

Transcript

  1. User-centred design of a student facing
    learning analytics dashboard
    Dr Ed de Quincey & Chris Briggs School of Computing and Mathematics, Keele University

    View Slide

  2. Dr Ed de Quincey @eddequincey
    Senior Lecturer in Computer Science, UG and PG Course Director
    School of Computing and Mathematics, Keele University
    Senior Fellow of the HEA
    instagram.com/eddequincey

    View Slide

  3. Chris Briggs @confusedmatrix
    Research Software Engineer in Learning Analytics
    School of Computing and Mathematics, Keele University
    instagram.com/confusedmatrix/

    View Slide

  4. Learning Analytics
    has been defined as a method for
    “deciphering trends
    and patterns from
    educational big data
    … to further the
    advancement of a
    personalized,
    supportive system
    of higher education.”
    (Johnson et al., 2013)
    Co-authorship network map of physicians publishing on hepatitis C (detail)
    Source: http://www.flickr.com/photos/speedoflife/8274993170/

    View Slide



  5. View Slide

  6. WP1. KLE Data Review


    • Inconsistent formats
    • Reports take a long time to run
    • No way to bulk download all “raw” data
    • Student Overview for a Single Course is the most useful BUT
    have to run for each student
    Around 11 “reports” available
    9 Course Reports, a Performance Dashboard and a Retention Centre

    View Slide

  7. Blackboard Analytics

    View Slide

  8. Apple Lisa
    PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,
    USING USER CENTRIC QUERIES, FORMATS and METAPHORS

    View Slide

  9. User-Centered Design Process Map
    http://www.usability.gov/how-to-and-tools/resources/ucd-map.html

    View Slide

  10. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in matched funding from Keele)
    Title: Learner Centred Design for Learning Analytics
    This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the
    user experience, and therefore the ability to interpret and use the information. By sharing the data directly
    with students, using student-centred representations of their learning activity, this project aims to facilitate a
    common understanding of the learning experience between lecturers and students. Expanding on a successful
    teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and
    personalise the learning experience of cohorts with differing levels of technical experience and levels of digital
    literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele
    University and incorporate them into the delivery of relevant modules with the key aims of
    increasing engagement, making the VLE a more active space for learning and teaching and bridging the current
    gap between physical and digital spaces.

    View Slide

  11. Outcomes for Students
    Students in lecture hall ©Jirka Matousek via Flickr
    • Access to personalised notifications and support
    e.g. highlighting/suggesting resources that have not been viewed.
    • Increased levels of engagement, in particular VLE usage.
    • Personalisation of cohort module delivery by the lecturer
    • Real-time feedback for students enabling them to judge
    their progress during a module using a different metric
    than current models of formative and summative feedback.
    • Direct involvement with the development of tools that
    support their learning.

    View Slide

  12. 1 Being the best you can be/Effort (ability to maintain effort)
    2 Build self-confidence
    3 Career/Vocation/Job prospects
    4 Industry
    5 Giving yourself options
    6 Grades/Marks/Qualifications
    7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually
    8 Mentoring/Family
    9 Money
    10 Part of a Professional community
    11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy)
    12
    Sense of connectedness with others with similar goals/ Success as a group of
    peers
    13 Social Prestige/Recognition
    Motivators for Studying in HE(Literature)
    Student Ambassadors

    View Slide

  13. Laddering Sessions
    run by Student Ambassadors
    10

    View Slide

  14. Looks how they feel
    Shows how hard they are
    trying
    No point in spending
    £9,000 if you’re not going
    to try hard and do well
    A B


    “When thinking about what motivates you to study your
    degree, which of these do you prefer and why?”.
    “Why..?”
    “Why..?”
    Laddering

    View Slide

  15. 14 Supporting family/Home life
    15 Negative imagery
    16 Rising above circumstances
    17 Financial security
    18 Controlling own destiny
    19 Individuality
    20 Opportunity to travel
    21 Fear of wasting University opportunity
    Additional Motivators for Studying in HE
    Laddering Sessions
    run by Student Ambassadors
    10

    View Slide

  16. Revised Motivators
    (21 grouped into 9)

    View Slide

  17. View Slide

  18. View Slide

  19. Timeline
    Formed basis of 6 Focus Groups with 20 students
    organised and run by Student Ambassadors

    View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. Professional Theme

    View Slide

  25. Personal/Personified Theme

    View Slide

  26. Personal/Personified Theme

    View Slide

  27. Personal/Personified Theme

    View Slide

  28. Personal/Personified Theme

    View Slide

  29. Personal/Personified Theme

    View Slide

  30. Both Themes

    View Slide

  31. View Slide

  32. Mobile Optimised

    View Slide

  33. Weekly Emails

    View Slide

  34. 14 Features used to create model
    Total number of content accesses
    Total duration of content accesses
    Average duration of content accesses
    Number of days per week content is accessed
    Average length of time between content accesses
    Average group size of simultaneous content accesses
    Number of times specific content is accessed
    Number of clicks on content of a certain type
    Number of clicks grouped by parent folder
    Seen/unseen content
    Total number of lecture capture views
    Number of times specific lectures are viewed
    Grades
    Absences
    From KLE click data
    From lecture capture data
    From SCIMS data

    View Slide

  35. Model to predict “Grade” using Decision Trees
    5 = 1st
    4 = 2:1
    3 = 2:2
    2 = 3rd
    1 = Fail

    View Slide

  36. Interventions
    Text based on decision trees

    View Slide

  37. Lecturer Dashboard

    View Slide

  38. Learning Analytics in the classroom?
    Students in lecture hall ©Jirka Matousek via Flickr

    View Slide

  39. What motivates students to study at University?
    ESC-20030 Regional Landscapes

    View Slide

  40. 2:2
    38% of students
    2:1
    62% of students
    Predicted attainment
    ESC-20030 Regional Landscapes
    (as of week 10)

    View Slide

  41. 0
    5
    10
    15
    20
    25
    30
    35
    40
    45
    1 2 3 4 5 6 7 8 9 10
    # students
    week
    Content type clicks by week
    ESC-20030 Regional Landscapes
    Assessments Lectures Playback Seminars

    View Slide

  42. Most popular content
    ESC-20030 Regional Landscapes
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38
    Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33
    Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18
    Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38
    Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33

    View Slide

  43. Least popular content
    ESC-20030 Regional Landscapes
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-
    ACL/Week 11 - Thermokarst Lakes
    0 0 0 0 0 0 0 0 0 0 0 0 0
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos
    0 0 0 0 0 1 0 0 0 0 0.1 1 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts
    0 0 0 1 0 1 0 0 0 1 0.3 3 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers
    0 0 0 2 0 2 0 0 0 0 0.4 4 2
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature
    0 0 0 0 4 1 0 1 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River
    0 0 0 4 2 0 0 0 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-
    ACL/Week 11 - Proglacial lakes 2
    0 0 0 0 0 0 0 0 0 7 0.7 7 7
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-9-Active-Layer-Azonal-Processes/Supplementary reading
    0 0 0 3 0 2 0 3 0 0 0.8 8 3
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-6-7-Desert-Environments-RIW/Lecture
    14 - Desert landsystems & climate change
    0 0 0 0 0 0 0 4 3 1 0.8 8 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-
    RIW/Lecture-7-An-Introduction-to-Permafrost-Environments/Videos
    0 0 0 8 0 1 0 0 0 0 0.9 9 8

    View Slide

  44. Think Aloud Sessions
    run by Student Ambassadors and Chris
    10

    View Slide

  45. V2

    View Slide

  46. V2

    View Slide

  47. V2

    View Slide

  48. V2

    View Slide

  49. V2

    View Slide

  50. Comparison with peers
    Next Stage

    View Slide

  51. 2:2
    38% of students
    2:1
    62% of students
    Did it work?
    94 students in Stage 1
    Next Stage

    View Slide

  52. Dr Ed de Quincey e.de [email protected]
    Chris Briggs [email protected]
    KEELE UNIVERSITY SCHOOL OF COMPUTING
    MOTIVATION METRICS V6
    ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK
    CEO

    View Slide