Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Co-designing, Developing and Integrating a Student Facing Learning Analytics System

Co-designing, Developing and Integrating a Student Facing Learning Analytics System

Dr Ed de Quincey, Chris Briggs & James Mitchell
School of Computing and Mathematics, Keele University
https://showtime.gre.ac.uk/index.php/ecentre/apt2018/paper/viewPaper/1291

Ed de Quincey

June 28, 2018
Tweet

More Decks by Ed de Quincey

Other Decks in Technology

Transcript

  1. Co-designing, Developing and Integrating a
    Student Facing Learning Analytics System
    Dr Ed de Quincey, Chris Briggs & James Mitchell
    School of Computing and Mathematics, Keele University

    View full-size slide

  2. Dr Ed de Quincey @eddequincey
    Senior Lecturer in Computer Science, UG and PG Course Director
    School of Computer Science and Mathematics, Keele University
    Senior Fellow of the HEA
    instagram.com/eddequincey

    View full-size slide

  3. Chris Briggs @confusedmatrix
    Research Software Engineer in Learning Analytics
    School of Computer Science and Mathematics, Keele University
    instagram.com/confusedmatrix/

    View full-size slide

  4. James Mitchell @mitchelljames
    PhD Student and Student Ambassador on the Project
    School of Computer Science and Mathematics, Keele University
    instagram.com/mitchelljames84/

    View full-size slide

  5. •  Inconsistent formats
    •  Reports take a long time to run
    •  No way to bulk download all “raw” data
    •  Student Overview for a Single Course is the most useful BUT
    have to run for each student
    KLE Data Review

    Around 11 “reports” available
    9 Course Reports, a Performance Dashboard and a Retention Centre

    View full-size slide

  6. Blackboard Analytics

    View full-size slide

  7. User-Centered Design Process Map
    http://www.usability.gov/how-to-and-tools/resources/ucd-map.html

    View full-size slide

  8. Apple Lisa
    PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,
    USING USER CENTRIC QUERIES, FORMATS and METAPHORS

    View full-size slide

  9. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in matched funding from Keele)
    Title: Learner Centred Design for Learning Analytics
    This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the
    user experience, and therefore the ability to interpret and use the information. By sharing the data directly
    with students, using student-centred representations of their learning activity, this project aims to facilitate a
    common understanding of the learning experience between lecturers and students. Expanding on a successful
    teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and
    personalise the learning experience of cohorts with differing levels of technical experience and levels of digital
    literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele
    University and incorporate them into the delivery of relevant modules with the key aims of
    increasing engagement, making the VLE a more active space for learning and teaching and bridging the current
    gap between physical and digital spaces.

    View full-size slide

  10. Outcomes for Students
    Students in lecture hall ©Jirka Matousek via Flickr
    •  Access to personalised notifications and support
    e.g. highlighting/suggesting resources that have not been viewed.
    •  Increased levels of engagement, in particular VLE usage.
    •  Personalisation of cohort module delivery by the lecturer
    •  Real-time feedback for students enabling them to judge
    their progress during a module using a different metric
    than current models of formative and summative feedback.
    •  Direct involvement with the development of tools that
    support their learning.

    View full-size slide

  11. 1 Being the best you can be/Effort (ability to maintain effort)
    2 Build self-confidence
    3 Career/Vocation/Job prospects
    4 Industry
    5 Giving yourself options
    6 Grades/Marks/Qualifications
    7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually
    8 Mentoring/Family
    9 Money
    10 Part of a Professional community
    11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy)
    12
    Sense of connectedness with others with similar goals/ Success as a group of
    peers
    13 Social Prestige/Recognition
    Motivators for Studying in HE (Literature)
    Student Ambassadors

    View full-size slide

  12. Laddering Sessions
    run by Student Ambassadors
    10

    View full-size slide

  13. Looks how they feel
    Shows how hard they are
    trying
    No point in spending
    £9,000 if you’re not going
    to try hard and do well


    “When thinking about what motivates you to study
    your degree, which of these do you prefer and why?”.
    “Why..?”
    “Why..?”
    Laddering
    A B

    View full-size slide

  14. 14 Supporting family/Home life
    15 Negative imagery
    16 Rising above circumstances
    17 Financial security
    18 Controlling own destiny
    19 Individuality
    20 Opportunity to travel
    21 Fear of wasting University opportunity
    Additional Motivators for Studying in HE
    Laddering Sessions
    run by Student Ambassadors
    10

    View full-size slide

  15. Revised Motivators
    (21 grouped into 9)
    Career/Industry
    Mastery
    Money
    Attainment
    Options
    Self Development
    Family
    Fear of …
    Professional Community

    View full-size slide

  16. Timeline
    Formed basis of 6 Focus Groups with 20 students
    organised and run by Student Ambassadors

    View full-size slide

  17. 14 Features used to create model
    Total number of content accesses
    Total duration of content accesses
    Average duration of content accesses
    Number of days per week content is accessed
    Average length of time between content accesses
    Average group size of simultaneous content accesses
    Number of times specific content is accessed
    Number of clicks on content of a certain type
    Number of clicks grouped by parent folder
    Seen/unseen content
    Total number of lecture capture views
    Number of times specific lectures are viewed
    Grades
    Absences
    From KLE click data
    From lecture capture data
    From SCIMS data

    View full-size slide

  18. Model to predict “Grade” using Decision Trees
    5 = 1st
    4 = 2:1
    3 = 2:2
    2 = 3rd
    1 = Fail

    View full-size slide

  19. Career/Industry
    Grade
    Absences
    Total clicks
    Money
    Grade
    Absences
    Total clicks
    Attainment Grade
    Mastery
    % of content seen
    Grade
    Variety of content clicked
    Options
    Grade
    Absences
    Total clicks
    Professional community Simultaneous users
    Family
    Grade
    Absences
    Total clicks
    Self development
    Total clicks
    Total duration
    Variety of content clicked
    Fear of failure Absences
    % of content seen
    Creating Scores for each Motivator

    View full-size slide

  20. V2
    Version 2 based on student feedback on Version
    1 from 10 one-to-one contextual interviews
    V1

    View full-size slide

  21. Personal/Personified Theme

    View full-size slide

  22. Representing Motivator Scores
    1 2 3 4 5

    View full-size slide

  23. Comparison with peers
    Both Themes

    View full-size slide

  24. Mobile Optimised

    View full-size slide

  25. The LA system was successfully
    integrated into the delivery of
    4 Undergraduate modules
    in Computer Science & Geography
    169 student sign-ups
    48% of students on the 4 modules

    View full-size slide

  26. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
    career
    money
    attainment
    self development
    fear of failure
    mastery
    options
    family
    professional community

    View full-size slide

  27. Weekly Emails

    View full-size slide

  28. Interventions
    Text based on decision trees

    View full-size slide

  29. Lecturer Dashboard

    View full-size slide

  30. Learning Analytics in the Classroom
    Using the visualisations to start discussions

    View full-size slide

  31. Most/least popular content
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38
    Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33
    Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18
    Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38
    Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-ACL/
    Week 11 - Thermokarst Lakes
    0 0 0 0 0 0 0 0 0 0 0 0 0
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos
    0 0 0 0 0 1 0 0 0 0 0.1 1 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts
    0 0 0 1 0 1 0 0 0 1 0.3 3 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers
    0 0 0 2 0 2 0 0 0 0 0.4 4 2
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature
    0 0 0 0 4 1 0 1 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River
    0 0 0 4 2 0 0 0 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-ACL/
    Week 11 - Proglacial lakes 2
    0 0 0 0 0 0 0 0 0 7 0.7 7 7
    Learning Analytics for Curriculum Design/Module Review?

    View full-size slide

  32. Students showed increased engagement with all modules
    compared to students from the previous year’s cohort by:
    clicking more in the VLE (between 4% and 20% more)
    having fewer absences (150% - 200% improvement)
    Students showed improved engagement with version 2:
    87% and 89% of students on semester 2 modules logged into the
    dashboard at least once vs 71% and 74% in semester 1
    Evaluation Results 169 students sign-ups
    Average weekly usage:
    51% of students viewed the email
    53% of students (who viewed the email) clicked through to the VLE
    22% of students viewed the report on the KLE

    View full-size slide

  33. Q1. Would you say a decline in your weekly score would
    influence your behaviour in the following weeks?
    Evaluation Results
    Yes - 57%
    No - 20%
    Maybe - 23%
    Questionnaire - 35 students
    “It did upset me, so I tried harder”
    “I tend to get lazier as the semester
    goes on, having a reminder that I am
    actually doing less work is good.”
    “Feels like someone is watching how
    much work I am doing”
    “I want to feel the satisfaction of a
    good week”
    “It helps you realise roughly how
    much work you are doing”
    “Don't want to simply be a perfunctory
    student.”
    “I didn't see a change in score.”
    “I think it is discouraging if the analytics
    indicate a predicted grade of lower
    than what you expect (and know) you
    will achieve”
    “I’m aware of when and why I stop
    interacting with the VLE, analytics cant
    understand real world circumstances”
    “I don’t think measuring performance
    based on VLE activity alone is a good
    way to measure it”

    View full-size slide

  34. Q2. Do you believe the data that is presented in the
    dashboard accurately reflects your engagement?
    Evaluation Results
    Strongly agree
    Agree
    Slightly agree
    Neither agree or disagree
    Slightly disagree
    Disagree
    Strongly disagree
    0% 5% 10% 15% 20% 25% 30%
    Questionnaire - 35 students

    View full-size slide

  35. Q3. Do you feel you can use the system as a tool to enable you
    to improve your grade performance ?
    Evaluation Results
    Yes - 54%
    No - 23%
    Maybe - 23%
    Questionnaire - 35 students
    “Gives weekly adjusted feedback,
    keeps you on your toes academically”
    “Tells me what I’m doing and what I
    can do to improve (i.e spend more
    time looking at certain resources etc.)”
    “Gives a clear idea on what you need
    to do more of when studying”
    “It somewhat held me accountable”
    “Results show you how well you are
    performing and if you need to get
    more work done.”
    “If there were a metric measuring
    attention during lectures, I would feel
    more assured.”
    “The feedback given is not particularly
    in-depth and as such it would be
    difficult to improve specific areas of my
    course.”
    “Needs more inputs perhaps a way for
    a lecturer to perhaps give there
    thoughts on that students performance
    every now and then.
    “Needs better recommendations”
    “Whilst the use of the KLE is a way of
    showing work; it doesn’t necessarily reveal
    how much other work you have done.“

    View full-size slide

  36. Q. Before using the system, were you aware of your engagement with the
    KLE and your attendance?
    Most students aware of attendance but not VLE data.
    Q. Can you describe your understanding of how you can influence the
    outcome of your learning analytics? How would you improve?
    All students said that viewing more content/regular VLE interaction and
    attending more sessions would improve scores.
    Q. What information have you found most useful when using the learning
    analytics system?
    -  Number of clicks and the cohort comparison
    -  Scores/Predictions/Stats breakdown
    -  % content seen
    -  Week-to-week outlook
    Evaluation Results 10 one-to-one Contextual Interviews
    whilst viewing their own data in the LA tool

    View full-size slide

  37. Q. What information have you found least useful when using the learning
    analytics system?
    Most students were unsure about the accuracy of the grade predictions
    and weekly “swings” in prediction.
    Q. Can you describe how the learning analytics system impacted your
    motivation?
    -  Made students reflect on a weekly/regular basis instead of only
    working hard around assessment time.
    -  When saw grades go down made them wanted to improve grade/
    gave extra boost (although one student said “Seeing high scores during the middle of the semester incentivised them to ‘cruise’.”)
    -  Made them more keen to go on the VLE and look around to see
    missed content/make themselves a better student.
    -  Reinforced effort e.g. ”good seeing how making an effort yielded good results for a
    specific week, showing that if you put the effort in, you would likely perform better”.
    Evaluation Results 10 one-to-one Contextual Interviews
    whilst viewing their own data in the LA tool

    View full-size slide

  38. Q. How has the learning analytics system changed your
    awareness of your progress?
    -  Gives feedback throughout the semester/gives weekly visibility meaning you
    don’t have to wait until the end of the module.
    -  The lecturer visualisation slides that were shown in class were useful - made
    them aware of resources that they hadn’t looked at.
    -  Being able to compare activity to the cohort was useful e.g. “when the average
    dipped but I maintained consistency.”
    -  Seeing a drop in performance on on the dashboard “kick-started” them to do
    more. e.g. “Without a system like this, although I would maybe know I wasn’t engaging as
    much as I should, I probably wouldn’t act on it."
    Q. In general, do you feel that the learning analytics system has had a
    direct effect on your performance on the module? How?
    Most students said positive (although “slightly”). Mentioned that it would
    have more impact on “harder” modules.
    Evaluation Results 10 one-to-one Contextual Interviews
    whilst viewing their own data in the LA tool

    View full-size slide

  39. Academics
    4
    Student Ambassadors
    4
    Students in User Research Sessions
    40
    Evaluation Questionnaire Respondents
    35
    Sign-ups to the LA Tool
    169

    View full-size slide

  40. Dr Ed de Quincey e.de [email protected]
    Chris Briggs [email protected]
    KEELE UNIVERSITY SCHOOL OF COMPUTING
    MOTIVATION METRICS V6
    ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK
    CEO

    View full-size slide