Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Co-designing, Developing and Integrating a Student Facing Learning Analytics System

Co-designing, Developing and Integrating a Student Facing Learning Analytics System

Dr Ed de Quincey, Chris Briggs & James Mitchell
School of Computing and Mathematics, Keele University
https://showtime.gre.ac.uk/index.php/ecentre/apt2018/paper/viewPaper/1291

Ed de Quincey

June 28, 2018
Tweet

More Decks by Ed de Quincey

Other Decks in Technology

Transcript

  1. Co-designing, Developing and Integrating a
    Student Facing Learning Analytics System
    Dr Ed de Quincey, Chris Briggs & James Mitchell
    School of Computing and Mathematics, Keele University

    View Slide

  2. Dr Ed de Quincey @eddequincey
    Senior Lecturer in Computer Science, UG and PG Course Director
    School of Computer Science and Mathematics, Keele University
    Senior Fellow of the HEA
    instagram.com/eddequincey

    View Slide

  3. Chris Briggs @confusedmatrix
    Research Software Engineer in Learning Analytics
    School of Computer Science and Mathematics, Keele University
    instagram.com/confusedmatrix/

    View Slide

  4. James Mitchell @mitchelljames
    PhD Student and Student Ambassador on the Project
    School of Computer Science and Mathematics, Keele University
    instagram.com/mitchelljames84/

    View Slide


  5. View Slide

  6. •  Inconsistent formats
    •  Reports take a long time to run
    •  No way to bulk download all “raw” data
    •  Student Overview for a Single Course is the most useful BUT
    have to run for each student
    KLE Data Review

    Around 11 “reports” available
    9 Course Reports, a Performance Dashboard and a Retention Centre

    View Slide

  7. Blackboard Analytics

    View Slide

  8. User-Centered Design Process Map
    http://www.usability.gov/how-to-and-tools/resources/ucd-map.html

    View Slide

  9. Apple Lisa
    PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,
    USING USER CENTRIC QUERIES, FORMATS and METAPHORS

    View Slide

  10. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in matched funding from Keele)
    Title: Learner Centred Design for Learning Analytics
    This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the
    user experience, and therefore the ability to interpret and use the information. By sharing the data directly
    with students, using student-centred representations of their learning activity, this project aims to facilitate a
    common understanding of the learning experience between lecturers and students. Expanding on a successful
    teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and
    personalise the learning experience of cohorts with differing levels of technical experience and levels of digital
    literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele
    University and incorporate them into the delivery of relevant modules with the key aims of
    increasing engagement, making the VLE a more active space for learning and teaching and bridging the current
    gap between physical and digital spaces.

    View Slide

  11. Outcomes for Students
    Students in lecture hall ©Jirka Matousek via Flickr
    •  Access to personalised notifications and support
    e.g. highlighting/suggesting resources that have not been viewed.
    •  Increased levels of engagement, in particular VLE usage.
    •  Personalisation of cohort module delivery by the lecturer
    •  Real-time feedback for students enabling them to judge
    their progress during a module using a different metric
    than current models of formative and summative feedback.
    •  Direct involvement with the development of tools that
    support their learning.

    View Slide

  12. 1 Being the best you can be/Effort (ability to maintain effort)
    2 Build self-confidence
    3 Career/Vocation/Job prospects
    4 Industry
    5 Giving yourself options
    6 Grades/Marks/Qualifications
    7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually
    8 Mentoring/Family
    9 Money
    10 Part of a Professional community
    11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy)
    12
    Sense of connectedness with others with similar goals/ Success as a group of
    peers
    13 Social Prestige/Recognition
    Motivators for Studying in HE (Literature)
    Student Ambassadors

    View Slide

  13. Laddering Sessions
    run by Student Ambassadors
    10

    View Slide

  14. Looks how they feel
    Shows how hard they are
    trying
    No point in spending
    £9,000 if you’re not going
    to try hard and do well


    “When thinking about what motivates you to study
    your degree, which of these do you prefer and why?”.
    “Why..?”
    “Why..?”
    Laddering
    A B

    View Slide

  15. 14 Supporting family/Home life
    15 Negative imagery
    16 Rising above circumstances
    17 Financial security
    18 Controlling own destiny
    19 Individuality
    20 Opportunity to travel
    21 Fear of wasting University opportunity
    Additional Motivators for Studying in HE
    Laddering Sessions
    run by Student Ambassadors
    10

    View Slide

  16. Revised Motivators
    (21 grouped into 9)
    Career/Industry
    Mastery
    Money
    Attainment
    Options
    Self Development
    Family
    Fear of …
    Professional Community

    View Slide

  17. View Slide

  18. View Slide

  19. Timeline
    Formed basis of 6 Focus Groups with 20 students
    organised and run by Student Ambassadors

    View Slide

  20. View Slide

  21. 14 Features used to create model
    Total number of content accesses
    Total duration of content accesses
    Average duration of content accesses
    Number of days per week content is accessed
    Average length of time between content accesses
    Average group size of simultaneous content accesses
    Number of times specific content is accessed
    Number of clicks on content of a certain type
    Number of clicks grouped by parent folder
    Seen/unseen content
    Total number of lecture capture views
    Number of times specific lectures are viewed
    Grades
    Absences
    From KLE click data
    From lecture capture data
    From SCIMS data

    View Slide

  22. Model to predict “Grade” using Decision Trees
    5 = 1st
    4 = 2:1
    3 = 2:2
    2 = 3rd
    1 = Fail

    View Slide

  23. Career/Industry
    Grade
    Absences
    Total clicks
    Money
    Grade
    Absences
    Total clicks
    Attainment Grade
    Mastery
    % of content seen
    Grade
    Variety of content clicked
    Options
    Grade
    Absences
    Total clicks
    Professional community Simultaneous users
    Family
    Grade
    Absences
    Total clicks
    Self development
    Total clicks
    Total duration
    Variety of content clicked
    Fear of failure Absences
    % of content seen
    Creating Scores for each Motivator

    View Slide

  24. V2
    Version 2 based on student feedback on Version
    1 from 10 one-to-one contextual interviews
    V1

    View Slide

  25. V2

    View Slide

  26. V2

    View Slide

  27. V2

    View Slide

  28. Personal/Personified Theme

    View Slide

  29. V2

    View Slide

  30. V2

    View Slide

  31. V2

    View Slide

  32. V2

    View Slide

  33. V2

    View Slide

  34. Representing Motivator Scores
    1 2 3 4 5

    View Slide

  35. V2

    View Slide

  36. Comparison with peers
    Both Themes

    View Slide

  37. Mobile Optimised

    View Slide

  38. The LA system was successfully
    integrated into the delivery of
    4 Undergraduate modules
    in Computer Science & Geography
    169 student sign-ups
    48% of students on the 4 modules

    View Slide

  39. View Slide

  40. 41% 59%

    View Slide

  41. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
    career
    money
    attainment
    self development
    fear of failure
    mastery
    options
    family
    professional community

    View Slide

  42. Weekly Emails

    View Slide

  43. Interventions
    Text based on decision trees

    View Slide

  44. Lecturer Dashboard

    View Slide

  45. Learning Analytics in the Classroom
    Using the visualisations to start discussions

    View Slide

  46. Most/least popular content
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38
    Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33
    Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18
    Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38
    Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24
    Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33
    week
    Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-ACL/
    Week 11 - Thermokarst Lakes
    0 0 0 0 0 0 0 0 0 0 0 0 0
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos
    0 0 0 0 0 1 0 0 0 0 0.1 1 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts
    0 0 0 1 0 1 0 0 0 1 0.3 3 1
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers
    0 0 0 2 0 2 0 0 0 0 0.4 4 2
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature
    0 0 0 0 4 1 0 1 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/
    Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River
    0 0 0 4 2 0 0 0 0 0 0.6 6 4
    Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-ACL/
    Week 11 - Proglacial lakes 2
    0 0 0 0 0 0 0 0 0 7 0.7 7 7
    Learning Analytics for Curriculum Design/Module Review?

    View Slide

  47. Students showed increased engagement with all modules
    compared to students from the previous year’s cohort by:
    clicking more in the VLE (between 4% and 20% more)
    having fewer absences (150% - 200% improvement)
    Students showed improved engagement with version 2:
    87% and 89% of students on semester 2 modules logged into the
    dashboard at least once vs 71% and 74% in semester 1
    Evaluation Results 169 students sign-ups
    Average weekly usage:
    51% of students viewed the email
    53% of students (who viewed the email) clicked through to the VLE
    22% of students viewed the report on the KLE

    View Slide

  48. Q1. Would you say a decline in your weekly score would
    influence your behaviour in the following weeks?
    Evaluation Results
    Yes - 57%
    No - 20%
    Maybe - 23%
    Questionnaire - 35 students
    “It did upset me, so I tried harder”
    “I tend to get lazier as the semester
    goes on, having a reminder that I am
    actually doing less work is good.”
    “Feels like someone is watching how
    much work I am doing”
    “I want to feel the satisfaction of a
    good week”
    “It helps you realise roughly how
    much work you are doing”
    “Don't want to simply be a perfunctory
    student.”
    “I didn't see a change in score.”
    “I think it is discouraging if the analytics
    indicate a predicted grade of lower
    than what you expect (and know) you
    will achieve”
    “I’m aware of when and why I stop
    interacting with the VLE, analytics cant
    understand real world circumstances”
    “I don’t think measuring performance
    based on VLE activity alone is a good
    way to measure it”

    View Slide

  49. Q2. Do you believe the data that is presented in the
    dashboard accurately reflects your engagement?
    Evaluation Results
    Strongly agree
    Agree
    Slightly agree
    Neither agree or disagree
    Slightly disagree
    Disagree
    Strongly disagree
    0% 5% 10% 15% 20% 25% 30%
    Questionnaire - 35 students

    View Slide

  50. Q3. Do you feel you can use the system as a tool to enable you
    to improve your grade performance ?
    Evaluation Results
    Yes - 54%
    No - 23%
    Maybe - 23%
    Questionnaire - 35 students
    “Gives weekly adjusted feedback,
    keeps you on your toes academically”
    “Tells me what I’m doing and what I
    can do to improve (i.e spend more
    time looking at certain resources etc.)”
    “Gives a clear idea on what you need
    to do more of when studying”
    “It somewhat held me accountable”
    “Results show you how well you are
    performing and if you need to get
    more work done.”
    “If there were a metric measuring
    attention during lectures, I would feel
    more assured.”
    “The feedback given is not particularly
    in-depth and as such it would be
    difficult to improve specific areas of my
    course.”
    “Needs more inputs perhaps a way for
    a lecturer to perhaps give there
    thoughts on that students performance
    every now and then.
    “Needs better recommendations”
    “Whilst the use of the KLE is a way of
    showing work; it doesn’t necessarily reveal
    how much other work you have done.“

    View Slide

  51. Q. Before using the system, were you aware of your engagement with the
    KLE and your attendance?
    Most students aware of attendance but not VLE data.
    Q. Can you describe your understanding of how you can influence the
    outcome of your learning analytics? How would you improve?
    All students said that viewing more content/regular VLE interaction and
    attending more sessions would improve scores.
    Q. What information have you found most useful when using the learning
    analytics system?
    -  Number of clicks and the cohort comparison
    -  Scores/Predictions/Stats breakdown
    -  % content seen
    -  Week-to-week outlook
    Evaluation Results 10 one-to-one Contextual Interviews
    whilst viewing their own data in the LA tool

    View Slide

  52. Q. What information have you found least useful when using the learning
    analytics system?
    Most students were unsure about the accuracy of the grade predictions
    and weekly “swings” in prediction.
    Q. Can you describe how the learning analytics system impacted your
    motivation?
    -  Made students reflect on a weekly/regular basis instead of only
    working hard around assessment time.
    -  When saw grades go down made them wanted to improve grade/
    gave extra boost (although one student said “Seeing high scores during the middle of the semester incentivised them to ‘cruise’.”)
    -  Made them more keen to go on the VLE and look around to see
    missed content/make themselves a better student.
    -  Reinforced effort e.g. ”good seeing how making an effort yielded good results for a
    specific week, showing that if you put the effort in, you would likely perform better”.
    Evaluation Results 10 one-to-one Contextual Interviews
    whilst viewing their own data in the LA tool

    View Slide

  53. Q. How has the learning analytics system changed your
    awareness of your progress?
    -  Gives feedback throughout the semester/gives weekly visibility meaning you
    don’t have to wait until the end of the module.
    -  The lecturer visualisation slides that were shown in class were useful - made
    them aware of resources that they hadn’t looked at.
    -  Being able to compare activity to the cohort was useful e.g. “when the average
    dipped but I maintained consistency.”
    -  Seeing a drop in performance on on the dashboard “kick-started” them to do
    more. e.g. “Without a system like this, although I would maybe know I wasn’t engaging as
    much as I should, I probably wouldn’t act on it."
    Q. In general, do you feel that the learning analytics system has had a
    direct effect on your performance on the module? How?
    Most students said positive (although “slightly”). Mentioned that it would
    have more impact on “harder” modules.
    Evaluation Results 10 one-to-one Contextual Interviews
    whilst viewing their own data in the LA tool

    View Slide

  54. Academics
    4
    Student Ambassadors
    4
    Students in User Research Sessions
    40
    Evaluation Questionnaire Respondents
    35
    Sign-ups to the LA Tool
    169

    View Slide

  55. Dr Ed de Quincey e.de [email protected]
    Chris Briggs [email protected]
    KEELE UNIVERSITY SCHOOL OF COMPUTING
    MOTIVATION METRICS V6
    ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK
    CEO

    View Slide