Co-designing, Developing and Integrating a Student Facing Learning Analytics System

Co-designing, Developing and Integrating a Student Facing Learning Analytics System

Dr Ed de Quincey, Chris Briggs & James Mitchell
School of Computing and Mathematics, Keele University
https://showtime.gre.ac.uk/index.php/ecentre/apt2018/paper/viewPaper/1291

Dafb8db748341892bda61117ecd16a43?s=128

Ed de Quincey

June 28, 2018
Tweet

Transcript

  1. Co-designing, Developing and Integrating a Student Facing Learning Analytics System

    Dr Ed de Quincey, Chris Briggs & James Mitchell School of Computing and Mathematics, Keele University
  2. Dr Ed de Quincey @eddequincey Senior Lecturer in Computer Science,

    UG and PG Course Director School of Computer Science and Mathematics, Keele University Senior Fellow of the HEA instagram.com/eddequincey
  3. Chris Briggs @confusedmatrix Research Software Engineer in Learning Analytics School

    of Computer Science and Mathematics, Keele University instagram.com/confusedmatrix/
  4. James Mitchell @mitchelljames PhD Student and Student Ambassador on the

    Project School of Computer Science and Mathematics, Keele University instagram.com/mitchelljames84/
  5. •  Inconsistent formats •  Reports take a long time to

    run •  No way to bulk download all “raw” data •  Student Overview for a Single Course is the most useful BUT have to run for each student KLE Data Review ✓ Around 11 “reports” available 9 Course Reports, a Performance Dashboard and a Retention Centre
  6. Blackboard Analytics

  7. User-Centered Design Process Map http://www.usability.gov/how-to-and-tools/resources/ucd-map.html

  8. Apple Lisa PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,

    USING USER CENTRIC QUERIES, FORMATS and METAPHORS
  9. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in

    matched funding from Keele) Title: Learner Centred Design for Learning Analytics This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the user experience, and therefore the ability to interpret and use the information. By sharing the data directly with students, using student-centred representations of their learning activity, this project aims to facilitate a common understanding of the learning experience between lecturers and students. Expanding on a successful teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and personalise the learning experience of cohorts with differing levels of technical experience and levels of digital literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele University and incorporate them into the delivery of relevant modules with the key aims of increasing engagement, making the VLE a more active space for learning and teaching and bridging the current gap between physical and digital spaces.
  10. Outcomes for Students Students in lecture hall ©Jirka Matousek via

    Flickr •  Access to personalised notifications and support e.g. highlighting/suggesting resources that have not been viewed. •  Increased levels of engagement, in particular VLE usage. •  Personalisation of cohort module delivery by the lecturer •  Real-time feedback for students enabling them to judge their progress during a module using a different metric than current models of formative and summative feedback. •  Direct involvement with the development of tools that support their learning.
  11. 1 Being the best you can be/Effort (ability to maintain

    effort) 2 Build self-confidence 3 Career/Vocation/Job prospects 4 Industry 5 Giving yourself options 6 Grades/Marks/Qualifications 7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually 8 Mentoring/Family 9 Money 10 Part of a Professional community 11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy) 12 Sense of connectedness with others with similar goals/ Success as a group of peers 13 Social Prestige/Recognition Motivators for Studying in HE (Literature) Student Ambassadors
  12. Laddering Sessions run by Student Ambassadors 10

  13. Looks how they feel Shows how hard they are trying

    No point in spending £9,000 if you’re not going to try hard and do well ↑ ↑ “When thinking about what motivates you to study your degree, which of these do you prefer and why?”. “Why..?” “Why..?” Laddering A B
  14. 14 Supporting family/Home life 15 Negative imagery 16 Rising above

    circumstances 17 Financial security 18 Controlling own destiny 19 Individuality 20 Opportunity to travel 21 Fear of wasting University opportunity Additional Motivators for Studying in HE Laddering Sessions run by Student Ambassadors 10
  15. Revised Motivators (21 grouped into 9) Career/Industry Mastery Money Attainment

    Options Self Development Family Fear of … Professional Community
  16. None
  17. None
  18. Timeline Formed basis of 6 Focus Groups with 20 students

    organised and run by Student Ambassadors
  19. None
  20. 14 Features used to create model Total number of content

    accesses Total duration of content accesses Average duration of content accesses Number of days per week content is accessed Average length of time between content accesses Average group size of simultaneous content accesses Number of times specific content is accessed Number of clicks on content of a certain type Number of clicks grouped by parent folder Seen/unseen content Total number of lecture capture views Number of times specific lectures are viewed Grades Absences From KLE click data From lecture capture data From SCIMS data
  21. Model to predict “Grade” using Decision Trees 5 = 1st

    4 = 2:1 3 = 2:2 2 = 3rd 1 = Fail
  22. Career/Industry Grade Absences Total clicks Money Grade Absences Total clicks

    Attainment Grade Mastery % of content seen Grade Variety of content clicked Options Grade Absences Total clicks Professional community Simultaneous users Family Grade Absences Total clicks Self development Total clicks Total duration Variety of content clicked Fear of failure Absences % of content seen Creating Scores for each Motivator
  23. V2 Version 2 based on student feedback on Version 1

    from 10 one-to-one contextual interviews V1
  24. V2

  25. V2

  26. V2

  27. Personal/Personified Theme

  28. V2

  29. V2

  30. V2

  31. V2

  32. V2

  33. Representing Motivator Scores 1 2 3 4 5

  34. V2

  35. Comparison with peers Both Themes

  36. Mobile Optimised

  37. The LA system was successfully integrated into the delivery of

    4 Undergraduate modules in Computer Science & Geography 169 student sign-ups 48% of students on the 4 modules
  38. None
  39. 41% 59%

  40. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

    100% career money attainment self development fear of failure mastery options family professional community
  41. Weekly Emails

  42. Interventions Text based on decision trees

  43. Lecturer Dashboard

  44. Learning Analytics in the Classroom Using the visualisations to start

    discussions
  45. Most/least popular content week Content 1 2 3 4 5

    6 7 8 9 10 Mean Total Max Lectures/Part 2 (weeks 4-11): Landsystems case studies 0 0 0 38 35 34 29 35 23 30 22.4 224 38 Lectures/Part 1 (weeks 2-3): Introduction to Geomorphology (PGK) 21 33 32 12 8 2 8 5 1 5 12.7 127 33 Assessments/Blog resources 4 4 13 16 18 15 10 12 7 17 11.6 116 18 Seminars/Week 4: Permafrost Environments 0 0 32 42 0 6 1 2 1 4 8.8 88 42 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 6-7: Desert Environments (RIW) 0 0 0 0 0 30 23 12 7 4 7.6 76 30 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 4-5: Permafrost Environments (RIW) 0 0 0 38 0 13 8 7 2 5 7.3 73 38 Lectures/Week 1: Introduction to the module and landsystems (RIW) 27 12 11 7 0 4 5 2 0 2 7 70 27 Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 1st Lecture: Geomorphology 1 17 24 12 4 0 1 5 3 0 2 6.8 68 24 Lectures/Part-1-weeks-2-3-Introduction-to-Geomorphology-PGK/PGK's 2nd Lecture: Geomorphology 2 8 29 15 5 0 0 4 3 0 1 6.5 65 29 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks 8-9: Dry Tropical Environments (MM) 0 0 0 0 0 0 0 33 20 10 6.3 63 33 week Content 1 2 3 4 5 6 7 8 9 10 Mean Total Max Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-ACL/ Week 11 - Thermokarst Lakes 0 0 0 0 0 0 0 0 0 0 0 0 0 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/ Lecture-10-Geotechnical-Challenges-Engineering-Solutions/YouTube videos 0 0 0 0 0 1 0 0 0 0 0.1 1 1 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/ Lecture-9-Active-Layer-Azonal-Processes/Permafrost Coasts 0 0 0 1 0 1 0 0 0 1 0.3 3 1 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/ Lecture-9-Active-Layer-Azonal-Processes/Permafrost Rivers 0 0 0 2 0 2 0 0 0 0 0.4 4 2 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/ Lecture-10-Geotechnical-Challenges-Engineering-Solutions/NICOP literature 0 0 0 0 4 1 0 1 0 0 0.6 6 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-4-5-Permafrost-Environments-RIW/ Lecture-7-An-Introduction-to-Permafrost-Environments/Videos/Take off from Clyde River 0 0 0 4 2 0 0 0 0 0 0.6 6 4 Lectures/Part-2-weeks-4-11-Landsystems-case-studies/Weeks-10-11-Lacustrine-Environments-ACL/ Week 11 - Proglacial lakes 2 0 0 0 0 0 0 0 0 0 7 0.7 7 7 Learning Analytics for Curriculum Design/Module Review?
  46. Students showed increased engagement with all modules compared to students

    from the previous year’s cohort by: clicking more in the VLE (between 4% and 20% more) having fewer absences (150% - 200% improvement) Students showed improved engagement with version 2: 87% and 89% of students on semester 2 modules logged into the dashboard at least once vs 71% and 74% in semester 1 Evaluation Results 169 students sign-ups Average weekly usage: 51% of students viewed the email 53% of students (who viewed the email) clicked through to the VLE 22% of students viewed the report on the KLE
  47. Q1. Would you say a decline in your weekly score

    would influence your behaviour in the following weeks? Evaluation Results Yes - 57% No - 20% Maybe - 23% Questionnaire - 35 students “It did upset me, so I tried harder” “I tend to get lazier as the semester goes on, having a reminder that I am actually doing less work is good.” “Feels like someone is watching how much work I am doing” “I want to feel the satisfaction of a good week” “It helps you realise roughly how much work you are doing” “Don't want to simply be a perfunctory student.” “I didn't see a change in score.” “I think it is discouraging if the analytics indicate a predicted grade of lower than what you expect (and know) you will achieve” “I’m aware of when and why I stop interacting with the VLE, analytics cant understand real world circumstances” “I don’t think measuring performance based on VLE activity alone is a good way to measure it”
  48. Q2. Do you believe the data that is presented in

    the dashboard accurately reflects your engagement? Evaluation Results Strongly agree Agree Slightly agree Neither agree or disagree Slightly disagree Disagree Strongly disagree 0% 5% 10% 15% 20% 25% 30% Questionnaire - 35 students
  49. Q3. Do you feel you can use the system as

    a tool to enable you to improve your grade performance ? Evaluation Results Yes - 54% No - 23% Maybe - 23% Questionnaire - 35 students “Gives weekly adjusted feedback, keeps you on your toes academically” “Tells me what I’m doing and what I can do to improve (i.e spend more time looking at certain resources etc.)” “Gives a clear idea on what you need to do more of when studying” “It somewhat held me accountable” “Results show you how well you are performing and if you need to get more work done.” “If there were a metric measuring attention during lectures, I would feel more assured.” “The feedback given is not particularly in-depth and as such it would be difficult to improve specific areas of my course.” “Needs more inputs perhaps a way for a lecturer to perhaps give there thoughts on that students performance every now and then. “Needs better recommendations” “Whilst the use of the KLE is a way of showing work; it doesn’t necessarily reveal how much other work you have done.“
  50. Q. Before using the system, were you aware of your

    engagement with the KLE and your attendance? Most students aware of attendance but not VLE data. Q. Can you describe your understanding of how you can influence the outcome of your learning analytics? How would you improve? All students said that viewing more content/regular VLE interaction and attending more sessions would improve scores. Q. What information have you found most useful when using the learning analytics system? -  Number of clicks and the cohort comparison -  Scores/Predictions/Stats breakdown -  % content seen -  Week-to-week outlook Evaluation Results 10 one-to-one Contextual Interviews whilst viewing their own data in the LA tool
  51. Q. What information have you found least useful when using

    the learning analytics system? Most students were unsure about the accuracy of the grade predictions and weekly “swings” in prediction. Q. Can you describe how the learning analytics system impacted your motivation? -  Made students reflect on a weekly/regular basis instead of only working hard around assessment time. -  When saw grades go down made them wanted to improve grade/ gave extra boost (although one student said “Seeing high scores during the middle of the semester incentivised them to ‘cruise’.”) -  Made them more keen to go on the VLE and look around to see missed content/make themselves a better student. -  Reinforced effort e.g. ”good seeing how making an effort yielded good results for a specific week, showing that if you put the effort in, you would likely perform better”. Evaluation Results 10 one-to-one Contextual Interviews whilst viewing their own data in the LA tool
  52. Q. How has the learning analytics system changed your awareness

    of your progress? -  Gives feedback throughout the semester/gives weekly visibility meaning you don’t have to wait until the end of the module. -  The lecturer visualisation slides that were shown in class were useful - made them aware of resources that they hadn’t looked at. -  Being able to compare activity to the cohort was useful e.g. “when the average dipped but I maintained consistency.” -  Seeing a drop in performance on on the dashboard “kick-started” them to do more. e.g. “Without a system like this, although I would maybe know I wasn’t engaging as much as I should, I probably wouldn’t act on it." Q. In general, do you feel that the learning analytics system has had a direct effect on your performance on the module? How? Most students said positive (although “slightly”). Mentioned that it would have more impact on “harder” modules. Evaluation Results 10 one-to-one Contextual Interviews whilst viewing their own data in the LA tool
  53. Academics 4 Student Ambassadors 4 Students in User Research Sessions

    40 Evaluation Questionnaire Respondents 35 Sign-ups to the LA Tool 169
  54. Dr Ed de Quincey e.de quincey@keele.ac.uk Chris Briggs c.briggs@keele.ac.uk KEELE

    UNIVERSITY SCHOOL OF COMPUTING MOTIVATION METRICS V6 ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK CEO