Student-centred design of LA systems

Presentation at the 11th UK Learning Analytics Network meeting, Aston University, Birmingham, 5th Sept 2017.

The interaction and interface design of Learning Analytics (LA) systems is often based upon the ability of the developer to extract information from disparate sources and not on the types of data and interpretive needs of the user. Current systems also tend to focus on the educator’s view and very rarely involve students in the development process. In this project we will be placing the learner at the centre by training student ambassadors in user-centred design techniques to find out what motivates their peers to study and how this can be incorporated into the design of a LA tool. Data Mining techniques will be used to build models of student behavior from VLE usage data and other relevant sources so that a LA tool can be developed and trialed on modules across the faculty. Students will be given access to a personalised representation of their progress in real-time, taking into account what motivates them to study. We will also investigate how LA can be incorporated into the delivery of modules with the key aims of increasing engagement, making the VLE a more active space for learning and teaching and bridging the current gap between physical and digital spaces.

Transcript

1. Student-centred design of LA systems
   Dr Ed de Quincey & Chris Briggs School of Computing and Mathematics, Keele University
   11th UK Learning Analytics Network Meeting
   

   View Slide

2. Dr Ed de Quincey @eddequincey
   Senior Lecturer in Computer Science, UG and PG Course Director
   School of Computing and Mathematics, Keele University
   Senior Fellow of the HEA
   instagram.com/eddequincey
   

   View Slide

3. Chris Briggs @jooldesign
   Research Software Engineer in Learning Analytics
   School of Computing and Mathematics, Keele University
   instagram.com/jooldesign/
   

   View Slide

4. View Slide

5. 
   
   

   View Slide

6. View Slide

7. WP1. KLE Data Review
   
   
   • Inconsistent formats
   • Reports take a long time to run
   • No way to bulk download all “raw” data
   • Student Overview for a Single Course is the most useful BUT
   have to run for each student
   Around 11 “reports” available
   9 Course Reports, a Performance Dashboard and a Retention Centre
   

   View Slide

8. Blackboard Analytics
   

   View Slide

9. Apple Lisa
   PRESENTING STUDENT DATA BACK TO STUDENTS and LECTURERS,
   USING USER CENTRIC QUERIES, FORMATS and METAPHORS
   

   View Slide

10. Identified and Reviewed 22 Learning Analytics Systems
    

    View Slide

11. Review of Existing Systems
    Few systems available for general use
    
    Primarily targeted at educators. Only 5 of the 22
    systems being designed purely for use by students
    Very few studies gathered the requirements for the
    systems directly from students
    Currently analysing visualisation techniques used
    and types/sources of data used
    

    View Slide

12. User-Centered Design Process Map
    http://www.usability.gov/how-to-and-tools/resources/ucd-map.html
    

    View Slide

13. WP3. Requirements Gathering
    Set as Coursework Case Study:
    Identify GUI metaphors that will engage and motivate
    them as learners and personalise their own learning
    experience
    2nd Year Computing Module - 82 students in 14 groups
    Deliverables included:
    Sets of User Persona, analysed results of requirements
    elicitation sessions and annotated screen mock-ups of
    potential LA Dashboards
    (highlighting 5 key features along with objective justifications wherever possible).
    

    View Slide

14. 'My Timeline': Past events which have been completed successfully provide
    gratification to the user in the form of positive icons such as thumbs up or smiley faces.
    The past timeline balances with the upcoming events to try and alleviate future workload
    stress by demonstrating positive success at the same time.
    

    View Slide

15. Course/Module Progress Bar: a progress bar based upon the how far in the course
    users are. Indicating how much of the course they should know and how much time is
    left until the course finishes.
    

    View Slide

16. Score Indicator: The score indicator is a metric derived from an algorithm
    which would track a student’s engagement with course materials and
    other important areas of engagement that students should be utilising
    (comments, discussion).
    

    View Slide

17. Degree Classification Requirements: Panel showing the Percentage
    needed in Future Assignments to get certain Degree Classifications.
    

    View Slide

18. Comparison with peers: The chance for the user to see how
    they are comparing with the top 20% of the class and how they
    are doing compared with the average mark.
    

    View Slide

19. Goals and Trophies: The student will get a list of goals or tasks
    sent to their dashboard and there they can also add more tasks
    of their own.
    

    View Slide

20. HEFCE Catalyst Grant £99,790 (£49,988 from Catalyst Fund, £49,802 in matched funding from Keele)
    Title: Learner Centred Design for Learning Analytics
    This project aims to avoid the common problem in Learning Analytics (LA) of the technology and data driving the
    user experience, and therefore the ability to interpret and use the information. By sharing the data directly
    with students, using student-centred representations of their learning activity, this project aims to facilitate a
    common understanding of the learning experience between lecturers and students. Expanding on a successful
    teaching innovation project at Keele University interface metaphors for LA will be identified that motivate and
    personalise the learning experience of cohorts with differing levels of technical experience and levels of digital
    literacy. We will then produce appropriate visualisations of student activity based on the data available at Keele
    University and incorporate them into the delivery of relevant modules with the key aims of
    increasing engagement, making the VLE a more active space for learning and teaching and bridging the current
    gap between physical and digital spaces.
    

    View Slide

21. Outcomes for Students
    Students in lecture hall ©Jirka Matousek via Flickr
    • Access to personalised notifications and support
    e.g. highlighting/suggesting resources that have not been viewed.
    • Increased levels of engagement, in particular VLE usage.
    • Personalisation of cohort module delivery by the lecturer
    • Real-time feedback for students enabling them to judge
    their progress during a module using a different metric
    than current models of formative and summative feedback.
    • Direct involvement with the development of tools that
    support their learning.
    

    View Slide

22. Student Ambassadors
    

    View Slide

23. 1 Being the best you can be/Effort (ability to maintain effort)
    2 Build self-confidence
    3 Career/Vocation/Job prospects
    4 Industry
    5 Giving yourself options
    6 Grades/Marks/Qualifications
    7 Mastery of a subject/Interest in Subject/Stretch themselves intellectually
    8 Mentoring/Family
    9 Money
    10 Part of a Professional community
    11 Self-efficacy/ Helplessness (this might be the opposite of self-efficacy)
    12
    Sense of connectedness with others with similar goals/ Success as a group of
    peers
    13 Social Prestige/Recognition
    Initial Identified Motivators for Studying in HE
    

    View Slide

24. View Slide

25. View Slide

26. Looks how they feel
    Shows how hard they are
    trying
    No point in spending
    £9,000 if you’re not going
    to try hard and do well
    A B
    ↑
    ↑
    “When thinking about what motivates you to study your
    degree, which of these do you prefer and why?”.
    “Why..?”
    “Why..?”
    Laddering
    

    View Slide

27. 14 Supporting family/Home life
    15 Negative imagery
    16 Rising above circumstances
    17 Financial security
    18 Controlling own destiny
    19 Individuality
    20 Opportunity to travel
    21 Fear of wasting University opportunity
    Additional Identified Motivators for
    Studying in HE from Laddering Sessions
    

    View Slide

28. Revised Motivators
    (21 grouped into 9)
    

    View Slide

29. Timeline
    Formed basis of 6 Focus Groups organised and run by Student Ambassadors
    

    View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. Professional Theme
    

    View Slide

35. Personal/Personified Theme
    

    View Slide

36. Personal/Personified Theme
    

    View Slide

37. Mobile Optimised
    

    View Slide

38. Work in Progress
    

    View Slide

39. Learning Analytics in the classroom?
    Students in lecture hall ©Jirka Matousek via Flickr
    

    View Slide

40. Dr Ed de Quincey e.de quincey@keele.ac.uk
    Chris Briggs c.briggs@keele.ac.uk
    KEELE UNIVERSITY SCHOOL OF COMPUTING
    MOTIVATION METRICS V6
    ALL DESIGNS ARE COPYRIGHT 2017 OF MORE THAN JUST DESIGN LIMITED WWW.MORETHANJUSTDESIGN.CO.UK
    CEO
    

    View Slide