$30 off During Our Annual Pro Sale. View Details »

A service design approach to improving student experience

A service design approach to improving student experience

Research had shown us that University of Edinburgh students were frustrated by inconsistencies between different courses in their virtual learning environment. So we embarked on a programme of user research with students. However, for most of our findings about students, we found that something about staff was driving that experience. By adopting a service design approach, we have built a stronger understanding of this complex ecosystem of interactions, leaving us better placed to improve student experience.

For more detailed blog posts see https://edin.ac/uxlf

Presented at UCD Gathering virtual conference.
https://www.ucdgathering.net/programme/service-design-approach-improving-student-experience

Duncan Stephen

October 15, 2020
Tweet

More Decks by Duncan Stephen

Other Decks in Education

Transcript

  1. Duncan Stephen
    A service design
    approach to improving
    student experience
    User Experience Manager
    The University of Edinburgh
    duncanstephen.net — @DuncanBSS

    View Slide

  2. I will be happy to take
    questions at the end
    duncanstephen.net — @DuncanBSS

    View Slide

  3. duncanstephen.net — @DuncanBSS
    Foundations
    Delivering an enhanced student learning experience online

    View Slide

  4. Students are frustrated by
    inconsistencies across
    digital systems
    duncanstephen.net — @DuncanBSS

    View Slide

  5. duncanstephen.net — @DuncanBSS
    Research by Headscape, 2016

    View Slide

  6. With course materials online…
    Individual schools
    Individual lecturers
    Individual approaches
    duncanstephen.net — @DuncanBSS

    View Slide

  7. Some schools already had
    their own templates
    These templates
    had 168 different
    top-level
    menu items
    duncanstephen.net — @DuncanBSS
    Analysis by Paul Smyth

    View Slide

  8. Answer =
    “Make it consistent”
    …Obviously
    duncanstephen.net — @DuncanBSS

    View Slide

  9. Consistency
    duncanstephen.net — @DuncanBSS
    Source image: moojj
    https://imgur.com/gallery/EFA8dLh

    View Slide

  10. 45,000 students
    8,000 academic staff
    24 schools and deaneries
    …all with different requirements
    duncanstephen.net — @DuncanBSS

    View Slide

  11. Solution focus
    ⟶Checklist
    ⟶Terminology
    ⟶Templates
    ⟶Training
    ⟶Automation
    ⟶QA and audit
    duncanstephen.net — @DuncanBSS

    View Slide

  12. A comprehensive
    programme of user research,
    adopting multiple methods
    duncanstephen.net — @DuncanBSS

    View Slide

  13. Overall goal to improve
    student experience
    But we knew that
    staff experience was
    also fundamental
    duncanstephen.net — @DuncanBSS

    View Slide

  14. Interviews with
    students and staff
    duncanstephen.net — @DuncanBSS

    View Slide

  15. A semester in the life of
    students using Learn
    duncanstephen.net — @DuncanBSS

    View Slide

  16. A semester in the life of
    students using Learn
    duncanstephen.net — @DuncanBSS
    Week 1
    Getting your
    head around
    Learn
    Week 6
    Assignment
    submission
    Week 7
    Different
    lecturers doing
    different things
    Week 8
    Getting
    feedback

    View Slide

  17. Seeing the
    connections
    with staff
    members’
    use of Learn
    duncanstephen.net — @DuncanBSS

    View Slide

  18. duncanstephen.net — @DuncanBSS
    Students’ experience
    Frustrations
    around being
    made to use
    templates
    Love using Learn
    as a comms tool
    Time consuming
    to give good
    feedback
    Staff members’ experience
    Lack of trust in
    existing
    templates
    Overwhelmed by
    notifications
    Inconsistencies
    around
    assignments and
    feedback

    View Slide

  19. Programme of targeted
    quantitative research
    duncanstephen.net — @DuncanBSS

    View Slide

  20. Quantitative research
    specifically designed to
    inform the development of
    the new template
    duncanstephen.net — @DuncanBSS

    View Slide

  21. duncanstephen.net — @DuncanBSS
    Sketch and plan by Neil Allison

    View Slide

  22. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students

    View Slide

  23. Phenomenal response rates
    Almost 5,000 responses
    across five studies
    duncanstephen.net — @DuncanBSS

    View Slide

  24. We learnt a lot about
    user research incentives
    Give people a small chance of
    a medium-sized incentive
    duncanstephen.net — @DuncanBSS

    View Slide

  25. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students

    View Slide

  26. Top tasks
    methodology
    Gerry McGovern
    duncanstephen.net — @DuncanBSS
    Long tail
    Long neck

    View Slide

  27. duncanstephen.net — @DuncanBSS
    0
    100
    200
    300
    400
    500
    600
    Students’ top tasks –
    the long neck and long tail

    View Slide

  28. Students’ top tasks
    duncanstephen.net — @DuncanBSS
    Lecture slides
    Lecture notes/outlines/handouts
    Lecture recordings
    Past exam papers
    Top 3 tasks
    >25% of votes
    Next 4 tasks
    Almost half
    Next 7 tasks
    Final 31 tasks 1. Lecture slides
    2. Lecture notes /
    outlines / handouts
    3. Lecture recordings
    4. Past exam papers
    5. Grades
    6. Reading lists / study
    resources
    7. Assignment deadlines
    8. Assignment feedback / feed
    forward

    View Slide

  29. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students

    View Slide

  30. 775 responses from students
    Every response was unique
    duncanstephen.net — @DuncanBSS

    View Slide

  31. duncanstephen.net — @DuncanBSS
    Course materials
    Assessment
    Course information
    Help and support

    View Slide

  32. duncanstephen.net — @DuncanBSS
    Course materials
    Assessment
    Course information
    Help and support

    View Slide

  33. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students

    View Slide

  34. Gerry McGovern on testing a
    new structure for the first time:
    “You will have maybe a
    60% success rate
    if you do it really well.”
    duncanstephen.net — @DuncanBSS

    View Slide

  35. Our success rates ranged
    from 67% to 93%
    duncanstephen.net — @DuncanBSS

    View Slide

  36. Worst performing task
    You need to contact a teacher.
    Where would you find their email address?
    67% success
    59% directness
    duncanstephen.net — @DuncanBSS

    View Slide

  37. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students

    View Slide

  38. Research suggests that
    when users get the first click right,
    87% ultimately succeed in a task
    versus 46% who get the first click wrong
    Jeff Sauro, Getting the first click right
    https://measuringu.com/first-click/
    duncanstephen.net — @DuncanBSS

    View Slide

  39. We A/B tested the
    new prototype against
    existing Learn courses
    duncanstephen.net — @DuncanBSS

    View Slide

  40. These results were
    less positive
    But we learned some really
    valuable information
    duncanstephen.net — @DuncanBSS

    View Slide

  41. It’s coming up to exam time and you’d like to see some
    mock exam papers. Where will you find them?
    30% selected
    Course materials
    21% selected
    Resource list
    42% selected
    Assessment
    duncanstephen.net — @DuncanBSS

    View Slide

  42. Past papers puzzle
    duncanstephen.net — @DuncanBSS

    View Slide

  43. Past papers
    were the
    4th top task
    duncanstephen.net — @DuncanBSS

    View Slide

  44. But they were
    not clearly
    grouped in
    the card sort
    duncanstephen.net — @DuncanBSS

    View Slide

  45. We placed it in the
    Assessment
    section
    But students
    were not
    finding it
    duncanstephen.net — @DuncanBSS

    View Slide

  46. Method triangulation
    to the rescue
    duncanstephen.net — @DuncanBSS

    View Slide

  47. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    Ongoing usability testing

    View Slide

  48. duncanstephen.net — @DuncanBSS

    View Slide

  49. How usability testing is
    shaping the changes to our
    virtual learning environment
    duncanstephen.net — @DuncanBSS

    View Slide

  50. Monthly cycle of
    usability testing
    duncanstephen.net — @DuncanBSS

    View Slide

  51. The most successful way to
    improve your design is to
    watch your users
    duncanstephen.net — @DuncanBSS

    View Slide

  52. “The closest thing we’ve found to a
    silver bullet when it comes to reliably
    improving the designs teams produce.”
    Jared Spool
    https://articles.uie.com/user_exposure_hours/
    duncanstephen.net — @DuncanBSS

    View Slide

  53. “User research is a team sport”
    “All team members should watch real
    users interacting with your service and
    talking about it — ideally for at least 2
    hours every 6 weeks.”
    Dipa Shah
    Government Digital Service User research community
    www.gov.uk/service-manual/user-research/how-user-research-improves-service-design
    duncanstephen.net — @DuncanBSS

    View Slide

  54. Usability testing has brought
    staff closer to our students
    duncanstephen.net — @DuncanBSS

    View Slide

  55. But usability testing
    isn’t all plain sailing
    duncanstephen.net — @DuncanBSS

    View Slide

  56. Usability testing is
    pretty easy to get
    your head around
    But there are still
    some challenges
    duncanstephen.net — @DuncanBSS

    View Slide

  57. Challenge 1
    Getting the go-ahead from
    your bosses to spend time on
    usability testing
    duncanstephen.net — @DuncanBSS

    View Slide

  58. Challenge 2
    Getting your colleagues
    to take on board
    what is uncovered
    by usability testing
    duncanstephen.net — @DuncanBSS

    View Slide

  59. Challenge 3
    Getting fixes to those
    problems implemented
    duncanstephen.net — @DuncanBSS

    View Slide

  60. How our approach to
    usability testing solves
    those challenges
    duncanstephen.net — @DuncanBSS

    View Slide

  61. Step 1
    Conduct usability testing
    Each cycle we recorded
    four or five students or
    staff members attempting
    to complete some
    common, representative
    tasks in Learn
    duncanstephen.net — @DuncanBSS

    View Slide

  62. We test with as many people as we can
    — which is not many
    But that’s OK because
    usability testing is a
    qualitative approach
    “The most striking truth of
    the curve is that zero users
    give zero insights.”
    Jakob Nielsen
    www.nngroup.com/articles/
    why-you-only-need-to-test-with-5-users/
    duncanstephen.net — @DuncanBSS

    View Slide

  63. Step 2
    Showcase the videos
    Open invite across the
    university to raise
    awareness among our
    community of learning
    technologists
    duncanstephen.net — @DuncanBSS

    View Slide

  64. Step 3
    Attendees witness usability issues
    Session attendees note
    down the usability issues
    they see while watching
    the videos
    Focus on issues, not
    potential solution ideas yet
    duncanstephen.net — @DuncanBSS

    View Slide

  65. Step 4
    Discuss each video in groups
    After each video,
    participants discuss for
    5 minutes in small groups
    People decide on their
    top 3 issues from each video
    duncanstephen.net — @DuncanBSS

    View Slide

  66. Step 5
    Collaboratively prioritise issues
    Across the whole room,
    people raise the usability
    issues they’ve seen
    Then together we work each
    issue through a
    prioritisation flowchart
    duncanstephen.net — @DuncanBSS

    View Slide

  67. Red route prioritisation
    How to prioritise
    usability problems
    David Travis
    www.userfocus.co.uk/articles/prioritise.html
    duncanstephen.net — @DuncanBSS

    View Slide

  68. Red route prioritisation
    Red routes are roads that
    are too important to get
    clogged up
    Think about key users
    and their top tasks
    in the same way
    duncanstephen.net — @DuncanBSS

    View Slide

  69. 3 yes/no questions
    Does the problem occur on a red route?
    Is the problem difficult for the user to overcome?
    Is the problem persistent?
    Yes to 3 questions = Critical usability issue
    Yes to 2 questions = Serious
    Yes to 1 question = Medium
    Yes to 0 questions = Low
    duncanstephen.net — @DuncanBSS

    View Slide

  70. duncanstephen.net — @DuncanBSS

    View Slide

  71. Step 6
    Impact/effort matrix
    Now we can start thinking
    about solutions, focusing on
    the most critical problems
    duncanstephen.net — @DuncanBSS

    View Slide

  72. Step 6
    Impact/effort matrix
    duncanstephen.net — @DuncanBSS
    Minor issue
    for users
    Major issue
    for users
    Easy
    solution
    available
    No easy
    solution
    available
    Tackle these first

    View Slide

  73. Why we prioritise in this way
    duncanstephen.net — @DuncanBSS

    View Slide

  74. “Running usability
    testing has been
    compared with
    taking a drink from
    a fire hydrant”
    David Travis
    www.userfocus.co.uk/articles/prioritise.html
    duncanstephen.net — @DuncanBSS

    View Slide

  75. The red route
    method moves us
    beyond prioritising
    by gut feel
    duncanstephen.net — @DuncanBSS

    View Slide

  76. Outcomes of usability testing
    duncanstephen.net — @DuncanBSS

    View Slide

  77. 31 usability tests with students and staff
    7 different Learn environments
    9 showcase sessions
    20–30 attendees at most sessions
    102 usability issues identified and prioritised
    duncanstephen.net — @DuncanBSS

    View Slide

  78. “A really valuable and informative
    process that we are hoping to embed
    across the school and the upcoming
    projects we are undertaking.”
    Emma Forrest
    Student Services Project Officer, School of GeoSciences
    duncanstephen.net — @DuncanBSS

    View Slide

  79. “Being able to see what works well and
    what is more challenging for users
    enables us to prioritise what really
    matters and ensure our design is
    reflective of our users’ needs.”
    Lee-Ann Simpson
    Learn Foundations Project Manager
    duncanstephen.net — @DuncanBSS

    View Slide

  80. Qualitative research was an
    essential complement to
    quantitative research
    duncanstephen.net — @DuncanBSS

    View Slide

  81. Past papers puzzle
    duncanstephen.net — @DuncanBSS

    View Slide

  82. Past papers
    were the
    4th top task
    duncanstephen.net — @DuncanBSS

    View Slide

  83. But they were
    not clearly
    grouped in
    the card sort
    duncanstephen.net — @DuncanBSS

    View Slide

  84. We placed it in the
    Assessment
    section
    But students
    were not
    finding it
    duncanstephen.net — @DuncanBSS

    View Slide

  85. Method triangulation
    to the rescue
    duncanstephen.net — @DuncanBSS

    View Slide

  86. By watching just 4 students through
    usability testing, we understood that
    they don’t look for past papers in Learn
    They Google for them
    duncanstephen.net — @DuncanBSS

    View Slide

  87. Without using a mixture of
    research methods, we would
    never have understood this
    duncanstephen.net — @DuncanBSS

    View Slide

  88. A service design approach
    duncanstephen.net — @DuncanBSS

    View Slide

  89. duncanstephen.net — @DuncanBSS
    Audience Front stage Backstage Behind-the-
    scenes
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c

    View Slide

  90. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c

    View Slide

  91. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c

    View Slide

  92. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    We’re rolling out
    enhanced
    announcements
    functionality!
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c

    View Slide

  93. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    We’re rolling out
    enhanced
    announcements
    functionality!
    I have an
    announcement!
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c

    View Slide

  94. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    We’re rolling out
    enhanced
    announcements
    functionality!
    I have an
    announcement!
    Not another
    announcement…
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c

    View Slide

  95. By conducting research
    across this entire
    ecosystem, we are
    better placed to improve
    students’ experience
    duncanstephen.net — @DuncanBSS

    View Slide

  96. Then… coronavirus
    duncanstephen.net — @DuncanBSS

    View Slide

  97. Schools that had adopted the
    Learn Foundations template
    early adapted to online
    delivery more easily
    Other schools needed more support
    duncanstephen.net — @DuncanBSS

    View Slide

  98. Learn Foundations rollout 2020/2021
    Plan:
    10 schools
    After coronavirus:
    20 schools (out of 24)
    duncanstephen.net — @DuncanBSS

    View Slide

  99. “Consistent templates paved the way
    for a seamless move into hybridity in a
    time of crisis.”
    Jon Jack
    Learning Technology Team Manager
    duncanstephen.net — @DuncanBSS

    View Slide

  100. Find out more on our blog:
    edin.ac/uxlf
    duncanstephen.net — @DuncanBSS

    View Slide

  101. Duncan Stephen
    Thank you
    edin.ac/uxlf
    duncanstephen.net — @DuncanBSS

    View Slide

  102. Duncan Stephen
    Questions?
    edin.ac/uxlf
    duncanstephen.net — @DuncanBSS

    View Slide