A service design approach to improving student experience

Research had shown us that University of Edinburgh students were frustrated by inconsistencies between different courses in their virtual learning environment. So we embarked on a programme of user research with students. However, for most of our findings about students, we found that something about staff was driving that experience. By adopting a service design approach, we have built a stronger understanding of this complex ecosystem of interactions, leaving us better placed to improve student experience.

For more detailed blog posts see https://edin.ac/uxlf

Presented at UCD Gathering virtual conference.
https://www.ucdgathering.net/programme/service-design-approach-improving-student-experience

Transcript

1. Duncan Stephen
   A service design
   approach to improving
   student experience
   User Experience Manager
   The University of Edinburgh
   duncanstephen.net — @DuncanBSS
   ⟶
   

   View Slide

2. I will be happy to take
   questions at the end
   duncanstephen.net — @DuncanBSS
   

   View Slide

3. duncanstephen.net — @DuncanBSS
   Foundations
   Delivering an enhanced student learning experience online
   

   View Slide

4. Students are frustrated by
   inconsistencies across
   digital systems
   duncanstephen.net — @DuncanBSS
   

   View Slide

5. duncanstephen.net — @DuncanBSS
   Research by Headscape, 2016
   

   View Slide

6. With course materials online…
   Individual schools
   Individual lecturers
   Individual approaches
   duncanstephen.net — @DuncanBSS
   

   View Slide

7. Some schools already had
   their own templates
   These templates
   had 168 different
   top-level
   menu items
   duncanstephen.net — @DuncanBSS
   Analysis by Paul Smyth
   

   View Slide

8. Answer =
   “Make it consistent”
   …Obviously
   duncanstephen.net — @DuncanBSS
   

   View Slide

9. Consistency
   duncanstephen.net — @DuncanBSS
   Source image: moojj
   https://imgur.com/gallery/EFA8dLh
   

   View Slide

10. 45,000 students
    8,000 academic staff
    24 schools and deaneries
    …all with different requirements
    duncanstephen.net — @DuncanBSS
    

    View Slide

11. Solution focus
    ⟶Checklist
    ⟶Terminology
    ⟶Templates
    ⟶Training
    ⟶Automation
    ⟶QA and audit
    duncanstephen.net — @DuncanBSS
    

    View Slide

12. A comprehensive
    programme of user research,
    adopting multiple methods
    duncanstephen.net — @DuncanBSS
    

    View Slide

13. Overall goal to improve
    student experience
    But we knew that
    staff experience was
    also fundamental
    duncanstephen.net — @DuncanBSS
    

    View Slide

14. Interviews with
    students and staff
    duncanstephen.net — @DuncanBSS
    

    View Slide

15. A semester in the life of
    students using Learn
    duncanstephen.net — @DuncanBSS
    

    View Slide

16. A semester in the life of
    students using Learn
    duncanstephen.net — @DuncanBSS
    Week 1
    Getting your
    head around
    Learn
    Week 6
    Assignment
    submission
    Week 7
    Different
    lecturers doing
    different things
    Week 8
    Getting
    feedback
    

    View Slide

17. Seeing the
    connections
    with staff
    members’
    use of Learn
    duncanstephen.net — @DuncanBSS
    

    View Slide

18. duncanstephen.net — @DuncanBSS
    Students’ experience
    Frustrations
    around being
    made to use
    templates
    Love using Learn
    as a comms tool
    Time consuming
    to give good
    feedback
    Staff members’ experience
    Lack of trust in
    existing
    templates
    Overwhelmed by
    notifications
    Inconsistencies
    around
    assignments and
    feedback
    

    View Slide

19. Programme of targeted
    quantitative research
    duncanstephen.net — @DuncanBSS
    

    View Slide

20. Quantitative research
    specifically designed to
    inform the development of
    the new template
    duncanstephen.net — @DuncanBSS
    

    View Slide

21. duncanstephen.net — @DuncanBSS
    Sketch and plan by Neil Allison
    

    View Slide

22. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    

    View Slide

23. Phenomenal response rates
    Almost 5,000 responses
    across five studies
    duncanstephen.net — @DuncanBSS
    

    View Slide

24. We learnt a lot about
    user research incentives
    Give people a small chance of
    a medium-sized incentive
    duncanstephen.net — @DuncanBSS
    

    View Slide

25. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    

    View Slide

26. Top tasks
    methodology
    Gerry McGovern
    duncanstephen.net — @DuncanBSS
    Long tail
    Long neck
    

    View Slide

27. duncanstephen.net — @DuncanBSS
    0
    100
    200
    300
    400
    500
    600
    Students’ top tasks –
    the long neck and long tail
    

    View Slide

28. Students’ top tasks
    duncanstephen.net — @DuncanBSS
    Lecture slides
    Lecture notes/outlines/handouts
    Lecture recordings
    Past exam papers
    Top 3 tasks
    >25% of votes
    Next 4 tasks
    Almost half
    Next 7 tasks
    Final 31 tasks 1. Lecture slides
    2. Lecture notes /
    outlines / handouts
    3. Lecture recordings
    4. Past exam papers
    5. Grades
    6. Reading lists / study
    resources
    7. Assignment deadlines
    8. Assignment feedback / feed
    forward
    

    View Slide

29. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    

    View Slide

30. 775 responses from students
    Every response was unique
    duncanstephen.net — @DuncanBSS
    

    View Slide

31. duncanstephen.net — @DuncanBSS
    Course materials
    Assessment
    Course information
    Help and support
    

    View Slide

32. duncanstephen.net — @DuncanBSS
    Course materials
    Assessment
    Course information
    Help and support
    

    View Slide

33. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    

    View Slide

34. Gerry McGovern on testing a
    new structure for the first time:
    “You will have maybe a
    60% success rate
    if you do it really well.”
    duncanstephen.net — @DuncanBSS
    

    View Slide

35. Our success rates ranged
    from 67% to 93%
    duncanstephen.net — @DuncanBSS
    

    View Slide

36. Worst performing task
    You need to contact a teacher.
    Where would you find their email address?
    67% success
    59% directness
    duncanstephen.net — @DuncanBSS
    

    View Slide

37. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Visual design
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    

    View Slide

38. Research suggests that
    when users get the first click right,
    87% ultimately succeed in a task
    versus 46% who get the first click wrong
    Jeff Sauro, Getting the first click right
    https://measuringu.com/first-click/
    duncanstephen.net — @DuncanBSS
    

    View Slide

39. We A/B tested the
    new prototype against
    existing Learn courses
    duncanstephen.net — @DuncanBSS
    

    View Slide

40. These results were
    less positive
    But we learned some really
    valuable information
    duncanstephen.net — @DuncanBSS
    

    View Slide

41. It’s coming up to exam time and you’d like to see some
    mock exam papers. Where will you find them?
    30% selected
    Course materials
    21% selected
    Resource list
    42% selected
    Assessment
    duncanstephen.net — @DuncanBSS
    

    View Slide

42. Past papers puzzle
    duncanstephen.net — @DuncanBSS
    

    View Slide

43. Past papers
    were the
    4th top task
    duncanstephen.net — @DuncanBSS
    

    View Slide

44. But they were
    not clearly
    grouped in
    the card sort
    duncanstephen.net — @DuncanBSS
    

    View Slide

45. We placed it in the
    Assessment
    section
    But students
    were not
    finding it
    duncanstephen.net — @DuncanBSS
    

    View Slide

46. Method triangulation
    to the rescue
    duncanstephen.net — @DuncanBSS
    

    View Slide

47. The elements of user experience
    duncanstephen.net — @DuncanBSS
    Conception
    Completion
    Diagram adapted from Jesse James Garrett:
    www.jjg.net/elements/pdf/elements.pdf
    User needs and
    business objectives
    Functional and
    content requirements
    Content design and
    interface design
    IA and
    interaction design
    Top tasks surveys with students and staff
    Card sorting with students
    Tree testing with students
    First click testing with students
    Ongoing usability testing
    

    View Slide

48. duncanstephen.net — @DuncanBSS
    

    View Slide

49. How usability testing is
    shaping the changes to our
    virtual learning environment
    duncanstephen.net — @DuncanBSS
    

    View Slide

50. Monthly cycle of
    usability testing
    duncanstephen.net — @DuncanBSS
    

    View Slide

51. The most successful way to
    improve your design is to
    watch your users
    duncanstephen.net — @DuncanBSS
    

    View Slide

52. “The closest thing we’ve found to a
    silver bullet when it comes to reliably
    improving the designs teams produce.”
    Jared Spool
    https://articles.uie.com/user_exposure_hours/
    duncanstephen.net — @DuncanBSS
    

    View Slide

53. “User research is a team sport”
    “All team members should watch real
    users interacting with your service and
    talking about it — ideally for at least 2
    hours every 6 weeks.”
    Dipa Shah
    Government Digital Service User research community
    www.gov.uk/service-manual/user-research/how-user-research-improves-service-design
    duncanstephen.net — @DuncanBSS
    

    View Slide

54. Usability testing has brought
    staff closer to our students
    duncanstephen.net — @DuncanBSS
    

    View Slide

55. But usability testing
    isn’t all plain sailing
    duncanstephen.net — @DuncanBSS
    

    View Slide

56. Usability testing is
    pretty easy to get
    your head around
    But there are still
    some challenges
    duncanstephen.net — @DuncanBSS
    

    View Slide

57. Challenge 1
    Getting the go-ahead from
    your bosses to spend time on
    usability testing
    duncanstephen.net — @DuncanBSS
    

    View Slide

58. Challenge 2
    Getting your colleagues
    to take on board
    what is uncovered
    by usability testing
    duncanstephen.net — @DuncanBSS
    

    View Slide

59. Challenge 3
    Getting fixes to those
    problems implemented
    duncanstephen.net — @DuncanBSS
    

    View Slide

60. How our approach to
    usability testing solves
    those challenges
    duncanstephen.net — @DuncanBSS
    

    View Slide

61. Step 1
    Conduct usability testing
    Each cycle we recorded
    four or five students or
    staff members attempting
    to complete some
    common, representative
    tasks in Learn
    duncanstephen.net — @DuncanBSS
    

    View Slide

62. We test with as many people as we can
    — which is not many
    But that’s OK because
    usability testing is a
    qualitative approach
    “The most striking truth of
    the curve is that zero users
    give zero insights.”
    Jakob Nielsen
    www.nngroup.com/articles/
    why-you-only-need-to-test-with-5-users/
    duncanstephen.net — @DuncanBSS
    

    View Slide

63. Step 2
    Showcase the videos
    Open invite across the
    university to raise
    awareness among our
    community of learning
    technologists
    duncanstephen.net — @DuncanBSS
    

    View Slide

64. Step 3
    Attendees witness usability issues
    Session attendees note
    down the usability issues
    they see while watching
    the videos
    Focus on issues, not
    potential solution ideas yet
    duncanstephen.net — @DuncanBSS
    

    View Slide

65. Step 4
    Discuss each video in groups
    After each video,
    participants discuss for
    5 minutes in small groups
    People decide on their
    top 3 issues from each video
    duncanstephen.net — @DuncanBSS
    

    View Slide

66. Step 5
    Collaboratively prioritise issues
    Across the whole room,
    people raise the usability
    issues they’ve seen
    Then together we work each
    issue through a
    prioritisation flowchart
    duncanstephen.net — @DuncanBSS
    

    View Slide

67. Red route prioritisation
    How to prioritise
    usability problems
    David Travis
    www.userfocus.co.uk/articles/prioritise.html
    duncanstephen.net — @DuncanBSS
    

    View Slide

68. Red route prioritisation
    Red routes are roads that
    are too important to get
    clogged up
    Think about key users
    and their top tasks
    in the same way
    duncanstephen.net — @DuncanBSS
    

    View Slide

69. 3 yes/no questions
    Does the problem occur on a red route?
    Is the problem difficult for the user to overcome?
    Is the problem persistent?
    Yes to 3 questions = Critical usability issue
    Yes to 2 questions = Serious
    Yes to 1 question = Medium
    Yes to 0 questions = Low
    duncanstephen.net — @DuncanBSS
    

    View Slide

70. duncanstephen.net — @DuncanBSS
    

    View Slide

71. Step 6
    Impact/effort matrix
    Now we can start thinking
    about solutions, focusing on
    the most critical problems
    duncanstephen.net — @DuncanBSS
    

    View Slide

72. Step 6
    Impact/effort matrix
    duncanstephen.net — @DuncanBSS
    Minor issue
    for users
    Major issue
    for users
    Easy
    solution
    available
    No easy
    solution
    available
    Tackle these first
    

    View Slide

73. Why we prioritise in this way
    duncanstephen.net — @DuncanBSS
    

    View Slide

74. “Running usability
    testing has been
    compared with
    taking a drink from
    a fire hydrant”
    David Travis
    www.userfocus.co.uk/articles/prioritise.html
    duncanstephen.net — @DuncanBSS
    

    View Slide

75. The red route
    method moves us
    beyond prioritising
    by gut feel
    duncanstephen.net — @DuncanBSS
    

    View Slide

76. Outcomes of usability testing
    duncanstephen.net — @DuncanBSS
    

    View Slide

77. 31 usability tests with students and staff
    7 different Learn environments
    9 showcase sessions
    20–30 attendees at most sessions
    102 usability issues identified and prioritised
    duncanstephen.net — @DuncanBSS
    

    View Slide

78. “A really valuable and informative
    process that we are hoping to embed
    across the school and the upcoming
    projects we are undertaking.”
    Emma Forrest
    Student Services Project Officer, School of GeoSciences
    duncanstephen.net — @DuncanBSS
    

    View Slide

79. “Being able to see what works well and
    what is more challenging for users
    enables us to prioritise what really
    matters and ensure our design is
    reflective of our users’ needs.”
    Lee-Ann Simpson
    Learn Foundations Project Manager
    duncanstephen.net — @DuncanBSS
    

    View Slide

80. Qualitative research was an
    essential complement to
    quantitative research
    duncanstephen.net — @DuncanBSS
    

    View Slide

81. Past papers puzzle
    duncanstephen.net — @DuncanBSS
    

    View Slide

82. Past papers
    were the
    4th top task
    duncanstephen.net — @DuncanBSS
    

    View Slide

83. But they were
    not clearly
    grouped in
    the card sort
    duncanstephen.net — @DuncanBSS
    

    View Slide

84. We placed it in the
    Assessment
    section
    But students
    were not
    finding it
    duncanstephen.net — @DuncanBSS
    

    View Slide

85. Method triangulation
    to the rescue
    duncanstephen.net — @DuncanBSS
    

    View Slide

86. By watching just 4 students through
    usability testing, we understood that
    they don’t look for past papers in Learn
    They Google for them
    duncanstephen.net — @DuncanBSS
    

    View Slide

87. Without using a mixture of
    research methods, we would
    never have understood this
    duncanstephen.net — @DuncanBSS
    

    View Slide

88. A service design approach
    duncanstephen.net — @DuncanBSS
    

    View Slide

89. duncanstephen.net — @DuncanBSS
    Audience Front stage Backstage Behind-the-
    scenes
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
    

    View Slide

90. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
    

    View Slide

91. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
    

    View Slide

92. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    We’re rolling out
    enhanced
    announcements
    functionality!
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
    

    View Slide

93. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    We’re rolling out
    enhanced
    announcements
    functionality!
    I have an
    announcement!
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
    

    View Slide

94. duncanstephen.net — @DuncanBSS
    Students Lecturers School-based
    support staff
    Learn service,
    central support
    We’d like to
    make it easier
    to post
    announcements
    We’re rolling out
    enhanced
    announcements
    functionality!
    I have an
    announcement!
    Not another
    announcement…
    Based on an illustration by Megan Erin Miller and Erik Flowers
    https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
    

    View Slide

95. By conducting research
    across this entire
    ecosystem, we are
    better placed to improve
    students’ experience
    duncanstephen.net — @DuncanBSS
    

    View Slide

96. Then… coronavirus
    duncanstephen.net — @DuncanBSS
    

    View Slide

97. Schools that had adopted the
    Learn Foundations template
    early adapted to online
    delivery more easily
    Other schools needed more support
    duncanstephen.net — @DuncanBSS
    

    View Slide

98. Learn Foundations rollout 2020/2021
    Plan:
    10 schools
    After coronavirus:
    20 schools (out of 24)
    duncanstephen.net — @DuncanBSS
    

    View Slide

99. “Consistent templates paved the way
    for a seamless move into hybridity in a
    time of crisis.”
    Jon Jack
    Learning Technology Team Manager
    duncanstephen.net — @DuncanBSS
    

    View Slide

100. Find out more on our blog:
     edin.ac/uxlf
     duncanstephen.net — @DuncanBSS
     

     View Slide

101. Duncan Stephen
     Thank you
     edin.ac/uxlf
     duncanstephen.net — @DuncanBSS
     

     View Slide

102. Duncan Stephen
     Questions?
     edin.ac/uxlf
     duncanstephen.net — @DuncanBSS
     

     View Slide