Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A service design approach to improving student experience

A service design approach to improving student experience

Research had shown us that University of Edinburgh students were frustrated by inconsistencies between different courses in their virtual learning environment. So we embarked on a programme of user research with students. However, for most of our findings about students, we found that something about staff was driving that experience. By adopting a service design approach, we have built a stronger understanding of this complex ecosystem of interactions, leaving us better placed to improve student experience.

For more detailed blog posts see https://edin.ac/uxlf

Presented at UCD Gathering virtual conference.
https://www.ucdgathering.net/programme/service-design-approach-improving-student-experience

08911a018fe9e8412434d165d8474187?s=128

Duncan Stephen

October 15, 2020
Tweet

Transcript

  1. Duncan Stephen A service design approach to improving student experience

    User Experience Manager The University of Edinburgh duncanstephen.net — @DuncanBSS ⟶
  2. I will be happy to take questions at the end

    duncanstephen.net — @DuncanBSS
  3. duncanstephen.net — @DuncanBSS Foundations Delivering an enhanced student learning experience

    online
  4. Students are frustrated by inconsistencies across digital systems duncanstephen.net —

    @DuncanBSS
  5. duncanstephen.net — @DuncanBSS Research by Headscape, 2016

  6. With course materials online… Individual schools Individual lecturers Individual approaches

    duncanstephen.net — @DuncanBSS
  7. Some schools already had their own templates These templates had

    168 different top-level menu items duncanstephen.net — @DuncanBSS Analysis by Paul Smyth
  8. Answer = “Make it consistent” …Obviously duncanstephen.net — @DuncanBSS

  9. Consistency duncanstephen.net — @DuncanBSS Source image: moojj https://imgur.com/gallery/EFA8dLh

  10. 45,000 students 8,000 academic staff 24 schools and deaneries …all

    with different requirements duncanstephen.net — @DuncanBSS
  11. Solution focus ⟶Checklist ⟶Terminology ⟶Templates ⟶Training ⟶Automation ⟶QA and audit

    duncanstephen.net — @DuncanBSS
  12. A comprehensive programme of user research, adopting multiple methods duncanstephen.net

    — @DuncanBSS
  13. Overall goal to improve student experience But we knew that

    staff experience was also fundamental duncanstephen.net — @DuncanBSS
  14. Interviews with students and staff duncanstephen.net — @DuncanBSS

  15. A semester in the life of students using Learn duncanstephen.net

    — @DuncanBSS
  16. A semester in the life of students using Learn duncanstephen.net

    — @DuncanBSS Week 1 Getting your head around Learn Week 6 Assignment submission Week 7 Different lecturers doing different things Week 8 Getting feedback
  17. Seeing the connections with staff members’ use of Learn duncanstephen.net

    — @DuncanBSS
  18. duncanstephen.net — @DuncanBSS Students’ experience Frustrations around being made to

    use templates Love using Learn as a comms tool Time consuming to give good feedback Staff members’ experience Lack of trust in existing templates Overwhelmed by notifications Inconsistencies around assignments and feedback
  19. Programme of targeted quantitative research duncanstephen.net — @DuncanBSS

  20. Quantitative research specifically designed to inform the development of the

    new template duncanstephen.net — @DuncanBSS
  21. duncanstephen.net — @DuncanBSS Sketch and plan by Neil Allison

  22. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion

    Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
  23. Phenomenal response rates Almost 5,000 responses across five studies duncanstephen.net

    — @DuncanBSS
  24. We learnt a lot about user research incentives Give people

    a small chance of a medium-sized incentive duncanstephen.net — @DuncanBSS
  25. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion

    Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
  26. Top tasks methodology Gerry McGovern duncanstephen.net — @DuncanBSS Long tail

    Long neck
  27. duncanstephen.net — @DuncanBSS 0 100 200 300 400 500 600

    Students’ top tasks – the long neck and long tail
  28. Students’ top tasks duncanstephen.net — @DuncanBSS Lecture slides Lecture notes/outlines/handouts

    Lecture recordings Past exam papers Top 3 tasks >25% of votes Next 4 tasks Almost half Next 7 tasks Final 31 tasks 1. Lecture slides 2. Lecture notes / outlines / handouts 3. Lecture recordings 4. Past exam papers 5. Grades 6. Reading lists / study resources 7. Assignment deadlines 8. Assignment feedback / feed forward
  29. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion

    Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
  30. 775 responses from students Every response was unique duncanstephen.net —

    @DuncanBSS
  31. duncanstephen.net — @DuncanBSS Course materials Assessment Course information Help and

    support
  32. duncanstephen.net — @DuncanBSS Course materials Assessment Course information Help and

    support
  33. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion

    Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
  34. Gerry McGovern on testing a new structure for the first

    time: “You will have maybe a 60% success rate if you do it really well.” duncanstephen.net — @DuncanBSS
  35. Our success rates ranged from 67% to 93% duncanstephen.net —

    @DuncanBSS
  36. Worst performing task You need to contact a teacher. Where

    would you find their email address? 67% success 59% directness duncanstephen.net — @DuncanBSS
  37. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion

    Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
  38. Research suggests that when users get the first click right,

    87% ultimately succeed in a task versus 46% who get the first click wrong Jeff Sauro, Getting the first click right https://measuringu.com/first-click/ duncanstephen.net — @DuncanBSS
  39. We A/B tested the new prototype against existing Learn courses

    duncanstephen.net — @DuncanBSS
  40. These results were less positive But we learned some really

    valuable information duncanstephen.net — @DuncanBSS
  41. It’s coming up to exam time and you’d like to

    see some mock exam papers. Where will you find them? 30% selected Course materials 21% selected Resource list 42% selected Assessment duncanstephen.net — @DuncanBSS
  42. Past papers puzzle duncanstephen.net — @DuncanBSS

  43. Past papers were the 4th top task duncanstephen.net — @DuncanBSS

  44. But they were not clearly grouped in the card sort

    duncanstephen.net — @DuncanBSS
  45. We placed it in the Assessment section But students were

    not finding it duncanstephen.net — @DuncanBSS
  46. Method triangulation to the rescue duncanstephen.net — @DuncanBSS

  47. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion

    Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students Ongoing usability testing
  48. duncanstephen.net — @DuncanBSS

  49. How usability testing is shaping the changes to our virtual

    learning environment duncanstephen.net — @DuncanBSS
  50. Monthly cycle of usability testing duncanstephen.net — @DuncanBSS

  51. The most successful way to improve your design is to

    watch your users duncanstephen.net — @DuncanBSS
  52. “The closest thing we’ve found to a silver bullet when

    it comes to reliably improving the designs teams produce.” Jared Spool https://articles.uie.com/user_exposure_hours/ duncanstephen.net — @DuncanBSS
  53. “User research is a team sport” “All team members should

    watch real users interacting with your service and talking about it — ideally for at least 2 hours every 6 weeks.” Dipa Shah Government Digital Service User research community www.gov.uk/service-manual/user-research/how-user-research-improves-service-design duncanstephen.net — @DuncanBSS
  54. Usability testing has brought staff closer to our students duncanstephen.net

    — @DuncanBSS
  55. But usability testing isn’t all plain sailing duncanstephen.net — @DuncanBSS

  56. Usability testing is pretty easy to get your head around

    But there are still some challenges duncanstephen.net — @DuncanBSS
  57. Challenge 1 Getting the go-ahead from your bosses to spend

    time on usability testing duncanstephen.net — @DuncanBSS
  58. Challenge 2 Getting your colleagues to take on board what

    is uncovered by usability testing duncanstephen.net — @DuncanBSS
  59. Challenge 3 Getting fixes to those problems implemented duncanstephen.net —

    @DuncanBSS
  60. How our approach to usability testing solves those challenges duncanstephen.net

    — @DuncanBSS
  61. Step 1 Conduct usability testing Each cycle we recorded four

    or five students or staff members attempting to complete some common, representative tasks in Learn duncanstephen.net — @DuncanBSS
  62. We test with as many people as we can —

    which is not many But that’s OK because usability testing is a qualitative approach “The most striking truth of the curve is that zero users give zero insights.” Jakob Nielsen www.nngroup.com/articles/ why-you-only-need-to-test-with-5-users/ duncanstephen.net — @DuncanBSS
  63. Step 2 Showcase the videos Open invite across the university

    to raise awareness among our community of learning technologists duncanstephen.net — @DuncanBSS
  64. Step 3 Attendees witness usability issues Session attendees note down

    the usability issues they see while watching the videos Focus on issues, not potential solution ideas yet duncanstephen.net — @DuncanBSS
  65. Step 4 Discuss each video in groups After each video,

    participants discuss for 5 minutes in small groups People decide on their top 3 issues from each video duncanstephen.net — @DuncanBSS
  66. Step 5 Collaboratively prioritise issues Across the whole room, people

    raise the usability issues they’ve seen Then together we work each issue through a prioritisation flowchart duncanstephen.net — @DuncanBSS
  67. Red route prioritisation How to prioritise usability problems David Travis

    www.userfocus.co.uk/articles/prioritise.html duncanstephen.net — @DuncanBSS
  68. Red route prioritisation Red routes are roads that are too

    important to get clogged up Think about key users and their top tasks in the same way duncanstephen.net — @DuncanBSS
  69. 3 yes/no questions Does the problem occur on a red

    route? Is the problem difficult for the user to overcome? Is the problem persistent? Yes to 3 questions = Critical usability issue Yes to 2 questions = Serious Yes to 1 question = Medium Yes to 0 questions = Low duncanstephen.net — @DuncanBSS
  70. duncanstephen.net — @DuncanBSS

  71. Step 6 Impact/effort matrix Now we can start thinking about

    solutions, focusing on the most critical problems duncanstephen.net — @DuncanBSS
  72. Step 6 Impact/effort matrix duncanstephen.net — @DuncanBSS Minor issue for

    users Major issue for users Easy solution available No easy solution available Tackle these first
  73. Why we prioritise in this way duncanstephen.net — @DuncanBSS

  74. “Running usability testing has been compared with taking a drink

    from a fire hydrant” David Travis www.userfocus.co.uk/articles/prioritise.html duncanstephen.net — @DuncanBSS
  75. The red route method moves us beyond prioritising by gut

    feel duncanstephen.net — @DuncanBSS
  76. Outcomes of usability testing duncanstephen.net — @DuncanBSS

  77. 31 usability tests with students and staff 7 different Learn

    environments 9 showcase sessions 20–30 attendees at most sessions 102 usability issues identified and prioritised duncanstephen.net — @DuncanBSS
  78. “A really valuable and informative process that we are hoping

    to embed across the school and the upcoming projects we are undertaking.” Emma Forrest Student Services Project Officer, School of GeoSciences duncanstephen.net — @DuncanBSS
  79. “Being able to see what works well and what is

    more challenging for users enables us to prioritise what really matters and ensure our design is reflective of our users’ needs.” Lee-Ann Simpson Learn Foundations Project Manager duncanstephen.net — @DuncanBSS
  80. Qualitative research was an essential complement to quantitative research duncanstephen.net

    — @DuncanBSS
  81. Past papers puzzle duncanstephen.net — @DuncanBSS

  82. Past papers were the 4th top task duncanstephen.net — @DuncanBSS

  83. But they were not clearly grouped in the card sort

    duncanstephen.net — @DuncanBSS
  84. We placed it in the Assessment section But students were

    not finding it duncanstephen.net — @DuncanBSS
  85. Method triangulation to the rescue duncanstephen.net — @DuncanBSS

  86. By watching just 4 students through usability testing, we understood

    that they don’t look for past papers in Learn They Google for them duncanstephen.net — @DuncanBSS
  87. Without using a mixture of research methods, we would never

    have understood this duncanstephen.net — @DuncanBSS
  88. A service design approach duncanstephen.net — @DuncanBSS

  89. duncanstephen.net — @DuncanBSS Audience Front stage Backstage Behind-the- scenes Based

    on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
  90. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service,

    central support Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
  91. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service,

    central support We’d like to make it easier to post announcements Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
  92. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service,

    central support We’d like to make it easier to post announcements We’re rolling out enhanced announcements functionality! Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
  93. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service,

    central support We’d like to make it easier to post announcements We’re rolling out enhanced announcements functionality! I have an announcement! Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
  94. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service,

    central support We’d like to make it easier to post announcements We’re rolling out enhanced announcements functionality! I have an announcement! Not another announcement… Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
  95. By conducting research across this entire ecosystem, we are better

    placed to improve students’ experience duncanstephen.net — @DuncanBSS
  96. Then… coronavirus duncanstephen.net — @DuncanBSS

  97. Schools that had adopted the Learn Foundations template early adapted

    to online delivery more easily Other schools needed more support duncanstephen.net — @DuncanBSS
  98. Learn Foundations rollout 2020/2021 Plan: 10 schools After coronavirus: 20

    schools (out of 24) duncanstephen.net — @DuncanBSS
  99. “Consistent templates paved the way for a seamless move into

    hybridity in a time of crisis.” Jon Jack Learning Technology Team Manager duncanstephen.net — @DuncanBSS
  100. Find out more on our blog: edin.ac/uxlf duncanstephen.net — @DuncanBSS

  101. Duncan Stephen Thank you edin.ac/uxlf duncanstephen.net — @DuncanBSS

  102. Duncan Stephen Questions? edin.ac/uxlf duncanstephen.net — @DuncanBSS