A service design approach to improving student experience

Research had shown us that University of Edinburgh students were frustrated by inconsistencies between different courses in their virtual learning environment. So we embarked on a programme of user research with students. However, for most of our findings about students, we found that something about staff was driving that experience. By adopting a service design approach, we have built a stronger understanding of this complex ecosystem of interactions, leaving us better placed to improve student experience.

For more detailed blog posts see https://edin.ac/uxlf

Presented at UCD Gathering virtual conference.
https://www.ucdgathering.net/programme/service-design-approach-improving-student-experience

Transcript

1. Duncan Stephen A service design approach to improving student experience User Experience Manager The University of Edinburgh duncanstephen.net — @DuncanBSS ⟶
2. I will be happy to take questions at the end duncanstephen.net — @DuncanBSS
3. duncanstephen.net — @DuncanBSS Foundations Delivering an enhanced student learning experience online
4. Students are frustrated by inconsistencies across digital systems duncanstephen.net — @DuncanBSS
5. duncanstephen.net — @DuncanBSS Research by Headscape, 2016
6. With course materials online… Individual schools Individual lecturers Individual approaches duncanstephen.net — @DuncanBSS
7. Some schools already had their own templates These templates had 168 different top-level menu items duncanstephen.net — @DuncanBSS Analysis by Paul Smyth
8. Answer = “Make it consistent” …Obviously duncanstephen.net — @DuncanBSS
9. Consistency duncanstephen.net — @DuncanBSS Source image: moojj https://imgur.com/gallery/EFA8dLh
10. 45,000 students 8,000 academic staff 24 schools and deaneries …all with different requirements duncanstephen.net — @DuncanBSS
11. Solution focus ⟶Checklist ⟶Terminology ⟶Templates ⟶Training ⟶Automation ⟶QA and audit duncanstephen.net — @DuncanBSS
12. A comprehensive programme of user research, adopting multiple methods duncanstephen.net — @DuncanBSS
13. Overall goal to improve student experience But we knew that staff experience was also fundamental duncanstephen.net — @DuncanBSS
14. Interviews with students and staff duncanstephen.net — @DuncanBSS
15. A semester in the life of students using Learn duncanstephen.net — @DuncanBSS
16. A semester in the life of students using Learn duncanstephen.net — @DuncanBSS Week 1 Getting your head around Learn Week 6 Assignment submission Week 7 Different lecturers doing different things Week 8 Getting feedback
17. Seeing the connections with staff members’ use of Learn duncanstephen.net — @DuncanBSS
18. duncanstephen.net — @DuncanBSS Students’ experience Frustrations around being made to use templates Love using Learn as a comms tool Time consuming to give good feedback Staff members’ experience Lack of trust in existing templates Overwhelmed by notifications Inconsistencies around assignments and feedback
19. Programme of targeted quantitative research duncanstephen.net — @DuncanBSS
20. Quantitative research specifically designed to inform the development of the new template duncanstephen.net — @DuncanBSS
21. duncanstephen.net — @DuncanBSS Sketch and plan by Neil Allison
22. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
23. Phenomenal response rates Almost 5,000 responses across five studies duncanstephen.net — @DuncanBSS
24. We learnt a lot about user research incentives Give people a small chance of a medium-sized incentive duncanstephen.net — @DuncanBSS
25. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
26. Top tasks methodology Gerry McGovern duncanstephen.net — @DuncanBSS Long tail Long neck
27. duncanstephen.net — @DuncanBSS 0 100 200 300 400 500 600 Students’ top tasks – the long neck and long tail
28. Students’ top tasks duncanstephen.net — @DuncanBSS Lecture slides Lecture notes/outlines/handouts Lecture recordings Past exam papers Top 3 tasks >25% of votes Next 4 tasks Almost half Next 7 tasks Final 31 tasks 1. Lecture slides 2. Lecture notes / outlines / handouts 3. Lecture recordings 4. Past exam papers 5. Grades 6. Reading lists / study resources 7. Assignment deadlines 8. Assignment feedback / feed forward
29. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
30. 775 responses from students Every response was unique duncanstephen.net — @DuncanBSS
31. duncanstephen.net — @DuncanBSS Course materials Assessment Course information Help and support
32. duncanstephen.net — @DuncanBSS Course materials Assessment Course information Help and support
33. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
34. Gerry McGovern on testing a new structure for the first time: “You will have maybe a 60% success rate if you do it really well.” duncanstephen.net — @DuncanBSS
35. Our success rates ranged from 67% to 93% duncanstephen.net — @DuncanBSS
36. Worst performing task You need to contact a teacher. Where would you find their email address? 67% success 59% directness duncanstephen.net — @DuncanBSS
37. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Visual design Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students
38. Research suggests that when users get the first click right, 87% ultimately succeed in a task versus 46% who get the first click wrong Jeff Sauro, Getting the first click right https://measuringu.com/first-click/ duncanstephen.net — @DuncanBSS
39. We A/B tested the new prototype against existing Learn courses duncanstephen.net — @DuncanBSS
40. These results were less positive But we learned some really valuable information duncanstephen.net — @DuncanBSS
41. It’s coming up to exam time and you’d like to see some mock exam papers. Where will you find them? 30% selected Course materials 21% selected Resource list 42% selected Assessment duncanstephen.net — @DuncanBSS
42. Past papers puzzle duncanstephen.net — @DuncanBSS
43. Past papers were the 4th top task duncanstephen.net — @DuncanBSS
44. But they were not clearly grouped in the card sort duncanstephen.net — @DuncanBSS
45. We placed it in the Assessment section But students were not finding it duncanstephen.net — @DuncanBSS
46. Method triangulation to the rescue duncanstephen.net — @DuncanBSS
47. The elements of user experience duncanstephen.net — @DuncanBSS Conception Completion Diagram adapted from Jesse James Garrett: www.jjg.net/elements/pdf/elements.pdf User needs and business objectives Functional and content requirements Content design and interface design IA and interaction design Top tasks surveys with students and staff Card sorting with students Tree testing with students First click testing with students Ongoing usability testing
48. duncanstephen.net — @DuncanBSS
49. How usability testing is shaping the changes to our virtual learning environment duncanstephen.net — @DuncanBSS
50. Monthly cycle of usability testing duncanstephen.net — @DuncanBSS
51. The most successful way to improve your design is to watch your users duncanstephen.net — @DuncanBSS
52. “The closest thing we’ve found to a silver bullet when it comes to reliably improving the designs teams produce.” Jared Spool https://articles.uie.com/user_exposure_hours/ duncanstephen.net — @DuncanBSS
53. “User research is a team sport” “All team members should watch real users interacting with your service and talking about it — ideally for at least 2 hours every 6 weeks.” Dipa Shah Government Digital Service User research community www.gov.uk/service-manual/user-research/how-user-research-improves-service-design duncanstephen.net — @DuncanBSS
54. Usability testing has brought staff closer to our students duncanstephen.net — @DuncanBSS
55. But usability testing isn’t all plain sailing duncanstephen.net — @DuncanBSS
56. Usability testing is pretty easy to get your head around But there are still some challenges duncanstephen.net — @DuncanBSS
57. Challenge 1 Getting the go-ahead from your bosses to spend time on usability testing duncanstephen.net — @DuncanBSS
58. Challenge 2 Getting your colleagues to take on board what is uncovered by usability testing duncanstephen.net — @DuncanBSS
59. Challenge 3 Getting fixes to those problems implemented duncanstephen.net — @DuncanBSS
60. How our approach to usability testing solves those challenges duncanstephen.net — @DuncanBSS
61. Step 1 Conduct usability testing Each cycle we recorded four or five students or staff members attempting to complete some common, representative tasks in Learn duncanstephen.net — @DuncanBSS
62. We test with as many people as we can — which is not many But that’s OK because usability testing is a qualitative approach “The most striking truth of the curve is that zero users give zero insights.” Jakob Nielsen www.nngroup.com/articles/ why-you-only-need-to-test-with-5-users/ duncanstephen.net — @DuncanBSS
63. Step 2 Showcase the videos Open invite across the university to raise awareness among our community of learning technologists duncanstephen.net — @DuncanBSS
64. Step 3 Attendees witness usability issues Session attendees note down the usability issues they see while watching the videos Focus on issues, not potential solution ideas yet duncanstephen.net — @DuncanBSS
65. Step 4 Discuss each video in groups After each video, participants discuss for 5 minutes in small groups People decide on their top 3 issues from each video duncanstephen.net — @DuncanBSS
66. Step 5 Collaboratively prioritise issues Across the whole room, people raise the usability issues they’ve seen Then together we work each issue through a prioritisation flowchart duncanstephen.net — @DuncanBSS
67. Red route prioritisation How to prioritise usability problems David Travis www.userfocus.co.uk/articles/prioritise.html duncanstephen.net — @DuncanBSS
68. Red route prioritisation Red routes are roads that are too important to get clogged up Think about key users and their top tasks in the same way duncanstephen.net — @DuncanBSS
69. 3 yes/no questions Does the problem occur on a red route? Is the problem difficult for the user to overcome? Is the problem persistent? Yes to 3 questions = Critical usability issue Yes to 2 questions = Serious Yes to 1 question = Medium Yes to 0 questions = Low duncanstephen.net — @DuncanBSS
70. duncanstephen.net — @DuncanBSS
71. Step 6 Impact/effort matrix Now we can start thinking about solutions, focusing on the most critical problems duncanstephen.net — @DuncanBSS
72. Step 6 Impact/effort matrix duncanstephen.net — @DuncanBSS Minor issue for users Major issue for users Easy solution available No easy solution available Tackle these first
73. Why we prioritise in this way duncanstephen.net — @DuncanBSS
74. “Running usability testing has been compared with taking a drink from a fire hydrant” David Travis www.userfocus.co.uk/articles/prioritise.html duncanstephen.net — @DuncanBSS
75. The red route method moves us beyond prioritising by gut feel duncanstephen.net — @DuncanBSS
76. Outcomes of usability testing duncanstephen.net — @DuncanBSS
77. 31 usability tests with students and staff 7 different Learn environments 9 showcase sessions 20–30 attendees at most sessions 102 usability issues identified and prioritised duncanstephen.net — @DuncanBSS
78. “A really valuable and informative process that we are hoping to embed across the school and the upcoming projects we are undertaking.” Emma Forrest Student Services Project Officer, School of GeoSciences duncanstephen.net — @DuncanBSS
79. “Being able to see what works well and what is more challenging for users enables us to prioritise what really matters and ensure our design is reflective of our users’ needs.” Lee-Ann Simpson Learn Foundations Project Manager duncanstephen.net — @DuncanBSS
80. Qualitative research was an essential complement to quantitative research duncanstephen.net — @DuncanBSS
81. Past papers puzzle duncanstephen.net — @DuncanBSS
82. Past papers were the 4th top task duncanstephen.net — @DuncanBSS
83. But they were not clearly grouped in the card sort duncanstephen.net — @DuncanBSS
84. We placed it in the Assessment section But students were not finding it duncanstephen.net — @DuncanBSS
85. Method triangulation to the rescue duncanstephen.net — @DuncanBSS
86. By watching just 4 students through usability testing, we understood that they don’t look for past papers in Learn They Google for them duncanstephen.net — @DuncanBSS
87. Without using a mixture of research methods, we would never have understood this duncanstephen.net — @DuncanBSS
88. A service design approach duncanstephen.net — @DuncanBSS
89. duncanstephen.net — @DuncanBSS Audience Front stage Backstage Behind-the- scenes Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
90. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service, central support Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
91. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service, central support We’d like to make it easier to post announcements Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
92. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service, central support We’d like to make it easier to post announcements We’re rolling out enhanced announcements functionality! Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
93. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service, central support We’d like to make it easier to post announcements We’re rolling out enhanced announcements functionality! I have an announcement! Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
94. duncanstephen.net — @DuncanBSS Students Lecturers School-based support staff Learn service, central support We’d like to make it easier to post announcements We’re rolling out enhanced announcements functionality! I have an announcement! Not another announcement… Based on an illustration by Megan Erin Miller and Erik Flowers https://blog.practicalservicedesign.com/the-difference-between-a-journey-map-and-a-service-blueprint-31a6e24c4a6c
95. By conducting research across this entire ecosystem, we are better placed to improve students’ experience duncanstephen.net — @DuncanBSS
96. Then… coronavirus duncanstephen.net — @DuncanBSS
97. Schools that had adopted the Learn Foundations template early adapted to online delivery more easily Other schools needed more support duncanstephen.net — @DuncanBSS
98. Learn Foundations rollout 2020/2021 Plan: 10 schools After coronavirus: 20 schools (out of 24) duncanstephen.net — @DuncanBSS
99. “Consistent templates paved the way for a seamless move into hybridity in a time of crisis.” Jon Jack Learning Technology Team Manager duncanstephen.net — @DuncanBSS
100. Find out more on our blog: edin.ac/uxlf duncanstephen.net — @DuncanBSS
101. Duncan Stephen Thank you edin.ac/uxlf duncanstephen.net — @DuncanBSS
102. Duncan Stephen Questions? edin.ac/uxlf duncanstephen.net — @DuncanBSS