Natalie Rowland & Lee Ryan

Transcript

1. 1 www.captionslive.com.au | captionslive@outlook.com | 0425 904 255 UX AUSTRALIA Design Research 2020 Day 2 Friday, 20 March 2020 Captioned by: Gail Kearney & Rebekah Goulevitch
2. 2 NATALIE ROWLAND: It's the home stretch and it has been a new conference experience like no other. We are here to share a fairly recent research, we call it an adventure. I'm Natalie, in Marrickville in Sydney and I would love to acknowledge the Gatigal people, the traditional custodians of this land and pay my respects to elders, past, present and emerging. We are three researchers, Marieke and I have worked together for over a decade and are super tuned in with the way of working, we both work remotely and are working parents. Lee is someone we have admired from afar with lots of connections and sought opportunities to collaborate. Lee is in our sister city, in Auckland LEE RYAN: Hi! (Speaks Maori), it is technically beer o'clock here. And you might not know it but we can all see a colour pallet on the front of your screen. NATALIE ROWLAND: I'm going to try to get rid of it, I hope you can cope. I'll keep it moving. It is a remote research team running remote research. Essentially a diary study like no other. We have been running diary studies for years, when we were building our own word press sites to run them. But this is a project we did back in October, in 2018. And we were thinking of presenting this at the design research last year. But to be honest, we weren't quite sure whether we were going to pull it off. And it was really the recruitment that was the key thing that was keeping us up at night, as all researchers can relate to, the stress of finding the right people to recruit. So, I should just set the scene and say that Marieke and I had done a prototype of the project at a smaller scale where we were looking at understanding what is the on boarding experience for people starting to use Trello? This project had 12 participants and it was for their first 10 days. The type of project that Lee, Marieke and I did is a different scale. Critical, we were having conversations that were participant-led is we built the trust right-from the beginning. Right from the start of our conversations. We had to establish respectful ways of working together as a team, together with our client and together with our participants. We were really nervous about whether or not people wanted to participate in the research process? What if they wanted to drop out? You can imagine our nerves once we knew it was commissioned. The other important context, we were researching within a business-to-business context. We were trying to understand the experience of teams evaluating and on boarding on to Jira. We needed to involve stakeholders from many functions, from design, product managers, mobile app developers and across locations. We had a client in America, Europe and Australia. And we needed to find ways to bring them on this journey with us as we've heard with the other presentations, bringing your stakeholders in early is critical. So, we saw some serious scale with Shaun's presentation and ours is in that zone except it was a different pace. We had 55 participants across the globe who we had over 160 conversations with. We haven't actually added them all up. Each conversation was at least 30 minutes. By Zoom, and for some people who stayed engaged with this particular methodology, we met them over eight times over four to six months. So, on boarding. It's a really tricky thing to understand and to research. We needed to think about what's the moment to start a conversation. How do we get as close to the start as possible. So, we deployed an intercept, to invite people to participate. And this intercept popped up before people had committed to
3. 3 signing up for a trial. And this was important. We didn't want to interrupt that process. Each journey was unique. We needed to be unobtrusive and some wanted to sign up and get started right away while others were essentially browsing and curious to start in a week's time. It was a big commitment. We were asking our participants to commit to at least three conversations with us. So, it was important to really set up those expectations early. We wanted to define what the ask was. And provide a really great indication as to how enthusiastic they were about participating. We told them what the ask was and they said fantastic, I'd love to get involved, I'm not sure, or, no, this isn't for me. And then we had another question, are you able to chat right now? We wanted to get as close to the first moments inside the product as possible. What do you think that meant for the research team? We had to be like water, as Jax said, to flow, adapt, reflect. We had to be available, to hand participants to different researchers who were ready, online, ready to go. LEE RYAN: So, being ready as a researcher is also about how do we immerse ourselves in the domain and sensitise ourselves to the most useful questions? So, it was just serendipity that as the project ramped up I was attending Epic, a global conference on ethnographic business. It was the ability to have conversations with people who are not only involved with projects right now that involve anthropology, business users and software from firms like IBM or Microsoft, but they were also able to point me to key articles. And there's a group called CECW, which is Computer Supported Collaborative Work. For 25 years it is has been a default mechanism to intervene in the design of work place software. I am just wondering, a possibility, I don't think I'm showing up? I don't know if we've switched views? I don't know, I have seen Tina's question whether we can close the colour pallet because it's still covering the screen. There I am but now we've got no presentation! OK. So, NATALIE ROWLAND: I have no idea how to get rid of a colour pallet. Sorry. Guys, I'm just conscious of the next presenter and I don't want to intrude. LEE RYAN: We're just pausing and I'll talk while Natalie carries on. No rush ses UX Events. I was at UX, and Ben mentioned yesterday about Lucie who is a giant in software and machines and understanding work place practices. She famously used video to understand how people interact with machines in situ. And is partly responsible for us getting a big green button on all those photo copiers. So, her comment about workplace practice, like the way in which people work is not always apparent and that assumptions are made at how tasks are performed rather than unearthing the underlying work practices. So by making work visible desires create a more intimate view of work place landscape. Which is what I just said. Now you'll see people, a beautiful screen and no colour pallet. Fantastic! NATALIE ROWLAND: Your worst nightmare! LEE RYAN: The questions how to use work practices to create an intimate review of work place realities. A part of the challenge was also creating a rhythm that works for both the organisation and the participants. So, setting that rhythm was partly about developing a cadence, that really
4. 4 worked for Atalassian. So, initially it was the classic two-week sprints within an organisation and we were bringing stakeholders every two weeks and later we expanded that out for three weeks. But we wanted it also to work for participants knowing that there were known milestones in the on boarding experience, if it's a seven-day trial, we wanted to be approximate to those dates. So, this kind of shows how we went around about it. So, the left side is the orange circles, which illustrate the first conversation. So, if we opened up Zoom and made that connection, sometimes we were spending time with people getting them as other people had said talking about themselves, their context, their team, the tools they're using. But if they opened up and really wanted to get going then we wanted to be responsive to what they wanted to do. We would be for with them as they got into the chair for the first time and their trial. Looking at their flow, the causes, the questions they were asking. If that was what happened when we wrapped up, we would circle back to the questions about team, tools and expectations. The circles on the right in yellow, they are like the check-ins we did. So, we did the first interview just as they started on Zoom and observing what they did and then we did check ins 24 hours later, three our four days, seven to 10 days and then weeks. We were setting appointments with calendar and trying to get a time when they would be naturally spending more time and energy figuring things out. We would try and check in see what's changed, what have you been up to? Can you show us the highs and lows? Sometimes they had great experiences and wanted to show what they had created. And then we were spending time together. If people said, "Hmm, usually I would Google this" and then paused, "But I'm not going to do it right now". We would go, "Great! Let's learn together". So that real shared, being there with them as they worked their way through and had the questions that they would naturally have. So, in all of this, therefore, it really means that you need to design a lot of care in the process, because people are quite vulnerable because they're literally being open about learning and trialling and making mistakes, with the software in front of you. So, we asked ourselves - you might hear my dog barking in the background who is just being harassed by the neighbour's cat... we asked ourselves how can we design a process that will orient ourselves and all the stakeholders coming in into doing the right thing as well protecting participant privacy, their data and all their work stuff that they were doing. So, one of the things that we did is we wrote out some principles which really helped us, the stakeholders and the participants. You can see that cat's quite annoying for the dog (dog barks). The principles as you can see here that we wrote up, how can we be great conversational partners, how can we minimise that feeling of being watched or judged? How do we take care to protect their privacy and their reputation? So, writing this up flowed through to the conversation guides that we wrote about how we talked to participants. But it also meant that when stakeholders came to us and said, "We saw that person, can we get X, Y, Z and track them down?" We could go, "Actually, we can't. Here are the principles", so we had something to fall back on. We not only developed some principles but we also developed practices. So, before any of the stakeholders who worked with us, who we called collaborators, before they could access any raw video footage they were briefed and reviewed and signed an observer consent form. We tied in to the Atalassian values so it connected up to what Atalassian looked to do with their customers as a value. And then following on from that, we used
5. 5 Google forms to set up observations. So that participants would say, once they signed the sheet, they could go in and view and then we were asking them to - what are you noticing? What might it mean? And trying to help them who were collaborating with us, as other people have mentioned, it is very easy to get stuck in your silos or the issues that are important for your team, so how can you think beyond your focus and tune into what this participant is saying and what it can be telling us and helping us as the researchers on the team to see what are we seeing? NATALIE ROWLAND: We know with any of these conferences, everyone wants a window under the hood to see what are the tools in the researcher's tool kit. I'm super curious, before we share ours, I wonder who in the audience is out researching when audio tapes were still the go? They were still there for the very early days of my career, my traineeship. And it is such a different way of researching. We would send the tapes off to transcribed and wait for printouts of the transcript to come back. This is in the '90s. Come forward to today: Luckily, the recruitment took a bit of time so we could have the chance to explore what were going to be the tools that we would use to manage this kind of project. So, we started by looking at the strengths and weaknesses of the existing tools, the ones we used for Trello. Post the Trello experience, we identified that we wanted to be able to see what each other was seeing, to easily dive in and dive out of each other's data, to start making those comparisons as we go, to help see patterns and possibilities earlier. And to also learn from each other's style and approach. It's a really important practice to review other researchers' interviews. Look at how they approach the same types of questions and explore and create space for observation. And that was a really wonderful thing that happened as a result of these tools. Welcome Dovetail! It was an elegant combination of Google dox and Trello and Lee will explain how we embraced Dovetail. It was important to eliminate the horror tasks and scheduling important on different time zones. We brought on Calendly and honestly it is worth every American dollar, even now. We thought how are we going to cut the videos together and get them to our stakeholders? Our I movie didn't have blurring capability. And goodness, we didn't have time for a three-week course. Luckily we discovered Screenflow. We tried it out and we learned the way it worked together, which then brings in loom, which helped us easily capture and share the things we were learning about our own tools. We used Google dox to create our own instructions for one another. Top tip: I know Becky is listening - whatever you do, set up your Zoom recordings to optimise for third-party video editor if you ever want to use your videos in another editing tool. Otherwise you're up for the green screen of death. So, we were working out how the - how to set up the frame, what dimensions as well as what kinds of blurring and blocking and tools we had to protect participant information. Then we had the challenge of managing the data. How do we make sure that we had in control of the data? In terms of who has access and for how long? And drop box gives you these kinds of controls. We were using our clients' drop box account, which had two-factor authentication, by setting it as team members, only people who had this capability could look at the videos. And the links could be set to expire at a particular time. And we could also disable downloads, which is also really important. Once someone downloads a video, it's on their desktop, you've lost it. Alright. So, like many other researchers today, we love using Zoom to meet and
6. 6 spend time with our participants. Zoom is familiar, you can use it on PC or mobile, and the lovely thing with Zoom is that participants are empowered to have their video on or off. And this is really important to building that respect. So, some days participants wouldn't want to have their video on. And that's completely OK. It's really intimate. We're right there with each other. In their physical spaces, in all kinds of spaces in their eco tech system. We were with participants as their clicked, closed, paused or gave up. When they had the video on, those emotions, those expressions were recorded. Certainly, our participants reacted in different ways. Some of them would rather talk about what they had done, they didn't feel as comfortable exploring learning, being vulnerable with us watching and that was OK. Others were happy to go on the journey together. And we adapted to meet their preferences and their comfort zone. As you can imagine, our participants were busy and snatching moments of time to connect and be with them as they explored. It was really challenging. Honestly, we just had to surrender and allow them to set the pace. This project was participant-led. All of our participants are their own unique journeys. Some had set everything up by day three. Others hadn't dared create anything until they watched how-to videos. Naturally, some are further along in their journeys at different points. We were really worried that people would want to drop out. What was really interesting with this study is that people wanted to keep talking. And we asked ourselves, why? Why did they want to engage with us and continue setting up the meetings and chatting? We were thinking, ultimately, the opportunity to talk to a complete stranger on the other side of the world about your work, about what matters to you, reflect on your own practice and what you need and why you're looking to this tool to solve a particular problem was actually somewhat therapeutic. It gave them the chance to pause, check in, and use the interviews as a chance to take a moment and take a breath from their work as well. Obviously, with Zoom, people opened their things up in a whole range of places. And we as researchers did. We were opening up, doing Zoom meetings from 7 o'clock in the morning to 9 o'clock at night. In the kitchen as people were unpacking or packing to go to Hawaii. We were in our cars, hiding from small kids and dogs. Anyone who has done or run a study knows there is a moment in time when you need to say goodbye. You have to let go, be grateful for the time you have spent together. I think all three of us are still wondering how some of our participants are going. The key part of this project that was really important was to build cross sections and break down silos. To build that was a key objective for this programme at work. We wanted to have a shared understanding of the customer experience. To learn together. So, as Lee said, we recruited key participants and core collaborators. With our core collaborators, we ran sharing story sessions. Every couple of weeks we hosted informal sessions with these collaborators and they watched at least an hour of participant, inside the product. And the value of these sharing stories sessions was certainly about connecting with the customers, but it was also about teams, someone in Poland connecting with someone in San Francisco and understanding what were the initiatives that each of them were working on and how it connected back to the customer as well. These became great team cross collaborations which were anchored in the experience of our customers. LEE RYAN: As you might imagine, we had a lot of data, a lot of
7. 7 conversations that we had with people from different countries, ongoing, so, hundreds of hours. One of the things that was helpful was using Dovetail bring a whole lot of information together and to embrace it as a data repository. So, something that worked really well for us, particularly in our initial interviews, that we would typically spend, have a conversation with participants and then after that review the Zoom video and then write up in Dovetail quite a lot of detail what had happened, we put screen shots in there, capture key quotes. It was quite a detailed summary. And we would create codes in the software for key things we thought had happened. And also, because we had - the observers were watching and putting comments in the Google forms afterwards, we created the double bracket. The double bracket indicated here is an observer comment, what they had noticed but also the language they used to talk about it and what they thought it might mean. We were able to integrate both our observations and theirs. This meant that Dovetail was really an ongoing data repository. The observer, stuff that had been captured in Google forms and we had new forms for each sprint. And that told us what it might mean. By creating category codes and adjusting them and keeping them general enough which meant at any time later if someone was interested in a particular issue or feature we could quickly capture all the interviews that related to that. Because we also had time codes where we had a hunch about something that was really important, it was easy for someone to go, "Oh, yeah, these are all the participants. Here are some key quotes. This is the time stamp that we can go and then edit in ScreenFlow". It was like a catalogue, all those interviews. NATALIE ROWLAND: When doing business-to-business research, taking care of your participants also means taking care of their team members, their company, as well as any of their clients. It's really important when working with video to choose and protect your heroes. You can imagine the kind of information people might reveal about their work, their company, when they feel comfortable. And we needed to protect this information, because it could easily be taken out of context. Choosing the characters to tell different stories is an art form. And what we learned was that telling the most challenging, the most painful stories, are best told by your most endearing characters, the ones that are likable, who are going to be taken seriously, by internal stakeholders. And also keeping in mind the need to be inclusive and diverse in terms of the perspectives. So, we used a lot of subtitling to really help ensure that evocative language was heard. Video's powerful. It needs to be used carefully. If there's one research tool that can be weaponised, it is videos. They need to be more than sound bytes, telling multiple stories. So people don't latch on to one thing. And this is a quote from the wonderful, local ethnographer, Charlie Cochrane. I'll let you read that. It is an absolute burden of responsibility to tell authentic stories. Video storytelling. The editing of the video is an immersive process in itself and it contributes to the sense-making. You see and hear new details every time you review a recording. It needs context, it needs details and it needs a rhythm. Righto, we are at time so we will wrap and share a few lessons learned. Lee, do you want to talk through these or shall I? LEE RYAN: Just thinking... um NATALIE ROWLAND: I can go
8. 8 LEE RYAN: Part of is about experimenting and learning together. It's great to really try, there are a lot more software that we tried and said, "Yes, we can do that, that will work" or we discarded, you know, like Screen Flow was a great find compared to other ones that we were trialling. We also created our own Google dox where we could share like what was working or developing a process so we could go in and add stuff as we went. Casual core team chats. It was great to kind of catch-up on a regular basis and make it free flowing about what is emerging which could be something really big that we thought the client needed to know or a stakeholder or it could be a thread that was going through the research. Making... NATALIE ROWLAND: Sorry, for us to jump in, and just compare what we were finding confusing, unusual, or even, indeed, wonderful. I'll just jump in on the making data visible. It was very much about helping our stakeholders tune in and understand what is significant for our participants early, to help them understand what do our participants care about. So, sharing as you go, was really critical, especially for all of us who hadn't used the product ourselves. Which was handy when people would ask us questions, because we didn't know the answers, but it wasn't handy when it came to try to understand what was going on. We would lean on stakeholders to help us with our interpretations. It was also really important on the practical front to wrap up to take care of each other and we heard about this and particularly working in social research spaces. Even with this project, checking in after any difficult, tricky conversations. And we'll wrap it there.