Toward Research Literacy

While understanding specific research methods helps us conduct user experience research, we do our best work when we understand the foundations of research itself.

In this talk, we consider the context of research, answering: How do we even know anything? How do we pick the method we need to understand something in particular? And, how do we make sure we stay within the strengths of the chosen method?

This big picture view helps the researcher and the research consumer alike to better understand and take action based on research findings.

Transcript

1. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “There’s so much hokum in focus groups, at times bordering on fraud.” Who can name who said this? (No Googling!)
2. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “There’s so much hokum in focus groups, at times bordering on fraud.” Robert Merton, the inventor of focus groups In The New Yorker “The Word Lab” by Nicholas Lemann, 16 October 2000
3. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “Even when the subjects are well selected, focus groups are supposed to be merely sources of ideas that need to be researched.” Robert Merton, credited with inventing focus groups In The New Yorker “The Word Lab” by Nicholas Lemann, 16 October 2000
4. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Toward Research Literacy Veronica Erb | UX Camp California | 27 July 2018
5. Working at NPR Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
6. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part One: What is finding out? Part Two: How can we find out? Part Three: How do we stay focused?
7. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com The players The Researcher The Research Consumer The Research Participant
8. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part One: What is finding out? Part Two: How can we find out? Part Three: How do we stay focused?
9. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part One: What is finding out?
10. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com A history of philosophies of truth Academic perspective
11. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Three ways to answer: When is a statement true? As described by Fennell and summarized by Erb
12. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com The statement matches the world. Correspondence theory of truth Descartes et al, 17th Century As described by Fennell and summarized by Erb
13. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com We have experienced that .the statement matches the world. Verificationist theory of truth Ayer, Carnap, 20th Century As described by Fennell and summarized by Erb
14. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com We have experienced that the statement matches the world over time. Fallablist theory of truth James, 20th Century As described by Fennell and summarized by Erb
15. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com We have experienced that .the statement matches the world. over time. Falliblist theory of truth James, 20th Century As described by Fennell and summarized by Erb Humans are fallible.
16. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “We have to live today by what truth we can get today, and be ready tomorrow to call it falsehood.” William James in “Pragmatism’s Conception of Truth,” March 14, 1907
17. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “...this fallibilist conception of truth appears to be the one that science operates with.” John Fennell, “Three Philosophical Theories of Truth: “Correspondence, Verificationist, and Fallibilist” 30 May 2018
18. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com There is no “authoritative.” What might this mean for user experience research?
19. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Prioritize research throughout design. Strategy
20. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Example: Selecting Station Staff SSO research Where are we starting? We interviewed the product owner and one developer for each product. These interviews helped us know what challenges adoption would face. How is our plan going? We did usability testing with a paper prototype of the account migration flow. It let us check whether the flow was likely to get in the way of accessing the tool. Nice! Help is on the way. Cool, I know what to do. I noticed a problem! Here’s how to fix what happened. — The System How did our plan go? The automated ticketing that we built into the new flow enables us to monitor how many people have needed help. When combined with account creations, we can notice scale.
21. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Own the uncertainty. Responsibilities
22. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Lead by example • Not everything needs to be researched. • Don’t overstate authority — it’s only going to bite you later. • Avoiding trust problems with research is a recipe for disappointment!
23. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part One: What is finding out? Although human perception is fallible, user experience research can, over time, inform our understanding.
24. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part Two: How can we find out?
25. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com The difficulty with declaring ways of understanding Academic perspective
26. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “I want to examine the instability of our current media ecosystem to then return to the question of: what kind of media literacy should we be working towards?” danah boyd, “You Think You Want Media Literacy, Do You?” 9 March 2018
27. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Media literacy encourages active inquiry and critical thinking about the messages we receive and create. Renee Hobbs, as quoted by danah boyd in “You Think You Want Media Literacy, Do You?” 9 March 2018
28. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “Evangelicals apply the same critique they use for the Bible, arguably a postmodern method of unpacking a text, to mainstream media — favoring their own research on topics rather than trusting media authorities.” Francesca Tripodi, “Alternative Facts, Alternative Truths”, 23 February 2018
29. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “We’re not living through a crisis about what is true, we’re living through a crisis about how we know whether something is true. We’re not disagreeing about facts, we’re disagreeing about epistemology.” Cory Doctorow, “Three kinds of propaganda, and what to do about them” 25 February 2017
30. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “But you can’t resolve fundamental epistemological differences through compromise.” danah boyd, “You Think You Want Media Literacy, Do You?” 9 March 2018
31. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com We each have unique perspectives. What might this mean for user experience research?
32. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Numbers! Stories!
33. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Numbers! Stories!
34. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Embrace methods for their strengths. Strategy
35. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “Often when we go out in the world and list that statistic, we just leave it at that… There’s a lot of context that we could be providing, that we don’t always give.” Nicole Beechum, “Equity in Learning” 16 April 2018 Interviewed by Carrie Neill
36. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Example: Evaluating new intranet designs Treejack by Optimal Sort Tree testing To test the navigation of the new intranet, we used tree testing, which isolates the information architecture and allowed current staff to try to find the relevant pages. Usability testing To test the interface of the new intranet, we did usability testing with current staff members.
37. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Own the research design phase. Responsibilities
38. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Lead by example • Get to know the researchers around you and their methodological skills. Once you do, choose your peer reviewers based on method. • Interview product colleagues to find out what information will help them, not to ask them how the research should be done. • Don’t let questions better answered by other methods sneak into your research; instead, plan additional research.
39. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part Two: How can we find out? Work toward a mixed methods approach, using each for what it does best.
40. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part Three: How do we stay focused?
41. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Noticing the complexity of the world Academic perspective
42. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Systems thinking is to the present era what the scientific method was for the Renaissance. It is a new way of analyzing problems and experimenting. Jessica Kerr, “The Origins of Opera and the Future of Programming” 15 April 2018
43. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Reductionism lurks around every corner; mocking the complexity of the living world we are part of. Nora Bateson, “Symmathesy: A Word in Progress” 2015 November 03
44. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com In fact, I would go so far as to say that there is no model or diagram that can effectively illustrate the learning within the context. Nora Bateson, “Symmathesy: A Word in Progress” 2015 November 03
45. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “How we can improve our perception of the complexity we live within, so we may improve our interaction with the world?” A description of Bateson’s work on the About page of her blog, retrieved 21 July 2018
46. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Humans affect the system. What might this mean for user experience research?
47. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Each person affects the results of research. The Researcher Interprets the data The Research Consumer Interprets the report The Research Participant Interprets themselves
48. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Confirm results each step of the way. Strategy
49. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com But human brains are wired to jump to solutions. A lot of the process is me pulling the reins and asking people to slow down, work together, and make sure we understand deeply and authentically what users are dealing with. Yasmine Khan, Curating Context, 5 February 2018 Interviewed by Carrie Neill
50. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Example: Podcast survey prezo v. SSO results prezo Included findings that could be misinterpreted Required slide transition for meaning Includes only relevant, supported findings Slide stands on its own [A slide that is so terrible that it cannot be let into the wild!]
51. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Own the meaning and scope of results. Responsibilities
52. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Lead by example • Explain the method. • Present findings with their context. • Save hypotheses for research planning.
53. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Part Three: How do we stay focused? Recognize that researchers and research findings intermingle. Just as our results must be updated over time, so must our skills and methods.
54. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Let’s work toward research literacy!
55. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Truth is perceived over time. Prioritize research throughout design. Own the uncertainty. Part One: What is finding out?
56. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com We can’t dictate understanding. Embrace methods for their strengths. Own the research design phase. Part Two: How can we find out?
57. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com The world is complicated. Confirm results each step of the way. Own the meaning and scope of results. Part Three: How do we stay focused?
58. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Own the uncertainty. Own the research design phase. Own the meaning and scope of results. Responsibilities
59. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com “Our qualitative adjuncts to the experimental design soon convinced that brilliant designer of experiments Carl Hovland that both kinds of data were required for sound conclusions…” Robert Merton, The Focussed Interview and Focus Groups: Continuities and Discontinuities, Winter 1987
60. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com Research isn’t a set of methods. It’s our dynamic process for understanding. It’s our responsibility to lead finding out.