$30 off During Our Annual Pro Sale. View Details »

Toward Research Literacy

Toward Research Literacy

While understanding specific research methods helps us conduct user experience research, we do our best work when we understand the foundations of research itself.

In this talk, we consider the context of research, answering: How do we even know anything? How do we pick the method we need to understand something in particular? And, how do we make sure we stay within the strengths of the chosen method?

This big picture view helps the researcher and the research consumer alike to better understand and take action based on research findings.

Veronica Erb

July 27, 2018
Tweet

More Decks by Veronica Erb

Other Decks in Research

Transcript

  1. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “There’s so much hokum in focus groups,
    at times bordering on fraud.”
    Who can name who said this? (No Googling!)

    View Slide

  2. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “There’s so much hokum in focus groups,
    at times bordering on fraud.”
    Robert Merton, the inventor of focus groups
    In The New Yorker “The Word Lab” by Nicholas Lemann, 16 October 2000

    View Slide

  3. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “Even when the
    subjects are well selected,
    focus groups are supposed to be
    merely sources of ideas that
    need to be researched.”
    Robert Merton, credited with inventing focus groups
    In The New Yorker “The Word Lab” by Nicholas Lemann, 16 October 2000

    View Slide

  4. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Toward Research Literacy
    Veronica Erb | UX Camp California | 27 July 2018

    View Slide

  5. Working at NPR
    Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com

    View Slide

  6. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part One: What is finding out?
    Part Two: How can we find out?
    Part Three: How do we stay focused?

    View Slide

  7. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    The players
    The Researcher
    The Research
    Consumer
    The Research
    Participant

    View Slide

  8. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part One: What is finding out?
    Part Two: How can we find out?
    Part Three: How do we stay focused?

    View Slide

  9. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part One: What is finding out?

    View Slide

  10. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    A history of philosophies of truth
    Academic perspective

    View Slide

  11. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Three ways to answer:
    When is a statement true?
    As described by Fennell and summarized by Erb

    View Slide

  12. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    The statement matches the world.
    Correspondence theory of truth
    Descartes et al, 17th Century
    As described by Fennell and summarized by Erb

    View Slide

  13. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    We have experienced that
    .the statement matches the world.
    Verificationist theory of truth
    Ayer, Carnap, 20th Century
    As described by Fennell and summarized by Erb

    View Slide

  14. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    We have experienced that
    the statement matches the world
    over time.
    Fallablist theory of truth
    James, 20th Century
    As described by Fennell and summarized by Erb

    View Slide

  15. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    We have experienced that
    .the statement matches the world.
    over time.
    Falliblist theory of truth
    James, 20th Century
    As described by Fennell and summarized by Erb
    Humans are fallible.

    View Slide

  16. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “We have to live today
    by what truth we can get today,
    and be ready tomorrow
    to call it falsehood.”
    William James in “Pragmatism’s Conception of Truth,” March 14, 1907

    View Slide

  17. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “...this fallibilist conception of
    truth appears to be the
    one that science operates with.”
    John Fennell, “Three Philosophical Theories of Truth:
    “Correspondence, Verificationist, and Fallibilist” 30 May 2018

    View Slide

  18. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    There is no “authoritative.”
    What might this mean for user experience research?

    View Slide

  19. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Prioritize research throughout design.
    Strategy

    View Slide

  20. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Example: Selecting Station Staff SSO research
    Where are we starting?
    We interviewed the product owner
    and one developer for each product.
    These interviews helped us know
    what challenges adoption would face.
    How is our plan going?
    We did usability testing with a paper prototype
    of the account migration flow. It let us check
    whether the flow was likely to get in the way
    of accessing the tool.
    Nice! Help
    is on the
    way.
    Cool, I
    know what
    to do.
    I noticed a
    problem!
    Here’s how to
    fix what
    happened.
    — The System
    How did our plan go?
    The automated ticketing that we built into the
    new flow enables us to monitor how many
    people have needed help. When combined
    with account creations, we can notice scale.

    View Slide

  21. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Own the uncertainty.
    Responsibilities

    View Slide

  22. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Lead by example
    ● Not everything needs to be researched.
    ● Don’t overstate authority — it’s only going to bite you later.
    ● Avoiding trust problems with research is a recipe for disappointment!

    View Slide

  23. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part One: What is finding out?
    Although human perception is fallible,
    user experience research can, over time,
    inform our understanding.

    View Slide

  24. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part Two: How can we find out?

    View Slide

  25. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    The difficulty with declaring ways of
    understanding
    Academic perspective

    View Slide

  26. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “I want to examine the instability of our
    current media ecosystem to then return
    to the question of: what kind of media
    literacy should we be working towards?”
    danah boyd, “You Think You Want Media Literacy, Do You?” 9 March 2018

    View Slide

  27. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Media literacy encourages active inquiry
    and critical thinking about the messages
    we receive and create.
    Renee Hobbs, as quoted by danah boyd in
    “You Think You Want Media Literacy, Do You?” 9 March 2018

    View Slide

  28. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “Evangelicals apply the same critique
    they use for the Bible, arguably a
    postmodern method of unpacking a text,
    to mainstream media — favoring their own
    research on topics rather than trusting
    media authorities.”
    Francesca Tripodi, “Alternative Facts, Alternative Truths”, 23 February 2018

    View Slide

  29. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “We’re not living through a crisis about
    what is true, we’re living through a crisis
    about how we know whether something
    is true. We’re not disagreeing about
    facts, we’re disagreeing about
    epistemology.”
    Cory Doctorow, “Three kinds of propaganda, and what to do about them”
    25 February 2017

    View Slide

  30. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “But you can’t resolve fundamental
    epistemological differences
    through compromise.”
    danah boyd, “You Think You Want Media Literacy, Do You?” 9 March 2018

    View Slide

  31. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    We each have unique perspectives.
    What might this mean for user experience research?

    View Slide

  32. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Numbers!
    Stories!

    View Slide

  33. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Numbers!
    Stories!

    View Slide

  34. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Embrace methods for their strengths.
    Strategy

    View Slide

  35. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “Often when we go out in the world and
    list that statistic, we just leave it at that…
    There’s a lot of context that we could be
    providing, that we don’t always give.”
    Nicole Beechum, “Equity in Learning” 16 April 2018
    Interviewed by Carrie Neill

    View Slide

  36. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Example: Evaluating new intranet designs
    Treejack by Optimal Sort
    Tree testing
    To test the navigation of the new intranet, we used tree testing, which isolates
    the information architecture and allowed current staff to try to find the relevant
    pages.
    Usability testing
    To test the interface of the new intranet, we did usability
    testing with current staff members.

    View Slide

  37. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Own the research design phase.
    Responsibilities

    View Slide

  38. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Lead by example
    ● Get to know the researchers around you and their methodological skills. Once
    you do, choose your peer reviewers based on method.
    ● Interview product colleagues to find out what information will help them, not to
    ask them how the research should be done.
    ● Don’t let questions better answered by other methods sneak into your
    research; instead, plan additional research.

    View Slide

  39. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part Two: How can we find out?
    Work toward a mixed methods approach,
    using each for what it does best.

    View Slide

  40. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part Three: How do we stay focused?

    View Slide

  41. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Noticing the complexity of the world
    Academic perspective

    View Slide

  42. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Systems thinking is to the present era
    what the scientific method was for the
    Renaissance. It is a new way of analyzing
    problems and experimenting.
    Jessica Kerr, “The Origins of Opera and the Future of
    Programming” 15 April 2018

    View Slide

  43. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Reductionism lurks around every
    corner; mocking the complexity of the
    living world we are part of.
    Nora Bateson, “Symmathesy: A Word in Progress” 2015
    November 03

    View Slide

  44. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    In fact, I would go so far as to say that
    there is no model or diagram that can
    effectively illustrate the learning within
    the context.
    Nora Bateson, “Symmathesy: A Word in Progress” 2015 November 03

    View Slide

  45. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “How we can improve our perception of
    the complexity we live within, so we may
    improve our interaction with the world?”
    A description of Bateson’s work on the About page of her blog,
    retrieved 21 July 2018

    View Slide

  46. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Humans affect the system.
    What might this mean for user experience research?

    View Slide

  47. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Each person affects the results of research.
    The
    Researcher
    Interprets the data
    The
    Research
    Consumer
    Interprets the report
    The
    Research
    Participant
    Interprets themselves

    View Slide

  48. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Confirm results each step of the way.
    Strategy

    View Slide

  49. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    But human brains are wired to jump to
    solutions. A lot of the process is me
    pulling the reins and asking people to
    slow down, work together, and make
    sure we understand deeply and
    authentically what users are dealing with.
    Yasmine Khan, Curating Context, 5 February 2018
    Interviewed by Carrie Neill

    View Slide

  50. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Example: Podcast survey prezo v. SSO results prezo
    Included findings that could be misinterpreted
    Required slide transition for meaning
    Includes only relevant, supported findings
    Slide stands on its own
    [A slide that is so terrible that
    it cannot be let into the wild!]

    View Slide

  51. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Own the meaning and scope of results.
    Responsibilities

    View Slide

  52. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Lead by example
    ● Explain the method.
    ● Present findings with their context.
    ● Save hypotheses for research planning.

    View Slide

  53. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Part Three: How do we stay focused?
    Recognize that researchers and research
    findings intermingle. Just as our results
    must be updated over time, so must our
    skills and methods.

    View Slide

  54. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Let’s work toward research literacy!

    View Slide

  55. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Truth is perceived over time.
    Prioritize research throughout design.
    Own the uncertainty.
    Part One: What is finding out?

    View Slide

  56. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    We can’t dictate understanding.
    Embrace methods for their strengths.
    Own the research design phase.
    Part Two: How can we find out?

    View Slide

  57. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    The world is complicated.
    Confirm results each step of the way.
    Own the meaning and scope of results.
    Part Three: How do we stay focused?

    View Slide

  58. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Own the uncertainty.
    Own the research design phase.
    Own the meaning and scope of results.
    Responsibilities

    View Slide

  59. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    “Our qualitative adjuncts to the
    experimental design soon convinced that
    brilliant designer of experiments Carl
    Hovland that both kinds of data were
    required for sound conclusions…”
    Robert Merton, The Focussed Interview and Focus Groups:
    Continuities and Discontinuities, Winter 1987

    View Slide

  60. Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
    Research isn’t a set of methods. It’s our
    dynamic process for understanding.
    It’s our responsibility to lead finding out.

    View Slide