Toward Research Literacy

Toward Research Literacy | 27 August 2018 | Veronica Erb | veronicaerb.com
“There’s so much hokum in focus groups,
at times bordering on fraud.”
Who can name who said this? (No Googling!)

View Slide

“There’s so much hokum in focus groups,
at times bordering on fraud.”
Robert Merton, the inventor of focus groups
In The New Yorker “The Word Lab” by Nicholas Lemann, 16 October 2000

View Slide

“Even when the
subjects are well selected,
focus groups are supposed to be
merely sources of ideas that
need to be researched.”
Robert Merton, credited with inventing focus groups
In The New Yorker “The Word Lab” by Nicholas Lemann, 16 October 2000

View Slide

Toward Research Literacy
Veronica Erb | UX Camp California | 27 July 2018

View Slide

Working at NPR

View Slide

Part One: What is finding out?
Part Two: How can we find out?
Part Three: How do we stay focused?

View Slide

The players
The Researcher
The Research
Consumer
The Research
Participant

View Slide


View Slide

A history of philosophies of truth
Academic perspective

View Slide

Three ways to answer:
When is a statement true?
As described by Fennell and summarized by Erb

View Slide

The statement matches the world.
Correspondence theory of truth
Descartes et al, 17th Century

View Slide

We have experienced that
.the statement matches the world.
Verificationist theory of truth
Ayer, Carnap, 20th Century

View Slide

the statement matches the world
over time.
Fallablist theory of truth
James, 20th Century

View Slide

.the statement matches the world.
over time.
Falliblist theory of truth
James, 20th Century
Humans are fallible.

View Slide

“We have to live today
by what truth we can get today,
and be ready tomorrow
to call it falsehood.”
William James in “Pragmatism’s Conception of Truth,” March 14, 1907

View Slide

“...this fallibilist conception of
truth appears to be the
one that science operates with.”
John Fennell, “Three Philosophical Theories of Truth:
“Correspondence, Verificationist, and Fallibilist” 30 May 2018

View Slide

There is no “authoritative.”
What might this mean for user experience research?

View Slide

Prioritize research throughout design.
Strategy

View Slide

Example: Selecting Station Staff SSO research
Where are we starting?
We interviewed the product owner
and one developer for each product.
These interviews helped us know
what challenges adoption would face.
How is our plan going?
We did usability testing with a paper prototype
of the account migration flow. It let us check
whether the flow was likely to get in the way
of accessing the tool.
Nice! Help
is on the
way.
Cool, I
know what
to do.
I noticed a
problem!
Here’s how to
fix what
happened.
— The System
How did our plan go?
The automated ticketing that we built into the
new flow enables us to monitor how many
people have needed help. When combined
with account creations, we can notice scale.

View Slide

Own the uncertainty.
Responsibilities

View Slide

Lead by example
● Not everything needs to be researched.
● Don’t overstate authority — it’s only going to bite you later.
● Avoiding trust problems with research is a recipe for disappointment!

View Slide

Although human perception is fallible,
user experience research can, over time,
inform our understanding.

View Slide


View Slide

The difficulty with declaring ways of
understanding

View Slide

“I want to examine the instability of our
current media ecosystem to then return
to the question of: what kind of media
literacy should we be working towards?”
danah boyd, “You Think You Want Media Literacy, Do You?” 9 March 2018

View Slide

Media literacy encourages active inquiry
and critical thinking about the messages
we receive and create.
Renee Hobbs, as quoted by danah boyd in
“You Think You Want Media Literacy, Do You?” 9 March 2018

View Slide

“Evangelicals apply the same critique
they use for the Bible, arguably a
postmodern method of unpacking a text,
to mainstream media — favoring their own
research on topics rather than trusting
media authorities.”
Francesca Tripodi, “Alternative Facts, Alternative Truths”, 23 February 2018

View Slide

“We’re not living through a crisis about
what is true, we’re living through a crisis
about how we know whether something
is true. We’re not disagreeing about
facts, we’re disagreeing about
epistemology.”
Cory Doctorow, “Three kinds of propaganda, and what to do about them”
25 February 2017

View Slide

“But you can’t resolve fundamental
epistemological differences
through compromise.”
danah boyd, “You Think You Want Media Literacy, Do You?” 9 March 2018

View Slide

We each have unique perspectives.

View Slide

Numbers!
Stories!

View Slide

Embrace methods for their strengths.
Strategy

View Slide

“Often when we go out in the world and
list that statistic, we just leave it at that…
There’s a lot of context that we could be
providing, that we don’t always give.”
Nicole Beechum, “Equity in Learning” 16 April 2018
Interviewed by Carrie Neill

View Slide

Example: Evaluating new intranet designs
Treejack by Optimal Sort
Tree testing
To test the navigation of the new intranet, we used tree testing, which isolates
the information architecture and allowed current staff to try to find the relevant
pages.
Usability testing
To test the interface of the new intranet, we did usability
testing with current staff members.

View Slide

Own the research design phase.
Responsibilities

View Slide

Lead by example
● Get to know the researchers around you and their methodological skills. Once
you do, choose your peer reviewers based on method.
● Interview product colleagues to find out what information will help them, not to
ask them how the research should be done.
● Don’t let questions better answered by other methods sneak into your
research; instead, plan additional research.

View Slide

Work toward a mixed methods approach,
using each for what it does best.

View Slide


View Slide

Noticing the complexity of the world

View Slide

Systems thinking is to the present era
what the scientific method was for the
Renaissance. It is a new way of analyzing
problems and experimenting.
Jessica Kerr, “The Origins of Opera and the Future of
Programming” 15 April 2018

View Slide

Reductionism lurks around every
corner; mocking the complexity of the
living world we are part of.
Nora Bateson, “Symmathesy: A Word in Progress” 2015
November 03

View Slide

In fact, I would go so far as to say that
there is no model or diagram that can
effectively illustrate the learning within
the context.
Nora Bateson, “Symmathesy: A Word in Progress” 2015 November 03

View Slide

“How we can improve our perception of
the complexity we live within, so we may
improve our interaction with the world?”
A description of Bateson’s work on the About page of her blog,
retrieved 21 July 2018

View Slide

Humans affect the system.

View Slide

Each person affects the results of research.
The
Researcher
Interprets the data
The
Research
Consumer
Interprets the report
The
Research
Participant
Interprets themselves

View Slide

Confirm results each step of the way.
Strategy

View Slide

But human brains are wired to jump to
solutions. A lot of the process is me
pulling the reins and asking people to
slow down, work together, and make
sure we understand deeply and
authentically what users are dealing with.
Yasmine Khan, Curating Context, 5 February 2018
Interviewed by Carrie Neill

View Slide

Example: Podcast survey prezo v. SSO results prezo
Included findings that could be misinterpreted
Required slide transition for meaning
Includes only relevant, supported findings
Slide stands on its own
[A slide that is so terrible that
it cannot be let into the wild!]

View Slide

Own the meaning and scope of results.
Responsibilities

View Slide

Lead by example
● Explain the method.
● Present findings with their context.
● Save hypotheses for research planning.

View Slide

Recognize that researchers and research
findings intermingle. Just as our results
must be updated over time, so must our
skills and methods.

View Slide

Let’s work toward research literacy!

View Slide

Truth is perceived over time.
Prioritize research throughout design.

View Slide

We can’t dictate understanding.
Embrace methods for their strengths.

View Slide

The world is complicated.
Confirm results each step of the way.

View Slide

Responsibilities

View Slide

“Our qualitative adjuncts to the
experimental design soon convinced that
brilliant designer of experiments Carl
Hovland that both kinds of data were
required for sound conclusions…”
Robert Merton, The Focussed Interview and Focus Groups:
Continuities and Discontinuities, Winter 1987

View Slide

Research isn’t a set of methods. It’s our
dynamic process for understanding.
It’s our responsibility to lead finding out.

View Slide

Toward Research Literacy

Toward Research Literacy

Veronica Erb

More Decks by Veronica Erb

Other Decks in Research

Featured

Transcript