DR2023 Laura Ryan & Brendan Reed - Design Research and the Evaluation Frontier

Transcript

1. Design Research
   and the Evaluation
   Frontier
   Laura Ryan — Partner, Strategy
   Brendan Reed — Product Strategist futurefriendly.team
   

   View Slide

2. | 2 |
   | Future | Friendly | Three key questions |
   01. Why should measurement matter to me?
   02. How do I decide what to measure?
   03. How do I put it into practice?
   Three
   key
   questions
   

   View Slide

3. | 3 |
   | Future | Friendly |
   Chapter 1
   Why does
   measurement
   matter?
   

   View Slide

4. | 4 |
   Every day,
   digital services
   have the power
   to change
   people's lives and
   build a better
   future.
   | Future | Friendly |
   Testing a prototype of the Cost of Living Savings Finder for ServiceNSW
   

   View Slide

5. | 5 |
   A cadence of user testing
   is critical for de-risking
   during concept
   development.
   | Future | Friendly |
   Usability testing helps us
   build confidence with
   new feature releases.
   

   View Slide

6. | 6 |
   | Future | Friendly |
   How certain are you that
   the digital product you
   researched, is delivering
   positive outcomes for
   people in the real world?
   

   View Slide

7. | 7 |
   “If you cannot measure it,
   you cannot improve it.”
   — Lord Kelvin
   | Future | Friendly |
   

   View Slide

8. | 8 |
   | Future | Friendly |
   Chapter 2
   Measurement,
   Evaluation &
   Learning
   Framework
   

   View Slide

9. | 9 |
   | Future | Friendly |
   Align stakeholders on what
   is important to measure.
   Be highly actionable. Be easily consumed.
   We designed this measurement and evaluation framework to:
   

   View Slide

10. | 10 |
    Share an example of it in practice.
    Introduce the MEL framework.
    | Future | Friendly |
    The Measurement, Evaluation &
    Learning (MEL) Framework
    

    View Slide

11. | 11 |
    Mission
    Our purpose; the change we are seeking to make.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |
    

    View Slide

12. | 12 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |
    

    View Slide

13. | 13 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |
    

    View Slide

14. | 14 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    Metrics
    The metrics we’ll use to answer
    these questions.
    Collection Methods
    The methods we’ll use to collect
    the data.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |
    

    View Slide

15. | 15 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    Metrics
    The metrics we’ll use to answer
    these questions.
    Collection Methods
    The methods we’ll use to collect
    the data.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |
    

    View Slide

16. | 16 |
    | Future | Friendly |
    NSW state government was
    looking to create a career
    development tool supporting
    citizens to build a digital
    career in government.
    MEL Framework case study — Digital Careers Pathway
    Questionnaire Career advice
    Two key features:
    

    View Slide

17. | 17 |
    | Future | Friendly |
    Having conducted over
    50 hours of concept and
    usability testing, it was
    time to pilot the product
    in the real world.
    

    View Slide

18. Mission
    | 18 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    | Future | Friendly |
    

    View Slide

19. Metric
    Collection method
    Mission
    Evaluation Questions
    | 19 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    User Interview and Exit Survey
    

    View Slide

20. Metric
    Collection method
    Mission
    Evaluation Questions
    | 20 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    Digital career mobility, progression and
    capability development.
    Proportion of managers who feel better
    equipped in career progression discussion
    Did it lead to more strategic career
    conversations between managers and
    users of the tool?
    User Interview and Exit Survey Manager Interview and Survey
    

    View Slide

21. Metric
    Collection method
    Mission
    Evaluation Questions
    | 21 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    Digital career mobility, progression and
    capability development.
    Proportion of managers who feel better
    equipped in career progression discussion
    Did it lead to more strategic career
    conversations between managers and
    users of the tool?
    NSW public sector is seen as an employer
    of choice for digital specialists
    User Interview and Exit Survey Manager Interview and Survey Analytics
    Proportion of users who were not working
    for the public service
    Did people use the tool, inside
    and outside the public service?
    

    View Slide

22. Metric
    Collection method
    Mission
    Evaluation Questions
    | 22 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    Digital career mobility, progression and
    capability development.
    Proportion of managers who feel better
    equipped in career progression discussion
    Did it lead to more strategic career
    conversations between managers and
    users of the tool?
    NSW public sector is seen as an employer
    of choice for digital specialists
    User Interview and Exit Survey Manager Interview and Survey Analytics
    Proportion of users who were not working
    for the public service
    Did people use the tool, inside
    and outside the public service?
    

    View Slide

23. | 23 |
    | Future | Friendly |
    Evaluation
    principles
    Chapter 3
    

    View Slide

24. | 24 |
    | Future | Friendly |
    Start with what
    matters — not
    what's easy.
    Take a top-down, outcomes-focused approach
    This allows us to;
    — Be led by what’s important, rather than what exists
    — Evaluate product performance against purpose
    — Measure and report on what matters
    How can we realise our mission
    through our products and
    services?
    How can we
    measure this?
    ↓ ↓
    

    View Slide

25. | 25 |
    | Future | Friendly |
    Tell your story
    simply.
    The language of evaluation is complex and confusing.
    This makes frameworks difficult to interpret.
    Having evaluation questions allows us to;
    — Improve organisational comprehension and
    ownership
    — Create actionable reports with believable
    recommendations
    Outcome?
    Success metric?
    Indicator?
    Output?
    Did it lead to more strategic career conversations
    between managers and users of the tool?
    ↓
    

    View Slide

26. | 26 |
    | Future | Friendly |
    Minimum Viable
    Metrics.
    There are myriad ways to measure a key evaluation
    question. The more you have, the more complex it
    becomes to determine performance and ultimately to
    report.
    Aim for no more than two metrics per evaluation question
    - one qualitative and quantitative.
    This allows us to;
    — Mitigate the risk of relying on one point of data, or
    too much data
    — Have enough data to gain insight into patterns and
    performance
    

    View Slide

27. | 27 |
    | Future | Friendly |
    Create space for
    unintended
    consequences.
    Unintended consequences are outcomes generated by
    the product that weren’t predicted and therefore, aren’t
    included for evaluation in the MEL framework. They can
    be positive or negative; either way they are an important
    learning opportunity.
    Consider having an impact area dedicated to capturing
    successes and missteps that don’t fit neatly within the
    MEL.
    Creating space to identify and evaluate them allows us to;
    — Celebrate unprecedented wins
    — Address and mitigate negative outcomes in the
    future
    — Iterate on the product to recreate and capture those
    positive outcomes more intentionally in the future
    …
    …
    …
    ↓
    ↓
    …?
    

    View Slide

28. | 28 |
    | Future | Friendly |
    Don’t create in
    isolation.
    In order to believe that something is right, key
    stakeholders need to be involved in the creation and
    co-design of the framework.
    Involve them early and often, to:
    — Increase understanding and alignment of the goals
    this framework is trying to achieve
    — Get their input on what questions they want
    answered
    — Increase buy-in that the evaluation questions and
    metrics, and thus the results we have are correct.
    

    View Slide

29. Evaluation principles — MEL Framework
    | 29 |
    | Future | Friendly |
    Start with what
    matters — not
    what's easy.
    Tell your story
    simply.
    Minimum
    Viable Metrics.
    Create space
    for unintended
    consequences.
    Don’t create in
    isolation.
    Mission
    Measures
    ↓ ↓
    Outcome?
    Success
    metric? Indicator?
    Output? …?
    Take a top down
    approach to evaluate
    product performance
    against purpose.
    Take a mixed methods
    approach to minimise
    the risk of too much,
    or too little data.
    Include an impact area
    that captures
    successes and
    missteps that don’t fit
    neatly within the MEL.
    Co-create framework
    with key stakeholders
    to create believability
    and buy-in.
    Translate impact areas
    into evaluation
    questions.
    

    View Slide

30. | 30 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    Metrics
    The metrics we’ll use to answer
    these questions.
    Collection Methods
    The methods we’ll use to collect
    the data.
    The Measurement, Evaluation & Learning (MEL) Framework
    

    View Slide

31. | 31 |
    | Future | Friendly |
    Thank you.
    → futurefriendly.team
    

    View Slide