seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. Metrics The metrics we’ll use to answer these questions. Collection Methods The methods we’ll use to collect the data. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. Metrics The metrics we’ll use to answer these questions. Collection Methods The methods we’ll use to collect the data. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
government was looking to create a career development tool supporting citizens to build a digital career in government. MEL Framework case study — Digital Careers Pathway Questionnaire Career advice Two key features:
NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? User Interview and Exit Survey
NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? Digital career mobility, progression and capability development. Proportion of managers who feel better equipped in career progression discussion Did it lead to more strategic career conversations between managers and users of the tool? User Interview and Exit Survey Manager Interview and Survey
NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? Digital career mobility, progression and capability development. Proportion of managers who feel better equipped in career progression discussion Did it lead to more strategic career conversations between managers and users of the tool? NSW public sector is seen as an employer of choice for digital specialists User Interview and Exit Survey Manager Interview and Survey Analytics Proportion of users who were not working for the public service Did people use the tool, inside and outside the public service?
NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? Digital career mobility, progression and capability development. Proportion of managers who feel better equipped in career progression discussion Did it lead to more strategic career conversations between managers and users of the tool? NSW public sector is seen as an employer of choice for digital specialists User Interview and Exit Survey Manager Interview and Survey Analytics Proportion of users who were not working for the public service Did people use the tool, inside and outside the public service?
what matters — not what's easy. Take a top-down, outcomes-focused approach This allows us to; — Be led by what’s important, rather than what exists — Evaluate product performance against purpose — Measure and report on what matters How can we realise our mission through our products and services? How can we measure this? ↓ ↓
story simply. The language of evaluation is complex and confusing. This makes frameworks difficult to interpret. Having evaluation questions allows us to; — Improve organisational comprehension and ownership — Create actionable reports with believable recommendations Outcome? Success metric? Indicator? Output? Did it lead to more strategic career conversations between managers and users of the tool? ↓
Metrics. There are myriad ways to measure a key evaluation question. The more you have, the more complex it becomes to determine performance and ultimately to report. Aim for no more than two metrics per evaluation question - one qualitative and quantitative. This allows us to; — Mitigate the risk of relying on one point of data, or too much data — Have enough data to gain insight into patterns and performance
for unintended consequences. Unintended consequences are outcomes generated by the product that weren’t predicted and therefore, aren’t included for evaluation in the MEL framework. They can be positive or negative; either way they are an important learning opportunity. Consider having an impact area dedicated to capturing successes and missteps that don’t fit neatly within the MEL. Creating space to identify and evaluate them allows us to; — Celebrate unprecedented wins — Address and mitigate negative outcomes in the future — Iterate on the product to recreate and capture those positive outcomes more intentionally in the future … … … ↓ ↓ …?
in isolation. In order to believe that something is right, key stakeholders need to be involved in the creation and co-design of the framework. Involve them early and often, to: — Increase understanding and alignment of the goals this framework is trying to achieve — Get their input on what questions they want answered — Increase buy-in that the evaluation questions and metrics, and thus the results we have are correct.
| Friendly | Start with what matters — not what's easy. Tell your story simply. Minimum Viable Metrics. Create space for unintended consequences. Don’t create in isolation. Mission Measures ↓ ↓ Outcome? Success metric? Indicator? Output? …? Take a top down approach to evaluate product performance against purpose. Take a mixed methods approach to minimise the risk of too much, or too little data. Include an impact area that captures successes and missteps that don’t fit neatly within the MEL. Co-create framework with key stakeholders to create believability and buy-in. Translate impact areas into evaluation questions.
seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. Metrics The metrics we’ll use to answer these questions. Collection Methods The methods we’ll use to collect the data. The Measurement, Evaluation & Learning (MEL) Framework
uxaustralia
shosira
nishame
shoko_seven11
centou
base
mottox2
usagimaru
godlingkogami
signull
ryu1013
lw
hjmkth
chriscoyier
seathinner
inesmontani
notwaldorf
sabderemane
mkilby
techseoconnect
greggifford
jnunemaker
shlominoach
jonoalderson
