Using diagnostic classification models to improve instructional decision-making

Transcript

1. The UNIVERSITY of KANSAS Accessible Teaching, Learning, and Assessment Systems (ATLAS) Using Diagnostic Classification Models to Improve Instructional Decision-Making W. Jake Thompson
2. Who am I? W. Jake Thompson, Ph.D. • Assistant Director of Psychometrics The UNIVERSITY of KANSAS • ATLAS | University of Kansas • Research: Applications of diagnostic psychometric models • Lead psychometrician and Co-PI for the Dynamic Learning Maps assessments • PI for an IES-funded project to develop software for diagnostic models @wjakethompson @wjakethompson.com 2026-10-01 2
3. The UNIVERSITY of KANSAS What are diagnostic models?
4. 2026-10-01 4 The UNIVERSITY of KANSAS
5. It's me (hi!) I'm the problem • Developing an assessment of musical knowledge, skills, and abilities • Each respondent is an album The UNIVERSITY of KANSAS • What we want to know: • What makes an album good or bad? • What could an album work on in order to improve? 2026-10-01 5
6. • Traditional assessments and psychometric models measure an overall skill or ability The UNIVERSITY of KANSAS • Assume a continuous latent trait 2026-10-01 6
7. The UNIVERSITY of KANSAS • The output is a weak ordering due to error in estimates • Confident Taylor Swift (debut) is the worst • Toward the middle of the distribution, the ordering is delicate 2026-10-01 7
8. The UNIVERSITY of KANSAS • Limited in the types of questions that can be answered • Why is Taylor Swift (debut) so low? • What aspects do an era demonstrate proficiency or competency of? • How much skill is "enough" to be competent? 2026-10-01 8
9. Diagnostic measurement • Designed to be multidimensional • No continuum of student achievement • Categorical constructs The UNIVERSITY of KANSAS • Usually binary (e.g., master/nonmaster, proficient/not proficient) • Several different names in the literature • Diagnostic classification models (DCMs) • Cognitive diagnostic models (CDMs) • Skills assessment models • Restricted latent class models 2026-10-01 9
10. Diagnostic music assessment The UNIVERSITY of KANSAS • Rather than measuring overall musical knowledge, we can break down music into a set of skill, or attributes Songwriting Production Vocals • Songwriting • Production • Vocals • Attributes are categorical, often dichotomous (e.g., proficient vs. not proficient) 2026-10-01 10
11. Benefits of DCMs • No scale, no overall "ability" SONGWRITING PRODUCTION VOCALS • DCMs probabilistically place individuals into classes The UNIVERSITY of KANSAS • Classes represented by skill profiles • High reliability with fewer items • Less information needed to classify than to place precisely along a scale 2026-10-01 11
12. Can I ask you a question? • Fine-grained, multidimensional results to answer more questions SONGWRITING PRODUCTION VOCALS • Why is Taylor Swift (debut) so low? The UNIVERSITY of KANSAS • Subpar songwriting, production, and vocals • What aspects are albums competent/proficient in? • DCMs provide classifications directly • How much skill is "enough" to be competent? • Empirically estimated by the model 2026-10-01 12
13. Fine-grained feedback • Distinguish between respondents who may have similar scale scores PRODUCTION VOCALS The UNIVERSITY of KANSAS SONGWRITING 2026-10-01 13
14. Profiles, not rankings • DCMs do not distinguish within classes PRODUCTION VOCALS The UNIVERSITY of KANSAS SONGWRITING 2026-10-01 14
15. When are DCMs appropriate? • Success depends on domain definitions • What are the attributes we're trying to measure? • Are the attributes measurable (e.g., with assessment items)? • Does the domain support categorical classifications? The UNIVERSITY of KANSAS • Alignment of assessment purpose and psychometric model • Are we interested in which respondent is "best/worst" or "highest/lowest"? • Or are we interested in the specific skills a respondent has mastered, and which could use additional support? 2026-10-01 15
16. The UNIVERSITY of KANSAS Using diagnostic models to improve student outcomes
17. Applications in educational assessment • Dynamic Learning Maps® QR code linking to: https://doi.org/10.1111/emip.12619 The UNIVERSITY of KANSAS • Achievement assessment for students with the most significant cognitive disabilities • Currently used in 25 states for accountability reporting to the U.S. Department of Education • Pathways for Instructionally Embedded Assessment • Competitive Grant for State Assessment award from the U.S. Department of Education • Partnership between ATLAS and the Missouri Department of Elementary and Secondary Education 2026-10-01 Thompson & Clark (2024): Improving instructional decision-making using diagnostic classification models 17
18. The UNIVERSITY of KANSAS
19. Dynamic Learning Maps (DLM) • Assessment items are aligned to extended content standards • Academic content available a multiple levels of complexity for each standard • Results are reported as a profile of skills mastered within each standard The UNIVERSITY of KANSAS • Teachers have flexibility to assign the level most appropriate for their student 2026-10-01 19
20. The UNIVERSITY of KANSAS
21. Using DLM assessment results • Skills within each standard follow a linear hierarchy • Efficiency in instruction and assessment administration to know where to focus The UNIVERSITY of KANSAS • Results are used throughout the year to inform individual education program (IEP) plans and instructional planning (Clark et al., 2023) 2026-10-01 21
22. The UNIVERSITY of KANSAS
23. Pathways for Instructionally Embedded Assessment (PIE) • Pathway levels were developed for grade 5 mathematics to identify precursor concepts and skills related to the learning target of each standard The UNIVERSITY of KANSAS • Students are assessed on each pathway level within the standard over the course of an instructional cycle • Baseline: Level 1 • Midway: Level 1 (retest) + Level 2 • End-of-unit: Level 2 (retest) + Level 3 • Teacher can use results to implement small group or individualized instructional strategies (ATLAS, 2025) 2026-10-01 23
24. PIE instructional cycle The UNIVERSITY of KANSAS Content Group (for one cycle/unit of instruction) Baseline Assessment Midway Assessment End-of-Unit Assessment Checks for EMERGING skills and understanding that lead up to the gradelevel standards Checks for understanding and skills APPROACHING the grade-level standards Checks for understanding and skills associated with the grade-level learning targets Begin again with next content group 2026-10-01 24
25. 2026-10-01 25 The UNIVERSITY of KANSAS
26. 2026-10-01 26 The UNIVERSITY of KANSAS
27. Additional research applications • Evaluate educator understanding of concepts to inform professional development opportunities (e.g., Bradshaw et al., 2014; Izsák et al., 2019) The UNIVERSITY of KANSAS • Understand profiles of mental health symptoms (e.g., Tan et al., 2023) • Evaluate the effect of interventions as transitions in proficiency status (e.g., Madison & Bradshaw, 2018) • How to get started… 2026-10-01 27
28. measr: Diagnostic models in R • Easily specify and estimate a DCM The UNIVERSITY of KANSAS • Wide variety of DCMs (e.g., LCDM, DINA, C-RUM) • Defined attribute relationships and dependencies • Supports maximum likelihood and full MCMC model estimation • Powerful model evaluation tools • Model fit using posterior predictive model checks • Model comparisons with leave-one-out cross validation • Classification accuracy and consistency metrics • Are you ready for it?: https://measr.r-dcm.org 2026-10-01 28
29. Thank you! Get in touch! • Interested in learning more about diagnostic models and measr? The UNIVERSITY of KANSAS • Today, 4:00–4:30pm • Bay QR code linking to: https://speakerdeck.com/wjakethompson/dcm-instructional-decisions Slides
30. The UNIVERSITY of KANSAS Acknowledgements The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305D210045 and R305D240032 to the University of Kansas Center for Research, Inc., ATLAS. The opinions expressed are those of the authors and do not represent the views of the the Institute or the U.S. Department of Education.