Discussion: Reconceptualizing Diagnostic Classification Models: Applications and New Developments

Invited discussion for the "Reconceptualizing Diagnostic Classification Models: Applications and New Developments" symposium, organized by the NCME Diagnostic Measurement SIGIMIE.

Transcript

1. Reconceptualizing Diagnostic Classification Models: Applications and New Developments Discussion W. Jake Thompson, Ph.D. 4/13/2024
2. Diagnostic assessment • Goal: Assessments can be used to improve student learning, increasing overall achievement • Diagnostic models are uniquely suited to this goal • Fine-grained, multi-dimensional results • High classification reliability with relatively few items • Proliferation of psychometric research in recent years 2
3. Today's session 3 • Five very strong papers describing recent advancements in diagnostic modeling • Incorporating new data sources • New statistical models • Three themes in diagnostic modeling research reflected in our session
5. Theme 1: New directions over applications 5 • The literature of DCMs has primarily focused on creating new models, incorporating new data, or developing new methods for evaluating model performance • Huang et al.; Lee & Gu; Mireles & Bao; Zhang • Relatively few studies examining the implementation of DCMs or using DCMs as a tool for answering substantive questions about constructs of interest • Lugu et al.
6. "Do we really need to … showcase … hyper-complex diagnostic measurement models for educational contexts if we can barely get to the point where even 'simpler' models are used meaningfully at scale?" 6 André A. Rupp Center for Assessment
7. Future directions? 7 • Focus not just on what the new method/model is • What are the intended uses, and who are the intended users? • Make friends with our colleagues in subject-area specialties • Collaborate on projects to implement the models that have been developed • Answer substantive questions with DCMs to add to the literature on student learning and instructional best practices
9. Theme 2: Accessibility to applied researchers 9 • Research describing new methods often does not offer access to applied researchers • Create bespoke scripts for specific analyses • Describe estimation algorithms, but do not provide code • Applied researchers may not have expertise to create their own implementation of methods • This creates a gap between the state of research and the state of practice
10. Future directions? 10 • New developments are only useful if they can be accessed by researchers • Make software for estimating new methods available • For application focused papers: • Include descriptions of software that was used • Make code available so that other researchers can see how to apply the methods
12. Theme 3: Incorporating new data sources 12 • Technology has opened up new data collection opportunities • Response times (Huang et al.; Lee & Gu; Mireles & Bao) • Eye tracking (Lee & Gu) • Clickstreams (Zhang) • We assume that these new indicators are indicative of the construct • Unintended consequences
13. Future directions? 13 • Additional research needed to support the alignment of new data sources to the intended construct • Rich literature for measuring the alignment of traditional items • Methods for evaluating invariance across different groups • Response time may be systematically different for students with extensive support needs, or students in rural areas with limited internet bandwidth
14. Summary 14 • Ultimately our goal is to improve learning or practice through diagnostic models • Need scaffolding to support applications • Theory tying data to the intended cognition • Accessible software • Case studies to illustrate usage • Discussions of appropriate and inappropriate uses
15. Get in touch! 15 Thank you! • Reception: Today! • 3:00–4:00pm • Convention Center 126A • SIGIMIE Website: https://www.ncme.org/community/ncme- sigimie/diagnostic-measurement