Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Discussion: Reconceptualizing Diagnostic Classification Models: Applications and New Developments

Discussion: Reconceptualizing Diagnostic Classification Models: Applications and New Developments

Invited discussion for the "Reconceptualizing Diagnostic Classification Models: Applications and New Developments" symposium, organized by the NCME Diagnostic Measurement SIGIMIE.

Jake Thompson

April 13, 2024

More Decks by Jake Thompson

Other Decks in Education


  1. Diagnostic assessment • Goal: Assessments can be used to improve

    student learning, increasing overall achievement • Diagnostic models are uniquely suited to this goal • Fine-grained, multi-dimensional results • High classification reliability with relatively few items • Proliferation of psychometric research in recent years 2
  2. Today's session 3 • Five very strong papers describing recent

    advancements in diagnostic modeling • Incorporating new data sources • New statistical models • Three themes in diagnostic modeling research reflected in our session
  3. 4

  4. Theme 1: New directions over applications 5 • The literature

    of DCMs has primarily focused on creating new models, incorporating new data, or developing new methods for evaluating model performance • Huang et al.; Lee & Gu; Mireles & Bao; Zhang • Relatively few studies examining the implementation of DCMs or using DCMs as a tool for answering substantive questions about constructs of interest • Lugu et al.
  5. "Do we really need to … showcase … hyper-complex diagnostic

    measurement models for educational contexts if we can barely get to the point where even 'simpler' models are used meaningfully at scale?" 6 André A. Rupp Center for Assessment
  6. Future directions? 7 • Focus not just on what the

    new method/model is • What are the intended uses, and who are the intended users? • Make friends with our colleagues in subject-area specialties • Collaborate on projects to implement the models that have been developed • Answer substantive questions with DCMs to add to the literature on student learning and instructional best practices
  7. 8

  8. Theme 2: Accessibility to applied researchers 9 • Research describing

    new methods often does not offer access to applied researchers • Create bespoke scripts for specific analyses • Describe estimation algorithms, but do not provide code • Applied researchers may not have expertise to create their own implementation of methods • This creates a gap between the state of research and the state of practice
  9. Future directions? 10 • New developments are only useful if

    they can be accessed by researchers • Make software for estimating new methods available • For application focused papers: • Include descriptions of software that was used • Make code available so that other researchers can see how to apply the methods
  10. 11

  11. Theme 3: Incorporating new data sources 12 • Technology has

    opened up new data collection opportunities • Response times (Huang et al.; Lee & Gu; Mireles & Bao) • Eye tracking (Lee & Gu) • Clickstreams (Zhang) • We assume that these new indicators are indicative of the construct • Unintended consequences
  12. Future directions? 13 • Additional research needed to support the

    alignment of new data sources to the intended construct • Rich literature for measuring the alignment of traditional items • Methods for evaluating invariance across different groups • Response time may be systematically different for students with extensive support needs, or students in rural areas with limited internet bandwidth
  13. Summary 14 • Ultimately our goal is to improve learning

    or practice through diagnostic models • Need scaffolding to support applications • Theory tying data to the intended cognition • Accessible software • Case studies to illustrate usage • Discussions of appropriate and inappropriate uses
  14. Get in touch! 15 Thank you! • Reception: Today! •

    3:00–4:00pm • Convention Center 126A • SIGIMIE Website: https://www.ncme.org/community/ncme- sigimie/diagnostic-measurement