Multitrait Multimethod and Latent State-Trait Analysis

These are the slides for the seminar "New Developments in Testtheory and Test Construction" (University of Mannheim, Master Psychology). Therein, newer models within the framework of classical test theory are presented, discussed, and illustrated with examples. More specifically, the first part deals with multitrait multimethod analysis, and the second part deals with models developed within the framework of latent state-trait theory (which is an extension of classical test theory).

Transcript

1. MTMM & LST H. Plieninger MTMM LST Appendix References Glossary Multitrait Multimethod and Latent State–Trait Analysis Hansjörg Plieninger University of Mannheim 1 / 92
2. MTMM & LST H. Plieninger MTMM LST Appendix References Glossary License • This is version 1.0.0 of this slidedeck. • Please report any errata. mail@hansjoerg.me hansjoerg_me https://www.hansjoerg.me • This work is licensed under a Creative Commons Attribution 4.0 International License. 2 / 92
3. MTMM & LST H. Plieninger MTMM LST Appendix References Glossary Table of Contents 1 Multitrait-Multimethod Analysis 2 Latent State–Trait Theory 3 Appendix 3 / 92
4. MTMM & LST H. Plieninger MTMM Validity MTMM LST Appendix References Glossary TOC: MTMM 1 Multitrait-Multimethod Analysis Validity MTMM 4 / 92
5. MTMM & LST H. Plieninger MTMM Validity MTMM LST Appendix References Glossary Validity Validity (Messick, 1995) “Validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions on the basis of test scores or other modes of assessment.” • Different aspects of validity; organized in many different ways • Here: Internal construct validity and external construct validity (Grimm & Widaman, 2012) 5 / 92
6. MTMM & LST H. Plieninger MTMM Validity MTMM LST Appendix References Glossary Internal Validity (Grimm & Widaman) Do the items measure the intended construct? • Content validity • Construct definition, objectives of the test, . . . • Dimensionality • Item difficulty, discrimination • Reliability • Measurement invariance, DIF 6 / 92
7. MTMM & LST H. Plieninger MTMM Validity MTMM LST Appendix References Glossary External Validity (Grimm & Widaman) Does the test measure the intended construct? • Criterion-related validity • Concurrent, predictive, postdictive • Convergent and discriminant validity • Change validity • Score interpretation 7 / 92
8. MTMM & LST H. Plieninger MTMM Validity MTMM LST Appendix References Glossary Test Theories and Validity • Both IRT and CTT can be used to assess validity. • IRT methods are especially well suited for aspects of internal validity: • Reliability, homogeneity, dimensionality, construct map, DIF, item fit, . . . • CTT methods are especially well suited for aspects of external validity: • MTMM: Convergent and discriminant validity • LST and growth modeling: Change validity → IRT and CTT methods should both be in our toolbox, for example: • First, IRT analyses on the item-/threshold-level • Second, CTT analyses on the (sub-) test-level 8 / 92
9. MTMM & LST H. Plieninger MTMM Validity MTMM LST Appendix References Glossary Convergent and Discriminant Validity • Convergent validity of a test/measure is high, if correlations with other tests measuring the same construct are high. • Discriminant validity of a test/measure is high, if correlations with other tests measuring different constructs are small. • What “small” means depends on theory about the constructs. 9 / 92
10. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Multitrait-Multimethod Analysis • Initially developed by Campbell and Fiske (1959). • Methods to assess the external validity of a test • Data for at least 2 (better 3) different traits are collected using at least 2 (better 3) different methods. • Each method is used for each trait. 10 / 92
11. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Method Effects I • We do not measure a trait but a systematic trait–method unit plus an unsystematic error component. • The observed data are (usually) interpreted as indicative of the trait, but they are also indicative of the method used. • Empirical results are confounded by method effects. • For example, factor correlations of a Big Five test indicate relationships among the traits, but they are also due to the fact that a common method (i.e., self-report) elicits shared variance. 11 / 92
12. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Method Effects II • Method effects are sources of systematic variance, e.g., due to: • Method, instrument, measurement device • Informant, rater, source • Occassion, context • To control for method effects, more than one method has to be used. • Goal of MTMM Analyses: • Assess convergent and discriminant validity (controlled for method effects) • Investigate influence of method effects and possibly explain them 12 / 92
13. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Example MTMM Matrix for methods A, B, C and traits 1, 2, 3: Method A Method B Method C Meth. Trait 1 2 3 1 2 3 1 2 3 A 1 Rel(A1) 2 rA2A1 Rel(A2) 3 rA3A1 rA3A2 Rel(A3) B 1 rB1A1 rB1A2 rB1A3 Rel(B1) 2 rB2A1 rB2A2 rB2A3 rB2B1 Rel(B2) 3 rB3A1 rB3A2 rB3A3 rB3B1 rB3B2 Rel(B3) C 1 rC1A1 rC1A2 rC1A3 rC1B1 rC1B2 rC1B3 Rel(C1) 2 rC2A1 rC2A2 rC2A3 rC2B1 rC2B2 rC2B3 rC2C1 Rel(C2) 3 rC3A1 rC3A2 rC3A3 rC3B1 rC3B2 rC3B3 rC3C1 rC3C2 Rel(C3) 13 / 92
14. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Reliabilities The reliabilities are on the main diagonal (monotrait-monomethod coefficients). Method A Method B Method C Meth. Trait 1 2 3 1 2 3 1 2 3 A 1 Rel(A1) 2 rA2A1 Rel(A2) 3 rA3A1 rA3A2 Rel(A3) B 1 rB1A1 rB1A2 rB1A3 Rel(B1) 2 rB2A1 rB2A2 rB2A3 rB2B1 Rel(B2) 3 rB3A1 rB3A2 rB3A3 rB3B1 rB3B2 Rel(B3) C 1 rC1A1 rC1A2 rC1A3 rC1B1 rC1B2 rC1B3 Rel(C1) 2 rC2A1 rC2A2 rC2A3 rC2B1 rC2B2 rC2B3 rC2C1 Rel(C2) 3 rC3A1 rC3A2 rC3A3 rC3B1 rC3B2 rC3B3 rC3C1 rC3C2 Rel(C3) 14 / 92
15. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Convergent Validity Correlations of variables measuring the same trait with different methods (monotrait-heteromethod correlations or validity diagonal) Method A Method B Method C Meth. Trait 1 2 3 1 2 3 1 2 3 A 1 Rel(A1) 2 rA2A1 Rel(A2) 3 rA3A1 rA3A2 Rel(A3) B 1 rB1A1 rB1A2 rB1A3 Rel(B1) 2 rB2A1 rB2A2 rB2A3 rB2B1 Rel(B2) 3 rB3A1 rB3A2 rB3A3 rB3B1 rB3B2 Rel(B3) C 1 rC1A1 rC1A2 rC1A3 rC1B1 rC1B2 rC1B3 Rel(C1) 2 rC2A1 rC2A2 rC2A3 rC2B1 rC2B2 rC2B3 rC2C1 Rel(C2) 3 rC3A1 rC3A2 rC3A3 rC3B1 rC3B2 rC3B3 rC3C1 rC3C2 Rel(C3) • Coefficients should be high (and significant). 15 / 92
16. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Discriminant Validity I Correlations of variables measuring different trait with the same method (heterotrait-monomethod correlations) Method A Method B Method C Meth. Trait 1 2 3 1 2 3 1 2 3 A 1 Rel(A1) 2 rA2A1 Rel(A2) 3 rA3A1 rA3A2 Rel(A3) B 1 rB1A1 rB1A2 rB1A3 Rel(B1) 2 rB2A1 rB2A2 rB2A3 rB2B1 Rel(B2) 3 rB3A1 rB3A2 rB3A3 rB3B1 rB3B2 Rel(B3) C 1 rC1A1 rC1A2 rC1A3 rC1B1 rC1B2 rC1B3 Rel(C1) 2 rC2A1 rC2A2 rC2A3 rC2B1 rC2B2 rC2B3 rC2C1 Rel(C2) 3 rC3A1 rC3A2 rC3A3 rC3B1 rC3B2 rC3B3 rC3C1 rC3C2 Rel(C3) • Coefficients should be “small” and smaller than the corresponding monotrait-heteromethod coefficients. • Indicative of method variance. • Using the same method for two different traits should yield lower correlations than using two different methods for the same trait. • Example for A1: (rB1A1 , rC1A1 ) > (rA2A1 , rA3A1 ) 16 / 92
17. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Discriminant Validity II Correlations of variables measuring different trait with different methods (heterotrait-heteromethod correlations) Method A Method B Method C Meth. Trait 1 2 3 1 2 3 1 2 3 A 1 Rel(A1) 2 rA2A1 Rel(A2) 3 rA3A1 rA3A2 Rel(A3) B 1 rB1A1 rB1A2 rB1A3 Rel(B1) 2 rB2A1 rB2A2 rB2A3 rB2B1 Rel(B2) 3 rB3A1 rB3A2 rB3A3 rB3B1 rB3B2 Rel(B3) C 1 rC1A1 rC1A2 rC1A3 rC1B1 rC1B2 rC1B3 Rel(C1) 2 rC2A1 rC2A2 rC2A3 rC2B1 rC2B2 rC2B3 rC2C1 Rel(C2) 3 rC3A1 rC3A2 rC3A3 rC3B1 rC3B2 rC3B3 rC3C1 rC3C2 Rel(C3) • Coefficients should be “small” and smaller than the corresponding monotrait-heteromethod coefficients. • Using two different methods for two different traits should yield lower correlations than using two different methods for the same trait. • Example for A1: rC1A1 > (rC2A1 , rC3A1 , rC1A2 , rC1A3 ) 17 / 92
18. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Example (Eid et al., 2006) ". . . data from an MTMM study exploring the relati- ons between self- and peer-rated frequency of nega- tive emotions. The traits were fear, anger, and sad- ness. The three methods were self-ratings, ratings by a good friend, and ratings by an acquaintance. The sample consisted of 172 triples of self- and peer raters. . . . While seated separately, all participants rated the frequency with which the target individual usually experienced different negative emotions using a four-category scale (from not at all to very often). Three scales, consisting of four emotion terms each, assessed fear, anger, and sadness" (Eid et al., 2006, pp. 288–289). Here, the variance–covariance matrix of nine sum scores (3 traits × 3 methods) is used. 18 / 92
19. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Example (Eid et al., 2006, p. 289) 19 / 92
20. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Interpretation • Campbell and Fiske (1959) based the interpretations directly on the coefficients in the MTMM matrix. • Problems: • Difficult, subjective, clear-cut rules missing • Based on manifest correlations; desired interpretation: latent • Descriptive (rather than inferential) comparisons 20 / 92
21. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Factor Models for MTMM Data • Confirmatory factor analysis (CFA) and structural equation models (SEM) for MTMM data overcome the problems with MTMM matrices. + Model-based; possible to test assumptions + Allows to differentiate systematic from unsystematic/error components + Latent variables, which may be related to each other and to external covariates • Many different models have been developed in recent years. • Here: Cursory, brief overview of different models without going into the technical details 21 / 92
22. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 22 / 92
23. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 • Discriminant validity: Factor correlations • Convergent validity: Factor loadings • Error variance: Measurement error + method variance (confounded) • No method effects estimated 23 / 92
24. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait Model: Example 0.4 0.6 0.5 0.4 0.6 0.4 0.3 0.6 0.5 0.6 1.0 0.6 SR.fe FR.fe AR.fe SR.an FR.an AR.an SR.sa FR.sa AR.sa Fear Anger Sadns • Bad fit (χ2 = 94.56, df = 24, p < .01, CFI = .67, RMSEA = .13) • Rather low factor loadings → mediocre convergent validity • Factor intercorrelations (spuriously!?!) high → low discriminant validity 24 / 92
25. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait/Correlated Uniqueness (CTCU) Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 25 / 92
26. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTCU Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 • Same-method residuals are allowed to correlate • Discriminant validity: Factor correlations • Convergent validity: Factor loadings • Error variance: Measurement error + method variance (confounded) 26 / 92
27. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTCU Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 − Unsystematic and systematic (method) variance confounded − Indicator reliability underestimated − No external covariates for method factors − No intercorrelations of method factors − Theoretically unattractive + No need to assume unidimensional method factor 27 / 92
28. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTCU Model: Example 0.4 0.6 0.5 0.5 0.4 0.4 0.3 0.4 0.6 0.3 0.3 0.1 0.2 0.4 0.4 0.3 0.3 0.3 0.4 0.7 0.1 SR.fe FR.fe AR.fe SR.an FR.an AR.an SR.sa FR.sa AR.sa Fear Anger Sadns • Good fit (χ2 = 19.94, df = 15, p = .17, CFI = .98, RMSEA = .04) • Low, heterogeneous factor loadings • Factor intercorrelations seem trustworthy 28 / 92
29. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait/Uncorrelated Method (CTUM) Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 M3 29 / 92
30. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTUM Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 M3 • Same-method indicators are modeled with a method factor • Discriminant validity: Factor correlations • Convergent validity: Trait factor loadings • Impact of methods: Method factor loadings • Restricted version of CTCU, because assumption of homogeneous method factors 30 / 92
31. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTUM Model: Example 0.4 0.6 0.5 0.5 0.4 0.4 0.3 0.4 0.6 1.1 0.2 0.2 0.3 0.4 0.8 0.4 0.5 0.4 0.4 0.7 0.1 SR.fe FR.fe AR.fe SR.an FR.an AR.an SR.sa FR.sa AR.sa Fear Anger Sadns SR FR AR • Good fit (χ2 = 19.94, df = 15, p = .17, CFI = .98, RMSEA = .04) • Low trait factor loadings • Relatively high method factor loadings • Loadings heterogeneous (→ makes interpretation difficult) • Factor intercorrelations seem trustworthy 31 / 92
32. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Reliability • Observed variance Var(Y ) is decomposed into trait variance Var(T), method variance Var(M), and error variance Var(E). • Reliability is the portion of systematic variance. Ypq = λTpqTp + λMpqMq + Epq Var(Y ) = λ2 Var(T) + λ2 Var(M) + Var(E) Rel(Ypq) = λ2 Tpq Var(Tp) + λ2 Mpq Var(Mq) Var(Ypq) = 1 − Var(Epq) Var(Ypq) 32 / 92
33. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Consistency and Specificity • Two sources of systematic variance → two “reliabilites”: • Consistency Con (“trait reliability”) • Method specificity MSpe (“method reliability”) Consistency and (Method-) Specificity Con(Ypq) = λ2 Tpq Var(Tp) Var(Ypq) MSpe(Ypq) = λ2 Mpq Var(Mq) Var(Ypq) • Con + MSpe = Rel • Note: Con and MSpe are sometimes defined relative to true score variance (i.e., Var(Y ) − Var(E)) rather than observed variance (i.e., Var(Y )), such that Con + MSpe = 1. 33 / 92
34. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTUM Model: Example 0.4 0.6 0.5 0.5 0.4 0.4 0.3 0.4 0.6 1.1 0.2 0.2 0.3 0.4 0.8 0.4 0.5 0.4 -0.3 0.5 0.6 0.7 0.7 0.6 0.9 0.1 0.4 1.0 1.0 1.0 1.0 1.0 1.0 0.4 0.7 0.1 SR.fe FR.fe AR.fe SR.an FR.an AR.an SR.sa FR.sa AR.sa Fear Anger Sadns SR FR AR • Consistency is indicative of convergent validity • For, e.g., variable AR.sa: • Con = .632∗1 1 = .40 • MSpe = .442∗1 1 = .19 • Rel = .40 + .19 = .59 34 / 92
35. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTUM Model: Summary Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 M3 + Method effects are explicitly modeled + No confounding of error and method variance + Method factors may be related to each other (CTCM) and to external covariates + Reasonable estimate of Rel(Ypq) + Interpretation of Con and MSpe − Empirical problems of non-convergence and Heywood cases (e.g., negative variance) − Theoretical difficulty of method-free traits − Method effects might be trait-specific 35 / 92
36. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTUM Model: Example Revisited 0.4 0.6 0.5 0.5 0.4 0.4 0.3 0.4 0.6 1.1 0.2 0.2 0.3 0.4 0.8 0.4 0.5 0.4 -0.3 0.5 0.6 0.7 0.7 0.6 0.9 0.1 0.4 1.0 1.0 1.0 1.0 1.0 1.0 0.4 0.7 0.1 SR.fe FR.fe AR.fe SR.an FR.an AR.an SR.sa FR.sa AR.sa Fear Anger Sadns SR FR AR • Negative error variance 36 / 92
37. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait/Correlated Method (CTCM) Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 M3 37 / 92
38. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTCM Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 M3 • Very similar to CTUM model • Method factors may now be correlated (e.g., mother and father rating may be positively correlated) − Is not globally identified, estimation problems − Indicators are now related via correlated traits and via correlated methods: Trait correlations may no longer be indicative of discriminant validity 38 / 92
39. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTCM Model: Example Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 M3 • Non-convergence 39 / 92
40. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Correlated Trait/Correlated Method Minus One Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 40 / 92
41. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTC(M−1) Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 • Eid (2000) • One method factor less than methods employed (→ M−1) • One method is chosen as the comparison standard • Trait factors now indicative of a trait-measured-by-the-comparison- standard • Method factors represent deviation from standard method 41 / 92
42. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTC(M−1) Model Y11 Y12 Y13 Y21 Y22 Y23 Y31 Y32 Y33 T1 T2 T3 M1 M2 + Aim is to overcome technical and theoretical limitations of previous approaches − A comparison standard has to be chosen (arbitrarily!?) − Model is not symmetric, fit changes when standard method is changed 42 / 92
43. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary CTC(M−1) Model: Example 0.9 0.3 0.2 0.8 0.3 0.3 0.6 0.3 0.4 0.6 0.5 0.7 0.6 0.4 0.6 0.4 0.7 0.1 0.4 SR.fe FR.fe AR.fe SR.an FR.an AR.an SR.sa FR.sa AR.sa Fear Anger Sadns FR AR • Good fit (χ2 = 24.57, df = 17, p = .10, CFI = .96, RMSEA = .05) • High trait factor loadings for SR • FR and AR exhibit high method specificity • Factor intercorrelations seem trustworthy • FR and AR substantially correlated 43 / 92
44. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Outlook • (Non-) interchangeable methods • Randomly selected raters are interchangeable methods. • Self, peer, and acquaintance are structurally different methods. • Interchangeable methods may/should be uncorrelated (→ CTCU, CTUM). • Multiple-indicator models • Method effects may not generalize across traits (i.e., trait-specific). • For example, peers may react differentially to socially desirable and neutral items. → Cannot be differentiated from error variance • Possible solution: Models that include multiple indicators for every trait–method unit 44 / 92
45. MTMM & LST H. Plieninger MTMM Validity MTMM MTMM Matrix MTMM CFA LST Appendix References Glossary Further Reading • Eid et al. (2006) • Moosbrugger and Kelava (2012). Chap. 14. • Empirical example: Biesanz and West (2004) 45 / 92
46. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary TOC: LST 2 Latent State–Trait Theory Theory Models Consistency and Specificity Example 46 / 92
47. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Latent State–Trait Theory • Measurement does not take place in a situational vacuum. • Measurement is not indicative of a person, but of a person-in-a-situation. • It is assumed that every measurement is influenced by • the person, • the situation, and • the person–situation interaction. • Trait: Stable over time, independent of situation (e.g., intelligence) • State: Variable/flexible, dependent on situation (e.g., mood) • Stability over time and relative influence of trait and state are matters concerning external validity. 47 / 92
48. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Classical Test Theory Extended • CTT assumes: yjp = τjp + jp • Reliability in CTT is the ratio of true-score and observed variance. • LST reconceptualizes CTT’s true score as a true score of person j on variable p at time t: τjpt . • This true score is called latent-state variable. • This state variable is comprised of trait- and state-parts (confounded). • Goal of LST: Decompose this latent-state variable 48 / 92
49. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Latent-State Variable • Basis of LST is the decomposition of the latent-state variable τ into a latent-trait variable ξ (xi) and a latent-state residual ζ (zeta). τpt ξpt ζpt Ypt pt yjpt = τjpt + jpt = ξjpt + ζjpt + jpt 49 / 92
50. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Latent-State Variable τpt ξpt ζpt Ypt pt • The latent-trait variable ξ: • Characteristic of the person; stable; independent of situation • Has an index t, because even traits may change over time. • The latent-state residual ζ: • Represents situational influences as well as person–situation interactions • Defined as the difference: τpt − ξpt 50 / 92
51. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary LST Models • Models for different purposes/research questions and of different complexity have been developed within LST. • In order to distinguish between state- and trait-effects • at least two variables (e.g., test halves) have to be measured at • at least two measurement occasions. • Important: The specific situations need not be known (contrarily to, e.g., experimental settings). • Certain models (i.e., τ-congeneric) are not identified with only 2 × 2 variables and therefore need further constraints (e.g., τ-equivalence). 51 / 92
52. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait Model 1 l21 l12 l22 Y11 Y21 Y12 Y22 t Time Point 1 Time Point 2 52 / 92
53. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait Model 1 l21 l12 l22 Y11 Y21 Y12 Y22 t Time Point 1 Time Point 2 • No situation or interaction effects modeled. • Good fit would indicate extreme stability. • Crude, strict model; not very plausible • Loadings may be free (i.e., τ-congeneric). 53 / 92
54. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Multistate Model 1 l21 1 l22 Y11 Y21 Y12 Y22 t 1 t 2 Time Point 1 Time Point 2 54 / 92
55. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Multistate Model 1 l21 1 l22 Y11 Y21 Y12 Y22 t 1 t 2 Time Point 1 Time Point 2 • A latent-state variable τ for every occasion • Model allows for trait as well as for state influences. • However, trait and state effects are confounded. • The correlation is indicative of the stability of the latent states. • Loadings may be free (i.e., τ-congeneric). 55 / 92
56. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Model 1 1 1 1 1 1 z1 z2 Y11 Y21 Y12 Y22 t 1 t 2 x Time Point 1 Time Point 2 56 / 92
57. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Model 1 1 1 1 1 1 z1 z2 Y11 Y21 Y12 Y22 t 1 t 2 x Time Point 1 Time Point 2 • A latent-state variable τ for every occasion • Observed variables are decomposed into error and τ. • τ is decomposed into the trait component ξ and the state-residual component ζ. • Relative size of variances is indicative of the stability/variability of the attribute. • τ-congeneric) model is just identified for two time points, loadings should be constrained • “real”, typical LST model 57 / 92
58. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Correlated Uniqueness Model 1 1 1 1 1 1 z1 z2 Y11 Y21 Y12 Y22 t 1 t 2 x Time Point 1 Time Point 2 58 / 92
59. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Correlated Uniqueness Model 1 1 1 1 1 1 z1 z2 Y11 Y21 Y12 Y22 t 1 t 2 x Time Point 1 Time Point 2 • The same instrument is administered at every time point. • Possible/plausible that this leads to shared variance, i.e., method effects. • Possible remedy: Correlated errors (or method factors) • Again, loadings have to be constrained. 59 / 92
60. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Consistency and Specificity τ-Equivalent Model • Because of the assumption Cov(ξpt, ζpt) = 0, the variance can be decomposed additively: Var(Ypt) = Var(τpt) + Var( pt) = Var(ξpt) + Var(ζpt) + Var( pt) Consistency and (Occasion-) Specificity Con(Ypt) = Var(ξpt) Var(Ypt) OSpe(Ypt) = Var(ζpt) Var(Ypt) Rel(Ypt) = Con(Ypt) + OSpe(Ypt) 60 / 92
61. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Consistency and Specificity τ-Congeneric Model • If the loadings are not equal to 1 (e.g., τ-congeneric) model), they have to be taken into account. • The respective equations (with reduced notation for convenience) are: Consistency and (Occasion-) Specificity Con(Y ) = λ2 τ λ2 ξ Var(ξ) Var(Y ) OSpe(Y ) = λ2 τ Var(ζ) Var(Y ) Rel(Y ) = Con(Y ) + OSpe(Y ) Var(Y ) = Var( ) + λ2 τ Var(ζ) + λ2 ξ Var(ξ) 61 / 92
62. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Consistency and Specificity • Consistency Con(Ypt) is that part of variance of variable p at time t that is due to the stable trait. • Specificity OSpe(Ypt) is that part of variance of variable p at time t that is due to the situation and the person–situation interaction. • Reliability Rel(Ypt) is that part of variance of variable p at time t that is due to the latent-state variable (i.e., systematic), not to error variance. 62 / 92
63. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Empirical Example • Steyer et al. (1989) administered the State-Trait Anxiety Inventory (STAI): two time points, two months apart, 64 students. • Both the state and the trait scale were comprised of 20 items and both were split in order to have two variables/halves. • Herein, results for the trait scale (Variables TApt ) will be presented. • The authors specified the very strict τ-parallel model with equal loadings and equal residual variances (which is possible but not necessary). 63 / 92
64. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait Model 1.0 1.0 1.0 1.0 3.8 3.8 3.8 3.8 16.6 TA11 TA21 TA12 TA22 t 64 / 92
65. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait Model 1.0 1.0 1.0 1.0 3.8 3.8 3.8 3.8 16.6 TA11 TA21 TA12 TA22 t • Rejected (χ2 = 23.41, df = 8, p = .00, CFI = .94, RMSEA = .18) 65 / 92
66. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Multistate Model 1.0 1.0 1.0 1.0 2.8 2.8 2.8 2.8 18.2 16.8 16.1 TA11 TA21 TA12 TA22 t 1 t 2 66 / 92
67. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Multistate Model 1.0 1.0 1.0 1.0 2.8 2.8 2.8 2.8 18.2 16.8 16.1 TA11 TA21 TA12 TA22 t 1 t 2 • Improved fit (χ2 = 12.38, df = 6, p = .05, CFI = .98, RMSEA = .13) • Latent correlation quite high (r = 16.1 √ 18.2∗ √ 16.8 = .92) 67 / 92
68. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Model 1.0 1.0 1.0 1.0 2.8 2.8 2.8 2.8 1.0 1.0 2.1 0.7 16.1 TA11 TA21 TA12 TA22 t 1 t 2 x 68 / 92
69. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Model 1.0 1.0 1.0 1.0 2.8 2.8 2.8 2.8 1.0 1.0 2.1 0.7 16.1 TA11 TA21 TA12 TA22 t 1 t 2 x • Same fit (χ2 = 12.38, df = 6, p = .05, CFI = .98, RMSEA = .13) • As expected, large trait variance compared to state-residual variance • Con(TAp1) = 16.1 16.1+2.1+2.8 = .76 • OSpe(TAp1) = 2.1 16.1+2.1+2.8 = .10 • Rel(TAp1) = .76 + .10 = .86 69 / 92
70. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Model Standardized Solution 0.93 0.93 0.92 0.92 0.13 0.13 0.14 0.14 0.94 0.98 0.12 0.04 1.00 TA11 TA21 TA12 TA22 t 1 t 2 x • Standardized solution, i.e., Var(Y ) = Var(τ) = Var(ξ) = 1 • Var(TA11) = 0.13+0.932∗ (0.12 + 0.942 ∗ 1) = 1 • Con(TAp1) = 0.932∗0.942∗1 1 = .76 • OSpe(TAp1) = 0.932∗0.12 1 = .10 70 / 92
71. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Correlated Uniqueness Model 1.0 1.0 1.0 1.0 2.8 2.8 2.8 2.8 1.0 1.0 -0.3 1.7 2.9 0.8 15.5 TA11 TA21 TA12 TA22 t 1 t 2 x 71 / 92
72. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Single-Trait/Multistate Correlated Uniqueness Model 1.0 1.0 1.0 1.0 2.8 2.8 2.8 2.8 1.0 1.0 -0.3 1.7 2.9 0.8 15.5 TA11 TA21 TA12 TA22 t 1 t 2 x • Good fit (χ2 = 2.10, df = 4, p = .72, CFI = 1.00, RMSEA = .00) • Con(TAp1) = 15.5 15.5+2.9+2.8 = .73 • OSpe(TAp1) = 2.9 15.5+2.9+2.8 = .14 • Rel(TAp1) = .73 + .14 = .87 72 / 92
73. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Exercise Now that you’ve seen the results for the trait variables, let’s look at the state variables. • What would you expect for the correlation in the multistate model? • What would you expect for the consistency and specificity in the single-trait/multistate model? • Develop a combined model for both the state and the trait variables. 73 / 92
74. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Results for State and Trait Scale State Scale Trait Scale (Steyer et al., 1989, p. 291) 74 / 92
75. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Results for State and Trait Scale (Steyer et al., 1989, p. 292) 75 / 92
76. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Multitrait/Multistate Model e1 e1 e1 e1 e2 e2 e2 e2 SA11 SA21 SA12 SA22 TA11 TA21 TA12 TA22 t 1 t 2 t 3 t 4 x1 x2 Time 1 Time 2 Time 1 Time 2 State Questionnaire Trait Questionnaire 76 / 92
77. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Multitrait/Multistate Model • What do you expect regarding the correlation between ξ1 and ξ2 ? 77 / 92
78. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Modified “Multitrait”/Multistate Model 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 3.2 3.2 3.2 3.2 2.9 2.9 2.9 2.9 1.0 1.0 1.0 1.0 0.8 0.7 0.1 1.6 27.2 9.9 2.4 1.2 15.1 SA11 SA21 SA12 SA22 TA11 TA21 TA12 TA22 t 1 t 2 t 3 t 4 x1 State Questionnaire Trait Questionnaire 78 / 92
79. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Modified “Multitrait”/Multistate Model • The correlation between ξ1 and ξ2 is so high (≈ 1) that the multitrait model leads to estimation problems. Modifying the model by using only one variable ξ resolves this. 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 3.2 3.2 3.2 3.2 2.9 2.9 2.9 2.9 1.0 1.0 1.0 1.0 0.8 0.7 0.1 1.6 27.2 9.9 2.4 1.2 15.1 SA11 SA21 SA12 SA22 TA11 TA21 TA12 TA22 t 1 t 2 t 3 t 4 x1 State Questionnaire Trait Questionnaire • Acceptable fit (χ2 = 34.42, df = 25, p = .10, CFI = .98, RMSEA = .08) • Con(SAp1) = 15.1 15.1+27.2+3.2 = .33 • Con(TAp1) = 15.1 15.1+2.4+2.9 = .74 • OSpe(SAp1) = 27.2 15.1+27.2+3.2 = .60 • OSpe(TAp1) = 2.4 15.1+2.4+2.9 = .12 79 / 92
80. MTMM & LST H. Plieninger MTMM LST Theory Models Consistency and Specificity Example Appendix References Glossary Further Reading • Steyer et al. (1999) • Steyer et al. (2012) • Moosbrugger and Kelava (2012): Chap. 15. 80 / 92
81. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA References Glossary TOC: Appendix 3 Appendix CTT CFA 81 / 92
82. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA References Glossary Classical Test Theory Axioms of Classical Test Theory τp = E(Yp) Yp = τp + p, with E( p) = 0 Cor(τp, p) = 0 Yp Random variable for test p, and yjp is a respective observation for person j on test p τ True score Error score 82 / 92
83. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA References Glossary Reliability Var(Yp) = Var(τp) + Var( p) Rel(Yp) = Var(τp) Var(Yp) = Var(τp) Var(τp) + Var( p) In case of weighted true score (e.g., in CFA): Y1 = λ1τ + 1 Y2 = λ2τ + 2 Rel(Yi ) = λ2 i Var(τ) Var(Yi ) (communality) 83 / 92
84. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA Measurement Models References Glossary Confirmatory Factor Analysis (CFA) • Based on the variance-covariance matrix Σ • Mean structure sometimes/often ignored, focus on centered variables • Goal: Explain manifest correlations in terms of latent variables • Means: Re-parametrization of Σ using fewer parameters • Linear model, no link function 84 / 92
85. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA Measurement Models References Glossary CFA Example l11 l21 l32 l42 Cov12 e 1 e 2 e 3 e 4 Var1 Var2 Y1 Y2 Y3 Y4 T1 T2 85 / 92
86. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA Measurement Models References Glossary CFA Example l11 l21 l32 l42 Cov12 e 1 e 2 e 3 e 4 Var1 Var2 Y1 Y2 Y3 Y4 T1 T2 Y1 = λ11T1 + 1 Y2 = λ21T1 + 2 Y3 = λ32T2 + 3 Y4 = λ42T2 + 4 86 / 92
87. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA Measurement Models References Glossary Measurement Models τ-Parallel Variables l1 l1 l1 Var1 Y1 Y2 Y3 T1 • Strictest model, least flexible, most df • Assumption of equal loadings and equal error variances: Y1 = λ1τ + 1 Y2 = λ2τ + 2 Y3 = λ3τ + 3 Var( 1 ) = Var( 2 ) = Var( 3 ) λ1 = λ2 = λ3 • Essential τ-parallelism: Intercepts may vary (not considered herein) 87 / 92
88. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA Measurement Models References Glossary Measurement Models τ-Equivalent Variables l1 l2 l3 Var1 Y1 Y2 Y3 T1 • More flexible than τ-parallel, more parameters, less df • Assumption of equal loadings: Y1 = λ1τ + 1 Y2 = λ2τ + 2 Y3 = λ3τ + 3 Var( 1 ) = Var( 2 ) = Var( 3 ) λ1 = λ2 = λ3 • Essential τ-equivalence: Intercepts may vary (not considered herein) 88 / 92
89. MTMM & LST H. Plieninger MTMM LST Appendix CTT CFA Measurement Models References Glossary Measurement Models τ-Congeneric Variables l1 l2 l3 Var1 Y1 Y2 Y3 T1 • Least restrictive model, most flexible, least df • Assumption: Y1 = λ1τ + 1 Y2 = λ2τ + 2 Y3 = λ3τ + 3 Var( 1 ) = Var( 2 ) = Var( 3 ) λ1 = λ2 = λ3 • The “typical” CFA model 89 / 92
90. MTMM & LST H. Plieninger MTMM LST Appendix References Glossary References I Biesanz, J. C., & West, S. G. (2004). Towards understanding assessments of the Big Five: Multitrait-multimethod analyses of convergent and discriminant validity across measurement occasion and type of observer. Journal of Personality, 72(4), 845–876. https://doi.org/10.1111/j.0022-3506.2004.00282.x Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105. https://doi.org/10.1037/h0046016 Eid, M. (2000). A multitrait-multimethod model with minimal assumptions. Psychometrika, 65(2), 241–261. https://doi.org/10.1007/BF02294377 Eid, M., Lischetzke, T., & Nussbeck, F. W. (2006). Structural equation models for multitrait-multimethod data. In M. Eid & E. Diener (Eds.), Handbook of multimethod measurement in psychology (pp. 283–299). APA. https://doi.org/10.1037/11383-020 Grimm, K. J., & Widaman, K. F. (2012). Construct validity. In H. Cooper (Ed.), APA Handbook of Research Methods in Psychology: Vol 1. Foundations, planning, measures, and psychometrics. (pp. 621–642). https://doi.org/10.1037/13619-033 90 / 92
91. MTMM & LST H. Plieninger MTMM LST Appendix References Glossary References II Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741 Moosbrugger, H., & Kelava, A. (Eds.). (2012). Testtheorie und Fragebogenkonstruktion (2nd ed.). https://doi.org/10.1007/978-3-642-20072-4 Steyer, R., Geiser, C., & Fiege, C. (2012). Latent state-trait models. In H. Cooper (Ed.), APA Handbook of Research Methods in Psychology: Vol. 3. Data analysis and research publication (pp. 291–308). APA. https://doi.org/10.1037/13621-014 Steyer, R., Majcen, A.-M., Schwenkmezger, P., & Buchner, A. (1989). A latent state-trait anxiety model and its application to determine consistency and specificity coefficients. Anxiety Research, 1(4), 281–299. https://doi.org/10.1080/08917778908248726 Steyer, R., Schmitt, M., & Eid, M. (1999). Latent state–trait theory and research in personality and individual differences. European Journal of Personality, 13(5), 389–408. https://doi.org/10.1002/(SICI)1099-0984(199909/10)13:5<389:: AID-PER361>3.0.CO;2-A 91 / 92
92. MTMM & LST H. Plieninger MTMM LST Appendix References Glossary Glossary CFI Comparative fit index; values ≥ 0.97 indicate good fit of a CFA model 24, 28, 31, 43, 65, 67, 69, 72, 79 CTT Classical test theory 8 DIF Differential item functioning (DIF): ICCs differ between groups 6, 8 LST Latent state–trait (theory) 8 RMSEA Root mean square error of approximation; values < 0.05 indicate good fit of a CFA model 24, 28, 31, 43, 65, 67, 69, 72, 79 92 / 92