Will Lowe
April 20, 2022
8

# Data Science and Decisions 2022: Week 6

April 20, 2022

## Transcript

1. ### DATA SCIENCE AND DECISION MAKING Decision making in and by

groups William Lowe Hertie School Data Science Lab 2022-04-20
2. ### PLAN 1 Why combine judgments? e view from data science

Bias, variance, and error Groupthink and other bad outcomes Multiverse models in science Motivating multiverses with Bayes e limits of model averaging Deliberation and social choice
3. ### WHY COMBINE? 2 Assume M models (or any decision makers)

each of whom predicts a target Y as ˆ Ym = Y + єm and compare this to a committee prediction ˆ Y Com = M M m ˆ Ym
4. ### WHY COMBINE? 2 Assume M models (or any decision makers)

each of whom predicts a target Y as ˆ Ym = Y + єm and compare this to a committee prediction ˆ Y Com = M M m ˆ Ym S Model m has mean squared error (MSE) Em = E[( ˆ Ym − Y) ] = E[єm] e average MSE error of M such models is E Av = M M m Em = M M m E[єm]
5. ### WHY COMBINE? 2 Assume M models (or any decision makers)

each of whom predicts a target Y as ˆ Ym = Y + єm and compare this to a committee prediction ˆ Y Com = M M m ˆ Ym S Model m has mean squared error (MSE) Em = E[( ˆ Ym − Y) ] = E[єm] e average MSE error of M such models is E Av = M M m Em = M M m E[єm] C E Com = E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ⎛ ⎝ M M m ˆ Ym ˆ YCom −Y ⎞ ⎠ ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ = E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ M M m єm ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ If model errors are E[єm] = (well-speci ed) E[єmєm′ ] = m ≠ m′ (uncorrelated) then E Com = M M m E[є ] = M EAv
6. ### WHY COMBINE? 2 Assume M models (or any decision makers)

each of whom predicts a target Y as ˆ Ym = Y + єm and compare this to a committee prediction ˆ Y Com = M M m ˆ Ym S Model m has mean squared error (MSE) Em = E[( ˆ Ym − Y) ] = E[єm] e average MSE error of M such models is E Av = M M m Em = M M m E[єm] C E Com = E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ ⎛ ⎝ M M m ˆ Ym ˆ YCom −Y ⎞ ⎠ ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ = E ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ M M m єm ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ If model errors are E[єm] = (well-speci ed) E[єmєm′ ] = m ≠ m′ (uncorrelated) then E Com = M M m E[є ] = M EAv Simply averaging gives us M times less error!

E
8. ### WHY COMBINE 3 C Any model committee is another model

E Model combination as a model design strategy is known as ensemble learning Important examples → Bagging (‘bootstrap aggregating’ Breiman, ) → Boosting (Freund & Schapire, ) Hint: When someone suggests Deep Learning, try XGBoost rst!
9. ### WHY COMBINE 3 C Any model committee is another model

E Model combination as a model design strategy is known as ensemble learning Important examples → Bagging (‘bootstrap aggregating’ Breiman, ) → Boosting (Freund & Schapire, ) Hint: When someone suggests Deep Learning, try XGBoost rst! W ? Model combination helps navigates the bias-variance decomposition → Individual models can a ord to be less biased and quite variable → because averaging reduces committee variance → e complete model has at least as little bias as its components Let’s brie y review the bias-variance tradeo ...
10. ### BIAS, VARIANCE, AND ERROR 4 −2 −1 0 1 2

0.00 0.25 0.50 0.75 1.00 Degree 0 1 2 3 6 8 MSE = variance + bias + noise → Model : no error, high variance, low bias → Model : high error, low variance, high bias → Model : lowest error, mid variance, mid bias 0.00 0.25 0.50 0.75 0 1 2 3 4 5 6 7 8 9 Degree RMSE Sample in out In-sample error is a good guide to out of sample performance...until it isn’t → is is the bias-variance tradeo
11. ### WHY IT WORKS 5 eoretically, we can guarantee that E

Com ≤ EAv Whether we get M times less error depends on our A → Correct model speci cations, e.g. no missing variables / unconfounded → Uncorrelated errors, e.g. errors independent in time and group What if these are not true? → Build a better model! → Learn the model error correlation matrix and make a weighted average Perfectly correlated judgment errors too
12. ### IN PEOPLE 6 H ? Risk factors for groupthink (according

to Janis): → Group cohesiveness (correlated errors) → Group insulation (model mispeci cation) → Leadership style (weighting, correlated errors) → Methodical information processing procedures S Increase committee diversity, e.g. board composition
13. ### IN PEOPLE 6 H ? Risk factors for groupthink (according

to Janis): → Group cohesiveness (correlated errors) → Group insulation (model mispeci cation) → Leadership style (weighting, correlated errors) → Methodical information processing procedures S Increase committee diversity, e.g. board composition O en thought to work because of → increased representation → new perspectives From our perspective, maybe because increased diversity → decorrelates errors → makes for better speci ed models (participants) Note: → ‘playing devil’s advocate’ (or WWJD?) fakes actual diversity by anti-correlating errors
14. ### ENSEMBLES IN RESEARCH 7 ere are usually lots of di

erent ways to study the same question Inter-researcher disagreement is o en driven by di erent → subsets of data, coding strategies → DV, IV, control choices → variable transformations, e.g. logs and cut-points → statistical assumptions, e.g. xed vs random e ects, interactions, and non-linearities
15. ### ENSEMBLES IN RESEARCH 7 ere are usually lots of di

erent ways to study the same question Inter-researcher disagreement is o en driven by di erent → subsets of data, coding strategies → DV, IV, control choices → variable transformations, e.g. logs and cut-points → statistical assumptions, e.g. xed vs random e ects, interactions, and non-linearities Identify models with decision makers How to make a nal inference? → Refuse! → Choose the best: model selection → Combine them: model averaging Multiverse analysis (Steegen et al., )

17. ### ENTER THE MULTIVERSE 9 I point estimates are above zero

and are below → Did they con rm hypothesis ? → Is the probability of hypothesis / ? How to combine these estimates?
18. ### BAYES AGAIN 10 Consider a prior over models M a.k.a.

preprocessing choices, of data D where Z is our z-score. M By the Law of Total Probability P(Z D) = M m P(Z, M = m D) = M m P(Z M = m, D)P(M = m D) P(M = m D) is sometimes called a ‘Bayes Factor’
19. ### BAYES AGAIN 10 Consider a prior over models M a.k.a.

preprocessing choices, of data D where Z is our z-score. M By the Law of Total Probability P(Z D) = M m P(Z, M = m D) = M m P(Z M = m, D)P(M = m D) P(M = m D) is sometimes called a ‘Bayes Factor’ If we don’t both to distinguish between how likely each model is in the light of the data then, maybe... P(M = m D) = P(M = m) = M (a ‘ at’ prior) What if we wanted to do things properly? Consider a model with parameters β Bayesian inferences about β should be based on P(β D, M = m) which is... P(D β, M = m)P(β M = m) P(D M = m) e marginal likelihood of the data is the denominator → How likely each M makes the data, averaging out uncertainty about β from which comes the Bayes Factor P(M = m D) = P(D M = m) P(M = m) P(D)
20. ### BAYES AGAIN 11 It’s fairly implausible that P(M = m

D) = P(M = m) = M (though not impossible) → If some model speci cations are a priori more likely than others P(M = m) ≠ M → If model speci cation choices are independent of D, i.e. you don’t learn anything about them by seeing the data P(M = m D) ≠ P(M = m)
21. ### BAYES AGAIN 11 It’s fairly implausible that P(M = m

D) = P(M = m) = M (though not impossible) → If some model speci cations are a priori more likely than others P(M = m) ≠ M → If model speci cation choices are independent of D, i.e. you don’t learn anything about them by seeing the data P(M = m D) ≠ P(M = m) M We can also use it to select a single model (but why?)
22. ### BAYES AGAIN 11 It’s fairly implausible that P(M = m

D) = P(M = m) = M (though not impossible) → If some model speci cations are a priori more likely than others P(M = m) ≠ M → If model speci cation choices are independent of D, i.e. you don’t learn anything about them by seeing the data P(M = m D) ≠ P(M = m) M We can also use it to select a single model (but why?) O ’ R → in exible model: few Ds but more likely → exible model: many Ds less likely Flexibility and data t trade o (Bishop, )
23. ### IN PEOPLE 12 H ? Decision weighting should re ect

expertise on this kind of data: P(M = m D) for my D → Small numbers of experts (but not one) → Not large numbers of uninformed Tetlock and Gardner ( ) o ers two cognitive styles → Hedgehogs (assign high probability to relatively few data sets) → Foxes (assign lower probability to a wider range) We should probably think of these as a continuum
24. ### THE LIMITS OF AVERAGING 13 Some things don’t make sense

to average D → R: rainfall → Y: crop yields → P: political con ict → F: famine Question: → What is the e ect of political con ict on famine? What variables to choose? (List, )
25. ### THE LIMITS OF AVERAGING 13 Some things don’t make sense

to average D → R: rainfall → Y: crop yields → P: political con ict → F: famine Question: → What is the e ect of political con ict on famine? What variables to choose? (List, ) In a model of F → Expert would condition on Y → Expert would not condition on Y → Expert would only condition on Y if they wanted the ‘direct e ect’ of P

27. ### THE LIMITS OF AVERAGING 14 D C Our experts can

t their models and we can average the predictions → Predictively this might be ne? → e marginal e ect of P on F is a mixture of very di erent ‘worlds’ We will be working with Expert ’s model (it’s a superset of the other two) But there’s no guarantee that the superset of more experts will make any sense (or even be acyclic!)

the course
30. ### REVISITING PAPAL CHOICE 17 S eorems from Arrow (Arrow, ),

Gibbard and Satterthwaite (Gibbard, ; Satterthwaite, ) suggest → Intuitive criteria for ‘voting’ cannot be simultaneously satis ed T [with deliberation] there would not be any need for an aggregating mechanism, since a rational discussion would tend to produce unanimous preferences Jon Elster, in Matravers and Pike ( ) Deliberation increases single peakedness, a.k.a. meta-agreement (Dryzek & List, ) Subsequent voting will be immune to Arrow-like problems
31. ### IDEOLOGY AND SOPHISTICATION 18 Broockman ( ) points out that

sophisticated respondents have low-D preferences → eir views on di erent policies are not independent → Equivalently, their e ective D is lower than the number of questions you ask them S Unsophisticated voters have nearly independent preferences → Many preference inference methods assume sophistication, e.g. averages of directed responses, scaling models, etc. → ese will put sophisticated voters in the right place, and unsophisticated voters in the middle anyway despite them being mostly elsewhere Ideology is a regularizer / dimensionality reducer / preference structurer... Deliberation increases sophistication!
32. ### PLAN 19 Why combine judgments? Bias, variance, and error Groupthink

and other bad outcomes Multiverse models in science Motivating multiverses with Bayes e limits of model averaging Deliberation and social choice
33. ### REFERENCES 20 Arrow, K. J. ( ). Social choice and

individual values. John Wiley & Sons. Bishop, C. M. ( ). Neural networks for pattern recognition. Oxford University Press. Breiman, L. ( ). Bagging predictors. Machine Learning, ( ), – . Broockman, D. E. ( ). Approaches to studying policy representation. Legislative Studies Quarterly, ( ), – . Dryzek, J. S., & List, C. ( ). Social choice theory and deliberative democracy: A reconciliation. British Journal of Political Science, ( ), – . Freund, Y., & Schapire, R. E. ( ). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, ( ), – . Gibbard, A. ( ). Manipulation of voting schemes: A general result. Econometrica, ( ), . Hornik, K., Stinchcombe, M., & White, H. ( ). Multilayer feedforward networks are universal approximators. Neural Networks, , – . List, C. ( ). e theory of judgment aggregation: An introductory review. Synthese, ( ), – . Matravers, D., & Pike, J. ( ). Debates in contemporary political philosophy: An anthology. Routledge.
34. ### REFERENCES 21 Rumelhart, D. E. (Ed.). ( ). Parallel distributed

processing. : Foundations. MIT Press. Satterthwaite, M. A. ( ). Strategy-proofness and Arrow’s conditions: Existence and correspondence theorems for voting procedures and social welfare functions. Journal of Economic eory, ( ), – . Schweinsberg, M., Feldman, M., Staub, N., van den Akker, O. R., van Aert, R. C. M., van Assen, M. A. L. M., Liu, Y., Altho , T., Heer, J., Kale, A., Mohamed, Z., Amireh, H., Venkatesh Prasad, V., Bernstein, A., Robinson, E., Snellman, K., Amy Sommer, S., Otner, S. M. G., Robinson, D., ... Luis Uhlmann, E. ( ). Same data, di erent conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis. Organizational Behavior and Human Decision Processes, , – . Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. ( ). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, ( ), – . Tetlock, P. E., & Gardner, D. ( ). Superforecasting: e art and science of prediction. Crown Publishers.
35. ### CRAZY FLEXIBLE MODELS 22 A old school multilayer perceptron (MLP,

Rumelhart, ) with one hidden layer: A universal approximator, due to the internal non-linearity (Hornik et al., ) A linear combination of J models E[Y X . . . XD] = J j βjϕj(X . . . XD) where ϕj is a non-linear function of a linear combination of inoput data ϕj(X . . . XD) = ( + exp(− d βjd Xd )) at’s → a model with D × J + J parameters → A regression on the output of J logistic regressions on the input data