Data Science and Decisions 2022: Week 1

Transcript

1. DATA SCIENCE AND DECISION MAKING Introducing decision theory Will Lowe Data Science Lab, Hertie School 2022-02-09
2. PLAN 1 Course logistics Visiting speakers Course goals Decision theory begins Alternatives Week sequence Your instructor
3. COURSE STRUCTURE 2 Structure → Moodle is the source of all course information → Evaluation: quizzes and a policy paper → ose quiz dates are still incorrect → With two exceptions, readings are for a er the lecture Expectations → Engagement, particularly with invited speakers → Willingness to share professional and/or personal experiences “Safety lights are for dudes”
4. PRACTITIONERS 3 Patrick Doupe Senior Economist, Zalando Caroline King Global Head, Business Development, Government A airs, SAP Jonathan Robinson Head of Analytics, Catalist Christopher Gandrud Head of Experiments and Economics, Zalando It is as yet unclear which speakers will join us in person and which remote For remote speakers we will all be remote. Robinson is in the US and will speak at pm on . . (We will record the session if you can’t make it)
5. GOALS 4 W I ’ An understanding and appreciation of → formal models of decision making, their strengths and limitations → how data science tools succeed (or fail) to realize this kind of decision making → e challenges of decision making for individuals, organizations (and individuals in organizations) → How this works in organizations, from people who have tried to make it work W ’ A data science / ML / AI course → We assume you already know enough about data science / ML/ AI and will concentrate on interpretation and implementation → So ware tools to take away and use in your next organization A collection of foolproof strategies for → embedding data science in decision making contexts → deciding how to make big life decisions
6. OR MAYBE YOU WILL... 5 Charles Darwin, April
7. DECISION THEORY: OVERVIEW 6 Decision making problems are ubiquitous Using decision theory to make decisions can nevertheless be controversial H Given actions {A}, consequences {C}, and a utility function U(., .) choose the A with highest value of E[U(A)] = i U(Ci , A)P(Ci A)
8. DECISION THEORY: OVERVIEW 6 Decision making problems are ubiquitous Using decision theory to make decisions can nevertheless be controversial H Given actions {A}, consequences {C}, and a utility function U(., .) choose the A with highest value of E[U(A)] = i U(Ci , A)P(Ci A) Moving parts: → e uncertainty part: P → e utility / loss function part: U(., .) → e expectation part E[.] Blaise Pascal, Living Wager Advocate
9. DECISION THEORY: OVERVIEW 7 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s (say, a proportion of the stake) would be fair to o er the player to forego (just) his th throw?
10. DECISION THEORY: OVERVIEW 7 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s (say, a proportion of the stake) would be fair to o er the player to forego (just) his th throw? P ’ In general the value of a bet is its expected value E[u] = i ui p(ui) A transaction is fair if it leaves the player’s expected values the same
11. DECISION THEORY: OVERVIEW 7 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s (say, a proportion of the stake) would be fair to o er the player to forego (just) his th throw? P ’ In general the value of a bet is its expected value E[u] = i ui p(ui) A transaction is fair if it leaves the player’s expected values the same E Flip a fair coin and get if heads and pay if tails → Expected value: So you would not pay to play this game...
12. FAIRNESS 8 P It’s a popular, but not uncontroversial idea “It’s not about what I want, it’s about what’s fair” Harvey Dent, Tosser
13. DECISION THEORY: OVERVIEW 9 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s as a portion of the stake would be fair to o er the player to forego (just) his th throw?
14. DECISION THEORY: OVERVIEW 9 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s as a portion of the stake would be fair to o er the player to forego (just) his th throw?
15. DECISION THEORY: OVERVIEW 9 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s as a portion of the stake would be fair to o er the player to forego (just) his th throw? F ’ Find the proportion s that equates the expected values of → Keeping the throw and forgoing the throw for a diminished stake
16. DECISION THEORY: OVERVIEW 9 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s as a portion of the stake would be fair to o er the player to forego (just) his th throw? F ’ Find the proportion s that equates the expected values of → Keeping the throw and forgoing the throw for a diminished stake E[Keeping the throw] u P(win on ) + u − P(lose th, win later) E[Forgoing th throw for su payo ] u × s + u − P(win later)
17. DECISION THEORY: OVERVIEW 9 P France : e Chevalier de Méré describes to Pascal two gambling problems, who discusses them with Fermat. Here’s the rst → A player has throws of a die to get a . → e player stakes u that he can do it → throws have happened without a What amount s as a portion of the stake would be fair to o er the player to forego (just) his th throw? F ’ Find the proportion s that equates the expected values of → Keeping the throw and forgoing the throw for a diminished stake E[Keeping the throw] u P(win on ) + u − P(lose th, win later) E[Forgoing th throw for su payo ] u × s + u − P(win later) To make them equal, set s =
18. ALTERNATIVES 10 International Criminal Court, Alternatives to the uncertainty part → Ignorance vs. uncertainty a.k.a. risk → Qualitative approaches, e.g. ‘beyond reasonable doubt’ (but c.f. ‘balance of probabilities’) → ‘Balancing considerations’, in human rights law
19. ALTERNATIVES 10 International Criminal Court, Alternatives to the uncertainty part → Ignorance vs. uncertainty a.k.a. risk → Qualitative approaches, e.g. ‘beyond reasonable doubt’ (but c.f. ‘balance of probabilities’) → ‘Balancing considerations’, in human rights law Alternatives to the utility / loss part → Refusal to quantify outcomes, e.g. ‘no price on human life’ → Procedures vs consequences, e.g. ‘inadmissible evidence’
20. ALTERNATIVES 10 International Criminal Court, Alternatives to the uncertainty part → Ignorance vs. uncertainty a.k.a. risk → Qualitative approaches, e.g. ‘beyond reasonable doubt’ (but c.f. ‘balance of probabilities’) → ‘Balancing considerations’, in human rights law Alternatives to the utility / loss part → Refusal to quantify outcomes, e.g. ‘no price on human life’ → Procedures vs consequences, e.g. ‘inadmissible evidence’ Alternatives to the expectation part → Rawlsian maximin & ‘precautionary principles’
21. DECISION THEORY, THE VERY IDEA 11 What kind of a theory is this? → Normative? → Descriptive? → Kinda both?
22. DECISION THEORY, THE VERY IDEA 11 What kind of a theory is this? → Normative? → Descriptive? → Kinda both? Two connected senses of normative: → Normative: You should decide this way → Intepretative: if you don’t decide this way we don’t really understand what you’re doing (Davidson, ; Dennett, ; Little, ; Quine, ) → Statistical: this is the way people usually make decisions
23. DECISION THEORY, THE VERY IDEA 11 What kind of a theory is this? → Normative? → Descriptive? → Kinda both? Two connected senses of normative: → Normative: You should decide this way → Intepretative: if you don’t decide this way we don’t really understand what you’re doing (Davidson, ; Dennett, ; Little, ; Quine, ) → Statistical: this is the way people usually make decisions Roughly Sellars’ ( ) distinction between the ‘Space of Reasons’ and ‘Space of Causes’ → In people, these can diverge – but not that far → In organizations more so, e.g. company mission statements vs company behavior
24. DECISION THEORY, THE VERY IDEA 12 Who is this theory about? → People → Parts of people → Machines → Groups, e.g. cabinets, governments, companies, religions One striking aspect of the theory is that it is agnostic about this question P → Aggregation → Strategy “I’m the decider. I decide what’s best” G. W. Bush,
25. WEEK SEQUENCE 13 → e big picture → e uncertainty part → e utility / loss function part → Deciding with a machine → Deciding with a brain → Fairness → Deciding in groups
26. THE UNCERTAINTY PART 14 In a newspaper said that the US Republican party candidate had a . chance of winning the national election → What does . mean here?
27. THE UNCERTAINTY PART 14 In a newspaper said that the US Republican party candidate had a . chance of winning the national election → What does . mean here? What is the probability that Iran re-enters the Nuclear Agreement in the next year? Consider one such probability estimate → What does it mean?
28. THE UNCERTAINTY PART 14 In a newspaper said that the US Republican party candidate had a . chance of winning the national election → What does . mean here? What is the probability that Iran re-enters the Nuclear Agreement in the next year? Consider one such probability estimate → What does it mean? Are you sure it should not be . higher? In the light of everyone else’s estimates, do you want to change yours? Why?
29. WEEK 2: THE UNCERTAINTY PART 15 Motivation → Why quantify uncertainty? → Why use probability to do it? eory: → Bayesian inference → Bayesian rationality → Expectations vs E[xpectations] Justi cation: → Consistency → ‘Dutch books’ and other arguments from embarrassment I’m all about the billiards
30. WEEK 2: THE UNCERTAINTY PART 16 Inevitability: → Representation theorems (Cox, ) → Axiomatic treatments of uncertainty measures Trouble: → Psychological realism → Computational tractability Alternatives: → Frequentism, e.g. Fisher and Neyman ere are two types of error and this isn’t either of them
31. THE CONSEQUENCES PART 17 You have a classi er that (or an expert who) makes predictions about war outbreaks → How much worse are false negatives than false positives? Is one year of healthy life better than two years of being chronically ill (Ogden, )
32. THE CONSEQUENCES PART 17 You have a classi er that (or an expert who) makes predictions about war outbreaks → How much worse are false negatives than false positives? Is one year of healthy life better than two years of being chronically ill (Ogden, ) Consider the utility/loss from following outcomes: → Global sea levels will rise cm by → Global sea levels will rise m by → Global sea levels will rise cm by
33. THE CONSEQUENCES PART 17 You have a classi er that (or an expert who) makes predictions about war outbreaks → How much worse are false negatives than false positives? Is one year of healthy life better than two years of being chronically ill (Ogden, ) Consider the utility/loss from following outcomes: → Global sea levels will rise cm by → Global sea levels will rise m by → Global sea levels will rise cm by General questions → What, if anything, do we owe future generations? → Some actions would reduce the size of those generations. → Does this matter? If so, how?
34. WEEK 3: THE CONSEQUENCES PART 18 eory: → of loss and utility → dominance, the ‘precautionary principle’, and other heuristics → How to discount the future Justi cation → Consistency (again) → Representation theorems (again) Implications → Political theory: Rawls vs Harsanyi → Mixing up probability and loss for good (and bad) reasons Savage in a bowtie
35. WEEK 5: THE RISE OF THE MACHINES 19 Machine learning → for probability estimation and predictions → forecast evaluation, calibration, and scoring Loss functions → fake ones, e.g. ‘log loss’ → real ones, e.g. reinforcement learning and Markov Decision Processes Really real losses → fairness and bias Computers exist. Probability doesn’t
36. WEEK 8: MEATSPACE 20 ‘Implementation’ details → Rationality in theory...and people → How is probability and loss represented in brains? → Risk aversion, non-exponential discounting → Cognitive heuristics and cognitive biases Performance → Forecasters and super-forecasters → Human decision making with machine learning in the mix
37. WEEK 10: GROUP DECISIONS 21 Decision making in and by organizations eory: → Group decisions and preference aggregation in humans and machines → Condorcet, Arrow, and the necessity of structure → Deliberation is great. Discuss. Practice: → Deliberation, ‘group think’, ‘ lter bubbles’ → Information design in bureaucracies and companies e Afghanisdag
38. WEEK 12: PUTTING IT ALL TOGETHER 22 Mixed human and machine decision making Implementation, risks, bene ts Whatever else we’d like to talk about, or that our speakers brought up. See you all next week!
39. REFERENCES 23 Cox, R. T. ( ). “Probability, frequency and reasonable expectation.” American Journal of Physics, ( ), – . Davidson, D. ( ). “Inquiries into truth and interpretation.” Clarendon Press. Dennett, D. C. ( ). Evolution, error, and intentionality. e Intentional Stance (pp. – ). MIT Press. Little, A. T. ( ). “Detecting motivated reasoning.” Ogden, J. ( ). “QALYs and their role in the NICE decision-making process.” Prescriber, ( ), – _eprint: https://onlinelibrary.wiley.com/doi/pdf/ . /psb. . Quine, W. v. O. ( ). “From a logical point of view ( nd Ed.). Harvard University Press. Sellars, W. ( ). “Empiricism and the philosophy of mind.” Harvard University Press.