Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman
Overview Deck of Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman - v 1.01
I N G ACC U R AT E LY A N D R O B U ST LY P R E D I C T S S U P R E M E CO U RT D E C I S I O N S DA N I E L M A RT I N KATZ | I L L I N O I S T E C H + STA N F O R D CO D E X M I C H A E L B O M M A R I TO | I L L I N O I S T E C H + STA N F O R D CO D E X J O S H B L AC K M A N | S O U T H T E X A S CO L L E G E O F L AW 0.4 0.5 0.6 0.7 0.8 0.9 2012 2013 2014 2015 2016 2017 Date of Decison Case Level Cumulative Accuracy By Model Type Model Type Model 1 Model 2 Model 3 Model 4 Null Model
I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | DA N I E L M A R T I N K AT Z . C O M C O R P | L E X P R E D I C T. C O M MICHAEL BOMMARITO E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | B O M M A R I TO L LC . C O M C O R P | L E X P R E D I C T. C O M JOSH BLACKMAN E D U | S O U T H T E X A S C O L L E G E O F L AW H O U S TO N B LO G | J O S H B L AC K M A N . C O M PAG E | J O S H B L AC K M A N . C O M / B LO G C O R P | L E X P R E D I C T. C O M
OF THE “RIGHT TO COUNSEL” CASES, 51 AMER. POL. SCI. REV. 1 (1957). 1957 S. SIDNEY ULMER, QUANTITATIVE ANALYSIS OF JUDICIAL PROCESSES: SOME PRACTICAL AND THEORETICAL APPLICATIONS, 28 LAW & CONTEMP. PROBS. 164 (1963). 1963 A CO U P L E O F E A R LY E F F O RT S
E C O U R T C A S E S PROBABILISTICALLY: THE SEARCH AND SEIZURE CASES, 1962-1981, 78 AMERICAN POLITICAL SCIENCE REVIEW 891 (1984) 1984 A N I M P O RTA N T L AT E R E F F O RT
Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project: B U T T H I S WA S T H E PA P E R T H AT I N S P I R E D O U R E F F O RT S 2004
SUPREME COURT FORECASTING USING BOOSTED DECISION TREES”. HTTP://J.MP/2NRJTO6 T H E R E A R E A L S O S O M E OT H E R N OTA B L E R E C E N T E F F O RT S
SEN.. “EMOTIONAL AROUSAL PREDICTS VOTING ON THE U.S. SUPREME COURT.” POLITICAL ANALYSIS (FORTHCOMING) T H E R E A R E A L S O S O M E OT H E R N OTA B L E R E C E N T E F F O RT S
N OT I N G T H AT P R E D I C T I O N O R I E N T E D PA P E R S A R E C U R R E N T LY S W I M M I N G AG A I N ST M A I N ST R E A M S O C I A L S C I E N C E ( A N D L AW )
T E D TO P O L I CY E VA LUAT I O N ( S U C H A S D O E S T H I S PA RT I C U L A R P O L I CY I N T E RV E N T I O N AC H I E V E I T S STAT E D O B J E C T I V E S )
Pauline T. Kim, Competing Approaches to Predicting Supreme Court Decision Making, 2 Perspectives on Politics 761 (2004). “the best test of an explanatory theory is its ability to predict future events. To the extent that scholars in both disciplines (social science and law) seek to explain court behavior, they ought to test their theories not only against cases already decided, but against future outcomes as well.”
to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown …. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.” Leo Breiman, Statistical modeling: The two cultures (with comments and a rejoinder by the author), 16 Statistical Science 199 (2001) Note: Leo Breiman Invented Random Forests
P O N S O R S H I P H AV E VA R I E D B U T T H E R E H AV E B E E N T E N S O F T H O U S A N D S O F $ $ A N D P R I Z E S D I ST R I B U T E D OV E R T H E Y E A R S
E S I N I N T E R E ST I N G WAY S A S S H O W N A B OV E ) U S E R S A R E F R E E TO C H A N G E T H E I R P R E D I C T I O N S U N T I L T H E DAT E O F F I N A L D E C I S I O N
Y M I Z E A N D T H E N O P E N S O U R C E P L AY E R P R E D I C T I O N DATA TO S U P P O RT ACA D E M I C R E S E A R C H I N TO P R E D I C T I O N , C R O W D S O U R C I N G , E TC .
O L K S C H A N G E T H E I R VOT E S 6 3 6 8 5 9 5 4 5 8 4 5 F I N A L P R E D I C T I O N S OV E R A L L P R E D I C T I O N S WO R K I N G W I T H R E A L DATA
H I S I S * N OT * R E A L LY S U RV I VO R S H I P B I A S B U T R AT H E R A R E V E L AT I O N M E C H A N I S M ( I . E . YO U ST I C K A R O U N D I F T H I N K YO U A R E G O O D AT T H E U N D E R LY I N G TA S K )
A L TY P E S T H E I N P U T S I G N A L S CA N A S S U M E M A N Y D I F F E R E N T F O R M S I N C LU D I N G F R O M M O D E L S O R I N D I V I D UA L S O R S E N S O R S ( O R S O M E CA S E S E V E N OT H E R C R O W D S )
humans but could be networked IT systems Decentralized Distributed Ledgers -or- Oracles -or- IOT sensors with Crowdsourcing Validation #Blockchain #InternetofThings #Crypto
I N G I S ‘ U N D I S C I P L I N E D ZO O O F M O D E L S ’ J E S S I CA F L AC K P R O F E S S O R S A N TA F E I N ST I T U T E D E C . 2 7 , 2 0 1 7 ( V I A T W I T T E R )
D E L S CA N A L S O TA K E I N TO ACCO U N T I N F O R M AT I O N ( S I G N A L S ) F R O M S O M E S U B S E T O F PA RT I C I PA N T S ( D E F I N E D U S I N G E I T H E R I N C LU S I O N R U L E S O R E XC LU S I O N R U L E S )
W E CO U L D I M P O S E A R E Q U I R E M E N T T H AT I N O R D E R TO B E I N C LU D E D I N T H E C R O W D, A P L AY E R M U ST H AV E E X P E R I E N C E - P R E D I C T E D AT L E A ST P CA S E S
N K T H E PA RT I C I PA N T S W I T H I N T H E OV E R A L L C R O W D O R A C R O W D S U B S E T ACCO R D I N G TO S O M E R A N K F U N C T I O N FOLLOW THE LEADER PLAYER N
E W E CO U L D I M P O S E A R E Q U I R E M E N T T H AT I N O R D E R TO B E I N C LU D E D I N T H E C R O W D - A P L AY E R M U ST H AV E AC H I E V E D S O M E M I N I M U M P E R F O R M A N C E T H R E S H O L D
S H O L D I N G W E CO U L D I M P O S E A R E Q U I R E M E N T T H AT I N O R D E R TO B E I N C LU D E D I N T H E C R O W D - A P L AY E R M U ST H AV E D E M O ST R AT E D STAT I ST I CA L LY S I G N I F I CA N T P E R F O R M A N C E
N T H E S I M P L E CA S E , W E CA N AV E R AG E . A LT E R N AT I V E LY, W E CA N S E L E C T A W E I G H T I N G S C H E M E W H I C H U P W E I G H T S P R E D I C TO R S B A S E D U P O N S O M E C R I T E R I A .
C T I O N R U L E S T H E I N T E R AC T I O N O F T H E S E R U L E S Y I E L D S * N OT * A N I N D I V I D UA L M O D E L B U T R AT H E R A M O D E L S PAC E
E 1 + ( 2 8 · 9 9 · 1 0 0 ) = 2 7 7 2 0 1 F I R ST T E R M I S T H E S I M P L E ST C R O W D S O U R C I N G M O D E L W I T H N O S U B S E T O R W E I G H T I N G R U L E S S E CO N D T E R M CO R R E S P O N D S TO 2 8 M O D E L S ( CO M B I N AT I O N O F P E R F O R M A N C E T H R E S H O L D / W E I G H T I N G R U L E S ) F O R E AC H CO M B I N AT I O N O F 9 9 R A N K A N D 1 0 0 E X P E R I E N C E T H R E S H O L D S
S A L A R G E M O D E L S PAC E W E D O H I G H L I G H T T H E P E R F O R M A N C E O F F O U R E X A M P L E M O D E L S ( A N D T H E N U L L M O D E L )
M U M ACC U R ACY ( T H E R E A R E AC T UA L LY S E V E R A L M O D E L CO N F I G U R AT I O N S W H I C H O F F E R R O U G H LY E Q U I VA L E N T P E R F O R M A N C E ) E X P E R I E N C E T H R E S H O L D O F 5 C R O W D S I Z E I S CA P P E D AT 2 2 E X P O N E N T I A L W E I G H T W I T H A L P H A O F 0 . 1
P LOT F L AT T E N S T H E D I M E N S I O N A L I TY O F T H E S PAC E ( E AC H C E L L I S T H E AV E R AG E M O D E L P E R F O R M A N C E OV E R A L L OT H E R M O D E L PA R A M E T E R AT E AC H E X P E R I E N C E , R A N K CO M B O )
streams of intelligence algorithm forecast ensemble method ENSEMBLE MODEL we can use machine learning methods and metadata such as case topic, lower court as well as crowd metadata to ‘learn’ the conditional weights to apply to the input signals
OT H E R T H I N G S L I N K S TO O U R I N T E R E ST # A B N O R M A L R E T U R N S A N D J U D I C I A L D E C I S I O N S https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
A N O N Y M I Z E A N D T H E N O P E N S O U R C E P L AY E R P R E D I C T I O N DATA TO S U P P O RT ACA D E M I C R E S E A R C H I N TO P R E D I C T I O N , C R O W D S O U R C I N G , E TC .
I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | DA N I E L M A R T I N K AT Z . C O M C O R P | L E X P R E D I C T. C O M MICHAEL BOMMARITO E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | B O M M A R I TO L LC . C O M C O R P | L E X P R E D I C T. C O M JOSH BLACKMAN E D U | S O U T H T E X A S C O L L E G E O F L AW H O U S TO N B LO G | J O S H B L AC K M A N . C O M PAG E | J O S H B L AC K M A N . C O M / B LO G C O R P | L E X P R E D I C T. C O M
I N G ACC U R AT E LY A N D R O B U ST LY P R E D I C T S S U P R E M E CO U RT D E C I S I O N S DA N I E L M A RT I N KATZ | I L L I N O I S T E C H + STA N F O R D CO D E X M I C H A E L B O M M A R I TO | I L L I N O I S T E C H + STA N F O R D CO D E X J O S H B L AC K M A N | S O U T H T E X A S CO L L E G E O F L AW 0.4 0.5 0.6 0.7 0.8 0.9 2012 2013 2014 2015 2016 2017 Date of Decison Case Level Cumulative Accuracy By Model Type Model Type Model 1 Model 2 Model 3 Model 4 Null Model