The Three Forms of (Legal) Prediction - Experts, Crowds + Algorithms
Professor Daniel Martin Katz presentation -- The Three Forms of (Legal) Prediction - Experts, Crowds + Algorithms as well as a discussion of SCOTUS and Associated Stock Market Returns aka "Law on the Market"
The Three Forms of (Legal) Prediction blog | ComputationalLegalStudies.com edu | illinois tech - chicago kent law lab | TheLawLab.com page | DanielMartinKatz.com
Legal Underwriting Data Driven EDiscovery/Due Diligence (Predictive Coding) #Predict Rogue Behavior Data Driven Compliance #Predict Contract Terms/Outcomes Data Driven Transactional Work #Predict Regulatory Outcomes Data Driven Lobbying, etc.
PA P E R A R T I C L E S , T E L E V I S I O N A N D RADIO TIME, CONFERENCE PANELS, BLOG P O S T S , A N D T W E E T S A R E D E V O T E D T O QUESTIONS SUCH AS: H O W W I L L T H E C O U R T R U L E I N T H I S PARTICULAR CASE? IN THE DIRECTION OF WHICH PARTY WILL AN INDIVIDUAL JUSTICE VOTE?
OF THE “RIGHT TO COUNSEL” CASES, 51 AMER. POL. SCI. REV. 1 (1957). 1957 S. SIDNEY ULMER, QUANTITATIVE ANALYSIS OF JUDICIAL PROCESSES: SOME PRACTICAL AND THEORETICAL APPLICATIONS, 28 LAW & CONTEMP. PROBS. 164 (1963). 1963 A CO U P L E O F E A R LY E F F O RT S
E C O U R T C A S E S PROBABILISTICALLY: THE SEARCH AND SEIZURE CASES, 1962-1981, 78 AMERICAN POLITICAL SCIENCE REVIEW 891 (1984) 1984 A N I M P O RTA N T L AT E R E F F O RT
Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project: B U T T H I S WA S T H E PA P E R T H AT I N S P I R E D O U R E F F O RT S 2004
Kim, Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project:
the completion time for a given task, the hours they projected differed by 71%, on average. When pathologists made two assessments of the severity of biopsy results, the correlation between their ratings was only .61 (out of a perfect 1.0), indicating that they made inconsistent diagnoses quite frequently. Judgments made by different people are even more likely to diverge.”
P O N S O R S H I P H AV E VA R I E D B U T T H E R E H AV E B E E N T E N S O F T H O U S A N D S O F $ $ A N D P R I Z E S D I ST R I B U T E D OV E R T H E Y E A R S
M E S I N I N T E R E ST I N G WAY S A S S H O W N A B OV E ) U S E R S A R E F R E E TO C H A N G E T H E I R P R E D I C T I O N S U N T I L T H E DAT E O F F I N A L D E C I S I O N
O L K S C H A N G E T H E I R VOT E S 6 3 6 8 5 9 5 4 5 8 4 5 F I N A L P R E D I C T I O N S OV E R A L L P R E D I C T I O N S WO R K I N G W I T H R E A L DATA
H I S I S * N OT * R E A L LY S U RV I VO R S H I P B I A S B U T R AT H E R A R E V E L AT I O N M E C H A N I S M ( I . E . YO U ST I C K A R O U N D I F T H I N K YO U A R E G O O D AT T H E U N D E R LY I N G TA S K )
A L TY P E S T H E I N P U T S I G N A L S CA N A S S U M E M A N Y D I F F E R E N T F O R M S I N C LU D I N G F R O M M O D E L S O R I N D I V I D UA L S O R S E N S O R S ( O R S O M E CA S E S E V E N OT H E R C R O W D S )
humans but could be networked IT systems Decentralized Distributed Ledgers -or- Oracles -or- IOT sensors with Crowdsourcing Validation #Blockchain #InternetofThings #Crypto
I N G I S ‘ U N D I S C I P L I N E D ZO O O F M O D E L S ’ J E S S I CA F L AC K P R O F E S S O R S A N TA F E I N ST I T U T E D E C . 2 7 , 2 0 1 7 ( V I A T W I T T E R )
D E L S CA N A L S O TA K E I N TO ACCO U N T I N F O R M AT I O N ( S I G N A L S ) F R O M S O M E S U B S E T O F PA RT I C I PA N T S ( D E F I N E D U S I N G E I T H E R I N C LU S I O N R U L E S O R E XC LU S I O N R U L E S )
C T I O N R U L E S T H E I N T E R AC T I O N O F T H E S E R U L E S Y I E L D S * N OT * A N I N D I V I D UA L M O D E L B U T R AT H E R A M O D E L S PAC E
E 1 + ( 2 8 · 9 9 · 1 0 0 ) = 2 7 7 2 0 1 F I R ST T E R M I S T H E S I M P L E ST C R O W D S O U R C I N G M O D E L W I T H N O S U B S E T O R W E I G H T I N G R U L E S S E CO N D T E R M CO R R E S P O N D S TO 2 8 M O D E L S ( CO M B I N AT I O N O F P E R F O R M A N C E T H R E S H O L D / W E I G H T I N G R U L E S ) F O R E AC H CO M B I N AT I O N O F 9 9 R A N K A N D 1 0 0 E X P E R I E N C E T H R E S H O L D S
S A L A R G E M O D E L S PAC E W E D O H I G H L I G H T T H E P E R F O R M A N C E O F F O U R E X A M P L E M O D E L S ( A N D T H E N U L L M O D E L )
M U M ACC U R ACY ( T H E R E A R E AC T UA L LY S E V E R A L M O D E L CO N F I G U R AT I O N S W H I C H O F F E R R O U G H LY E Q U I VA L E N T P E R F O R M A N C E ) E X P E R I E N C E T H R E S H O L D O F 5 C R O W D S I Z E I S CA P P E D AT 2 2 E X P O N E N T I A L W E I G H T W I T H A L P H A O F 0 . 1
P LOT F L AT T E N S T H E D I M E N S I O N A L I TY O F T H E S PAC E ( E AC H C E L L I S T H E AV E R AG E M O D E L P E R F O R M A N C E OV E R A L L OT H E R M O D E L PA R A M E T E R AT E AC H E X P E R I E N C E , R A N K CO M B O )
N OT I N G T H AT P R E D I C T I O N O R I E N T E D PA P E R S A R E C U R R E N T LY S W I M M I N G AG A I N ST M A I N ST R E A M S O C I A L S C I E N C E ( A N D L AW )
T E D TO P O L I CY E VA LUAT I O N ( S U C H A S D O E S T H I S PA RT I C U L A R P O L I CY I N T E RV E N T I O N AC H I E V E I T S STAT E D O B J E C T I V E S )
Pauline T. Kim, Competing Approaches to Predicting Supreme Court Decision Making, 2 Perspectives on Politics 761 (2004). “the best test of an explanatory theory is its ability to predict future events. To the extent that scholars in both disciplines (social science and law) seek to explain court behavior, they ought to test their theories not only against cases already decided, but against future outcomes as well.”
to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown …. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.” Leo Breiman, Statistical modeling: The two cultures (with comments and a rejoinder by the author), 16 Statistical Science 199 (2001) Note: Leo Breiman Invented Random Forests
streams of intelligence algorithm forecast ensemble method ENSEMBLE MODEL we can use machine learning methods and metadata such as case topic, lower court as well as crowd metadata to ‘learn’ the conditional weights to apply to the input signals
permit doctors and patients to understand the genetic basis of human disease and the role that genes play in the onset, progression and treatment of disease.”
WRDS Data which is *not* open source but can be obtained from Wharton https://github.com/mjbommar/law-on-the-market https://wrds-web.wharton.upenn.edu/wrds/
and seek to identify statistically significant deviations from that baseline We want to isolate the effect of the event from other general market movements