Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman

Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman

Overview Deck of Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman - v 1.01

0f2a473c07602f3dd53c5ed0de0c56b5?s=128

Daniel Martin Katz

January 17, 2018
Tweet

Transcript

  1. C R O W D S O U R C

    I N G ACC U R AT E LY A N D R O B U ST LY P R E D I C T S S U P R E M E CO U RT D E C I S I O N S DA N I E L M A RT I N KATZ | I L L I N O I S T E C H + STA N F O R D CO D E X M I C H A E L B O M M A R I TO | I L L I N O I S T E C H + STA N F O R D CO D E X J O S H B L AC K M A N | S O U T H T E X A S CO L L E G E O F L AW 0.4 0.5 0.6 0.7 0.8 0.9 2012 2013 2014 2015 2016 2017 Date of Decison Case Level Cumulative Accuracy By Model Type Model Type Model 1 Model 2 Model 3 Model 4 Null Model
  2. DANIEL MARTIN KATZ E D U | I L L

    I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | DA N I E L M A R T I N K AT Z . C O M C O R P | L E X P R E D I C T. C O M MICHAEL BOMMARITO E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | B O M M A R I TO L LC . C O M C O R P | L E X P R E D I C T. C O M JOSH BLACKMAN E D U | S O U T H T E X A S C O L L E G E O F L AW H O U S TO N B LO G | J O S H B L AC K M A N . C O M PAG E | J O S H B L AC K M A N . C O M / B LO G C O R P | L E X P R E D I C T. C O M
  3. H T T P S : / / A R

    X I V. O RG / A B S / 1712 . 0 3 84 6 H T T P S : / / PA P E R S . S S R N . C O M / S O L 3 / PA P E R S . C F M ? A B S T R AC T _ I D = 3 0 8 5710
  4. O U R PA P E R I S A

    B O U T P R E D I C T I N G T H E D E C I S I O N S O F T H E S U P R E M E CO U RT O F T H E U N I T E D STAT E S #SCOTUS
  5. JUDICIAL PREDICTION IS THE LAW + POLITICS GRAIL QUEST

  6. YOU COULD START WITH HOLMES AND THE LEGAL REALISTS BUT

    THESE WERE *NOT* REALLY SCIENTIFIC EFFORTS
  7. FRED KORT, PREDICTING SUPREME COURT DECISIONS MATHEMATICALLY: A QUANTITATIVE ANALYSIS

    OF THE “RIGHT TO COUNSEL” CASES, 51 AMER. POL. SCI. REV. 1 (1957). 1957 S. SIDNEY ULMER, QUANTITATIVE ANALYSIS OF JUDICIAL PROCESSES: SOME PRACTICAL AND THEORETICAL APPLICATIONS, 28 LAW & CONTEMP. PROBS. 164 (1963). 1963 A CO U P L E O F E A R LY E F F O RT S
  8. COMPUTERWORLD JULY 1971 PROGRAM WRITTEN IN FORTRAN (THE 91% PREDICTION

    MARK WAS LIMITED TO CERTAIN CASES) HAROLD SPAETH
  9. JEFFREY A. SEGAL, PREDICTING S U P R E M

    E C O U R T C A S E S PROBABILISTICALLY: THE SEARCH AND SEIZURE CASES, 1962-1981, 78 AMERICAN POLITICAL SCIENCE REVIEW 891 (1984) 1984 A N I M P O RTA N T L AT E R E F F O RT
  10. Columbia Law Review (2004) Theodore W. Ruger, Pauline T. Kim,

    Andrew D. Martin, Kevin M. Quinn Legal and Political Science Approaches to Predicting Supreme Court Decision Making The Supreme Court Forecasting Project: B U T T H I S WA S T H E PA P E R T H AT I N S P I R E D O U R E F F O RT S 2004
  11. http://journals.plos.org/plosone/article? id=10.1371/journal.pone.0174698 available at RESEARCH ARTICLE A general approach for

    predicting the behavior of the Supreme Court of the United States Daniel Martin Katz1,2*, Michael J. Bommarito II1,2, Josh Blackman3 1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston, Houston, TX, United States of America * dkatz3@kentlaw.iit.edu Abstract Building on developments in machine learning and prior work in the science of judicial pre- diction, we construct a model designed to predict the behavior of the Supreme Court of the United States in a generalized, out-of-sample context. To do so, we develop a time-evolving random forest classifier that leverages unique feature engineering to predict more than 240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015). Using only data available prior to decision, our model outperforms null (baseline) models at both the justice and case level under both parametric and non-parametric tests. Over nearly two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus- tice vote level. More recently, over the past century, we outperform an in-sample optimized null model by nearly 5%. Our performance is consistent with, and improves on the general level of prediction demonstrated by prior work; however, our model is distinctive because it can be applied out-of-sample to the entire past and future of the Court, not a single term. Our results represent an important advance for the science of quantitative legal prediction and portend a range of other potential applications. Introduction As the leaves begin to fall each October, the first Monday marks the beginning of another term for the Supreme Court of the United States. Each term brings with it a series of challenging, important cases that cover legal questions as diverse as tax law, freedom of speech, patent law, administrative law, equal protection, and environmental law. In many instances, the Court’s decisions are meaningful not just for the litigants per se, but for society as a whole. Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal and political observers. Every year, newspapers, television and radio pundits, academic jour- nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular case. Will the Justices vote based on the political preferences of the President who appointed them or form a coalition along other dimensions? Will the Court counter expectations with an unexpected ruling? PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18 a1111111111 a1111111111 a1111111111 a1111111111 a1111111111 OPEN ACCESS Citation: Katz DM, Bommarito MJ, II, Blackman J (2017) A general approach for predicting the behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698. https://doi. org/10.1371/journal.pone.0174698 Editor: Luı ´s A. Nunes Amaral, Northwestern University, UNITED STATES Received: January 17, 2017 Accepted: March 13, 2017 Published: April 12, 2017 Copyright: © 2017 Katz et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability Statement: Data and replication code are available on Github at the following URL: https://github.com/mjbommar/scotus-predict-v2/. Funding: The author(s) received no specific funding for this work. Competing interests: All Authors are Members of a LexPredict, LLC which provides consulting services to various legal industry stakeholders. We received no financial contributions from LexPredict or anyone else for this paper. This does not alter our adherence to PLOS ONE policies on sharing data and materials. DANIEL MARTIN KATZ, MICHAEL J. BOMMARITO II & JOSH BLACKMAN, A GENERAL APPROACH FOR PREDICTING THE BEHAVIOR OF THE SUPREME COURT OF THE UNITED STATES, PLOS ONE 12.4 (2017): E0174698. APRIL 2017 T H I S I S O U R A LG O PA P E R
  12. JULY 2017 AARON KAUFMAN, PETER KRAFT, AND MAYA SEN. “IMPROVING

    SUPREME COURT FORECASTING USING BOOSTED DECISION TREES”. HTTP://J.MP/2NRJTO6 T H E R E A R E A L S O S O M E OT H E R N OTA B L E R E C E N T E F F O RT S
  13. DECEMBER 2017 DIETRICH, BRYCE J., RYAN D. ENOS, AND MAYA

    SEN.. “EMOTIONAL AROUSAL PREDICTS VOTING ON THE U.S. SUPREME COURT.” POLITICAL ANALYSIS (FORTHCOMING) T H E R E A R E A L S O S O M E OT H E R N OTA B L E R E C E N T E F F O RT S
  14. T H E S O U R C E CO

    D E F O R O U R A LG O PA P E R I S AVA I L A B L E O N
  15. None
  16. W E CA L L O U R A LG

    O PA P E R A ‘ G E N E R A L’ A P P R OAC H
  17. B E CAU S E W E A R E

    N OT I N T E R E ST E D I N A LO CA L LY T U N E D M O D E L B U T R AT H E R A M O D E L T H AT CA N ‘ STA N D T H E T E ST O F T I M E ’
  18. G E N E R A L S COT U

    S P R E D I C T I O N 243,882 28,009 Case Outcomes Justice Votes 1816-2015
  19. W E P R E D I C T *

    N OT * A S I N G L E Y E A R B U T R AT H E R ~ 2 0 0 Y E A R S ( 1 8 1 6 - 2 0 1 5 ) O U T O F S A M P L E
  20. N O W I T I S WO RT H

    N OT I N G T H AT P R E D I C T I O N O R I E N T E D PA P E R S A R E C U R R E N T LY S W I M M I N G AG A I N ST M A I N ST R E A M S O C I A L S C I E N C E ( A N D L AW )
  21. CAU S A L I N F E R E

    N C E I S T H E H A L L M A R K O F M O ST Q UA N T O R I E N T E D L AW + S O C I A L S C I E N C E S C H O L A R S H I P
  22. I T I S B E ST S U I

    T E D TO P O L I CY E VA LUAT I O N ( S U C H A S D O E S T H I S PA RT I C U L A R P O L I CY I N T E RV E N T I O N AC H I E V E I T S STAT E D O B J E C T I V E S )
  23. O R I N STA N C E S W

    H E R E E STA B L I S H I N G L I N K S B E T W E E N CAU S E A N D E F F E C T A R E C R I T I CA L
  24. B U T T H E R E I S

    A N A LT E R N AT I V E PA R A D I G M # P R E D I C T I O N
  25. M AC H I N E L E A R

    N I N G P R E D I C T I V E A N A LY T I C S ‘ I N V E R S E ’ P R O B L E M B -S C H O O L CO M P S C I P H Y S I C S P R E D I C T I O N
  26. Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger &

    Pauline T. Kim, Competing Approaches to Predicting Supreme Court Decision Making, 2 Perspectives on Politics 761 (2004). “the best test of an explanatory theory is its ability to predict future events. To the extent that scholars in both disciplines (social science and law) seek to explain court behavior, they ought to test their theories not only against cases already decided, but against future outcomes as well.”
  27. https://www.computationallegalstudies.com/2017/08/28/legal-analytics-versus-empirical- legal-studies-causal-inference-vs-prediction/ https://www.slideshare.net/Danielkatz/legal-analytics-versus-empirical- legal-studies-or-causal-inference-vs-prediction-redux M O R E O N

    T H AT TO P I C H E R E
  28. None
  29. T H E R E I S G R O

    W I N G I N T E R E ST I N T H E P R E D I C T I O N C E N T R I C A P P R OAC H
  30. “There are two cultures in the use of statistical modeling

    to reach conclusions from data. One assumes that the data are generated by a given stochastic data model. The other uses algorithmic models and treats the data mechanism as unknown …. If our goal as a field is to use data to solve problems, then we need to move away from exclusive dependence on data models and adopt a more diverse set of tools.” Leo Breiman, Statistical modeling: The two cultures (with comments and a rejoinder by the author), 16 Statistical Science 199 (2001) Note: Leo Breiman Invented Random Forests
  31. None
  32. None
  33. None
  34. None
  35. None
  36. Susan Athey, The Impact of Machine Learning on Economics http://www.nber.org/chapters/c14009.pdf

  37. 3 5 5 S C I E N C E

    6 3 2 4 3 F E B R UA R Y 2 0 1 7 S P E C I A L I S S U E O N P R E D I C T I O N
  38. None
  39. T H E T H R E E F O

    R M S O F ( L E G A L ) P R E D I C T I O N
  40. T H E R E A R E O N

    LY T H R E E F O R M S O F P R E D I C T I O N E X P E RT S , C R O W D S , A LG O R I T H M S
  41. P R E D I C T I O N

    I S N OT N E C E S S A R I LY # M L A LO N E B U T R AT H E R S O M E E N S E M B L E O F E X P E RT S , C R O W D S + A LG O R I T H M S
  42. http://www.sciencemag.org/news/ 2017/05/artificial-intelligence-prevails- predicting-supreme-court-decisions Professor Katz noted that in the long

    term …“We believe the blend of experts, crowds, and algorithms is the secret sauce for the whole thing.” May 2nd 2017
  43. None
  44. I N T H I S PA P E R

    W E A R E F O C U S E D U P O N R E V E A L E D E X P E RT S A N D C R O W D S
  45. H T T P S : / / A R

    X I V. O RG / A B S / 1712 . 0 3 84 6 H T T P S : / / PA P E R S . S S R N . C O M / S O L 3 / PA P E R S . C F M ? A B S T R AC T _ I D = 3 0 8 5710
  46. None
  47. THE STRUCTURE FOR TODAY I. INTRODUCTION II. DATA + TOURNAMENT

    OVERVIEW III. CROWD CONSTRUCTION FRAMEWORK IV. MODEL TESTING + RESULTS V. CONCLUSION + FUTURE WORK
  48. DATA + TOURNAMENT OVERVIEW

  49. FA N TA SY S COT U S I S

    A S CO O L A S I T S O U N D S
  50. FA N TA SY S COT U S WA S

    F O U N D E D B Y J O S H B L AC K M A N I N 2 0 0 9
  51. P R I Z E S A N D S

    P O N S O R S H I P H AV E VA R I E D B U T T H E R E H AV E B E E N T E N S O F T H O U S A N D S O F $ $ A N D P R I Z E S D I ST R I B U T E D OV E R T H E Y E A R S
  52. None
  53. None
  54. U S E R S C R E AT E

    A LO G I N A N D ACC E S S T H E S I T E
  55. O N A CA S E B Y CA S

    E B A S I S , U S E R S CA N E N T E R T H E I R R E S P E C T I V E P R E D I C T I O N S
  56. U S E R S A R E F R

    E E TO C H A N G E T H E I R P R E D I C T I O N S U N T I L T H E DAT E O F F I N A L D E C I S I O N
  57. https://fivethirtyeight.com/features/obamacares- chances-of-survival-are-looking-better-and-better/ ( S O M E T I M

    E S I N I N T E R E ST I N G WAY S A S S H O W N A B OV E ) U S E R S A R E F R E E TO C H A N G E T H E I R P R E D I C T I O N S U N T I L T H E DAT E O F F I N A L D E C I S I O N
  58. F O R E AC H CA S E ,

    W E A R E A B L E TO T R AC K TO P E R F O R M A N C E O F P L AY E R S A N D CO M PA R E I T TO T H E O U TCO M E O F T H E CA S E S
  59. A S W E H AV E D O N

    E W I T H O U R L E G A L A . I . P R O D U C T S …
  60. W E W I L L A N O N

    Y M I Z E A N D T H E N O P E N S O U R C E P L AY E R P R E D I C T I O N DATA TO S U P P O RT ACA D E M I C R E S E A R C H I N TO P R E D I C T I O N , C R O W D S O U R C I N G , E TC .
  61. None
  62. S U M M A R Y STAT I ST

    I C S
  63. 6 S COT U S T E R M S

  64. 4 2 5 L I ST E D CA S

    E S 6 S COT U S T E R M S
  65. 7 2 8 4 U N I Q U E

    PA RT I C I PA N T S 4 2 5 L I ST E D CA S E S 6 S COT U S T E R M S
  66. 7 2 8 4 U N I Q U E

    PA RT I C I PA N T S 4 2 5 L I ST E D CA S E S 6 3 6 8 5 9 P R E D I C T I O N S 6 S COT U S T E R M S
  67. W E H AV E A S I G N

    I F I CA N T A M O U N T O F P L AY E R T U R N OV E R WO R K I N G W I T H R E A L DATA ( ~ 3 % O F M A X PA RT I C I PAT I O N )
  68. S O M E T I M E S F

    O L K S C H A N G E T H E I R VOT E S 6 3 6 8 5 9 5 4 5 8 4 5 F I N A L P R E D I C T I O N S OV E R A L L P R E D I C T I O N S WO R K I N G W I T H R E A L DATA
  69. T H E N U M B E R O

    F P L AY E R S H A S D E C L I N E D B U T T H E E N G AG E M E N T R AT E H A S I N C R E A S E D WO R K I N G W I T H R E A L DATA
  70. W E B E L I E V E T

    H I S I S * N OT * R E A L LY S U RV I VO R S H I P B I A S B U T R AT H E R A R E V E L AT I O N M E C H A N I S M ( I . E . YO U ST I C K A R O U N D I F T H I N K YO U A R E G O O D AT T H E U N D E R LY I N G TA S K )
  71. S U M M A R Y DATA

  72. None
  73. CROWD MODELING PRINCIPLES

  74. C R O W D S O U R C

    I N G C R O W D S O U R C I N G D O E S * N OT * R E F E R TO A S P E C I F I C T E C H N I Q U E O R A LG O R I T H M
  75. C R O W D S O U R C

    I N G G E N E R A L LY R E F E R S TO A P R O C E S S O F AG G R E G AT I O N A N D / O R S E G M E N TAT I O N O F I N F O R M AT I O N S I G N A L S
  76. VA R I O U S S I G N

    A L TY P E S T H E I N P U T S I G N A L S CA N A S S U M E M A N Y D I F F E R E N T F O R M S I N C LU D I N G F R O M M O D E L S O R I N D I V I D UA L S O R S E N S O R S ( O R S O M E CA S E S E V E N OT H E R C R O W D S )
  77. CROWD OF INDIVIDUALS The most well know approach involves extracting

    ‘wisdom’ from crowds where crowds are built from individual people
  78. CROWD OF SENSORS Note crowds need not be composed of

    humans but could be networked IT systems Decentralized Distributed Ledgers -or- Oracles -or- IOT sensors with Crowdsourcing Validation #Blockchain #InternetofThings #Crypto
  79. #CRYPTO CROWD Thanks to Team Augur for the Shoutout

  80. CRYPTO CROWDS? For most applications, crowdsourcing could produce *better* performance

    but the biggest challenge is the coordination costs
  81. CRYPTO CROWDS? Sometimes getting even a second opinion in medicine

    (or any professional service) is very challenging — obtaining a 100th opinion is practically impossible
  82. CRYPTO CROWDS? Among other things Crypto Infrastructure (Blockchain) might provide

    the mechanism necessary to lower the coordination costs in crowd constructuion
  83. MORE ON THAT TOPIC HERE BLOCKCHAINLAWCLASS.COM

  84. Random Forest Model Breiman, L.(2001). Random forests. Machine learning, 45(1),

    5-32. Grow a set of differentiated trees through bagging and random substrates (predict using a consensus mechanism) C R O W D O F M O D E L S
  85. C R O W D O F M O D

    E L S Each Poll has a slightly different methodology and historic performance (can be aggregated in various ways )
  86. Poll Aggregation (note not always successful) C R O W

    D O F M O D E L S Each Poll has a slightly different methodology and historic performance (can be aggregated in various ways )
  87. None
  88. A S W E R E V I E W

    E D T H E C R O W D S O U R C I N G L I T E R AT U R E …
  89. W E O B S E RV E D T

    H AT I T WA S D I F F I C U LT TO A P P LY T H E P R I N C I P L E S TO C R O W D S S U C H A S O U R S
  90. C R O W D S O U R C

    I N G I S ‘ U N D I S C I P L I N E D ZO O O F M O D E L S ’ J E S S I CA F L AC K P R O F E S S O R S A N TA F E I N ST I T U T E D E C . 2 7 , 2 0 1 7 ( V I A T W I T T E R )
  91. W E AG R E E … A N D

    T H U S I N T H E PA P E R W E J U ST STA RT E D OV E R …
  92. A N D AT T E M P T E

    D TO B U I L D C R O W D S F R O M F I R ST P R I N C I P L E S …
  93. None
  94. T H R E E M A J O R

    M O D E L I N G P R I N C I P L E S
  95. M U ST CO N F R O N T

    P R O P E R I T E S O F E M P I R I CA L DATA ( 1 )
  96. O U T O F S A M P L

    E M U ST O F F E R O N LY T H E H I STO R I CA L LY AVA I L A B L E I N F O R M AT I O N S E T TO T H E P R E D I C TO R S ( 2 )
  97. A P P LY A N E M P I

    R I CA L M O D E L T H I N K I N G A P P R OAC H M O D E L S PAC E S I N G L E M O D E L ( 3 ) >
  98. None
  99. CROWD CONSTRUCTION FRAMEWORK

  100. W E O U T L I N E A

    G E N E R A L F R A M E WO R K F O R CO N ST R U C T I N G C R O W D S F R O M F I R ST P R I N C I P L E S
  101. I N T H E C L A S S

    I C CO N D O R C E T J U R Y S E T T I N G , M O D E L S TY P I CA L LY U S E P R E D I C T I O N S F R O M A L L PA RT I C I PA N T S
  102. H O W E V E R , M O

    D E L S CA N A L S O TA K E I N TO ACCO U N T I N F O R M AT I O N ( S I G N A L S ) F R O M S O M E S U B S E T O F PA RT I C I PA N T S ( D E F I N E D U S I N G E I T H E R I N C LU S I O N R U L E S O R E XC LU S I O N R U L E S )
  103. None
  104. E X P E R I E N C E

    P E R F O R M A N C E R A N K STAT I ST I CA L T H R E S H O L D I N G W E I G H T I N G C R O W D CO N ST R U C T I O N R U L E S
  105. E X P E R I E N C E

    W E CO U L D I M P O S E A R E Q U I R E M E N T T H AT I N O R D E R TO B E I N C LU D E D I N T H E C R O W D, A P L AY E R M U ST H AV E E X P E R I E N C E - P R E D I C T E D AT L E A ST P CA S E S
  106. R A N K W E CA N R A

    N K T H E PA RT I C I PA N T S W I T H I N T H E OV E R A L L C R O W D O R A C R O W D S U B S E T ACCO R D I N G TO S O M E R A N K F U N C T I O N FOLLOW THE LEADER PLAYER N
  107. P E R F O R M A N C

    E W E CO U L D I M P O S E A R E Q U I R E M E N T T H AT I N O R D E R TO B E I N C LU D E D I N T H E C R O W D - A P L AY E R M U ST H AV E AC H I E V E D S O M E M I N I M U M P E R F O R M A N C E T H R E S H O L D
  108. STAT I ST I CA L T H R E

    S H O L D I N G W E CO U L D I M P O S E A R E Q U I R E M E N T T H AT I N O R D E R TO B E I N C LU D E D I N T H E C R O W D - A P L AY E R M U ST H AV E D E M O ST R AT E D STAT I ST I CA L LY S I G N I F I CA N T P E R F O R M A N C E
  109. W E I G H T I N G I

    N T H E S I M P L E CA S E , W E CA N AV E R AG E . A LT E R N AT I V E LY, W E CA N S E L E C T A W E I G H T I N G S C H E M E W H I C H U P W E I G H T S P R E D I C TO R S B A S E D U P O N S O M E C R I T E R I A .
  110. C R O W D CO N ST R U

    C T I O N R U L E S T H E I N T E R AC T I O N O F T H E S E R U L E S Y I E L D S * N OT * A N I N D I V I D UA L M O D E L B U T R AT H E R A M O D E L S PAC E
  111. T H E M O D E L S PAC

    E M O D E L S PAC E F E AT U R E S 2 7 7 , 2 0 1 P OT E N T I A L M O D E L S
  112. T H E M O D E L S PAC

    E 1 + ( 2 8 · 9 9 · 1 0 0 ) = 2 7 7 2 0 1 F I R ST T E R M I S T H E S I M P L E ST C R O W D S O U R C I N G M O D E L W I T H N O S U B S E T O R W E I G H T I N G R U L E S S E CO N D T E R M CO R R E S P O N D S TO 2 8 M O D E L S ( CO M B I N AT I O N O F P E R F O R M A N C E T H R E S H O L D / W E I G H T I N G R U L E S ) F O R E AC H CO M B I N AT I O N O F 9 9 R A N K A N D 1 0 0 E X P E R I E N C E T H R E S H O L D S
  113. None
  114. MODEL TESTING & RESULTS

  115. W E S I M U L AT E T

    H E P E R F O R M A N C E O F * E AC H * O F T H E 2 7 7 , 2 0 1 P OT E N T I A L C R O W D M O D E L S M O D E L ( S ) ACC U R ACY
  116. A LT H O U G H I T I

    S A L A R G E M O D E L S PAC E W E D O H I G H L I G H T T H E P E R F O R M A N C E O F F O U R E X A M P L E M O D E L S ( A N D T H E N U L L M O D E L )
  117. B A S E L I N E A LWAY

    S G U E S S R E V E R S E N U L L M O D E L
  118. M O D E L 1 A L L C

    R O W D S I M P L E M A J O R I TY
  119. M O D E L 2 F O L LO

    W T H E L E A D E R W I T H N O T H R E S H O L D I N G
  120. M O D E L 3 F O L LO

    W T H E L E A D E R W I T H E X P E R I E N C E T H R E S H O L D I N G ( X P = 5 )
  121. M O D E L 4 M A X I

    M U M ACC U R ACY ( T H E R E A R E AC T UA L LY S E V E R A L M O D E L CO N F I G U R AT I O N S W H I C H O F F E R R O U G H LY E Q U I VA L E N T P E R F O R M A N C E ) E X P E R I E N C E T H R E S H O L D O F 5 C R O W D S I Z E I S CA P P E D AT 2 2 E X P O N E N T I A L W E I G H T W I T H A L P H A O F 0 . 1
  122. CASE LEVEL CUMULATIVE ACCURACY

  123. JUSTICE LEVEL CUMULATIVE ACCURACY

  124. DISTRIBUTION OF JUSTICE LEVEL MODEL ACCURACY

  125. None
  126. W E S I M U L AT E T

    H E P E R F O R M A N C E O F * E AC H * O F T H E 2 7 7 , 2 0 1 P OT E N T I A L C R O W D M O D E L S R O B U ST N E S S O F P E R F O R M A N C E
  127. ROBUSTNESS VISUALIZED T H I S I S A L

    L R E L AT I V E TO T H E N U L L M O D E L ( O F A LWAY S G U E S S R E V E R S E )
  128. ROBUSTNESS VISUALIZED T H E CO N TO U R

    P LOT F L AT T E N S T H E D I M E N S I O N A L I TY O F T H E S PAC E ( E AC H C E L L I S T H E AV E R AG E M O D E L P E R F O R M A N C E OV E R A L L OT H E R M O D E L PA R A M E T E R AT E AC H E X P E R I E N C E , R A N K CO M B O )
  129. ROBUSTNESS VISUALIZED J U ST I C E L E

    V E L CA S E L E V E L
  130. None
  131. CONCLUSION + FUTURE WORK

  132. T H I S PA P E R O F

    F E R S CO N T R I B U T I O N S TO T H E G E N E R A L S C I E N T I F I C L I T E R AT U R E
  133. I N OT H E R WO R D S

    , I T O F F E R A F R A M E WO R K F O R C R O W D S O U R C I N G G E N E R A L LY W I T H # S COT U S A S A N A P P L I E D E X A M P L E
  134. T H E R E A R E A L

    S O M A N Y F O L LO W O N PA P E R S T H AT M I G H T B E G E N E R AT E D
  135. I N C LU D I N G B L

    E N D I N G O U R A LG O W I T H T H E C R O W D I N A N E N S E M B L E ( M E TA ) M O D E L
  136. crowd forecast learning problem is to discover how to blend

    streams of intelligence algorithm forecast ensemble method ENSEMBLE MODEL we can use machine learning methods and metadata such as case topic, lower court as well as crowd metadata to ‘learn’ the conditional weights to apply to the input signals
  137. W H I C H A M O N G

    OT H E R T H I N G S L I N K S TO O U R I N T E R E ST # A B N O R M A L R E T U R N S A N D J U D I C I A L D E C I S I O N S https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
  138. AG A I N W E W I L L

    A N O N Y M I Z E A N D T H E N O P E N S O U R C E P L AY E R P R E D I C T I O N DATA TO S U P P O RT ACA D E M I C R E S E A R C H I N TO P R E D I C T I O N , C R O W D S O U R C I N G , E TC .
  139. None
  140. DANIEL MARTIN KATZ E D U | I L L

    I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | DA N I E L M A R T I N K AT Z . C O M C O R P | L E X P R E D I C T. C O M MICHAEL BOMMARITO E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M PAG E | B O M M A R I TO L LC . C O M C O R P | L E X P R E D I C T. C O M JOSH BLACKMAN E D U | S O U T H T E X A S C O L L E G E O F L AW H O U S TO N B LO G | J O S H B L AC K M A N . C O M PAG E | J O S H B L AC K M A N . C O M / B LO G C O R P | L E X P R E D I C T. C O M
  141. H T T P S : / / A R

    X I V. O RG / A B S / 1712 . 0 3 84 6 H T T P S : / / PA P E R S . S S R N . C O M / S O L 3 / PA P E R S . C F M ? A B S T R AC T _ I D = 3 0 8 5710
  142. C R O W D S O U R C

    I N G ACC U R AT E LY A N D R O B U ST LY P R E D I C T S S U P R E M E CO U RT D E C I S I O N S DA N I E L M A RT I N KATZ | I L L I N O I S T E C H + STA N F O R D CO D E X M I C H A E L B O M M A R I TO | I L L I N O I S T E C H + STA N F O R D CO D E X J O S H B L AC K M A N | S O U T H T E X A S CO L L E G E O F L AW 0.4 0.5 0.6 0.7 0.8 0.9 2012 2013 2014 2015 2016 2017 Date of Decison Case Level Cumulative Accuracy By Model Type Model Type Model 1 Model 2 Model 3 Model 4 Null Model