$30 off During Our Annual Pro Sale. View Details »

Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman

Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman

Overview Deck of Crowdsourcing Accurately and Robustly Predicts Supreme Court Decisions - Professors Daniel Martin Katz, Michael Bommarito & Josh Blackman - v 1.01

Daniel Martin Katz
PRO

January 17, 2018
Tweet

More Decks by Daniel Martin Katz

Other Decks in Science

Transcript

  1. C R O W D S O U R C I N G ACC U R AT E LY
    A N D R O B U ST LY P R E D I C T S
    S U P R E M E CO U RT D E C I S I O N S
    DA N I E L M A RT I N KATZ | I L L I N O I S T E C H + STA N F O R D CO D E X
    M I C H A E L B O M M A R I TO | I L L I N O I S T E C H + STA N F O R D CO D E X
    J O S H B L AC K M A N | S O U T H T E X A S CO L L E G E O F L AW
    0.4
    0.5
    0.6
    0.7
    0.8
    0.9
    2012 2013 2014 2015 2016 2017
    Date of Decison
    Case Level Cumulative Accuracy By Model Type
    Model Type Model 1 Model 2 Model 3 Model 4 Null Model

    View Slide

  2. DANIEL MARTIN KATZ
    E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X
    B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M
    PAG E | DA N I E L M A R T I N K AT Z . C O M
    C O R P | L E X P R E D I C T. C O M
    MICHAEL BOMMARITO
    E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X
    B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M
    PAG E | B O M M A R I TO L LC . C O M
    C O R P | L E X P R E D I C T. C O M
    JOSH BLACKMAN
    E D U | S O U T H T E X A S C O L L E G E O F L AW H O U S TO N
    B LO G | J O S H B L AC K M A N . C O M
    PAG E | J O S H B L AC K M A N . C O M / B LO G
    C O R P | L E X P R E D I C T. C O M

    View Slide

  3. H T T P S : / / A R X I V. O RG / A B S / 1712 . 0 3 84 6
    H T T P S : / / PA P E R S . S S R N . C O M / S O L 3 /
    PA P E R S . C F M ? A B S T R AC T _ I D = 3 0 8 5710

    View Slide

  4. O U R PA P E R I S A B O U T
    P R E D I C T I N G T H E D E C I S I O N S
    O F T H E S U P R E M E CO U RT O F
    T H E U N I T E D STAT E S
    #SCOTUS

    View Slide

  5. JUDICIAL PREDICTION
    IS THE LAW + POLITICS
    GRAIL QUEST

    View Slide

  6. YOU COULD START
    WITH HOLMES AND
    THE LEGAL REALISTS
    BUT THESE WERE
    *NOT* REALLY
    SCIENTIFIC EFFORTS

    View Slide

  7. FRED KORT, PREDICTING SUPREME
    COURT DECISIONS MATHEMATICALLY:
    A QUANTITATIVE ANALYSIS OF THE
    “RIGHT TO COUNSEL” CASES, 51
    AMER. POL. SCI. REV. 1 (1957).
    1957
    S. SIDNEY ULMER, QUANTITATIVE
    ANALYSIS OF JUDICIAL PROCESSES:
    SOME PRACTICAL AND THEORETICAL
    APPLICATIONS, 28 LAW & CONTEMP.
    PROBS. 164 (1963).
    1963
    A CO U P L E O F E A R LY E F F O RT S

    View Slide

  8. COMPUTERWORLD
    JULY 1971
    PROGRAM WRITTEN IN FORTRAN
    (THE 91% PREDICTION MARK WAS
    LIMITED TO CERTAIN CASES)
    HAROLD SPAETH

    View Slide

  9. JEFFREY A. SEGAL, PREDICTING
    S U P R E M E C O U R T C A S E S
    PROBABILISTICALLY: THE SEARCH
    AND SEIZURE CASES, 1962-1981,
    78 AMERICAN POLITICAL SCIENCE
    REVIEW 891 (1984)
    1984
    A N I M P O RTA N T
    L AT E R E F F O RT

    View Slide

  10. Columbia Law Review (2004)
    Theodore W. Ruger, Pauline T. Kim,
    Andrew D. Martin, Kevin M. Quinn
    Legal and Political Science Approaches
    to Predicting Supreme Court Decision
    Making
    The Supreme Court
    Forecasting Project:
    B U T T H I S WA S T H E PA P E R
    T H AT I N S P I R E D O U R E F F O RT S
    2004

    View Slide

  11. http://journals.plos.org/plosone/article?
    id=10.1371/journal.pone.0174698
    available at
    RESEARCH ARTICLE
    A general approach for predicting the
    behavior of the Supreme Court of the United
    States
    Daniel Martin Katz1,2*, Michael J. Bommarito II1,2, Josh Blackman3
    1 Illinois Tech - Chicago-Kent College of Law, Chicago, IL, United States of America, 2 CodeX - The Stanford
    Center for Legal Informatics, Stanford, CA, United States of America, 3 South Texas College of Law Houston,
    Houston, TX, United States of America
    * [email protected]
    Abstract
    Building on developments in machine learning and prior work in the science of judicial pre-
    diction, we construct a model designed to predict the behavior of the Supreme Court of the
    United States in a generalized, out-of-sample context. To do so, we develop a time-evolving
    random forest classifier that leverages unique feature engineering to predict more than
    240,000 justice votes and 28,000 cases outcomes over nearly two centuries (1816-2015).
    Using only data available prior to decision, our model outperforms null (baseline) models at
    both the justice and case level under both parametric and non-parametric tests. Over nearly
    two centuries, we achieve 70.2% accuracy at the case outcome level and 71.9% at the jus-
    tice vote level. More recently, over the past century, we outperform an in-sample optimized
    null model by nearly 5%. Our performance is consistent with, and improves on the general
    level of prediction demonstrated by prior work; however, our model is distinctive because it
    can be applied out-of-sample to the entire past and future of the Court, not a single term.
    Our results represent an important advance for the science of quantitative legal prediction
    and portend a range of other potential applications.
    Introduction
    As the leaves begin to fall each October, the first Monday marks the beginning of another term
    for the Supreme Court of the United States. Each term brings with it a series of challenging,
    important cases that cover legal questions as diverse as tax law, freedom of speech, patent law,
    administrative law, equal protection, and environmental law. In many instances, the Court’s
    decisions are meaningful not just for the litigants per se, but for society as a whole.
    Unsurprisingly, predicting the behavior of the Court is one of the great pastimes for legal
    and political observers. Every year, newspapers, television and radio pundits, academic jour-
    nals, law reviews, magazines, blogs, and tweets predict how the Court will rule in a particular
    case. Will the Justices vote based on the political preferences of the President who appointed
    them or form a coalition along other dimensions? Will the Court counter expectations with an
    unexpected ruling?
    PLOS ONE | https://doi.org/10.1371/journal.pone.0174698 April 12, 2017 1 / 18
    a1111111111
    a1111111111
    a1111111111
    a1111111111
    a1111111111
    OPEN ACCESS
    Citation: Katz DM, Bommarito MJ, II, Blackman J
    (2017) A general approach for predicting the
    behavior of the Supreme Court of the United
    States. PLoS ONE 12(4): e0174698. https://doi.
    org/10.1371/journal.pone.0174698
    Editor: Luı
    ´s A. Nunes Amaral, Northwestern
    University, UNITED STATES
    Received: January 17, 2017
    Accepted: March 13, 2017
    Published: April 12, 2017
    Copyright: © 2017 Katz et al. This is an open
    access article distributed under the terms of the
    Creative Commons Attribution License, which
    permits unrestricted use, distribution, and
    reproduction in any medium, provided the original
    author and source are credited.
    Data Availability Statement: Data and replication
    code are available on Github at the following URL:
    https://github.com/mjbommar/scotus-predict-v2/.
    Funding: The author(s) received no specific
    funding for this work.
    Competing interests: All Authors are Members of
    a LexPredict, LLC which provides consulting
    services to various legal industry stakeholders. We
    received no financial contributions from LexPredict
    or anyone else for this paper. This does not alter
    our adherence to PLOS ONE policies on sharing
    data and materials.
    DANIEL MARTIN KATZ, MICHAEL J.
    BOMMARITO II & JOSH BLACKMAN,
    A GENERAL APPROACH FOR
    PREDICTING THE BEHAVIOR OF THE
    SUPREME COURT OF THE UNITED
    STATES, PLOS ONE 12.4 (2017):
    E0174698.
    APRIL
    2017
    T H I S I S O U R A LG O PA P E R

    View Slide

  12. JULY
    2017
    AARON KAUFMAN, PETER
    KRAFT, AND MAYA SEN.
    “IMPROVING SUPREME
    COURT FORECASTING
    USING BOOSTED
    DECISION TREES”.
    HTTP://J.MP/2NRJTO6
    T H E R E A R E A L S O S O M E OT H E R
    N OTA B L E R E C E N T E F F O RT S

    View Slide

  13. DECEMBER
    2017
    DIETRICH, BRYCE J.,
    RYAN D. ENOS, AND
    MAYA SEN.. “EMOTIONAL
    AROUSAL PREDICTS
    VOTING ON THE U.S.
    SUPREME COURT.”
    POLITICAL ANALYSIS
    (FORTHCOMING)
    T H E R E A R E A L S O S O M E OT H E R
    N OTA B L E R E C E N T E F F O RT S

    View Slide

  14. T H E S O U R C E CO D E F O R
    O U R A LG O PA P E R I S
    AVA I L A B L E O N

    View Slide

  15. View Slide

  16. W E CA L L O U R
    A LG O PA P E R A
    ‘ G E N E R A L’
    A P P R OAC H

    View Slide

  17. B E CAU S E W E A R E
    N OT I N T E R E ST E D
    I N A LO CA L LY
    T U N E D M O D E L
    B U T R AT H E R A
    M O D E L T H AT CA N
    ‘ STA N D T H E T E ST
    O F T I M E ’

    View Slide

  18. G E N E R A L S COT U S
    P R E D I C T I O N
    243,882
    28,009
    Case Outcomes
    Justice Votes
    1816-2015

    View Slide

  19. W E P R E D I C T
    * N OT * A S I N G L E
    Y E A R B U T R AT H E R
    ~ 2 0 0 Y E A R S
    ( 1 8 1 6 - 2 0 1 5 )
    O U T O F S A M P L E

    View Slide

  20. N O W I T I S WO RT H N OT I N G
    T H AT P R E D I C T I O N
    O R I E N T E D PA P E R S A R E
    C U R R E N T LY S W I M M I N G
    AG A I N ST M A I N ST R E A M
    S O C I A L S C I E N C E ( A N D L AW )

    View Slide

  21. CAU S A L I N F E R E N C E
    I S T H E H A L L M A R K O F
    M O ST Q UA N T O R I E N T E D
    L AW + S O C I A L S C I E N C E
    S C H O L A R S H I P

    View Slide

  22. I T I S B E ST S U I T E D TO
    P O L I CY E VA LUAT I O N
    ( S U C H A S D O E S T H I S
    PA RT I C U L A R P O L I CY
    I N T E RV E N T I O N AC H I E V E
    I T S STAT E D O B J E C T I V E S )

    View Slide

  23. O R I N STA N C E S W H E R E
    E STA B L I S H I N G L I N K S
    B E T W E E N CAU S E A N D
    E F F E C T A R E C R I T I CA L

    View Slide

  24. B U T T H E R E I S A N
    A LT E R N AT I V E
    PA R A D I G M
    # P R E D I C T I O N

    View Slide

  25. M AC H I N E L E A R N I N G
    P R E D I C T I V E A N A LY T I C S
    ‘ I N V E R S E ’ P R O B L E M
    B -S C H O O L
    CO M P S C I
    P H Y S I C S
    P R E D I C T I O N

    View Slide

  26. Andrew D. Martin, Kevin M. Quinn, Theodore W. Ruger & Pauline T.
    Kim, Competing Approaches to Predicting Supreme Court Decision
    Making, 2 Perspectives on Politics 761 (2004).
    “the best test of an explanatory theory is its
    ability to predict future events. To the extent
    that scholars in both disciplines (social
    science and law) seek to explain court
    behavior, they ought to test their theories
    not only against cases already decided, but
    against future outcomes as well.”

    View Slide

  27. https://www.computationallegalstudies.com/2017/08/28/legal-analytics-versus-empirical-
    legal-studies-causal-inference-vs-prediction/
    https://www.slideshare.net/Danielkatz/legal-analytics-versus-empirical-
    legal-studies-or-causal-inference-vs-prediction-redux
    M O R E O N T H AT
    TO P I C H E R E

    View Slide

  28. View Slide

  29. T H E R E I S
    G R O W I N G
    I N T E R E ST I N T H E
    P R E D I C T I O N
    C E N T R I C
    A P P R OAC H

    View Slide

  30. “There are two cultures in the use of
    statistical modeling to reach conclusions
    from data. One assumes that the data
    are generated by a given stochastic data
    model. The other uses algorithmic
    models and treats the data mechanism
    as unknown …. If our goal as a field is
    to use data to solve problems, then we
    need to move away from exclusive
    dependence on data models and adopt
    a more diverse set of tools.”
    Leo Breiman, Statistical modeling:
    The two cultures (with comments
    and a rejoinder by the author), 16
    Statistical Science 199 (2001)
    Note: Leo Breiman Invented
    Random Forests

    View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide

  36. Susan Athey, The Impact of Machine Learning on Economics
    http://www.nber.org/chapters/c14009.pdf

    View Slide

  37. 3 5 5 S C I E N C E 6 3 2 4
    3 F E B R UA R Y 2 0 1 7
    S P E C I A L I S S U E
    O N P R E D I C T I O N

    View Slide

  38. View Slide

  39. T H E T H R E E F O R M S
    O F ( L E G A L )
    P R E D I C T I O N

    View Slide

  40. T H E R E A R E O N LY
    T H R E E F O R M S O F
    P R E D I C T I O N
    E X P E RT S ,
    C R O W D S ,
    A LG O R I T H M S

    View Slide

  41. P R E D I C T I O N I S
    N OT N E C E S S A R I LY
    # M L A LO N E B U T
    R AT H E R S O M E
    E N S E M B L E O F
    E X P E RT S ,
    C R O W D S +
    A LG O R I T H M S

    View Slide

  42. http://www.sciencemag.org/news/
    2017/05/artificial-intelligence-prevails-
    predicting-supreme-court-decisions
    Professor Katz noted
    that in the long term
    …“We believe the
    blend of experts,
    crowds, and algorithms
    is the secret sauce for
    the whole thing.”
    May 2nd 2017

    View Slide

  43. View Slide

  44. I N T H I S PA P E R W E
    A R E F O C U S E D
    U P O N R E V E A L E D
    E X P E RT S A N D
    C R O W D S

    View Slide

  45. H T T P S : / / A R X I V. O RG / A B S / 1712 . 0 3 84 6
    H T T P S : / / PA P E R S . S S R N . C O M / S O L 3 /
    PA P E R S . C F M ? A B S T R AC T _ I D = 3 0 8 5710

    View Slide

  46. View Slide

  47. THE STRUCTURE FOR TODAY
    I. INTRODUCTION
    II. DATA + TOURNAMENT OVERVIEW
    III. CROWD CONSTRUCTION FRAMEWORK
    IV. MODEL TESTING + RESULTS
    V. CONCLUSION + FUTURE WORK

    View Slide

  48. DATA
    +
    TOURNAMENT
    OVERVIEW

    View Slide

  49. FA N TA SY S COT U S I S A S
    CO O L A S I T S O U N D S

    View Slide

  50. FA N TA SY S COT U S WA S F O U N D E D
    B Y J O S H B L AC K M A N I N 2 0 0 9

    View Slide

  51. P R I Z E S A N D
    S P O N S O R S H I P H AV E
    VA R I E D B U T T H E R E
    H AV E B E E N T E N S O F
    T H O U S A N D S O F $ $ A N D
    P R I Z E S D I ST R I B U T E D
    OV E R T H E Y E A R S

    View Slide

  52. View Slide

  53. View Slide

  54. U S E R S
    C R E AT E
    A LO G I N
    A N D
    ACC E S S
    T H E S I T E

    View Slide

  55. O N A CA S E B Y CA S E B A S I S ,
    U S E R S CA N E N T E R T H E I R
    R E S P E C T I V E P R E D I C T I O N S

    View Slide

  56. U S E R S A R E F R E E
    TO C H A N G E T H E I R
    P R E D I C T I O N S
    U N T I L T H E DAT E O F
    F I N A L D E C I S I O N

    View Slide

  57. https://fivethirtyeight.com/features/obamacares-
    chances-of-survival-are-looking-better-and-better/
    ( S O M E T I M E S I N
    I N T E R E ST I N G WAY S
    A S S H O W N A B OV E )
    U S E R S A R E F R E E
    TO C H A N G E T H E I R
    P R E D I C T I O N S
    U N T I L T H E DAT E O F
    F I N A L D E C I S I O N

    View Slide

  58. F O R E AC H CA S E , W E
    A R E A B L E TO T R AC K
    TO P E R F O R M A N C E
    O F P L AY E R S A N D
    CO M PA R E I T TO
    T H E O U TCO M E O F
    T H E CA S E S

    View Slide

  59. A S W E H AV E D O N E W I T H
    O U R L E G A L A . I . P R O D U C T S …

    View Slide

  60. W E W I L L A N O N Y M I Z E
    A N D T H E N O P E N S O U R C E
    P L AY E R P R E D I C T I O N
    DATA TO S U P P O RT
    ACA D E M I C R E S E A R C H
    I N TO P R E D I C T I O N ,
    C R O W D S O U R C I N G , E TC .

    View Slide

  61. View Slide

  62. S U M M A R Y
    STAT I ST I C S

    View Slide

  63. 6 S COT U S T E R M S

    View Slide

  64. 4 2 5 L I ST E D CA S E S
    6 S COT U S T E R M S

    View Slide

  65. 7 2 8 4 U N I Q U E PA RT I C I PA N T S
    4 2 5 L I ST E D CA S E S
    6 S COT U S T E R M S

    View Slide

  66. 7 2 8 4 U N I Q U E PA RT I C I PA N T S
    4 2 5 L I ST E D CA S E S
    6 3 6 8 5 9 P R E D I C T I O N S
    6 S COT U S T E R M S

    View Slide

  67. W E H AV E A
    S I G N I F I CA N T A M O U N T
    O F P L AY E R T U R N OV E R
    WO R K I N G W I T H
    R E A L DATA
    ( ~ 3 % O F M A X PA RT I C I PAT I O N )

    View Slide

  68. S O M E T I M E S F O L K S
    C H A N G E T H E I R VOT E S
    6 3 6 8 5 9
    5 4 5 8 4 5 F I N A L P R E D I C T I O N S
    OV E R A L L P R E D I C T I O N S
    WO R K I N G W I T H
    R E A L DATA

    View Slide

  69. T H E N U M B E R O F
    P L AY E R S H A S D E C L I N E D
    B U T T H E E N G AG E M E N T
    R AT E H A S I N C R E A S E D
    WO R K I N G W I T H
    R E A L DATA

    View Slide

  70. W E B E L I E V E T H I S I S * N OT *
    R E A L LY S U RV I VO R S H I P
    B I A S B U T R AT H E R A
    R E V E L AT I O N M E C H A N I S M
    ( I . E . YO U ST I C K A R O U N D I F
    T H I N K YO U A R E G O O D AT T H E
    U N D E R LY I N G TA S K )

    View Slide

  71. S U M M A R Y DATA

    View Slide

  72. View Slide

  73. CROWD
    MODELING
    PRINCIPLES

    View Slide

  74. C R O W D S O U R C I N G
    C R O W D S O U R C I N G D O E S
    * N OT * R E F E R TO A S P E C I F I C
    T E C H N I Q U E O R A LG O R I T H M

    View Slide

  75. C R O W D S O U R C I N G
    G E N E R A L LY R E F E R S TO A
    P R O C E S S O F AG G R E G AT I O N
    A N D / O R S E G M E N TAT I O N O F
    I N F O R M AT I O N S I G N A L S

    View Slide

  76. VA R I O U S S I G N A L TY P E S
    T H E I N P U T S I G N A L S CA N A S S U M E
    M A N Y D I F F E R E N T F O R M S
    I N C LU D I N G F R O M M O D E L S O R
    I N D I V I D UA L S O R S E N S O R S
    ( O R S O M E CA S E S E V E N OT H E R C R O W D S )

    View Slide

  77. CROWD OF INDIVIDUALS
    The most well know approach involves
    extracting ‘wisdom’ from crowds where crowds
    are built from individual people

    View Slide

  78. CROWD OF SENSORS
    Note crowds need not be composed of humans
    but could be networked IT systems
    Decentralized Distributed Ledgers -or- Oracles
    -or- IOT sensors with Crowdsourcing Validation
    #Blockchain
    #InternetofThings
    #Crypto

    View Slide

  79. #CRYPTO CROWD
    Thanks to Team Augur for the Shoutout

    View Slide

  80. CRYPTO CROWDS?
    For most applications,
    crowdsourcing could
    produce *better*
    performance but the
    biggest challenge is
    the coordination costs

    View Slide

  81. CRYPTO CROWDS?
    Sometimes getting even a
    second opinion in medicine
    (or any professional service)
    is very challenging —
    obtaining a 100th opinion is
    practically impossible

    View Slide

  82. CRYPTO CROWDS?
    Among other things Crypto
    Infrastructure (Blockchain)
    might provide the
    mechanism necessary to
    lower the coordination costs
    in crowd constructuion

    View Slide

  83. MORE
    ON THAT
    TOPIC
    HERE
    BLOCKCHAINLAWCLASS.COM

    View Slide

  84. Random Forest Model
    Breiman, L.(2001). Random forests.
    Machine learning, 45(1), 5-32.
    Grow a set of differentiated trees
    through bagging and random
    substrates (predict using a consensus
    mechanism)
    C R O W D O F M O D E L S

    View Slide

  85. C R O W D O F
    M O D E L S
    Each Poll has a slightly different
    methodology and historic performance
    (can be aggregated in various ways )

    View Slide

  86. Poll Aggregation
    (note not always successful)
    C R O W D O F
    M O D E L S
    Each Poll has a slightly different
    methodology and historic performance
    (can be aggregated in various ways )

    View Slide

  87. View Slide

  88. A S W E R E V I E W E D T H E
    C R O W D S O U R C I N G
    L I T E R AT U R E …

    View Slide

  89. W E O B S E RV E D T H AT I T
    WA S D I F F I C U LT TO A P P LY
    T H E P R I N C I P L E S TO
    C R O W D S S U C H A S O U R S

    View Slide

  90. C R O W D
    S O U R C I N G I S
    ‘ U N D I S C I P L I N E D
    ZO O O F M O D E L S ’
    J E S S I CA F L AC K
    P R O F E S S O R
    S A N TA F E I N ST I T U T E
    D E C . 2 7 , 2 0 1 7
    ( V I A T W I T T E R )

    View Slide

  91. W E AG R E E …
    A N D T H U S I N T H E PA P E R
    W E J U ST STA RT E D OV E R …

    View Slide

  92. A N D AT T E M P T E D TO
    B U I L D C R O W D S F R O M
    F I R ST P R I N C I P L E S …

    View Slide

  93. View Slide

  94. T H R E E M A J O R
    M O D E L I N G P R I N C I P L E S

    View Slide

  95. M U ST CO N F R O N T
    P R O P E R I T E S O F
    E M P I R I CA L DATA
    ( 1 )

    View Slide

  96. O U T O F S A M P L E
    M U ST O F F E R O N LY T H E
    H I STO R I CA L LY AVA I L A B L E
    I N F O R M AT I O N S E T TO
    T H E P R E D I C TO R S
    ( 2 )

    View Slide

  97. A P P LY A N
    E M P I R I CA L
    M O D E L T H I N K I N G
    A P P R OAC H
    M O D E L
    S PAC E
    S I N G L E
    M O D E L
    ( 3 )
    >

    View Slide

  98. View Slide

  99. CROWD
    CONSTRUCTION
    FRAMEWORK

    View Slide

  100. W E O U T L I N E A
    G E N E R A L F R A M E WO R K
    F O R CO N ST R U C T I N G
    C R O W D S F R O M F I R ST
    P R I N C I P L E S

    View Slide

  101. I N T H E C L A S S I C
    CO N D O R C E T J U R Y
    S E T T I N G , M O D E L S
    TY P I CA L LY U S E
    P R E D I C T I O N S F R O M
    A L L PA RT I C I PA N T S

    View Slide

  102. H O W E V E R , M O D E L S CA N
    A L S O TA K E I N TO ACCO U N T
    I N F O R M AT I O N ( S I G N A L S )
    F R O M S O M E S U B S E T O F
    PA RT I C I PA N T S
    ( D E F I N E D U S I N G E I T H E R I N C LU S I O N
    R U L E S O R E XC LU S I O N R U L E S )

    View Slide

  103. View Slide

  104. E X P E R I E N C E
    P E R F O R M A N C E
    R A N K
    STAT I ST I CA L T H R E S H O L D I N G
    W E I G H T I N G
    C R O W D CO N ST R U C T I O N R U L E S

    View Slide

  105. E X P E R I E N C E
    W E CO U L D I M P O S E A
    R E Q U I R E M E N T T H AT I N O R D E R
    TO B E I N C LU D E D I N T H E
    C R O W D, A P L AY E R M U ST H AV E
    E X P E R I E N C E - P R E D I C T E D AT
    L E A ST P CA S E S

    View Slide

  106. R A N K
    W E CA N R A N K T H E PA RT I C I PA N T S
    W I T H I N T H E OV E R A L L C R O W D O R
    A C R O W D S U B S E T ACCO R D I N G
    TO S O M E R A N K F U N C T I O N
    FOLLOW THE LEADER PLAYER N

    View Slide

  107. P E R F O R M A N C E
    W E CO U L D I M P O S E A
    R E Q U I R E M E N T T H AT I N O R D E R
    TO B E I N C LU D E D I N T H E
    C R O W D - A P L AY E R M U ST H AV E
    AC H I E V E D S O M E M I N I M U M
    P E R F O R M A N C E T H R E S H O L D

    View Slide

  108. STAT I ST I CA L T H R E S H O L D I N G
    W E CO U L D I M P O S E A
    R E Q U I R E M E N T T H AT I N O R D E R
    TO B E I N C LU D E D I N T H E
    C R O W D - A P L AY E R M U ST H AV E
    D E M O ST R AT E D STAT I ST I CA L LY
    S I G N I F I CA N T P E R F O R M A N C E

    View Slide

  109. W E I G H T I N G
    I N T H E S I M P L E CA S E , W E CA N
    AV E R AG E . A LT E R N AT I V E LY, W E CA N
    S E L E C T A W E I G H T I N G S C H E M E
    W H I C H U P W E I G H T S P R E D I C TO R S
    B A S E D U P O N S O M E C R I T E R I A .

    View Slide

  110. C R O W D CO N ST R U C T I O N R U L E S
    T H E I N T E R AC T I O N O F T H E S E
    R U L E S Y I E L D S * N OT * A N
    I N D I V I D UA L M O D E L B U T
    R AT H E R A M O D E L S PAC E

    View Slide

  111. T H E M O D E L S PAC E
    M O D E L S PAC E
    F E AT U R E S 2 7 7 , 2 0 1
    P OT E N T I A L M O D E L S

    View Slide

  112. T H E M O D E L S PAC E
    1 + ( 2 8 · 9 9 · 1 0 0 ) = 2 7 7 2 0 1
    F I R ST T E R M I S T H E S I M P L E ST
    C R O W D S O U R C I N G M O D E L W I T H N O S U B S E T
    O R W E I G H T I N G R U L E S
    S E CO N D T E R M CO R R E S P O N D S TO 2 8 M O D E L S
    ( CO M B I N AT I O N O F P E R F O R M A N C E
    T H R E S H O L D / W E I G H T I N G R U L E S )
    F O R E AC H CO M B I N AT I O N O F 9 9 R A N K A N D
    1 0 0 E X P E R I E N C E T H R E S H O L D S

    View Slide

  113. View Slide

  114. MODEL TESTING
    & RESULTS

    View Slide

  115. W E S I M U L AT E T H E
    P E R F O R M A N C E O F
    * E AC H * O F T H E
    2 7 7 , 2 0 1 P OT E N T I A L
    C R O W D M O D E L S
    M O D E L ( S )
    ACC U R ACY

    View Slide

  116. A LT H O U G H I T I S A
    L A R G E M O D E L S PAC E
    W E D O H I G H L I G H T T H E
    P E R F O R M A N C E O F F O U R
    E X A M P L E M O D E L S
    ( A N D T H E N U L L M O D E L )

    View Slide

  117. B A S E L I N E
    A LWAY S
    G U E S S
    R E V E R S E
    N U L L
    M O D E L

    View Slide

  118. M O D E L 1
    A L L
    C R O W D
    S I M P L E
    M A J O R I TY

    View Slide

  119. M O D E L 2
    F O L LO W T H E
    L E A D E R
    W I T H N O
    T H R E S H O L D I N G

    View Slide

  120. M O D E L 3
    F O L LO W T H E
    L E A D E R
    W I T H
    E X P E R I E N C E
    T H R E S H O L D I N G
    ( X P = 5 )

    View Slide

  121. M O D E L 4
    M A X I M U M
    ACC U R ACY
    ( T H E R E A R E AC T UA L LY S E V E R A L M O D E L
    CO N F I G U R AT I O N S W H I C H O F F E R R O U G H LY
    E Q U I VA L E N T P E R F O R M A N C E )
    E X P E R I E N C E T H R E S H O L D O F 5
    C R O W D S I Z E I S CA P P E D AT 2 2
    E X P O N E N T I A L W E I G H T W I T H A L P H A O F 0 . 1

    View Slide

  122. CASE
    LEVEL
    CUMULATIVE
    ACCURACY

    View Slide

  123. JUSTICE
    LEVEL
    CUMULATIVE
    ACCURACY

    View Slide

  124. DISTRIBUTION
    OF JUSTICE
    LEVEL MODEL
    ACCURACY

    View Slide

  125. View Slide

  126. W E S I M U L AT E T H E
    P E R F O R M A N C E O F
    * E AC H * O F T H E
    2 7 7 , 2 0 1 P OT E N T I A L
    C R O W D M O D E L S
    R O B U ST N E S S
    O F P E R F O R M A N C E

    View Slide

  127. ROBUSTNESS VISUALIZED
    T H I S I S A L L R E L AT I V E
    TO T H E N U L L M O D E L
    ( O F A LWAY S G U E S S
    R E V E R S E )

    View Slide

  128. ROBUSTNESS VISUALIZED
    T H E CO N TO U R P LOT
    F L AT T E N S T H E
    D I M E N S I O N A L I TY
    O F T H E S PAC E
    ( E AC H C E L L I S T H E AV E R AG E
    M O D E L P E R F O R M A N C E OV E R
    A L L OT H E R M O D E L
    PA R A M E T E R AT E AC H
    E X P E R I E N C E , R A N K CO M B O )

    View Slide

  129. ROBUSTNESS VISUALIZED
    J U ST I C E L E V E L
    CA S E L E V E L

    View Slide

  130. View Slide

  131. CONCLUSION
    +
    FUTURE WORK

    View Slide

  132. T H I S PA P E R O F F E R S
    CO N T R I B U T I O N S TO
    T H E G E N E R A L
    S C I E N T I F I C L I T E R AT U R E

    View Slide

  133. I N OT H E R WO R D S , I T
    O F F E R A F R A M E WO R K
    F O R C R O W D S O U R C I N G
    G E N E R A L LY W I T H
    # S COT U S A S A N
    A P P L I E D E X A M P L E

    View Slide

  134. T H E R E A R E A L S O M A N Y
    F O L LO W O N PA P E R S
    T H AT M I G H T B E
    G E N E R AT E D

    View Slide

  135. I N C LU D I N G B L E N D I N G
    O U R A LG O W I T H T H E
    C R O W D I N A N E N S E M B L E
    ( M E TA ) M O D E L

    View Slide

  136. crowd
    forecast
    learning problem is to discover how to blend streams of intelligence
    algorithm
    forecast
    ensemble method
    ENSEMBLE MODEL
    we can use
    machine learning
    methods and
    metadata such as
    case topic, lower
    court as well as
    crowd metadata
    to ‘learn’ the
    conditional
    weights to apply to
    the input signals

    View Slide

  137. W H I C H A M O N G OT H E R
    T H I N G S L I N K S TO O U R
    I N T E R E ST # A B N O R M A L
    R E T U R N S A N D
    J U D I C I A L D E C I S I O N S
    https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726

    View Slide

  138. AG A I N W E W I L L A N O N Y M I Z E
    A N D T H E N O P E N S O U R C E
    P L AY E R P R E D I C T I O N DATA
    TO S U P P O RT ACA D E M I C
    R E S E A R C H I N TO
    P R E D I C T I O N ,
    C R O W D S O U R C I N G , E TC .

    View Slide

  139. View Slide

  140. DANIEL MARTIN KATZ
    E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X
    B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M
    PAG E | DA N I E L M A R T I N K AT Z . C O M
    C O R P | L E X P R E D I C T. C O M
    MICHAEL BOMMARITO
    E D U | I L L I N O I S T E C H + S TA N F O R D C O D E X
    B LO G | C O M P U TAT I O N A L L E GA L S T U D I E S . C O M
    PAG E | B O M M A R I TO L LC . C O M
    C O R P | L E X P R E D I C T. C O M
    JOSH BLACKMAN
    E D U | S O U T H T E X A S C O L L E G E O F L AW H O U S TO N
    B LO G | J O S H B L AC K M A N . C O M
    PAG E | J O S H B L AC K M A N . C O M / B LO G
    C O R P | L E X P R E D I C T. C O M

    View Slide

  141. H T T P S : / / A R X I V. O RG / A B S / 1712 . 0 3 84 6
    H T T P S : / / PA P E R S . S S R N . C O M / S O L 3 /
    PA P E R S . C F M ? A B S T R AC T _ I D = 3 0 8 5710

    View Slide

  142. C R O W D S O U R C I N G ACC U R AT E LY
    A N D R O B U ST LY P R E D I C T S
    S U P R E M E CO U RT D E C I S I O N S
    DA N I E L M A RT I N KATZ | I L L I N O I S T E C H + STA N F O R D CO D E X
    M I C H A E L B O M M A R I TO | I L L I N O I S T E C H + STA N F O R D CO D E X
    J O S H B L AC K M A N | S O U T H T E X A S CO L L E G E O F L AW
    0.4
    0.5
    0.6
    0.7
    0.8
    0.9
    2012 2013 2014 2015 2016 2017
    Date of Decison
    Case Level Cumulative Accuracy By Model Type
    Model Type Model 1 Model 2 Model 3 Model 4 Null Model

    View Slide