Legal NLP - Breaking the Legal Language Barrier ?

Slide 1

Slide 1 text

Legal NLP—Breaking the Legal Language Barrier? Dirk Hartung & Daniel Martin Katz Presentation at Stanford CodeX Future Law 2022—Law, Education and Experience Talks (LEX)

Slide 2

Slide 2 text

Daniel Martin Katz Illinois Tech Chicago-Kent Bucerius Law School Stanford CodeX Dirk Hartung Bucerius Law School Stanford CodeX Your Presenters

Slide 3

Slide 3 text

A LEGAL COMPLEXITY PICTURE LAW LAW LAND Featuring NATURAL LANGUAGE and DOMAIN-SPECIFIC JARGON Where Natural Language is the Coin of Realm …

Slide 4

Slide 4 text

The Encoding of this Neural Net Begins Early On … Indeed, many of my law students consider their initial foray into the ﬁeld … as an exercise in learning a ‘new language’ Law School as a Neural Network Encoder 🧠

Slide 5

Slide 5 text

Specialized Dictionaries In Support of the Linguistic Immersion Program, We Offer Our Students a Steady Diet of Language Text Based Summaries Case Books

Slide 6

Slide 6 text

Law / Lawyering is (in part) an exercise in linguistic construction and interpretation But Law is Not Just About the Consumption of Natural Language

Slide 7

Slide 7 text

Text Production at a Massive Scale these are just some of the legal work product being produced on a daily basis across the world’s various legal systems …are massive producers of text Lawyers Judges Regulators Briefs Memos Statutes Opinions Regulations Contracts

Slide 8

Slide 8 text

Stanford CodeX TechIndex - 1800+ Companies and Counting … Signiﬁcant Growth in LegalTech Over the Last Decade techindex.law.stanford.edu

Slide 9

Slide 9 text

There are a wide variety of companies and solutions which have been developed … Tools to help individuals & organizations navigate the scale and complexity of the law … Helping Navigate the Scale and Complexity of the Law Perspectives on Legal Complexity from some of our other work

Slide 10

Slide 10 text

Many (most) of these software offerings had to have some account for natural language because …

Slide 11

Slide 11 text

In Law, most roads lead to a document … And that document is very likely to be expressed in natural language … All Most Roads in Law Lead to a Document

Slide 12

Slide 12 text

Despite laudable efforts to the move law away from natural language and toward code … We are unlikely to see natural language displaced as the method to encode the law for the foreseeable future … Law as Code, Code as Law and the (Stubborn) Persistence of Natural Language

Slide 13

Slide 13 text

So in order to make progress on certain problems we are likely to have to confront natural language … Good News is that there is a subﬁeld in CS / AI whose primary focus is at the intersection of language and computation … Some Good News from Comp Sci

Slide 14

Slide 14 text

YOU ARE HERE Language Computer Science Natural Language Processing (Computational Linguistics) NLP is a Branch of AI NLP as a Branch of AI

Slide 15

Slide 15 text

WHAT IS A ROUGH DEFINITION OF NLP ?

Slide 16

Slide 16 text

“It is the Statistical Representation of Language …

Slide 17

Slide 17 text

Semantic Methods (Fairly Difficult) Syntax Methods (Fairly Easy) Historically, Big Divide between Semantics and Syntax

Slide 18

Slide 18 text

“There have been a series of clever approaches to backdoor into semantics* … (*while also being scalable)

Slide 19

Slide 19 text

Semantic Methods (Fairly Difficult) Syntax Methods (Fairly Easy) Historically, Big Divide between Semantics and Syntax Quasi-Semantic Methods

Slide 20

Slide 20 text

The Age of ‘Neural’ NLP

Slide 21

Slide 21 text

Word2Vec (2013)

Slide 22

Slide 22 text

ELMO (2018)

Slide 23

Slide 23 text

BERT (2019)

Slide 24

Slide 24 text

The GPT Trilogy 2018 - Present

Slide 25

Slide 25 text

Big Bird 2021

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

“Okay that is general NLP but what about ‘LEGAL NLP’ … ?

Slide 29

Slide 29 text

PAST PRESENT FUTURE LEGAL NLP

Slide 30

Slide 30 text

Historically there were a fairly limited number of commercial applications which leveraged LEGAL NLP…

Slide 31

Slide 31 text

In this talk, Richard Susskind notes that when he entered the scene in the early 1980’s, there were fewer than 40 papers on AI+Law TOTAL (let alone NLP+Law) reinventlawchannel.com/richard-susskind- future-of-artiﬁcial-intelligence-and-law

Slide 32

Slide 32 text

Both the AI + Law Conference and the Jurix conference thereafter began to focus on these topics … With a strong focus on academic topics such as legal argumentation / legal reasoning …

Slide 33

Slide 33 text

The 2010’s is the Decade Where the Academic and Commercial Worlds Began to Really Collide …

Slide 34

Slide 34 text

But if we look at both the academic and commercial sphere, we still observe a fairly thin account for legal language … Certainly as compared to humans and expert lawyers ... But this not an uncommon issue across the NLP world

Slide 35

Slide 35 text

The need to understand Sub-Dialects of English is a familiar problem …

Slide 36

Slide 36 text

So the Scientiﬁc / Engineering task at hand is to improve the performance of Legal NLP Models … By further breaking down the legal language barrier By grafting broader NLP developments to Domain Speciﬁc Needs in Law

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

PAST PRESENT FUTURE LEGAL NLP

Slide 39

Slide 39 text

Present ● Most current NLP in Law Survey ○ Zhong et al. ○ ACL Main Conference 2020 ○ Embedding/Symbol-based ● 3 applications ○ Judgment prediction ○ Case similarity ○ Legal Q’n’A ● A good starting point to understand the state of the ﬁeld in 2020 aclanthology.org/2020.acl-main.466.pdf

Slide 40

Slide 40 text

Present ● Domain-Speciﬁc NLP toolkit ○ Bommarito et al. ○ 2018 ○ NLTK for law ● Python package ○ Standard NLP capabilities ○ Legal information extraction ○ Embeddings and classiﬁers ○ Legal lexica arxiv.org/abs/1806.03688

Slide 41

Slide 41 text

Present ● Led by Our Co-Author Ilias Chalkidis (who is one of the world’s most published Legal NLP experts) ● Legal-BERT is an effort to PreTrain on Legal Information ● 12 GB of diverse English legal text from several ﬁelds (e.g., legislation, court cases, contracts) scraped from publicly available resources ● Demonstrated Utility of Pre-Training on Model Performance arxiv.org/abs/2010.02559

Slide 42

Slide 42 text

Present ● Latest large data set contribution ○ Zheng et al ○ Stanford ○ ICAIL 2021 ● The role of legal language domain speciﬁcity ○ Legal NLP tasks are too easy ○ Problems often not meaningful for practitioners ○ Task similarity determines meaningfulness of pretraining dl.acm.org/doi/abs/10.1145/3462757.3466088

Slide 43

Slide 43 text

Foundations

Slide 44

Slide 44 text

Inspiration ● A diverse dataset ○ Tasks ○ Set Size ○ Text Genres ○ Degrees of Difficulty ● A diagnostic dataset to evaluate and analyze model performance ● Public Leaderboard and Visualization gluebenchmark.com

Slide 45

Slide 45 text

Inspiration (Super) GLUE

Slide 46

Slide 46 text

Our LexGLUE Colleagues! buceri.us/lexglue

Slide 47

Slide 47 text

Dataset Overview

Slide 48

Slide 48 text

Model Overview

Slide 49

Slide 49 text

Results per Data Set

Slide 50

Slide 50 text

Overall Aggregated Scores

Slide 51

Slide 51 text

How to engage? github.com/coastalcph/lex-glue

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

PAST PRESENT FUTURE LEGAL NLP

Slide 54

Slide 54 text

Future for LexGLUE Future for LegalNLP

Slide 55

Slide 55 text

Where is LexGLUE going strategically and tech-wise ? Extend the benchmark to datasets in other languages Reduce legal restrictions inhibiting the creation of datasets Develop better anonymization tools to free data Add human evaluation to ground truth Integrated submission environment Automatic evaluation and Leaderboard updates Incorporate approaches for Transformer-based models and long documents Build models which leverage document structure Curate a large-scale legal pre-training corpus Create even larger legal language models

Slide 56

Slide 56 text

Summary Points on the Future of LexGLUE ● Build Proper Infrastructure including a Leaderboard ● Extend the benchmark to datasets in other languages ● Expand to number of tasks ● Include More Difficult Tasks as part of a ‘Super LexGlue’ ● Curate a large-scale legal pre-training corpus (Legal C4) ○ (more on this in a second)

Slide 57

Slide 57 text

Future for LegalNLP

Slide 58

Slide 58 text

As a law professor I live this dualism … Training Two Forms of Neural Networks between training my student’s neural network and working with these neural networks …

Slide 59

Slide 59 text

Take a Look at the Information Diet And Compare this to the Info Diet of My Law Students

Slide 60

Slide 60 text

How Might One Optimize this Pre-Training Diet ? No Real Pre-Training Diet Optimization has been undertaken thus far … What is the Ideal Pre-Training Mixture ? General Language Legal Language Which pre-training ‘diet’ will perform best ? On a Per Task Basis On a Cross /Overall Task Basis

Slide 61

Slide 61 text

Daniel Martin Katz Illinois Tech Chicago-Kent Bucerius Law School Stanford CodeX Dirk Hartung Bucerius Law School Stanford CodeX Your Presenters

Slide 62

Slide 62 text

Our LexGLUE Colleagues! buceri.us/lexglue

Slide 63

Slide 63 text

Legal NLP—Breaking the Legal Language Barrier? Dirk Hartung & Daniel Martin Katz Presentation at Stanford CodeX Future Law 2022—Law, Education and Experience Talks (LEX)