Generative A.I. + Law - Background, Applications and Use Cases Including GPT-4 Passes the Bar Exam

Slide 1

Slide 1 text

G E N E R AT I V E A . I . + L AW @ computational professor daniel martin katz danielmartinkatz.com Illinois tech - chicago kent law 273Ventures.com B AC KG R O U N D, A P P L I CAT I O N S A N D U S E CA S E S I N C LU D I N G G P T- 4 PA S S E S T H E B A R E X A M https://bit.ly/3yxhY2e

Slide 2

Slide 2 text

NOVEMBER 30, 2022

Slide 3

Slide 3 text

I THINK THIS DAY WILL GO DOWN AS A VERY IMPORTANT DAY IN THE HISTORY OF TECHNOLOGY

Slide 4

Slide 4 text

NOVEMBER 30, 2022 IS THE DAY THAT CHATGPT WAS FIRST RELEASED BECAUSE

Slide 5

Slide 5 text

THIS WAS THE BEGINNING OF THE BROADER PUBLIC’S AWARENESS …

Slide 6

Slide 6 text

OF WHAT WAS ALREADY TRUE …

Slide 7

Slide 7 text

WHICH IS THAT IN THE BACKGROUND

Slide 8

Slide 8 text

THERE HAVE BEEN FUNDAMENTAL IMPROVEMENTS OCCURING WITHIN THE WORLD OF NATURAL LANGUAGE PROCESSING (NLP)

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

I PERSONALLY HAVE BEEN WORKING IN THIS AREA FOR A WHILE …

Slide 11

Slide 11 text

THINKING ABOUT HOW TO USE NLP MODELS IN THE LEGAL DOMAIN

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

https://sites.google.com/view/nllp/nllp-2019

Slide 14

Slide 14 text

https://law.mit.edu/pub/openedgar/release/1

Slide 15

Slide 15 text

https://aclanthology.org/2022.acl-long.297.pdf

Slide 16

Slide 16 text

https://speakerdeck.com/danielkatz/legal-nlp-breaking-the-legal-language-barrier

Slide 17

Slide 17 text

BUT IT HAS BEEN HARD TO COMMUNICATE TO MANY FOLKS IN THE LEGAL SECTOR …

Slide 18

Slide 18 text

ABOUT THE NATURE OF THE LANDSCAPE SHIFT IN THE FIELD …

Slide 19

Slide 19 text

MAYBE IT IS ROBOLAWYER FATIUGE …?

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

MAYBE IT IS THE ABSENCE OF A CLEAR DEMONSTRATION PROJECT …

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

SO NOW WE ARE JUST A FEW MONTHS INTO WHAT IS ARGUABLY THE MOST SUCCESSFUL PRODUCT LAUNCH IN HISTORY …

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

TODAY — I WOULD LIKE TO DISCUSS WHAT ALL OF THESE DEVELOPMENTS …

Slide 26

Slide 26 text

MIGHT MEAN FOR THE DELIVERY OF LEGAL SERVICES …

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

SO FOR MY OPENING STATEMENT PLEASE TAKE INTO ACCOUNT THE FOLLOWING FOUR CONSIDERATIONS…

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

< CONSIDERATION 1 > LANGUAGE IS THE ‘COIN OF THE REALM’ HERE IN THE WORLD OF LAW < CONSIDERATION 2 > THE MARCH OF LEGAL COMPLEXITY < CONSIDERATION 3 > MACHINES ARE INCREASINGLY IMPROVING IN LANGUAGE PROCESSING < CONSIDERATION 4 > LEGAL LANGUAGE != REGULAR LANGUAGE ?

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

< CONSIDERATION 1 >

Slide 33

Slide 33 text

LANGUAGE IS THE ‘COIN OF THE REALM’ HERE IN THE WORLD OF LAW

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

VIRTUALLY ALL ROADS IN LAW LEAD TO A DOCUMENT (CONSUMPTION, PRODUCTION OR BOTH)

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

LAWYERS, JUDGES AND REGULATORS ARE MASSIVE PRODUCERS OF TEXT

Slide 38

Slide 38 text

BRIEFS, MEMOS, STATUTES, OPINIONS, REGULATIONS, CONTRACTS, ETC.

Slide 39

Slide 39 text

ARE JUST SOME OF THE LEGAL WORK PRODUCT …

Slide 40

Slide 40 text

PRODUCED ON A DAILY BASIS ACROSS THE WORLD’S VARIOUS LEGAL SYSTEMS

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

NOT ONLY IS IT THE SHEER VOLUME OF TEXT BUT ALSO THESE MASSIVE VOLUMES OF TEXT ARE NOTORIOUSLY COMPLEX

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

< CONSIDERATION 2 >

Slide 45

Slide 45 text

THE MARCH OF LEGAL COMPLEXITY

Slide 46

Slide 46 text

W H E T H E R YO U A R E A L A R G E M U LT I N AT I O N A L CO R P O R AT I O N , A S M A L L B U S I N E S S O R A N I N D I V I D UA L C I T I Z E N …

Slide 47

Slide 47 text

L AW H A S A CO M P L E X I TY C H A L L E N G E …

Slide 48

Slide 48 text

IF YOU TAKE A STEP BACK

Slide 49

Slide 49 text

AND LOOK AT THE UNIT ECONOMICS OF LAW

Slide 50

Slide 50 text

THE REAL ANIMATING DRIVER IN THE DEMAND FOR UNITS OF LEGAL PRODUCTION

Slide 51

Slide 51 text

IS COMPLEXITY

Slide 52

Slide 52 text

SOCIAL, ECONOMIC AND POLITICAL COMPLEXITY

Slide 53

Slide 53 text

MANIFESTS ITSELF IN OUR DOMAIN AS LEGAL COMPLEXITY

Slide 54

Slide 54 text

WHILE IT IS SOMEWHAT HARD TO PRECISELY DEFINE …

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

VIRTUALLY EVERY WAY YOU MIGHT CONSIDER IT LEGAL COMPLEXITY HAS GROWN

Slide 57

Slide 57 text

DANIEL MARTIN KATZ, CORINNA COUPETTE, JANIS BECKEDORF & DIRK HARTUNG, COMPLEX SOCIETIES AND THE GROWTH OF THE LAW, 10 SCIENTIFIC REPORTS 18737 (2020) CORINNA COUPETTE, JANIS BECKEDORF, DIRK HARTUNG, MICHAEL BOMMARITO, & DANIEL MARTIN KATZ, MEASURING LAW OVER TIME: A NETWORK ANALYTICAL FRAMEWORK WITH AN APPLICATION TO STATUTES AND REGULATIONS IN THE UNITED STATES AND GERMANY, FRONT. PHYS. (2021 FORTHCOMING)

Slide 58

Slide 58 text

SIGNIFICANT GROWTH IN LAWS AND REGULATIONS

Slide 59

Slide 59 text

KEY TAKE AWAY - OVER PAST TWO DECADES ~50%+ MORE STATUTES ~200% MORE REGULATIONS

Slide 60

Slide 60 text

SO THE QUESTION IS HOW TO MATCH THAT COMPLEXITY WITH THE APPROPRIATE

Slide 61

Slide 61 text

COMPLEXITY MITIGATION TACTICS …

Slide 62

Slide 62 text

MIXTURE OF PEOPLE, PROCESS AND TECHNOLOGY

Slide 63

Slide 63 text

DESIGNING MORE SCALABLE SYSTEMS

Slide 64

Slide 64 text

http://prawfsblawg.blogs.com/prawfsblawg/2017/03/-complexity-mitigation-strategies- for-law-law-land-and-beyond-and-some-other-thoughts-on-had fi eld-su.html#more SOME THOUGHTS ON LEGAL COMPLEXITY MITIGATION STRATEGIES

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

< CONSIDERATION 3 >

Slide 67

Slide 67 text

MACHINES ARE INCREASINGLY IMPROVING IN THEIR ABILITY TO PROCESS AND ‘UNDERSTAND’ LANGUAGE

Slide 68

Slide 68 text

INITIAL PERFORMANCE AND HUMAN PERFORMANCE ARE NORMALIZED TO -1 AND 0 RESPECTIVELY (KIELA ET AL., 2021).

Slide 69

Slide 69 text

AND LEADING MODELS ARE RAPIDLY GROWING IN SCALE

Slide 70

Slide 70 text

SIZE OF LARGE LANGUAGE MODELS IN BILLIONS OF PARAMETERS

Slide 71

Slide 71 text

THE IMPLICATION IS THAT PREVIOUSLY UNTOUCHABLE PROBLEMS WILL BEGIN TO COME WITHIN REACH …

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

BECAUSE COMPLEXITY IS ARGUABLY SCALING FASTER THAN AVAILABLE RESOURCES …

Slide 74

Slide 74 text

WE ARGUABLY NEED A FORCE MULTIPLIER …

Slide 75

Slide 75 text

AND LANGUAGE IS CORE TO MOST (ALL) LEGAL TASKS

Slide 76

Slide 76 text

THE FACT THAT LANGUAGE TECHNOLOGY HAS BEEN MATERIALLY IMPROVING MIGHT BODE VERY WELL …

Slide 77

Slide 77 text

AS THE FORCE MULTIPLIER, PRODUCTIVITY ENHANCER …

Slide 78

Slide 78 text

THE FIELD ARGUABLY NEEDS …

Slide 79

Slide 79 text

TO SUPPORT THE QUANTITY, QUALITY, AND ACCESSIBILITY OF LEGAL SERVICES DEMANDED BY SOCIETY

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

< CONSIDERATION 4 >

Slide 82

Slide 82 text

SO WHAT ABOUT LEGAL LANGUAGE ?

Slide 83

Slide 83 text

LETS BE CLEAR — FOR AI / NLP TO MAKE A DEEP INCURSION INTO THIS FIELD …

Slide 84

Slide 84 text

IT NEEDS TO BE ABLE TO MORE FUNDAMENTALLY ENGAGE WITH LANGUAGE

Slide 85

Slide 85 text

AND NOT JUST ANY LANGUAGE …

Slide 86

Slide 86 text

BUT LEGAL LANGUAGE

Slide 87

Slide 87 text

THE PUBLIC HAS A DIM VIEW OF LEGAL LANGUAGE …

Slide 88

Slide 88 text

THEY DESCRIBE LEGAL DOCUMENTS AND ARGUMENTS USING TERMS SUCH AS ‘LEGALESE’ ‘LEGAL JARGON’ ‘LEGAL GOBBLEDYGOOK’

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

LEGAL ENGLISH != ENGLISH (AND THIS IS TRUE FOR MANY OTHER LANGUAGES)

Slide 91

Slide 91 text

INDEED MANY LAW STUDENTS CONSIDER THEIR INITIAL FORAY INTO THE FIELD …

Slide 92

Slide 92 text

AS AN EXERCISE IN ‘LEARNING A NEW LANGUAGE’

Slide 93

Slide 93 text

IN SUPPORT OF THIS TASK …

Slide 94

Slide 94 text

LAW FEATURES SPECIALIZED DICTIONARIES

Slide 95

Slide 95 text

TEXT BASED SUMMARIES

Slide 96

Slide 96 text

AND MANY OTHER RESOURCES DESIGNED TO SUPPORT …

Slide 97

Slide 97 text

THE DEVELOPMENT OF THE LINGUISTIC IMMERSION PROGRAM THAT WE CALL #LEGALEDUCATION

Slide 98

Slide 98 text

No content

Slide 99

Slide 99 text

NLP MODELS HAVE BEEN DEPLOYED IN THE LEGAL SECTOR …

Slide 100

Slide 100 text

BUT THE QUALITY AND PERFORMANCE HAS BEEN MIXED …

Slide 101

Slide 101 text

FRANKLY *NO* LEGAL TECH SOLUTION BUILT TO DATE HAS BEEN ABLE TO REALLY WORK WELL WITH NUANCES OF LEGAL LANGUAGE …

Slide 102

Slide 102 text

BUT THAT MIGHT BE ABOUT TO CHANGE …

Slide 103

Slide 103 text

No content

Slide 104

Slide 104 text

TODAY I WILL DIVIDE THE REMAINING PRESENTATION INTO FIVE PARTS

Slide 105

Slide 105 text

(1) THE (LONG) PATH TO GENERATIVE A.I. (2) WHAT IS GPT / CHATGPT / GPT-4 ? (3) BAR EXAM AS WINDOW INTO CAPABILITIES (4) LLMs IN THE DELIVERY OF LEGAL SERVICES (5) STRATEGIC CONSIDERATIONS AND THE ROAD AHEAD PRESENTATION IN FIVE PARTS

Slide 106

Slide 106 text

No content

Slide 107

Slide 107 text

(1) THE (LONG) PATH TO GENERATIVE A.I.

Slide 108

Slide 108 text

IN ORDER TO UNDERSTAND WHERE THIS MIGHT ALL BE GOING …

Slide 109

Slide 109 text

IT IS USEFUL TO UNDERSTAND THE ARC THAT HAS BROUGHT US TO THIS MOMENT …

Slide 110

Slide 110 text

SO LETS GO BACK TO THE EARLY DAYS …

Slide 111

Slide 111 text

BUT BEFORE WE TALK ABOUT ANYTHING TOO FANCY …

Slide 112

Slide 112 text

I WOULD LIKE TO REMIND EVERYONE …

Slide 113

Slide 113 text

BEFORE THERE WERE COMPUTERS

Slide 114

Slide 114 text

HUMANS ARE THE ORIGINAL COMPUTERS

Slide 115

Slide 115 text

HUMANS ONCE DID ALL OF THE COMPUTING

Slide 116

Slide 116 text

OVER THE COURSE OF MANY YEARS WE WERE ABLE TO LEVERAGE ALTERNATIVE FORMS OF COMPUTATION …

Slide 117

Slide 117 text

No content

Slide 118

Slide 118 text

THE FIELD OF ARTIFICIAL INTELLIGENCE IS ROUGHLY 75 YEARS OLD …

Slide 119

Slide 119 text

“BEFORE 1949, ‘COMPUTERS LACKED A KEY PREREQUISITE FOR INTELLIGENCE: THEY COULDN’T STORE COMMANDS, ONLY EXECUTE THEM …” http://sitn.hms.harvard.edu/ fl ash/2017/history-arti fi cial-intelligence/

Slide 120

Slide 120 text

SO WHAT IS ‘ARTIFICIAL INTELLIGENCE’ ?

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

No content

Slide 123

Slide 123 text

BIG IDEA IN AI IS TO DEVELOP IN MACHINES SOME LEVEL OF SYNTHETIC (OR ARTIFICIAL) REPRESENTATION OF A PREVIOUSLY HUMAN CENTERED PROCESS

Slide 124

Slide 124 text

ARTIFICIAL INTELLIGENCE IS A BROAD FIELD

Slide 125

Slide 125 text

No content

Slide 126

Slide 126 text

EVEN IN THE FOUNDATIONAL DAYS THERE WAS A VIEW ABOUT THE ROLE OF A ‘CONVERSATIONAL AGENT’ AS A MEANS TO EVALUATE THE QUALITY OF AN AI SYSTEM

Slide 127

Slide 127 text

WHICH IS TO SAY THAT LANGUAGE HAS ALWAYS BEEN A CORE TOPIC FOR THE FIELD OF A.I.

Slide 128

Slide 128 text

ALAN TURING IS ARGUABLY THE FATHER OF THE FIELD OF COMPUTER SCIENCE

Slide 129

Slide 129 text

AND THIS IDEA OF CONVERSATIONAL AGENT COMES FROM HIM …

Slide 130

Slide 130 text

NOTE THE MOVIE WAS MOSTLY ABOUT CRACKING ENIGMA BUT HERE IS WHERE IT GETS ITS TITLE THE TURING TEST

Slide 131

Slide 131 text

No content

Slide 132

Slide 132 text

IT IS IMPORTANT TO REMEMBER WITH TOPICS SUCH AS ARTIFICIAL INTELLIGENCE AND NATURAL LANGUAGE PROCESSING …

Slide 133

Slide 133 text

THE FIELD OF A.I. HAS HAD MANY FALSE STARTS AND A.I. WINTERS

Slide 134

Slide 134 text

EVEN SOME OF THE WORLD’S LEADING EXPERTS HAVE GOTTEN THINGS WRONG …

Slide 135

Slide 135 text

“MACHINES WILL BE CAPABLE, WITHIN TWENTY YEARS, OF DOING ANY WORK THAT A MAN CAN DO.” – Herbert Simon in 1965

Slide 136

Slide 136 text

“MACHINES WILL BE CAPABLE, WITHIN TWENTY YEARS, OF DOING ANY WORK THAT A MAN CAN DO.” – Herbert Simon in 1965 did not pan out by 1985 but did turn out okay for him …

Slide 137

Slide 137 text

SO I MENTION THIS ONLY SO THAT WE GROUND OURSELVES A BIT …

Slide 138

Slide 138 text

No content

Slide 139

Slide 139 text

A.I. BROADLY HAS MADE MAJOR PROGRESS ON MANY TOPICS …

Slide 140

Slide 140 text

No content

Slide 141

Slide 141 text

MACHINE LEARNING

Slide 142

Slide 142 text

No content

Slide 143

Slide 143 text

MACHINE LEARNING HAS BEEN USED TO UNDERTAKE VARIOUS FORMS OF PREDICTION IN LAW

Slide 144

Slide 144 text

EXAMPLE LITIGATION PREDICTION

Slide 145

Slide 145 text

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698 Katz DM, Bommarito MJ II, Blackman J (2017), A General Approach for Predicting the Behavior of the Supreme Court of the United States. PLoS ONE 12(4): e0174698.

Slide 146

Slide 146 text

http://www.sciencemag.org/news/ 2017/05/arti fi cial-intelligence-prevails- predicting-supreme-court-decisions Professor Katz noted that in the long term … “We believe the blend of experts, crowds, and algorithms is the secret sauce for the whole thing.” May 2nd 2017

Slide 147

Slide 147 text

https://www.wsj.com/articles/hidden- in-plain-sight-a-powerful-way-to- beat-the-market-1497367597 FT Big Read Feb 7, 2019

Slide 148

Slide 148 text

http://arxiv.org/abs/1508.05751 available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726

Slide 149

Slide 149 text

No content

Slide 150

Slide 150 text

LITIGATION FUNDING

Slide 151

Slide 151 text

No content

Slide 152

Slide 152 text

No content

Slide 153

Slide 153 text

No content

Slide 154

Slide 154 text

“Lawyers say the real value in mediation and arbitration might in the future come from large-scale data analysis of arbitrators and mediators themselves, in an effort to predict outcomes and potentially affect the course of settlements … Matthew Saunders, partner at Ashurst, notes that data analytics “could be extended to predicting which way arbitrators or a mediator might go”. Such technology may yet be some way off. In mediation, “a skilled facilitator helps the parties to explore where common ground can be found as the basis for an amicable settlement,” says James Freeman, arbitration partner at Allen & Overy. “The mediation process”, he adds, “is inherently a human one”.

Slide 155

Slide 155 text

DATA —> VARIABLES MODELS (STATISTICAL) OUTPUTS WITH VALIDATION

Slide 156

Slide 156 text

MUCH MORE COULD BE SAID ON THE LEGAL PREDICTION SIDE OF THIS …

Slide 157

Slide 157 text

No content

Slide 158

Slide 158 text

TODAY I AM MOSTLY GOING TO FOCUS ON GENERATIVE LANGUAGE MODELS …

Slide 159

Slide 159 text

A.I. IS A VERY BROAD FIELD

Slide 160

Slide 160 text

AND OF COURSE THERE ARE OTHER GENERATIVE MODELS MY SON AND I RECENTLY CREATED ‘BROCCOLI MAN’ GENERATIVE A.I. ART

Slide 161

Slide 161 text

No content

Slide 162

Slide 162 text

HISTORICALLY LANGUAGE HAS ALWAYS LAGGED OTHER ADVANCES IN A.I. …

Slide 163

Slide 163 text

BECAUSE OF THE UNDERLYING STATISTICAL DEPENDENCIES - LANGUAGE IS A *HARD* PROBLEM …

Slide 164

Slide 164 text

YET MAJOR ADVANCES ARE FINALLY COMING TO MARKET …

Slide 165

Slide 165 text

No content

Slide 166

Slide 166 text

HISTORICALLY NLP (LIKE THE REST OF A.I.) WAS RULES-BASED

Slide 167

Slide 167 text

RULES-BASED A.I. WAS ACTUALLY THE ONLY OPTION AT ONE POINT

Slide 168

Slide 168 text

Moore’s law !

Slide 169

Slide 169 text

Kryder’s law !

Slide 170

Slide 170 text

EVENTUALLY COMPUTER CHIPS WERE FAST ENOUGH AND DATA STORAGE WAS CHEAP ENOUGH …

Slide 171

Slide 171 text

THAT NLP COULD REALLY EMBRACE STATISTICAL MODELING …

Slide 172

Slide 172 text

NLP SLOWLY BECOMES A STATISTICAL FIELD IN THE 1990’S AND 2000’S

Slide 173

Slide 173 text

MARK LIBERMAN, "THE TREND TOWARDS STATISTICAL MODELS IN NATURAL LANGUAGE PROCESSING." NATURAL LANGUAGE AND SPEECH: SYMPOSIUM PROCEEDINGS BRUSSELS, SPRINGER, (1991)

Slide 174

Slide 174 text

BUT THE PRIMARY EFFORT HERE WAS ON SYNTAX BASED APPROACHES …

Slide 175

Slide 175 text

“There have been a series of clever approaches to backdoor into semantics* … (*while also being scalable) Semantic Methods (Fairly Difficult) Syntax Methods (Fairly Easy) Historically, Big Divide between Semantics and Syntax Quasi-Semantic Methods

Slide 176

Slide 176 text

No content

Slide 177

Slide 177 text

SYNTAX

Slide 178

Slide 178 text

WORDS WORD FREQUENCY PART OF SPEECH FREQUENCY ETC.

Slide 179

Slide 179 text

CTRL + F IS EXACT ‘STRING’ MATCHING

Slide 180

Slide 180 text

REGULAR EXPRESSION (REGEX) RULES-BASED METHOD(S) THAT CAN BE USED TO LOOK FOR WORD PATTERNS AND RETURN RESULTS FOR THOSE PATTERNS

Slide 181

Slide 181 text

TF - IDF TERM FREQUENCY INVERSE DOCUMENT FREQUENCY EXPLOITS THE FREQUENCY OF WORDS IN DOCUMENTS IN ORDER TO PROFILE THEM

Slide 182

Slide 182 text

No content

Slide 183

Slide 183 text

SEMANTICS

Slide 184

Slide 184 text

SEMANTICS IS ABOUT THE MEANING OF INDIVIDUAL WORDS SEMANTICS IS RELATIONSHIP BETWEEN WORDS THAT INTERACT TO PRODUCE HIGHER ORDER MEANING

Slide 185

Slide 185 text

LOTS OF THE TRAINING FOR LAWYERS IS ACTUALLY ABOUT THE DEEP SEMANTIC INTERPRETATION OF LANGUAGE CONTRACTS STATUTES REGULATIONS JUDICIAL DECISIONS EXAMPLES —>

Slide 186

Slide 186 text

SEMANTICS IS HARD FOR MACHINES

Slide 187

Slide 187 text

No content

Slide 188

Slide 188 text

BUT THE PAST DECADE THERE HAS BEGUN TO BE PROGRESS ON THIS FRONT …

Slide 189

Slide 189 text

WE HAVE SEEN THE EMERGENCE OF QUASI-SEMANTIC METHODS

Slide 190

Slide 190 text

QUASI-SEMANTIC METHODS WHICH CAN INCREASINGLY TAKE CONTEXT INTO ACCOUNT

Slide 191

Slide 191 text

WHAT WE SEE TODAY ARE FOLKS USING DEEP NEURAL NETWORKS TO WORK ON NLP PROBLEMS

Slide 192

Slide 192 text

GENERATIVE A.I. IN THE LINGUISTIC CONTEXT IS A MASH UP OF DEEP LEARNING + NLP

Slide 193

Slide 193 text

CLASSIC VS NEURAL NLP

Slide 194

Slide 194 text

No content

Slide 195

Slide 195 text

BTW WE SEE THIS SAME PROGRESSION IN LEGAL NLP AS WELL …

Slide 196

Slide 196 text

HERE WE ANALYZE TRENDS WITHIN EVERY LEGAL NLP PAPER (MORE THAN 600+ PAPERS) WRITTEN OVER THE PAST DECADE

Slide 197

Slide 197 text

GROWTH IN NUMBER OF PAPERS AND LANGUAGES

Slide 198

Slide 198 text

THE NEURAL AGENDA HAS COME TO LEGAL NLP

Slide 199

Slide 199 text

ALMOST NO TEXT GENERATION OR MACHINE SUMMARIZATION (BUT NOW THAT IS VERY LIKELY TO CHANGE)

Slide 200

Slide 200 text

No content

Slide 201

Slide 201 text

(2) WHAT IS GPT ? WHAT IS CHATGPT ? WHAT IS GPT-4 ?

Slide 202

Slide 202 text

GPT = GENERATIVE PRE-TRAINED TRANSFORMER (WE CAN TRY TO UNPACK ALL OF THIS IN A MINUTE)

Slide 203

Slide 203 text

GPT = IT IS A LARGE LANGUAGE MODEL (LLM) BUT FAR FROM THE ONLY ONE GENERATIVE PRE-TRAINED TRANSFORMER

Slide 204

Slide 204 text

TODAY I WILL MOSTLY LIMIT MY COMMENTS TO GPT BUT JUST WANTED TO FLAG THERE ARE OTHER MODELS OUT THERE

Slide 205

Slide 205 text

XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730 THE LEADING LLM MODELS

Slide 206

Slide 206 text

THE LEADING LLM MODELS XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730

Slide 207

Slide 207 text

No content

Slide 208

Slide 208 text

THE OPEN AI GPT MODELS COMBINE SEVERAL IDEAS TOGETHER FROM THE VARIOUS PAPERS IN THIS PROGRESSION …

Slide 209

Slide 209 text

Word2Vec (2013)

Slide 210

Slide 210 text

No content

Slide 211

Slide 211 text

No content

Slide 212

Slide 212 text

ELMO (2018)

Slide 213

Slide 213 text

BERT (2019)

Slide 214

Slide 214 text

No content

Slide 215

Slide 215 text

No content

Slide 216

Slide 216 text

No content

Slide 217

Slide 217 text

No content

Slide 218

Slide 218 text

No content

Slide 219

Slide 219 text

No content

Slide 220

Slide 220 text

No content

Slide 221

Slide 221 text

BTW IT SHOULD BE NOTED THAT …

Slide 222

Slide 222 text

CHATGPT / GPT-3.5 / GPT-4 ARE PROPRIETARY MODELS SO WE DO NOT KNOW FOR CERTAIN ALL OF THE DEEP DETAILS

Slide 223

Slide 223 text

No content

Slide 224

Slide 224 text

WE WILL NOT GET INTO THE SUPER TECHNICAL DETAILS TODAY …

Slide 225

Slide 225 text

FOR A VERY APPROACHABLE TREATMENT GO HERE https://writings.stephenwolfram.com/2023/02/ what-is-chatgpt-doing-and-why-does-it-work/

Slide 226

Slide 226 text

I MET STEPHEN WOLFRAM’S ROBOT PRESENCE IN HOUSTON IN 2011 HE APPEARED AT A COMPUTATIONAL LAW CONFERENCE VIA ROBOT (A REAL BOSS MOVE) STEPHEN AND YOURS TRULY TALKING ABOUT RULE 90 IT WAS A ‘BIG BANG’ MOMENT

Slide 227

Slide 227 text

No content

Slide 228

Slide 228 text

BETTER HARDWARE PARALLEL COMPUTING ATTENTION MECHANISM THE PATH TO MEGASCALE LARGE LANGUAGE MODELS HAS BEEN DRIVEN BY A MIXTURE OF

Slide 229

Slide 229 text

THE SAME TECH THAT HAS POWERED HIGHLY IMMERSIVE GAMING HAS ALSO PUSHED SCIENCE FORWARD … https://www.thegamer.com/pc-games-best-intense-graphics/

Slide 230

Slide 230 text

APRIL 5, 2023 NOT COMPARED DIRECTLY WITH THE H100 BUT THIS SHOWS YOU THE NATURE OF THE COMPETITION TAKING PLACE https://www.cnbc.com/2023/04/05/google- reveals-its-newest-ai-supercomputer- claims-it-beats-nvidia-.html

Slide 231

Slide 231 text

WOULD TAKE 300 YEARS TO TRAIN EVEN GPT-3 ON A SINGLE GPU BUT WE TRANSFORMER ARCHITECTURE ALLOWS FOR SIGNIFICANT PARALLELIZATION

Slide 232

Slide 232 text

TRAINED MODEL ON A BILLION PAIRS OF WORDS IN JUST 3.5 DAYS ON EIGHT NVIDIA GPUS … THIS APPROACH HIGHLIGHTED A PATH TO THE PRESENT WITH EVER LARGER FUTURE https://blogs.nvidia.com/blog/2022/03/25/ what-is-a-transformer-model/

Slide 233

Slide 233 text

NOW PARALLEL COMPUTING IS BEING DONE ON A MEGASCALE

Slide 234

Slide 234 text

No content

Slide 235

Slide 235 text

GPT-3 (RELEASED IN 2020) IS THIRD GENERATION OF THE GPT FAMILY

Slide 236

Slide 236 text

GPT-3 (RELEASED IN 2020) IS THIRD GENERATION OF THE GPT FAMILY GPT-3.5 IS AN INTERMEDIATE IMPROVEMENT ON ORIGINAL GPT-3

Slide 237

Slide 237 text

GPT-3 (RELEASED IN 2020) IS THIRD GENERATION OF THE GPT FAMILY GPT-4 IS THE MOST RECENT RELEASE AND IT IS LIKELY TO BE A ‘FAMILY OF MODELS’ GPT-3.5 IS AN INTERMEDIATE IMPROVEMENT ON ORIGINAL GPT-3

Slide 238

Slide 238 text

JUST LIKE GPT-3.5 IS NOW A FAMILY OF MODELS

Slide 239

Slide 239 text

SOME INPUTS TO GPT COMMON CRAWL WEBTEXT2 BOOKS1/2 WIKIPEDIA OPENAI LIKELY DID SOME SIGNIFICANT CLEANING / PREPROCESSING OF THESE SOURCES PRIOR TO TRAINING GPT-3 IS 175B PARAMETERS TRAINED WITH 499B TOKENS

Slide 240

Slide 240 text

TRANSFORMER ARCHITECTURE NEURAL NETWORK MODEL SCALE REINFORCEMENT LEARNING PRETRAINING JUST SOME KEY TERMINOLOGY YOU SHOULD LEARN CONTEXT WINDOW ATTENTION MECHANISM MODEL TUNING GRADIENT DESCENT

Slide 241

Slide 241 text

No content

Slide 242

Slide 242 text

TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3

Slide 243

Slide 243 text

COMBINE SPECIFIC TRAINING ON INSTRUCTIONS TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3

Slide 244

Slide 244 text

FURTHER TUNE ON INSTRUCTIONS COMBINE SPECIFIC TRAINING ON INSTRUCTIONS TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3

Slide 245

Slide 245 text

FURTHER TUNE ON INSTRUCTIONS SUPPORT WITH SPECIFIC REINFORCEMENT LEARNING FEEDBACK LOOP COMBINE SPECIFIC TRAINING ON INSTRUCTIONS TAKE ORIGINAL LLM GPT-3 https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent- Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1 CHATGPT IS AN OFFSHOOT OF GPT-3

Slide 246

Slide 246 text

THE GPT-4 TECHNICAL REPORT OFFERS SOME BUT NOT MOST OF THE KEY DETAILS

Slide 247

Slide 247 text

RLHF WAS CLEARLY PART OF THE BREW HERE BUT GENERALLY DOES NOT SEEM TO BE MOVING THE NEEDLE ALL THAT MUCH AT THIS POINT

Slide 248

Slide 248 text

SO THE POINT HERE IS TO MAKE CLEAR THAT THERE ARE MANY STEPS AND MOVING PARTS …

Slide 249

Slide 249 text

THERE ARE OTHER PLAYERS BESIDES OPENAI AND THEY WILL LIKELY PURSUE OTHER STEPS / MOVING PARTS … AND MANY MORE …

Slide 250

Slide 250 text

No content

Slide 251

Slide 251 text

(3) BAR EXAM AS WINDOW INTO INCREASED CAPABILITIES

Slide 252

Slide 252 text

AGAIN WE HAD BEEN TRYING TO DISCUSS / HIGHLIGHT THE INCREASING CAPABILITIES OF LANGUAGE MODELS FOR SOME TIME

Slide 253

Slide 253 text

WE THOUGHT THAT SOME SORT OF A PUBLIC DEMONSTRATION WOULD HELP MAKE THE POINT …

Slide 254

Slide 254 text

REMEMBER THIS IS A TASK THAT MANY WOULD THINK IS IMPOSSIBLE (LAST YEAR I WOULD HAVE SAID IT WOULD NOT OCCUR FOR MANY YEARS)

Slide 255

Slide 255 text

SOME TASKS LAWYERS REGULARLY UNDERTAKE ARE ACTUALLY WAY *EASIER* THAN THE BAR EXAM (AND OTHERS ARE HARDER) LET’S BE CLEAR …

Slide 256

Slide 256 text

TO HAVE A CHANCE ON THE BAR EXAM …

Slide 257

Slide 257 text

EXAMINEE MUST POSSESS A THRESHOLD AMOUNT OF LEGAL KNOWLEDGE AND READING COMPREHENSION SKILLS AND SEMANTIC AND SYNTACTIC COMMAND OF THE ENGLISH LANGUAGE

Slide 258

Slide 258 text

No content

Slide 259

Slide 259 text

RIGHT BEFORE THE HOLIDAYS ROUGHLY DEC 22, 2022

Slide 260

Slide 260 text

I CALLED MY FRIEND AND FREQUENT COLLABORATOR MIKE BOMMARITO

Slide 261

Slide 261 text

AND SAID LET’S EVALUATE GPT-3.5 (TEXT-DAVINCI-003) ON THE MULTIPLE CHOICE PART OF THE BAR EXAM

Slide 262

Slide 262 text

AND IT DID PRETTY WELL …

Slide 263

Slide 263 text

V1.01 - December 29, 2022 V2.01 - January 3, 2023

Slide 264

Slide 264 text

No content

Slide 265

Slide 265 text

No content

Slide 266

Slide 266 text

LITTLE DID WE KNOW AT THE TIME THAT THIS PAPER PUT US ON A PATH …

Slide 267

Slide 267 text

TO BE INVOLVED IN THE LAUNCH OF GPT-4* *OBVIOUSLY WE ARE JUST PLAYING A VERY SMALL ROLE IN THE BIG PICTURE (BUT WE WILL TAKE IT!)

Slide 268

Slide 268 text

GPT-4 BAR EXAM https://openai.com/research/gpt-4 March 14, 2023

Slide 269

Slide 269 text

CHATGPT BAR EXAM PASSES Paper Now Available on SSRN! March 15, 2023 - Version 1.01 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233

Slide 270

Slide 270 text

No content

Slide 271

Slide 271 text

No content

Slide 272

Slide 272 text

https://www.law.com/legaltechnews/ 2023/03/17/how-gpt-4-mastered-the- entire-bar-exam-and-why-that-matters/ abajournal.com/web/article/latest- version-of-chatgpt-aces-the-bar-exam- with-score-in-90th-percentile

Slide 273

Slide 273 text

https://www.lawnext.com/ 2023/03/gpt-takes-the-bar-exam- again-this-time-it-score-among- top-10-of-test-takers.html

Slide 274

Slide 274 text

https://www.youtube.com/watch?v=4pGG79VdNmU

Slide 275

Slide 275 text

No content

Slide 276

Slide 276 text

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233

Slide 277

Slide 277 text

< THE UBE > UNIFORM BAR EXAM

Slide 278

Slide 278 text

No content

Slide 279

Slide 279 text

No content

Slide 280

Slide 280 text

< THE MBE >

Slide 281

Slide 281 text

MULTISTATE BAR EXAM (MBE) SUBJECTS TESTED TORTS CONTRACTS EVIDENCE REAL PROPERTY CIVIL PROCEDURE CONSTITUTIONAL LAW CRIMINAL LAW AND PROCEDURE

Slide 282

Slide 282 text

No content

Slide 283

Slide 283 text

No content

Slide 284

Slide 284 text

No content

Slide 285

Slide 285 text

No content

Slide 286

Slide 286 text

< THE MEE >

Slide 287

Slide 287 text

MULTISTATE ESSAY EXAMINATION (MEE) SUBJECTS TESTED TORTS EVIDENCE CONTRACTS FAMILY LAW REAL PROPERTY CONFLICT OF LAWS TRUSTS AND ESTATES CONSTITUTIONAL LAW BUSINESS ASSOCIATIONS FEDERAL CIVIL PROCEDURE UNIFORM COMMERCIAL CODE CRIMINAL LAW AND PROCEDURE

Slide 288

Slide 288 text

MULTISTATE ESSAY EXAMINATION (MEE) https://github.com/mjbommar/gpt4-passes-the-bar/blob/main/data/MEE.md

Slide 289

Slide 289 text

‘REPRESENTATIVE GOOD ANSWERS’ https://mdcourts.gov/sites/default/ fi les/import/ble/ examanswers/2022/202207uberepgoodanswers.pdf SEVERAL STATE BARS RELEASE REPRESENTATIVE GOOD ANSWERS THESE ARE VERY HELPFUL FOR EVALUATION PURPOSES AS THEY ARE ACTUAL ANSWERS WHICH ARE ABOVE MERELY PASSING ANSWERS (BUT NOT NECESSARILY PERFECT)

Slide 290

Slide 290 text

No content

Slide 291

Slide 291 text

(GPT-3.0 was released in 2020)

Slide 292

Slide 292 text

(GPT-3.0 was released in 2020) This ain’t passing any bar exam

Slide 293

Slide 293 text

No content

Slide 294

Slide 294 text

No content

Slide 295

Slide 295 text

< THE MPT >

Slide 296

Slide 296 text

MULTISTATE PERFORMANCE TEST (MPT) SKILLS TESTED EXAMINEE MUST COMPLETE A PRACTICAL LAWYERING TASK SUCH AS LEGAL ANALYSIS, FACT ANALYSIS, PROBLEM SOLVING, ORGANIZATION AND MANAGEMENT OF INFORMATION, AND CLIENT COMMUNICATION

Slide 297

Slide 297 text

MULTISTATE PERFORMANCE EXAM (MPT) 10-15 PAGES OF MATERIALS THE FILE = THE FACTS THE LIBRARY = THE LAW ~5000 TOKEN INPUTS (ACCESS HERE) BIT.LY/40F3FQ2

Slide 298

Slide 298 text

No content

Slide 299

Slide 299 text

No content

Slide 300

Slide 300 text

OVERALL PERFORMANCE

Slide 301

Slide 301 text

SOME KEY SHORTCOMINGS OF GPT-4 ON THE BAR EXAM FAILED TO PROPERLY CALCULATE THE DISTRIBUTION OF ASSETS FROM A TESTAMENTARY TRUST WHICH HAS BEEN DEEMED TO BE INVALID PROVIDED AN INCORRECT ANSWER ON A CIVIL PROCEDURE QUESTION REGARDING DIVERSITY JURISDICTION AFTER THE JOINDER OF A NECESSARY PARTY PROVIDED IMPROPER ANALYSIS ON A REAL PROPERTY (REAL ESTATE) SUBQUESTION REGARDING BOTH THE PROPER DESIGNATION OF A FUTURE INTEREST AND THE APPLICATION OF THE RULE AGAINST PERPETUITIES

Slide 302

Slide 302 text

No content

Slide 303

Slide 303 text

No content

Slide 304

Slide 304 text

https://github.com/mjbommar/gpt4-passes-the-bar ACCESS THE FULL OUTPUT AND OTHER SUPPORTING MATERIALS HERE

Slide 305

Slide 305 text

No content

Slide 306

Slide 306 text

OUR BAR EXAM ANALYSIS IS WHAT IS CALLED A ‘ZERO SHOT’ ANALYSIS

Slide 307

Slide 307 text

ZERO SHOT ANALYSIS REFLECTS THE FLOOR AND NOT THE CEILING OF CURRENT CAPABILITIES

Slide 308

Slide 308 text

No content

Slide 309

Slide 309 text

ZERO SHOT ENTER PROMPT RECEIVE ANSWER ’PROMPT ENGINEERING’ IS ABOUT TUNING / REFINING PROMPTS TO OBTAIN BETTER ANSWERS

Slide 310

Slide 310 text

No content

Slide 311

Slide 311 text

THIS ZERO SHOT AND NOT EVEN THE LATEST MODEL (I.E. NOT GPT-4)

Slide 312

Slide 312 text

*NOT* A VERY SOPHISTICATED TAKE ON THIS SITUATION BUT REFLECTIVE OF THE MODAL PERSPECTIVE OF LAWYERS / LAW PROFS NOTE: ALL COMMERICAL RESEARCH TOOLS ALREADY HAVE ‘A.I.’ IN THEM

Slide 313

Slide 313 text

No content

Slide 314

Slide 314 text

ONE WAY TO DRAMATICALLY REDUCE THE LIKELIHOOD OF A HALLUCINATION IS TO MOVE OUT OF THE ‘ZERO SHOT’ PARADIGM

Slide 315

Slide 315 text

AND ANCHOR INITIAL OUTPUT AGAINST SOME ‘GROUND TRUTH’ …

Slide 316

Slide 316 text

ONE SHOT GET RESULT QUERY THAT RESULT AGAINST SOMETHING ELSE REFINE RESULT ENTER PROMPT OUTPUT FINAL ANSWER (RETRIEVAL AUGMENTATION) (IF NEEDED) EXAMPLES

Slide 317

Slide 317 text

No content

Slide 318

Slide 318 text

THE GENERALIZATION OF ALL OF THIS IS AN ORCHESTRATION LAYER TO BRING MULTIPLE STREAMS / TECHNOLOGIES TOGETHER

Slide 319

Slide 319 text

FEW SHOT (CHAIN OF THOUGHT PROMPTING)

Slide 320

Slide 320 text

HERE DAMIEN RIEHL FROM FASTCASE WILL WALK YOU THROUGH HOW TO BUILD MORE OF A BRIEF THROUGH SEQUENTIAL PROMPTING … HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

Slide 321

Slide 321 text

HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

Slide 322

Slide 322 text

FEW SHOT (LANGCHAIN / AUTOGPT PROMPTING)

Slide 323

Slide 323 text

FEW SHOT (LEGAL DOMAIN) (LANGCHAIN / AUTOGPT STYLE ORCHESTRATION LAYER) https://docs.kelvin.legal/docs/examples/due-dilligence/ https://docs.kelvin.legal/docs/examples/litigation-automation/

Slide 324

Slide 324 text

No content

Slide 325

Slide 325 text

THERE ARE LOTS OF OPPORTUNITIES OUTSIDE THE ZERO-SHOT CONTEXT

Slide 326

Slide 326 text

OR EVEN OTHER PLUGINS … FOR EXAMPLE WOLFRAM COULD HELP SOLVE FOR ISSUES WITH QUANTITATIVE REASONING

Slide 327

Slide 327 text

https://arxiv.org/pdf/2303.17651.pdf https://selfre fi ne.info/ https://github.com/madaan/self-re fi ne

Slide 328

Slide 328 text

GENERATIVE AGENTS

Slide 329

Slide 329 text

No content

Slide 330

Slide 330 text

No content

Slide 331

Slide 331 text

(4) LLMs IN THE DELIVERY OF LEGAL SERVICES

Slide 332

Slide 332 text

IN GENERAL …

Slide 333

Slide 333 text

DRAFTING + EDITING COMMUNICATIONS EMAILS REPORTS PRESENTATIONS MARKETING MATERIALS

Slide 334

Slide 334 text

DRAFTING + EDITING LEGAL DOCUMENTS CONTRACTS BRIEFS MEMOS INTERROGATORIES DEMAND LETTERS

Slide 335

Slide 335 text

CONDUCTING LEGAL RESEARCH EXTERNAL POINTS OF LAW INTERNAL KNOWLEDGE MGMT

Slide 336

Slide 336 text

SUMMARIZING LARGE BODIES OF TEXTUAL MATERIAL SUMMARIZING LEGAL DOCUMENTS

Slide 337

Slide 337 text

AND REMEMBER WE CAN COMBINE OUTPUTS WITH OTHER MACHINE LEARNING MODELS, ETC.

Slide 338

Slide 338 text

OR EVEN OTHER PLUGINS … FOR EXAMPLE WOLFRAM COULD HELP SOLVE FOR ISSUES WITH QUANTITATIVE REASONING

Slide 339

Slide 339 text

No content

Slide 340

Slide 340 text

BUT LETS LOOK AT SEVERAL CONCRETE EXAMPLES …

Slide 341

Slide 341 text

(1) DRAFT A MNDA

Slide 342

Slide 342 text

No content

Slide 343

Slide 343 text

No content

Slide 344

Slide 344 text

(2) DRAFT A REPLY BRIEF HERE DAMIEN RIEHL FROM FASTCASE WILL WALK YOU THROUGH HOW TO BUILD MORE OF A BRIEF THROUGH SEQUENTIAL PROMPTING … HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

Slide 345

Slide 345 text

No content

Slide 346

Slide 346 text

(3) DRAFT REPORT OR PRESENTATION

Slide 347

Slide 347 text

https://youtu.be/S7xTBa93TX8

Slide 348

Slide 348 text

No content

Slide 349

Slide 349 text

DUE DILIGENCE / M&A SUPPORT (4)

Slide 350

Slide 350 text

No content

Slide 351

Slide 351 text

https://docs.kelvin.legal/docs/examples/due-dilligence/

Slide 352

Slide 352 text

https://docs.kelvin.legal/docs/examples/due-dilligence/ DUE DILIGENCE /M&A SUPPORT

Slide 353

Slide 353 text

https://docs.kelvin.legal/docs/examples/due-dilligence/ DUE DILIGENCE /M&A SUPPORT

Slide 354

Slide 354 text

https://docs.kelvin.legal/docs/examples/due-dilligence/

Slide 355

Slide 355 text

No content

Slide 356

Slide 356 text

LITIGATION SUPPORT & TRIAGE (5)

Slide 357

Slide 357 text

No content

Slide 358

Slide 358 text

No content

Slide 359

Slide 359 text

(6) BUILD A LEGAL KNOWLEDGE GRAPH https://www.ibm.com/topics/knowledge-graph

Slide 360

Slide 360 text

(6) BUILD A LEGAL KNOWLEDGE GRAPH HTTPS://TAX-GRAPH.273VENTURES.COM/

Slide 361

Slide 361 text

(6) BUILD A LEGAL KNOWLEDGE GRAPH

Slide 362

Slide 362 text

(6) BUILD A LEGAL KNOWLEDGE GRAPH

Slide 363

Slide 363 text

No content

Slide 364

Slide 364 text

(7) LANGUAGE MODELS ARE ZERO-SHOT TAGGERS

Slide 365

Slide 365 text

https://www.linkedin.com/posts/danielkatz-3b001539_feedbackfriday-legaltech- legaldata-activity-7045070829428113409-fIZz

Slide 366

Slide 366 text

No content

Slide 367

Slide 367 text

https://www.law.com/ legaltechnews/2023/03/20/sali- harnesses-most-advanced-gpt- models-to-help-legal-industry- speak-the-same-data-language/ March 23, 2023 Michael Bommarito - Gentleman Farmer

Slide 368

Slide 368 text

https://sali-search.kelvin.legal

Slide 369

Slide 369 text

No content

Slide 370

Slide 370 text

(8) REGULATORY MONITORING

Slide 371

Slide 371 text

https://www.linkedin.com/posts/ bommarito_summarize-everything-published-in- the-federal-activity-7048621468422725632-CjuO April 3, 2023

Slide 372

Slide 372 text

No content

Slide 373

Slide 373 text

(9) SUBSTANTIVE LEGAL ANALYSIS (COPYRIGHT)

Slide 374

Slide 374 text

HTTPS://WWW.YOUTUBE.COM/WATCH?V=NQZCRHR8YPU&T=11S

Slide 375

Slide 375 text

No content

Slide 376

Slide 376 text

HERE ARE JUST A FEW MORE OF MANY OTHER POTENTIAL AREAS WHERE THIS CLASS OF TECH MIGHT BE USEFUL … E-DISCOVERY LEGAL BILLING DEALS DATABASES CONTRACT DRAFTING AND MANY MANY OTHER EXAMPLES

Slide 377

Slide 377 text

HERE A JUST A COUPLE OF RESOURCES THAT HAVE BEEN MADE AVAILABLE (THERE WILL BE MANY MORE) https://ssrn.com/abstract=4404017

Slide 378

Slide 378 text

https://thebrainyacts.beehiiv.com/

Slide 379

Slide 379 text

No content

Slide 380

Slide 380 text

NOW A QUICK WORD OF CAUTION

Slide 381

Slide 381 text

BECAUSE OF THE POTENTIAL FOR MODEL HALLUCINATION, ETC.

Slide 382

Slide 382 text

IT IS IMPORTANT TO USE THESE TOOLS (AT LEAST FOR NOW) AS PART OF A HUMAN IN THE LOOP PROCESS

Slide 383

Slide 383 text

IN OTHER WORDS, FOR MOST USE CASES HUMANS SHOULD ALWAYS BE PART OF THE ‘RETRIEVAL AUGMENTATION’ LAYER

Slide 384

Slide 384 text

AND OF COURSE OR MACHINES CAN AID IN RETRIEVAL AUGMENTATION / QUALITY ASSURANCE

Slide 385

Slide 385 text

CENTAUR CHESS HUMAN + MACHINE HUMAN OR MACHINE >

Slide 386

Slide 386 text

No content

Slide 387

Slide 387 text

(5) STRATEGIC CONSIDERATIONS AND THE ROAD AHEAD

Slide 388

Slide 388 text

FUD VS FEAR, UNCERTAINTY, & DOUBT FEAR OF MISSING OUT FOMO

Slide 389

Slide 389 text

FUD FEAR, UNCERTAINTY, & DOUBT HALLUCINATIONS CONFIDENTIALITY INFORMATION SECURITY

Slide 390

Slide 390 text

AMAZING CAPABILITIES KEEPING UP WITH THE MARKET NOT WANTING TO LOOK OUT OF STEP TO CLIENTS FEAR OF MISSING OUT FOMO

Slide 391

Slide 391 text

FUD VS FEAR, UNCERTAINTY, & DOUBT FEAR OF MISSING OUT FOMO ORGANIZATIONS / INDIVIDUALS TOGGLE BETWEEN THESE TWO POINTS OF VIEW (SOMETIME WITHIN THE SAME CONVERSATION)

Slide 392

Slide 392 text

No content

Slide 393

Slide 393 text

SEVERAL EARLY TOOLS IN THE MARKET …

Slide 394

Slide 394 text

No content

Slide 395

Slide 395 text

MAY 4, 2023 https://www.lexisnexis.com/en- us/products/lexis-plus-ai.page

Slide 396

Slide 396 text

MAY 23, 2023 https://www.lawnext.com/2023/05/ thomson-reuters-previews-its-plans-for- generative-ai-announces-integration-with- microsoft-365-copilot.html

Slide 397

Slide 397 text

No content

Slide 398

Slide 398 text

No content

Slide 399

Slide 399 text

OBVIOUSLY WE HAVE A COMPANY AND WE WORKED WITH CASETEXT ON THE BAR PAPER — SO PERHAPS WE ARE NOT 100% NEUTRAL … SHANG GAO PABLO ARREDONDO

Slide 400

Slide 400 text

BUT OF COURSE THESE WILL NOT BE THE LAST TOOLS AND IT REMAINS TO BE SEEN HOW IT WILL ALL SHAKE OUT …

Slide 401

Slide 401 text

I THINK IT IS CLEAR THAT THIS IS GOING TO BE A LONG RACE …

Slide 402

Slide 402 text

No content

Slide 403

Slide 403 text

WHAT IS THE DATA AND TECHNOLOGY STRATEGY THAT ORGANIZATIONS CAN EMPLOY IN LIGHT OF THESE CHANGE IN THE TECHNOLOGY LANDSCAPE ?

Slide 404

Slide 404 text

BUILD VS ASSEMBLE VS BUY BUILD IS LIKELY OFF THE TABLE FOR THE SHORT TO MEDIUM TERM ASSEMBLE HOWEVER IS A VIABLE OPTION SO CAREFUL PROCUREMENT IS GOING TO BE THE STRATEGIC PATH

Slide 405

Slide 405 text

DATA STRATEGY HOW TO COLLECT / REGULARIZE DATA FOR USE INSIDE THESE SYSTEMS OR AS A LAYER ON TOP OF THESE SYSTEMS

Slide 406

Slide 406 text

No content

Slide 407

Slide 407 text

SEVERAL OTHER STRATEGIC CONSIDERATIONS WHAT IS THE ROLE OF DOMAIN SPECIFIC TRAINING? HOW COULD I CUSTOMIZE THESE MODELS FOR USE WITHIN MY OWN ORGANIZATION ? WHICH MODELS / LLMS DO I LEVERAGE ? HOW DO I CHOOSE ? CAN I MIX AND MATCH ? WHAT IS THE DIFFERENTIAL IMPACT OF THESE MODELS BY LEGAL ORGANIZATION ? HOW DO I THINK ABOUT QUESTIONS OF PRIVACY AND INFORMATION SECURITY ? COPYRIGHT?

Slide 408

Slide 408 text

WHAT IS THE DIFFERENTIAL IMPACT OF THESE MODELS BY LEGAL ORGANIZATION ? (1)

Slide 409

Slide 409 text

I THINK THAT THERE ARE GOING TO BE DIFFERENTIAL IMPACTS BY TYPE OF LEGAL ORGANIZATION

Slide 410

Slide 410 text

MANAGERS SHOULD REORGANIZE JOBS SO ORGANIZATION CAN FURTHER CONCENTRATE TASKS WHICH CAN BE ENABLED BY A.I. MICRO LEVEL

Slide 411

Slide 411 text

LOOK TO PLACES WHERE THERE IS A CONCENTRATION OF LOW TO MEDIUM COMPLEXITY TASKS MACRO LEVEL ALSP, LPO, LEGAL OPS, SPECIALIZE LAW FIRMS, ETC.

Slide 412

Slide 412 text

TRAINING FOLKS TO USE THESE TOOLS TO THE MAXIMUM EXTENT POSSIBLE HUMAN CAPITAL (THIS WILL BE AN IMPORANT LIMITATION ON ORGANIZATIONAL SUCCESS)

Slide 413

Slide 413 text

No content

Slide 414

Slide 414 text

HOW SHOULD I THINK ABOUT QUESTIONS OF PRIVACY AND INFORMATION SECURITY ? COPYRIGHT? (2)

Slide 415

Slide 415 text

SHOULD / CAN I PUT CONFIDENTIAL CLIENT DATA IN THESE SYSTEMS ? MANY FOLKS HAVE ASKED ME —

Slide 416

Slide 416 text

PROCEED WITH CAUTION — I WOULD TELL YOU TO AUDIT PRECISELY WHAT IS HAPPENING AND WHERE IT IS HAPPENING SHOULD / CAN I PUT CONFIDENTIAL CLIENT DATA IN THESE SYSTEMS ? (CLIENT CONSENT + AUDITED SAFEGUARDS) MANY FOLKS HAVE ASKED ME —

Slide 417

Slide 417 text

THE ROGUE ASSOCIATE (OR PARTNER) PASTING STUFF INTO GPT OR ANOTHER LLM MODEL ONE VERSION OF THE CONCERN

Slide 418

Slide 418 text

https://mashable.com/article/samsung-chatgpt-leak-details

Slide 419

Slide 419 text

WE HAVE BEEN WORKING WITH A FEW FIRMS ON HOW TO NAVIGATE THE THORNY INFO SEC AND LEGAL ETHICS ISSUES

Slide 420

Slide 420 text

IF NOT, DOES VENDOR HAVE SOC2 / ISO2700 ? HOW MANY OTHER PARTIES ARE IN THE FLOW OF YOUR INFORMATION ? CONTRACTUAL REPRESENTATIONS FROM THOSE 3RD PARTIES? IS THE SOLUTION AVAILABLE ON PREM ? SOME VENDOR QUESTIONS WHAT CONTROLS DOES THE VENDOR APPLY TO THE TRANSITING OF YOUR DATA ?

Slide 421

Slide 421 text

LEVERAGING THESE TOOLS ON INTERNAL DATA IS REALLY THE HOLY GRAIL …. INTERNAL DATA WE ARE GOING TO GET THERE BUT IT REQUIRES A REAL THOUGHT OUT APPROACH HERE

Slide 422

Slide 422 text

WHAT IS GOING TO HAPPEN WITH COPYRIGHT ? MANY FOLKS HAVE ASKED ME —

Slide 423

Slide 423 text

WHAT IS GOING TO HAPPEN WITH COPYRIGHT ? I THINK THIS IS GOING TO LEAD TO A MAJOR DECISION IN COPYRIGHT AND/OR ACTION BY CONGRESS, ETC. MANY FOLKS HAVE ASKED ME —

Slide 424

Slide 424 text

https://arxiv.org/pdf/2303.15715.pdf THIS IS A VERY GOOD PAPER ON THE TOPIC

Slide 425

Slide 425 text

No content

Slide 426

Slide 426 text

WHAT IS THE ROLE OF DOMAIN SPECIFIC TRAINING? (3)

Slide 427

Slide 427 text

WILL A GENERAL MODEL BEAT A DOMAIN SPECIFIC MODEL ? SURE THAT IS POSSIBLE / LIKELY — IF THE GENERAL MODEL IS LARGE AND DOMAIN MODEL IS SMALL

Slide 428

Slide 428 text

AGAIN HERE IS AN EXAMPLE WITH ACTUAL RESULTS ATTACHED TO IT …

Slide 429

Slide 429 text

https://www.bloomberg.com/company/press/ bloomberggpt-50-billion-parameter-llm-tuned- fi nance/ https://arxiv.org/pdf/2303.17564.pdf

Slide 430

Slide 430 text

https://twitter.com/rasbt/status/1642880757566676992

Slide 431

Slide 431 text

SO HOW COULD THIS BE IMPROVED EVEN FURTHER ?

Slide 432

Slide 432 text

SOURCES OF LLM MODEL IMPROVEMENT USE DIFFERENT TRAINING DATA NEURAL NET ACCESS / IMPLEMENTATION INSTRUCTION MODULE FINE TUNING RETRIEVAL AUGMENTATION (RAG) PROMPT ENGINEERING CHAIN OF THOUGHT PROMPTING RLHF AGENT KNOWLEDGE GRAPH TRAVERSAL MODEL CREATOR MODEL USER (REINFORCEMENT LEARNING FROM HUMAN FEEDBACK)

Slide 433

Slide 433 text

SO WILL THERE BE A ‘LAW GPT’? PROBABLY SOON IT IS AN OPEN QUESTION WILL IT BE BETTER THAN GPT-4 ON LAW TASKS?

Slide 434

Slide 434 text

No content

Slide 435

Slide 435 text

HOW COULD I CUSTOMIZE THESE MODELS FOR USE WITHIN MY OWN ORGANIZATION ? (4)

Slide 436

Slide 436 text

HOW IS ALL OF YOUR DATA GOING TO GET INSIDE OF THESE SYSTEMS … ?

Slide 437

Slide 437 text

WE ARE LIKELY ON A PATH TO INTERNAL GPT STYLE OFFERINGS

Slide 438

Slide 438 text

FOLKS ARE EVENTUALLY GOING BUILD / TRAIN / TUNE THEIR OWN GPT STYLE MODELS…

Slide 439

Slide 439 text

TO DO SO FOLKS ARE GOING TO NEED TO STRUCTURE / PREPROCESS THEIR DATA

Slide 440

Slide 440 text

https://nationalmagazine.ca/en-ca/articles/legal-market/legal-tech/2023/chatting-with-gpt

Slide 441

Slide 441 text

CONNECTING THE THREE LAYERS https://nationalmagazine.ca/en-ca/articles/ legal-market/legal-tech/2023/chatting-with-gpt

Slide 442

Slide 442 text

No content

Slide 443

Slide 443 text

WHICH MODELS / LLMS DO I LEVERAGE ? HOW DO I CHOOSE ? CAN I MIX AND MATCH ? (5)

Slide 444

Slide 444 text

SO WE HAVE SEEN QUITE A BIT OF THIS SINCE THE LAUNCH OF CHATGPT NOW

Slide 445

Slide 445 text

THIS IS NOT INHERENTLY BAD BUT I THINK ORGANIZATIONS NEED TO THINK ABOUT THEIR PRECISE STRATEGY HERE NOW

Slide 446

Slide 446 text

SOURCES OF LLM MODEL IMPROVEMENT USE DIFFERENT TRAINING DATA NEURAL NET ACCESS / IMPLEMENTATION (ALPACA, ETC.) INSTRUCTION MODULE FINE TUNING RETRIEVAL AUGMENTATION (RAG) PROMPT ENGINEERING CHAIN OF THOUGHT PROMPTING (LANGCHAIN, ETC.) RLHF (E.G. BLOOMBERG GPT) (DOLLY, ETC.) (PLUGINS / UI / GROUNDING) MODEL CREATOR MODEL USER (VARIOUS FORMS) (VARIOUS FORMS) (REINFORCEMENT LEARNING FROM HUMAN FEEDBACK) (AGENTS BUILD / TRAVERSE GRAPHS) (MANY APPROACHES) AGENT KNOWLEDGE GRAPH TRAVERSAL

Slide 447

Slide 447 text

REMEMBER YOU ARE GOING TO HAVE DIRECT ACCESS …

Slide 448

Slide 448 text

SO WHAT IS THE VENDOR VALUE ADD OVER BASE LLM MODEL? TRUST BUT VERIFY THERE IS GOING TO BE SIGNIFICANT PRESSURE ON VENDORS TO OVERSTATE THE ACTUAL CONTRIBUTION OF THEIR OFFERING ABOVE JUST AN LLM API CALL I SUGGEST YOU TRUST BUT VERIFY ALL CLAIMS

Slide 449

Slide 449 text

CAN YOU DETERMINE WHETHER A VENDOR IS SIMPLY SELLING YOU A UI/ UX WRAPPER ON TOP OF AN LLM ? AGAIN THIS MIGHT BE OKAY BUT DO NOT PAY AN EXCESSIVE PREMIUM …

Slide 450

Slide 450 text

IT WOULD BE A BIG MISTAKE TO GO ALL IN ON ANY VENDOR OR SPECIFIC LLM AT THIS POINT PLACE A SMALL TO MEDIUM BET

Slide 451

Slide 451 text

AS THE YEAR PROGRESSES, I BELIEVE THIS WILL *NOT* JUST BE A GPT STORY …

Slide 452

Slide 452 text

IT IS A HIGHLY COMPETITVE LANDSCAPE AND OTHER FOLKS WILL BE ENTERING THE MARKET WITH ADDITIONAL OFFERINGS

Slide 453

Slide 453 text

THERE ARE OTHER PLAYERS BESIDES OPENAI AND THEY WILL LIKELY PURSUE OTHER STEPS / MOVING PARTS … AND MANY MORE …

Slide 454

Slide 454 text

REMEMBER THAT GOOGLE ACTUALLY INVENTED MUCH OF THE TECH THAT BROUGHT YOU GPT https://www.forbes.com/sites/richardnieva/ 2023/02/08/google-openai-chatgpt-microsoft- bing-ai/ https://www.nytimes.com/2023/01/20/ technology/google-chatgpt-arti fi cial- intelligence.html

Slide 455

Slide 455 text

CAN YOU BOTH RAPIDLY AND RIGOROUSLY EVALUATE THE QUALITY OF MODELS OR FIGURE OUT HOW TO MIX AND MATCH THEM … ? (ENSEMBLE LEARNING ANYONE?)

Slide 456

Slide 456 text

No content

Slide 457

Slide 457 text

FINAL THOUGHTS

Slide 458

Slide 458 text

THIS IS SOMETHING OF A DIFFERENT FLAVOR FROM THE AVERAGE TECHNOLOGICAL INNOVATION …

Slide 459

Slide 459 text

No content

Slide 460

Slide 460 text

NOTE THIS IS A CHATGPT PAPER NOT EVEN CONSIDERING POTENTIAL ADDITIONAL GAINS IN GPT-4 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4375283 “TIME TAKEN DECREASES BY 0.8 SDS AND OUTPUT QUALITY RISES BY 0.4 SDS.”

Slide 461

Slide 461 text

OVER 50% PRODUCTIVITY GAINS FROM GPT-3.5 VERSION OF COPILOT (USED FOR PROGRAMMING) https://arxiv.org/pdf/2302.06590 COPILOT X RELEASED MARCH 23RD 2023 https://github.blog/2023-03-22-github-copilot- x-the-ai-powered-developer-experience/ FEBRUARY 13 2023

Slide 462

Slide 462 text

https://arxiv.org/abs/2303.10130 “OUR FINDINGS REVEAL THAT AROUND 80% OF THE U.S. WORKFORCE COULD HAVE AT LEAST 10% OF THEIR WORK TASKS AFFECTED BY THE INTRODUCTION OF LLMS, WHILE APPROXIMATELY 19% OF WORKERS MAY SEE AT LEAST 50% OF THEIR TASKS IMPACTED. WE DO NOT MAKE PREDICTIONS ABOUT THE DEVELOPMENT OR ADOPTION TIMELINE OF SUCH LLMS”

Slide 463

Slide 463 text

No content

Slide 464

Slide 464 text

TASKS VS JOBS WHAT IS BEING MADE POSSIBLE? WE RACE WITH THE MACHINE LABOR MARKET IMPLICATIONS (IT IS COMPLICATED) (IN A WORLD WHERE TEXT GENERATION IS FAR CHEAPER)

Slide 465

Slide 465 text

No content

Slide 466

Slide 466 text

OVERALL, THERE SEEMS TO BE NON-LINEAR PERFORMANCE PERHAPS AKIN TO BROADER LANGUAGE ACQUISTION ?

Slide 467

Slide 467 text

IS THERE IS SOMETHING MORE FUNDAMENTAL GOING ON EMERGENT BEHAVIOR ?

Slide 468

Slide 468 text

CUT+PASTE SAFETY FIRST THIS IS ALL KINDA BACK TO THE FUTURE FOR ME

Slide 469

Slide 469 text

https://research.google/pubs/pub52065/

Slide 470

Slide 470 text

https://futurism.com/gpt-4-sparks-of-agi https://www.youtube.com/watch?v=qbIk7-JPB2c

Slide 471

Slide 471 text

THIS IS AN ACTIVE DEBATE IN THE ACADEMIC LITERATURE I WOULD SAY THE JURY IS STILL OUT

Slide 472

Slide 472 text

AND WHILE THERE ARE LIMITS AND THINGS DO NOT GROW EXPONENTIALLY FOREVER

Slide 473

Slide 473 text

I THINK THIS IS GOING TO BE THE MOST EXCITING YEAR IN TECHNOLOGY IN A VERY LONG TIME !