Generative A.I. + Law - A Year in Review

Slide 1

Slide 1 text

G E N E R AT I V E A . I . + L AW @ computational professor daniel martin katz danielmartinkatz.com Illinois tech - chicago kent law 273Ventures.com A Y E A R I N R E V I E W I N C LU D I N G G P T PA S S E S T H E B A R & E L E V E N DAY S T H AT D E F I N E D T H E Y E A R I N ( L E G A L ) A I https://bit.ly/3yxhY2e

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

NOVEMBER 30, 2022

Slide 4

Slide 4 text

I THINK THIS DAY WILL GO DOWN AS A VERY IMPORTANT DAY IN THE HISTORY OF TECHNOLOGY

Slide 5

Slide 5 text

NOVEMBER 30, 2022 IS THE DAY THAT CHATGPT WAS FIRST RELEASED BECAUSE

Slide 6

Slide 6 text

THIS WAS THE BEGINNING OF THE BROADER PUBLIC’S AWARENESS …

Slide 7

Slide 7 text

OF WHAT WAS ALREADY TRUE …

Slide 8

Slide 8 text

WHICH IS THAT IN THE BACKGROUND

Slide 9

Slide 9 text

THERE HAVE BEEN FUNDAMENTAL IMPROVEMENTS OCCURRING WITHIN THE WORLD OF NATURAL LANGUAGE PROCESSING (NLP)

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

I PERSONALLY HAVE BEEN WORKING IN THIS AREA FOR A WHILE …

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

THINKING ABOUT HOW TO USE LANGUAGE MODELS HERE IN THE LEGAL SECTOR

Slide 14

Slide 14 text

BUT IT HAS BEEN HARD TO COMMUNICATE TO MANY FOLKS IN THE LEGAL AND RELATED SECTORS …

Slide 15

Slide 15 text

ABOUT THE NATURE OF THE LANDSCAPE SHIFT IN THE FIELD …

Slide 16

Slide 16 text

MAYBE IT IS ROBOLAWYER FATIGUE …?

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

MAYBE IT IS THE ABSENCE OF A CLEAR DEMONSTRATION PROJECT …

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

SO NOW WE ARE JUST 12 MONTHS INTO ARGUABLY THE MOST SUCCESSFUL PRODUCT LAUNCH IN HISTORY …

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

SO TO KICK THINGS OFF HERE AT THE TLTF SUMMIT

Slide 24

Slide 24 text

TODAY — I THOUGHT I WOULD OFFER SOME OPENING COMMENTS ON THIS MAGICAL YEAR …

Slide 25

Slide 25 text

STARTING WITH AN OVERVIEW …

Slide 26

Slide 26 text

AND THEN TRANSITIONING TO A ‘YEAR IN REVIEW’

Slide 27

Slide 27 text

A ‘YEAR IN REVIEW’ THROUGH THE LENS OF ELEVEN DATES THAT HELPED DEFINED THE PRESENT AND FUTURE FOR OUR FIELD…

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

BUT FIRST — HOW DID WE GET HERE?

Slide 30

Slide 30 text

ALL OF THIS GENERATIVE A.I. DID NOT COME OUT OF NOWHERE …

Slide 31

Slide 31 text

MACHINES ARE INCREASINGLY IMPROVING IN THEIR ABILITY TO PROCESS AND ‘UNDERSTAND’ LANGUAGE

Slide 32

Slide 32 text

INITIAL PERFORMANCE AND HUMAN PERFORMANCE ARE NORMALIZED TO -1 AND 0 RESPECTIVELY (KIELA ET AL., 2021).

Slide 33

Slide 33 text

AND LEADING MODELS ARE RAPIDLY GROWING IN SCALE

Slide 34

Slide 34 text

SIZE OF LARGE LANGUAGE MODELS IN BILLIONS OF PARAMETERS

Slide 35

Slide 35 text

THE IMPLICATION IS THAT PREVIOUSLY UNTOUCHABLE PROBLEMS WILL BEGIN TO COME WITHIN REACH …

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

THIS IS MOSTLY A STORY OF BRUTE FORCE COMPUTING …

Slide 38

Slide 38 text

Moore’s law !

Slide 39

Slide 39 text

Kryder’s law !

Slide 40

Slide 40 text

AND THAT BRUTE FORCE GETS APPLIED TO LANGUAGE AND PATTERNS IN LANGUAGE

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

WHAT IS GPT ? WHAT IS CHATGPT ? WHAT IS GPT-4 ?

Slide 43

Slide 43 text

GPT = GENERATIVE PRE-TRAINED TRANSFORMER (WE CAN TRY TO UNPACK ALL OF THIS IN A MINUTE)

Slide 44

Slide 44 text

GPT = IT IS A LARGE LANGUAGE MODEL (LLM) BUT FAR FROM THE ONLY ONE GENERATIVE PRE-TRAINED TRANSFORMER

Slide 45

Slide 45 text

XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730 THE LEADING LLM MODELS

Slide 46

Slide 46 text

BETTER HARDWARE PARALLEL COMPUTING ATTENTION MECHANISM THE PATH TO MEGASCALE LARGE LANGUAGE MODELS HAS BEEN DRIVEN BY A MIXTURE OF

Slide 47

Slide 47 text

THE SAME TECH THAT HAS POWERED HIGHLY IMMERSIVE GAMING HAS ALSO PUSHED SCIENCE FORWARD … https://www.thegamer.com/pc-games-best-intense-graphics/

Slide 48

Slide 48 text

APRIL 5, 2023 NOT COMPARED DIRECTLY WITH THE H100 BUT THIS SHOWS YOU THE NATURE OF THE COMPETITION TAKING PLACE https://www.cnbc.com/2023/04/05/google- reveals-its-newest-ai-supercomputer- claims-it-beats-nvidia-.html

Slide 49

Slide 49 text

WOULD TAKE 300 YEARS TO TRAIN EVEN GPT-3 ON A SINGLE GPU BUT WE TRANSFORMER ARCHITECTURE ALLOWS FOR SIGNIFICANT PARALLELIZATION

Slide 50

Slide 50 text

TRAINED MODEL ON A BILLION PAIRS OF WORDS IN JUST 3.5 DAYS ON EIGHT NVIDIA GPUS … THIS APPROACH HIGHLIGHTED A PATH TO THE PRESENT WITH EVER LARGER FUTURE https://blogs.nvidia.com/blog/2022/03/25/ what-is-a-transformer-model/

Slide 51

Slide 51 text

NOW PARALLEL COMPUTING IS BEING DONE ON A MEGASCALE

Slide 52

Slide 52 text

SOME INPUTS TO GPT COMMON CRAWL WEBTEXT2 BOOKS1/2 WIKIPEDIA OPENAI LIKELY DID SOME SIGNIFICANT CLEANING / PREPROCESSING OF THESE SOURCES PRIOR TO TRAINING GPT-3 IS 175B PARAMETERS TRAINED WITH 499B TOKENS

Slide 53

Slide 53 text

THE POINT HERE IS TO MAKE CLEAR THAT THERE ARE MANY STEPS AND MOVING PARTS …

Slide 54

Slide 54 text

THERE ARE OTHER PLAYERS BESIDES OPENAI AND THEY WILL LIKELY PURSUE OTHER STEPS / MOVING PARTS … AND MANY MORE …

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

NOW WHY IS ALL OF THIS INTERESTING HERE IN THE WORLD OF LAW?

Slide 57

Slide 57 text

LANGUAGE IS THE ‘COIN OF THE REALM’ HERE IN THE LEGAL WORLD

Slide 58

Slide 58 text

VIRTUALLY ALL ROADS IN LAW LEAD TO A DOCUMENT (CONSUMPTION, PRODUCTION OR BOTH)

Slide 59

Slide 59 text

IF MACHINES CAN ‘UNDERSTAND’ NOT ONLY GENERAL LANGUAGE BUT COMPLEX ‘LEGALESE’ …

Slide 60

Slide 60 text

THEN THEY MIGHT BE ABLE TO REALLY HELP OUR FIELD …

Slide 61

Slide 61 text

BUT HOW WELL CAN LEADING MODELS PERFORM ON COMPLEX PROFESSIONAL TASKS ?

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

RIGHT BEFORE THE HOLIDAYS ROUGHLY DEC 22, 2022

Slide 64

Slide 64 text

I CALLED MY FRIEND AND FREQUENT COLLABORATOR MIKE BOMMARITO

Slide 65

Slide 65 text

AND SAID LET’S EVALUATE CHATGPT ON THE MULTIPLE CHOICE PART OF THE BAR EXAM

Slide 66

Slide 66 text

AND IT DID PRETTY WELL …

Slide 67

Slide 67 text

V1.01 - December 29, 2022 V2.01 - January 3, 2023

Slide 68

Slide 68 text

LITTLE DID WE KNOW AT THE TIME THAT THIS PAPER PUT US ON A PATH …

Slide 69

Slide 69 text

TO BE INVOLVED IN THE LAUNCH OF GPT-4* *OBVIOUSLY WE ARE JUST PLAYING A VERY SMALL ROLE IN THE BIG PICTURE (BUT WE WILL TAKE IT!)

Slide 70

Slide 70 text

GPT-4 BAR EXAM https://openai.com/research/gpt-4 March 14, 2023

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

https://www.youtube.com/watch?v=4pGG79VdNmU

Slide 73

Slide 73 text

CHATGPT BAR EXAM PASSES Paper Now Available on SSRN! March 15, 2023 - Version 1.01 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

AND IT CAN DO THE ESSAYS AS WELL …

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

(GPT-3.0 was released in 2020)

Slide 79

Slide 79 text

(GPT-3.0 was released in 2020) This ain’t passing any bar exam

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

No content

Slide 82

Slide 82 text

OBVIOUSLY THE BAR EXAM IS *NOT* THE ‘BE ALL END ALL’ BUT IT IS SOME MEASURE OF CAPABILITY

Slide 83

Slide 83 text

BUT THE BROADER QUESTION IS HOW TO INDUSTRIALIZE THESE RAW CAPABILITIES WITHIN SPECIFIC LEGAL WORKSTREAMS / USE CASES

Slide 84

Slide 84 text

OVER THE PAST YEAR …

Slide 85

Slide 85 text

MANY (LEGAL) ORGANIZATIONS HAVE HAD TO CONSIDER HOW TO INCORPORATE AND/ OR METABOLIZE THESE DEVELOPMENTS

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

FROM ‘OUT OF THE BOX’ RAW CAPABILITIES TO INDUSTRIAL APPLICATIONS

Slide 88

Slide 88 text

OUR BAR EXAM ANALYSIS IS REALLY AN EVALUATION OF THE ‘OUT OF THE BOX’ CAPABILITIES OF GPT-4 (ALSO CALLED A ‘ZERO-SHOT’ ANALYSIS OF THE CAPABILITIES)

Slide 89

Slide 89 text

IN OTHER WORDS, WE ARE SHOWING THE BARE MINIMUM OF THE CAPABILITIES HERE

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

‘OUT OF THE BOX’ ENTER PROMPT RECEIVE ANSWER (ZERO-SHOT)

Slide 92

Slide 92 text

‘AD HOC PROMPT ENGINEERING’ ENTER PROMPT RECEIVE ANSWER ’PROMPT ENGINEERING’ IS ABOUT TUNING / REFINING PROMPTS TO OBTAIN BETTER ANSWERS (HALF-SHOT)

Slide 93

Slide 93 text

MUCH MORE CAN BE DONE WITH ADDITIONAL ENGINEERING LAYERS BUILT ON TOP OF THESE RAW CAPABILITIES

Slide 94

Slide 94 text

No content

Slide 95

Slide 95 text

HERE ARE TEN EXAMPLES OF LLMs APPLICATIONS / USE CASES IN THE LEGAL SECTOR

Slide 96

Slide 96 text

LOOK AT THESE EXAMPLES OF THE CURRENT ‘ART OF THE POSSIBLE’

Slide 97

Slide 97 text

BECAUSE SOME OF THE APPROACHES / FRAMEWORKS CAN BE RETROFITTED TO OTHER PROBLEMS

Slide 98

Slide 98 text

No content

Slide 99

Slide 99 text

(1) DRAFT A BRIEF HERE DAMIEN RIEHL FROM VLEX WALKS THROUGH HOW TO BUILD MORE OF A BRIEF THROUGH SEQUENTIAL PROMPTING … HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

Slide 100

Slide 100 text

HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT- LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/

Slide 101

Slide 101 text

No content

Slide 102

Slide 102 text

CONTRACT NEGOTIATION (2) (AUTOMATIC REDLINE AGAINST A PLAYBOOK)

Slide 103

Slide 103 text

No content

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

(3) REGULATORY MONITORING

Slide 107

Slide 107 text

MANY (ALL) ORGANIZATIONS MUST MONITOR THE CHANGING REGULATORY ENVIRONMENT

Slide 108

Slide 108 text

PARTICULARLY ON A GLOBAL BASIS — THE IDENTIFICATION AND ROUTING OF ALL OF THIS WORK IS A GOOD CANDIDATE FOR MACHINES …

Slide 109

Slide 109 text

THINK OF AN ‘AIR TRAFFIC CONTROL SYSTEM’ BUT FOR REGULATORY MANAGEMENT

Slide 110

Slide 110 text

1. IDENTIFY NEW REGULATIONS 2. IDENTIFY ANY IMPACTED INTERNAL POLICIES / PROCEDURES 3. IDENTIFY INDIVIDUAL / TEAM RESPONSIBLE FOR COMPLIANCE WITH NEW REGULATIONS 4. ISSUE TICKET TO THE INDIVIDUAL / TEAM TO UPDATE POLICY AND COMPLIANCE PLAN APPROXIMATE WORKFLOW FOR REGULATORY MONITORING** **NOTE: EXACT INTERNAL PROCESS MAY DIFFER BY ORGANIZATION

Slide 111

Slide 111 text

No content

Slide 112

Slide 112 text

No content

Slide 113

Slide 113 text

(4) PATENT DRAFTING

Slide 114

Slide 114 text

No content

Slide 115

Slide 115 text

No content

Slide 116

Slide 116 text

No content

Slide 117

Slide 117 text

(5) LEGAL RESEARCH

Slide 118

Slide 118 text

No content

Slide 119

Slide 119 text

No content

Slide 120

Slide 120 text

THOSE ARE JUST A FEW MORE OF MANY OTHER POTENTIAL AREAS WHERE THIS CLASS OF TECH MIGHT BE USEFUL … AND MANY MANY OTHER EXAMPLES 6. E-DISCOVERY 7. DEALS DATABASES 8. LITIGATION MGMT 9. BUSINESS OF LAW 10. COMPLEX FINANCIAL DOCS

Slide 121

Slide 121 text

No content

Slide 122

Slide 122 text

GOOD NEWS YOU ARE NOT GOING TO HAVE TO IMAGINE ALL OF THE POSSIBILITIES

Slide 123

Slide 123 text

BECAUSE MORE THAN 30+ COMPANIES ARE GOING TO PRESENT OVER THE NEXT FEW DAYS

Slide 124

Slide 124 text

STARTUP SHOWCASE COMPANIES

Slide 125

Slide 125 text

SCALE STAGE COMPANIES

Slide 126

Slide 126 text

No content

Slide 127

Slide 127 text

BUT TO SET THE STAGE FOR THE DAYS TO COME …

Slide 128

Slide 128 text

ELEVEN DAYS THAT HELPED DEFINE THE PRESENT AND FUTURE FOR GEN (LEGAL) A.I. THE YEAR IN REVIEW

Slide 129

Slide 129 text

FOR EACH OF THESE ELEVEN DATES, I WILL OFFER THOUGHTS ON THE IMPLICATIONS OF THESE DATES TO THE FUTURE OF (LEGAL) AI

Slide 130

Slide 130 text

No content

Slide 131

Slide 131 text

NOVEMBER 30, 2022 TWO DATES HIGHLIGHTED SO FAR #1 OF 11 IMPLICATION GPT / LLMS GO MAINSTREAM

Slide 132

Slide 132 text

IMPLICATION MARCH 15, 2023 #2 OF 11 GPT-4 IS A SERIOUS UPGRADE OVER PRIOR VERSIONS TWO DATES HIGHLIGHTED SO FAR

Slide 133

Slide 133 text

No content

Slide 134

Slide 134 text

MARCH 30, 2023 THE RISE OF VERTICALIZED LLMS #3 OF 11

Slide 135

Slide 135 text

FOLKS OFTEN ASK ME — WILL THERE BE A ‘LAW GPT’? PROBABLY SOON

Slide 136

Slide 136 text

HERE IS PROBABLY THE CLOSEST EXAMPLE… USING ~300+ BILLION FINANCE TOKENS**

Slide 137

Slide 137 text

https://www.bloomberg.com/company/press/ bloomberggpt-50-billion-parameter-llm-tuned- fi nance/ https://arxiv.org/pdf/2303.17564.pdf

Slide 138

Slide 138 text

NOT ONLY COMPETITION AMONG FOUNDATIONAL MODELS IMPLICATION BUT ALSO VERTICALIZED GPTS (E.G. LAW GPTS) (GPT, CLAUDE , GEMINI / BARD, ETC.)

Slide 139

Slide 139 text

No content

Slide 140

Slide 140 text

APRIL 6, 2023 INFO SEC, GEN AI POLICIES AND THE RISE PRIVATE INSTANCES #4 OF 11

Slide 141

Slide 141 text

https://mashable.com/article/samsung-chatgpt-leak-details

Slide 142

Slide 142 text

THE ROGUE ASSOCIATE (OR PARTNER) PASTING STUFF INTO GPT OR ANOTHER LLM MODEL LAW VERSION OF THE CONCERN

Slide 143

Slide 143 text

FUD VS FEAR, UNCERTAINTY, & DOUBT FEAR OF MISSING OUT FOMO ORGANIZATIONS / INDIVIDUALS TOGGLE BETWEEN THESE TWO POINTS OF VIEW (SOMETIME WITHIN THE SAME CONVERSATION)

Slide 144

Slide 144 text

ORGANIZATIONS LAUNCH PRIVATE INSTANCES OF GPT-4, ETC.

Slide 145

Slide 145 text

https://legaltechtrends.substack.com/p/legal-tech-trends-newsletter-14 ORGANIZATIONS LAUNCH PRIVATE INSTANCES OF GPT-4, ETC.

Slide 146

Slide 146 text

FEAR / DOUBT IS ULTIMATELY GOING TO BE MANAGED AS THESE CAPABILITIES BECOME BAKED INTO ENTERPRISE GRADE OFFERINGS IMPLICATION

Slide 147

Slide 147 text

No content

Slide 148

Slide 148 text

MAY 27, 2023 THE NEW YORK CHATGPT LAWYER #5 OF 11

Slide 149

Slide 149 text

No content

Slide 150

Slide 150 text

BECAUSE OF THE POTENTIAL FOR MODEL HALLUCINATION, ETC.

Slide 151

Slide 151 text

IT IS IMPORTANT TO USE THESE TOOLS (AT LEAST FOR NOW) AS PART OF A HUMAN IN THE LOOP PROCESS

Slide 152

Slide 152 text

ONE WAY TO DRAMATICALLY REDUCE THE LIKELIHOOD OF A HALLUCINATION IS TO ANCHOR OUTPUT AGAINST SOME ‘GROUND TRUTH’ …

Slide 153

Slide 153 text

ONE SHOT ENHANCE THIS QUERY USING CONTEXTUALLY RELEVANT INFORMATION ORIGINAL QUERY OUTPUT* (RETRIEVAL AUGMENTATION) (COULD BE CONSTRUCTED ON VIA ‘PROMPT ENGINEERING’) ENTERED INTO A FOUNDATIONAL MODEL (VARIOUS TECHNIQUES ARE POSSIBLE SUCH AS USING EMBEDDINGS AND STORED IN A VECTOR DATABASE) ( QUERY & CONTEXTUAL INFO) REVISED QUERY **NOTE OUTPUT CAN BE FURTHER ENHANCED IN FEW SHOT EXAMPLES

Slide 154

Slide 154 text

THERE IS A BROADER ENGINEERING ARC WHICH WE WILL SEE AS TOOLS / OFFERINGS MATURE

Slide 155

Slide 155 text

LAYERS OF ENGINEERING ON TOP OF LLMS IS KEY TO TRULY UNLOCK COMPLEX VERTICALS LIKE LAW IMPLICATION

Slide 156

Slide 156 text

No content

Slide 157

Slide 157 text

JUNE 26, 2023 CASETEXT & THOMSON REUTERS DEAL #6 OF 11

Slide 158

Slide 158 text

https://techcrunch.com/2023/06/26/ thomson-reuters-buys-casetext-an-ai- legal-tech-startup-for-650m-in-cash/

Slide 159

Slide 159 text

EVERY STARTUP IN THIS ROOM OWES A DEBT OF GRATITUDE TO CASETEXT FOR SETTING A FAIRLY LARGE PUBLIC NUMBER

Slide 160

Slide 160 text

MASS APPEAL PRODUCT FIN(LEGAL)TECH DOCUSIGN LEGALZOOM LIT FINANCE RISK MGMT BUT EVEN LARGER $$ ARE POSSIBLE FOR THIS SPACE

Slide 161

Slide 161 text

MORE BROADLY, I THINK MANY LEGACY LEGALTECH PROVIDERS ARE GOING TO NEED HELP ON THEIR GEN AI TRANSITION

Slide 162

Slide 162 text

EVENTUALLY SOME WILL GIVE UP ON THEIR HALF BAKED INTERNAL EFFORTS AND SEEK TO INNOVATE VIA ACQUISITION

Slide 163

Slide 163 text

MORE M&A TRANSACTIONS ARE COMING IN 2024 IMPLICATION

Slide 164

Slide 164 text

No content

Slide 165

Slide 165 text

JULY 18, 2023 LLAMA 2 RELEASE #7 OF 11

Slide 166

Slide 166 text

No content

Slide 167

Slide 167 text

No content

Slide 168

Slide 168 text

WILL OPEN SOURCE MODEL ULTIMATELY MATCH OR EXCEED PERFORMANCE OF LEADING MODELS ?

Slide 169

Slide 169 text

THIS REMAINS TO BE SEEN …

Slide 170

Slide 170 text

OPEN SOURCE LLMS ARE IN A SERIOUS HORSE RACE WITH OPENAI, ETC. IMPLICATION SIGNIFICANT IMPLICATIONS FOR INVESTMENT, ENGINEERING, ETC.

Slide 171

Slide 171 text

No content

Slide 172

Slide 172 text

SEPTEMBER 15, 2023 GPT USE IN KNOWLEDGE WORK THE AGE OF CENTAUR CHESS #8 OF 11

Slide 173

Slide 173 text

BIG PICTURE HERE IS A BRAND NEW STUDY WHICH HIGHLIGHTS BOTH THE POTENTIAL PRODUCTIVITY AND QUALITY GAINS IN PROFESSIONAL SERVICES

Slide 174

Slide 174 text

“Our study conducted with Boston Consulting Group, a global management consulting fi rm, we examine the performance implications of AI on realistic, complex, and knowledge- intensive tasks. … [The] experiment involved 758 consultants … subjects were randomly assigned to one of three conditions: no AI access, GPT-4 AI access, or GPT-4 AI access with a prompt engineering overview. For each one of a set of 18 realistic consulting tasks within the frontier of AI capabilities, consultants using AI were signi fi cantly more productive (they completed 12.2% more tasks on average, and completed tasks 25.1% more quickly), and produced signi fi cantly higher quality results (more than 40% higher quality compared to a control group).” https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321

Slide 175

Slide 175 text

CENTAUR CHESS HUMAN + MACHINE HUMAN OR MACHINE >

Slide 176

Slide 176 text

AND REMEMBER THIS STUDY ONLY CONSIDERS GPT-4 WITH PROMPTING IMPLICATION GREATER PERFORMANCE WILL FOLLOW WHEN ENGINEERING ENHANCEMENTS ARE ADDED SIGNIFICANT QUALITY AND EFFICIENCY RESULTS

Slide 177

Slide 177 text

No content

Slide 178

Slide 178 text

NOVEMBER 6, 2023 OPEN AI DEV DAY AGENTS, AGENTS, AGENTS #9 OF 11

Slide 179

Slide 179 text

No content

Slide 180

Slide 180 text

GENERATIVE AGENTS

Slide 181

Slide 181 text

No content

Slide 182

Slide 182 text

WE HAVE SEEN QUITE A BIT OF THIS SINCE THE LAUNCH OF CHATGPT

Slide 183

Slide 183 text

WHAT IS THE PRODUCT VALUE ADD OVER BASELINE CAPABILITIES? TRUST BUT VERIFY THERE IS GOING TO BE SIGNIFICANT PRESSURE ON LEGAL TECH COMPANIES TO OVERSTATE THE ACTUAL CONTRIBUTION OF THEIR OFFERING ABOVE JUST AN LLM API CALL I SUGGEST YOU TRUST BUT VERIFY ALL CLAIMS

Slide 184

Slide 184 text

IMPLICATION MORE INNOVATION MORE DISRUPTION MORE NEED FOR ADAPTATION AND THIN PRODUCTS BUILT ON GPT WILL LIKELY NEED TO PIVOT …

Slide 185

Slide 185 text

No content

Slide 186

Slide 186 text

NOVEMBER 17, 2023 SUPPLY CHAIN RISK FOR THE WORLD OF GEN AI #10 OF 11

Slide 187

Slide 187 text

NOVEMBER 17 2023 https://www.nytimes.com/2023/11/17/technology/openai-sam-altman-ousted.html

Slide 188

Slide 188 text

No content

Slide 189

Slide 189 text

NOVEMBER 22, 2023 https://www.nytimes.com/2023/11/22/technology/openai-sam-altman-returns.html

Slide 190

Slide 190 text

THIS AMONG OTHER THINGS HIGHLIGHTS SUPPLY CHAIN RISK IN AI …

Slide 191

Slide 191 text

BUT ALSO A MORE FUNDAMENTAL TRUTH …

Slide 192

Slide 192 text

K-LLMS WHERE K IS SOME NUMBER GREATER THAN ONE THIS WAS ALWAYS GOING TO BE ABOUT MIXING AND MATCHING MODELS (FANCY WAY TO SAY MIX AND MATCH MODELS)

Slide 193

Slide 193 text

IMPLICATION COMPANIES NEED TO BE ABLE TO ADAPT AS TECH LANDSCAPE RAPIDLY EVOLVES COMPANIES WITH A SINGLE SOURCE LLM STRATEGY ARE AT REAL RISK

Slide 194

Slide 194 text

No content

Slide 195

Slide 195 text

FINAL THOUGHTS

Slide 196

Slide 196 text

ELEVEN DAYS THAT HELPED DEFINE THE PRESENT AND FUTURE FOR GEN (LEGAL) A.I.

Slide 197

Slide 197 text

SO FAR I HAVE MENTIONED TEN OF THE DATES THAT WILL HELP DEFINE THE FUTURE FOR OUR FIELD…

Slide 198

Slide 198 text

APRIL 6, 2023 JUNE 26, 2023 JULY 18, 2023 NOVEMBER 6, 2023 NOVEMBER 17, 2023 MARCH 30, 2023 SEPTEMBER 15, 2023 MAY 27, 2023 SAM ALTMAN FIRED OPEN AI DEV DAY LLAMA 2 RELEASE HBS STUDY RELEASE CASETEXT + TR DEAL NY CHATGPT LAWYER BLOOMBERG GPT RELEASE SAMSUNG CODE INCIDENT SUMMARY OF TEN KEY DATES FOR THE YEAR MARCH 15, 2023 GPT-4 RELEASE DAY NOVEMBER 30, 2022 CHATGPT RELEASE DAY

Slide 199

Slide 199 text

HIGHLIGHTS AI SUPPLY CHAIN RISKS OPEN SOURCE MODELS MIGHT WIN BIG GAINS IN PROFESSIONAL SERVICES MORE M&A ACTIVITY TO FOLLOW NEED LAYERS ON TOP BASE MODELS BLOOMBERG GPT SAMSUNG INCIDENT GPT-4 RELEASE 11.30.22 CHATGPT RELEASE GPT GOES MAINSTREAM 3.15.23 3.30.23 RISE OF VERTICALIZED LLMS GPT-4 IS A SERIOUS UPGRADE 4.06.23 ENTERPRISE DEPLOYMENT NEEDED SAM ALTMAN FIRED OPEN AI DEV DAY LLAMA 2 RELEASE HBS STUDY RELEASE CASETEXT + TR DEAL NY CHATGPT LAWYER TEN DAYS AND THEIR IMPLICATIONS 5.27.23 6.26.23 7.18.23 9.15.23 11.6.23 11.17.23 AGENTS - MORE INNOVATION COMING

Slide 200

Slide 200 text

BUT I THINK YOU ALREADY KNOW THE ELEVENTH DAY THAT IS GOING TO DEFINE THE FUTURE …

Slide 201

Slide 201 text

DECEMBER 4, 2023 BECAUSE THAT DAY IS TODAY #11 OF 11

Slide 202

Slide 202 text

WHAT AN AMAZING COLLECTION OF INDIVIDUALS AND ORGANIZATIONS …

Slide 203

Slide 203 text

THIS IS INDEED GOING TO BE THE ‘ROOM WHERE IT HAPPENS’ …

Slide 204

Slide 204 text

BECAUSE ALL OF YOU ARE GOING TO MAKE IT HAPPEN …

Slide 205

Slide 205 text

No content

Slide 206

Slide 206 text

NO DOUBT THIS HAS BEEN THE MOST EXCITING YEAR IN TECHNOLOGY IN A VERY LONG TIME

Slide 207

Slide 207 text

HERE WE GO BUT THE BEST IS YET TO COME ! !

Slide 208

Slide 208 text

No content

Slide 209

Slide 209 text

No content

Slide 210

Slide 210 text

No content

Slide 211

Slide 211 text

273VENTURES.COM KELVIN.LEGAL

Slide 212

Slide 212 text

https://www.linkedin.com/in/daniel-katz-3b001539/

Slide 213

Slide 213 text

G E N E R AT I V E A . I . + L AW @ computational professor daniel martin katz danielmartinkatz.com Illinois tech - chicago kent law 273Ventures.com Y E A R I N R E V I E W I N C LU D I N G G P T PA S S E S T H E B A R & E L E V E N DAY S T H AT D E F I N E D T H E Y E A R I N ( L E G A L ) A I https://bit.ly/3yxhY2e