Generative A.I. + Law - Background, Applications and Use Cases Including GPT-4 Passes the Bar Exam

Professor Daniel Martin Katz - Generative A.I. + Law - Background, Applications and Use Cases Including GPT-4 Passes the Bar Exam - Topics include some History of NLP, The Path to Generative A.I. (including ChatGPT and GPT-4), Some Applications / Use Cases in the Legal Sector, GPT-4 Passes the Bar Exam, Legal A.I. Operational Issues and the Path Forward ... (Updated 06.14.23)

Transcript

1. G E N E R AT I V E A . I . + L AW
   @ computational
   professor daniel martin katz danielmartinkatz.com
   Illinois tech - chicago kent law 273Ventures.com
   B AC KG R O U N D, A P P L I CAT I O N S A N D U S E CA S E S
   I N C LU D I N G G P T- 4 PA S S E S T H E B A R E X A M
   https://bit.ly/3yxhY2e
   

   View Slide

2. NOVEMBER 30, 2022
   

   View Slide

3. I THINK THIS DAY
   WILL GO DOWN AS
   A VERY IMPORTANT
   DAY IN THE HISTORY
   OF TECHNOLOGY
   

   View Slide

4. NOVEMBER 30, 2022
   IS THE DAY THAT CHATGPT
   WAS FIRST RELEASED
   BECAUSE
   

   View Slide

5. THIS WAS THE
   BEGINNING OF THE
   BROADER PUBLIC’S
   AWARENESS …
   

   View Slide

6. OF WHAT WAS
   ALREADY TRUE …
   

   View Slide

7. WHICH IS
   
   
   THAT IN THE
   BACKGROUND
   

   View Slide

8. THERE HAVE BEEN
   FUNDAMENTAL
   IMPROVEMENTS
   OCCURING WITHIN
   THE WORLD OF
   NATURAL LANGUAGE
   PROCESSING (NLP)
   

   View Slide

9. View Slide

10. I PERSONALLY HAVE
    BEEN WORKING IN THIS
    AREA FOR A WHILE …
    

    View Slide

11. THINKING ABOUT HOW
    TO USE NLP MODELS IN
    THE LEGAL DOMAIN
    

    View Slide

12. View Slide

13. https://sites.google.com/view/nllp/nllp-2019
    

    View Slide

14. https://law.mit.edu/pub/openedgar/release/1
    

    View Slide

15. https://aclanthology.org/2022.acl-long.297.pdf
    

    View Slide

16. https://speakerdeck.com/danielkatz/legal-nlp-breaking-the-legal-language-barrier
    

    View Slide

17. BUT IT HAS BEEN HARD
    TO COMMUNICATE TO
    MANY FOLKS IN THE
    LEGAL SECTOR …
    

    View Slide

18. ABOUT THE
    NATURE OF THE
    LANDSCAPE SHIFT
    IN THE FIELD …
    

    View Slide

19. MAYBE IT IS
    
    
    ROBOLAWYER
    
    
    FATIUGE …?
    

    View Slide

20. View Slide

21. MAYBE IT IS THE
    ABSENCE OF A CLEAR
    DEMONSTRATION
    PROJECT …
    

    View Slide

22. View Slide

23. SO NOW WE ARE JUST A FEW
    MONTHS INTO WHAT IS
    ARGUABLY THE MOST
    SUCCESSFUL PRODUCT
    LAUNCH IN HISTORY …
    

    View Slide

24. View Slide

25. TODAY —
    
    
    I WOULD LIKE TO DISCUSS
    WHAT ALL OF THESE
    DEVELOPMENTS …
    

    View Slide

26. MIGHT MEAN FOR
    THE DELIVERY OF
    LEGAL SERVICES …
    

    View Slide

27. View Slide

28. SO FOR MY OPENING
    STATEMENT PLEASE TAKE
    INTO ACCOUNT THE
    FOLLOWING FOUR
    CONSIDERATIONS…
    

    View Slide

29. View Slide

30. < CONSIDERATION 1 >
    LANGUAGE IS THE ‘COIN OF THE
    REALM’ HERE IN THE WORLD OF LAW
    < CONSIDERATION 2 >
    THE MARCH OF LEGAL COMPLEXITY
    < CONSIDERATION 3 >
    MACHINES ARE INCREASINGLY
    IMPROVING IN LANGUAGE PROCESSING
    < CONSIDERATION 4 >
    LEGAL LANGUAGE != REGULAR LANGUAGE ?
    

    View Slide

31. View Slide

32. < CONSIDERATION 1 >
    

    View Slide

33. LANGUAGE IS THE
    
    
    ‘COIN OF THE REALM’
    HERE IN THE
    
    
    WORLD OF LAW
    

    View Slide

34. View Slide

35. VIRTUALLY ALL ROADS IN
    LAW LEAD TO A DOCUMENT
    (CONSUMPTION, PRODUCTION OR BOTH)
    

    View Slide

36. View Slide

37. LAWYERS, JUDGES AND
    REGULATORS ARE MASSIVE
    PRODUCERS OF TEXT
    

    View Slide

38. BRIEFS,
    
    
    MEMOS,
    
    
    STATUTES,
    
    
    OPINIONS,
    
    
    REGULATIONS,
    
    
    CONTRACTS,
    
    
    ETC.
    

    View Slide

39. ARE JUST SOME OF THE
    LEGAL WORK PRODUCT …
    

    View Slide

40. PRODUCED ON A DAILY
    BASIS ACROSS THE
    WORLD’S VARIOUS
    LEGAL SYSTEMS
    

    View Slide

41. View Slide

42. NOT ONLY IS IT THE SHEER
    VOLUME OF TEXT BUT
    ALSO THESE MASSIVE
    VOLUMES OF TEXT ARE
    NOTORIOUSLY COMPLEX
    

    View Slide

43. View Slide

44. < CONSIDERATION 2 >
    

    View Slide

45. THE MARCH OF
    LEGAL COMPLEXITY
    

    View Slide

46. W H E T H E R YO U A R E A
    L A R G E M U LT I N AT I O N A L
    CO R P O R AT I O N , A
    S M A L L B U S I N E S S O R A N
    I N D I V I D UA L C I T I Z E N …
    

    View Slide

47. L AW H A S A
    CO M P L E X I TY
    C H A L L E N G E …
    

    View Slide

48. IF YOU TAKE A STEP BACK
    

    View Slide

49. AND LOOK AT THE UNIT
    ECONOMICS OF LAW
    

    View Slide

50. THE REAL ANIMATING
    DRIVER IN THE DEMAND
    FOR UNITS OF LEGAL
    PRODUCTION
    

    View Slide

51. IS COMPLEXITY
    

    View Slide

52. SOCIAL, ECONOMIC AND
    POLITICAL COMPLEXITY
    

    View Slide

53. MANIFESTS ITSELF IN
    OUR DOMAIN AS
    LEGAL COMPLEXITY
    

    View Slide

54. WHILE IT IS
    SOMEWHAT HARD TO
    PRECISELY DEFINE …
    

    View Slide

55. View Slide

56. VIRTUALLY EVERY WAY
    YOU MIGHT CONSIDER IT
    LEGAL COMPLEXITY
    
    
    HAS GROWN
    

    View Slide

57. DANIEL MARTIN KATZ, CORINNA
    COUPETTE, JANIS BECKEDORF & DIRK
    HARTUNG, COMPLEX SOCIETIES AND THE
    GROWTH OF THE LAW, 10 SCIENTIFIC
    REPORTS 18737 (2020)
    CORINNA COUPETTE, JANIS BECKEDORF,
    DIRK HARTUNG, MICHAEL BOMMARITO, &
    DANIEL MARTIN KATZ, MEASURING LAW
    OVER TIME: A NETWORK ANALYTICAL
    FRAMEWORK WITH AN APPLICATION TO
    STATUTES AND REGULATIONS IN THE
    UNITED STATES AND GERMANY, FRONT.
    PHYS. (2021 FORTHCOMING)
    

    View Slide

58. SIGNIFICANT GROWTH IN LAWS AND REGULATIONS
    

    View Slide

59. KEY TAKE AWAY -
    
    
    OVER PAST TWO DECADES
    
    
    ~50%+ MORE STATUTES
    
    
    ~200% MORE REGULATIONS
    

    View Slide

60. SO THE QUESTION IS
    HOW TO MATCH THAT
    COMPLEXITY WITH THE
    APPROPRIATE
    

    View Slide

61. COMPLEXITY
    MITIGATION
    TACTICS …
    

    View Slide

62. MIXTURE OF PEOPLE,
    PROCESS AND
    TECHNOLOGY
    

    View Slide

63. DESIGNING MORE
    SCALABLE SYSTEMS
    

    View Slide

64. http://prawfsblawg.blogs.com/prawfsblawg/2017/03/-complexity-mitigation-strategies-
    for-law-law-land-and-beyond-and-some-other-thoughts-on-had
    fi
    eld-su.html#more
    SOME THOUGHTS ON
    
    
    LEGAL COMPLEXITY
    MITIGATION STRATEGIES
    

    View Slide

65. View Slide

66. < CONSIDERATION 3 >
    

    View Slide

67. MACHINES ARE
    INCREASINGLY
    IMPROVING IN
    THEIR ABILITY TO
    PROCESS AND
    ‘UNDERSTAND’
    LANGUAGE
    

    View Slide

68. INITIAL PERFORMANCE AND HUMAN PERFORMANCE ARE
    NORMALIZED TO -1 AND 0 RESPECTIVELY (KIELA ET AL., 2021).
    

    View Slide

69. AND LEADING
    MODELS ARE RAPIDLY
    GROWING IN SCALE
    

    View Slide

70. SIZE OF LARGE LANGUAGE MODELS IN
    BILLIONS OF PARAMETERS
    

    View Slide

71. THE IMPLICATION IS
    THAT PREVIOUSLY
    UNTOUCHABLE
    PROBLEMS WILL
    BEGIN TO COME
    WITHIN REACH …
    

    View Slide

72. View Slide

73. BECAUSE COMPLEXITY
    
    
    IS ARGUABLY SCALING
    FASTER THAN AVAILABLE
    RESOURCES …
    

    View Slide

74. WE ARGUABLY NEED A
    FORCE MULTIPLIER …
    

    View Slide

75. AND LANGUAGE IS
    CORE TO MOST (ALL)
    LEGAL TASKS
    

    View Slide

76. THE FACT THAT LANGUAGE
    TECHNOLOGY HAS BEEN
    MATERIALLY IMPROVING
    MIGHT BODE VERY WELL …
    

    View Slide

77. AS THE FORCE
    MULTIPLIER,
    PRODUCTIVITY
    ENHANCER …
    

    View Slide

78. THE FIELD
    ARGUABLY
    NEEDS …
    

    View Slide

79. TO SUPPORT THE
    QUANTITY, QUALITY,
    AND ACCESSIBILITY
    OF LEGAL SERVICES
    DEMANDED BY
    SOCIETY
    

    View Slide

80. View Slide

81. < CONSIDERATION 4 >
    

    View Slide

82. SO WHAT ABOUT
    LEGAL LANGUAGE ?
    

    View Slide

83. LETS BE CLEAR —
    
    
    FOR AI / NLP TO
    MAKE A DEEP
    INCURSION INTO
    THIS FIELD …
    

    View Slide

84. IT NEEDS TO BE ABLE TO
    MORE FUNDAMENTALLY
    ENGAGE WITH LANGUAGE
    

    View Slide

85. AND NOT JUST
    ANY LANGUAGE …
    

    View Slide

86. BUT LEGAL LANGUAGE
    

    View Slide

87. THE PUBLIC HAS A
    DIM VIEW OF LEGAL
    LANGUAGE …
    

    View Slide

88. THEY DESCRIBE LEGAL
    DOCUMENTS AND
    ARGUMENTS USING
    TERMS SUCH AS
    
    
    ‘LEGALESE’
    
    
    ‘LEGAL JARGON’
    
    
    ‘LEGAL GOBBLEDYGOOK’
    

    View Slide

89. View Slide

90. LEGAL ENGLISH
    
    
    !=
    
    
    ENGLISH
    (AND THIS IS TRUE FOR MANY
    OTHER LANGUAGES)
    

    View Slide

91. INDEED MANY
    LAW STUDENTS
    CONSIDER THEIR
    INITIAL FORAY
    INTO THE FIELD …
    

    View Slide

92. AS AN
    EXERCISE IN
    ‘LEARNING A
    NEW
    LANGUAGE’
    

    View Slide

93. IN SUPPORT OF
    THIS TASK …
    

    View Slide

94. LAW FEATURES
    SPECIALIZED
    DICTIONARIES
    

    View Slide

95. TEXT BASED
    SUMMARIES
    

    View Slide

96. AND MANY
    OTHER
    RESOURCES
    DESIGNED TO
    SUPPORT …
    

    View Slide

97. THE DEVELOPMENT OF
    THE LINGUISTIC
    IMMERSION PROGRAM
    THAT WE CALL
    #LEGALEDUCATION
    

    View Slide

98. View Slide

99. NLP MODELS HAVE BEEN
    DEPLOYED IN THE LEGAL
    SECTOR …
    

    View Slide

100. BUT THE QUALITY AND
     PERFORMANCE HAS
     BEEN MIXED …
     

     View Slide

101. FRANKLY *NO* LEGAL
     TECH SOLUTION BUILT
     TO DATE HAS BEEN ABLE
     TO REALLY WORK WELL
     WITH NUANCES OF
     LEGAL LANGUAGE …
     

     View Slide

102. BUT THAT MIGHT BE
     ABOUT TO CHANGE …
     

     View Slide

103. View Slide

104. TODAY I WILL
     DIVIDE THE
     REMAINING
     PRESENTATION
     INTO FIVE
     PARTS
     

     View Slide

105. (1) THE (LONG) PATH TO GENERATIVE A.I.
     
     
     (2) WHAT IS GPT / CHATGPT / GPT-4 ?
     
     
     (3) BAR EXAM AS WINDOW INTO CAPABILITIES
     
     
     (4) LLMs IN THE DELIVERY OF LEGAL SERVICES
     
     
     (5) STRATEGIC CONSIDERATIONS AND THE ROAD AHEAD
     
     
     PRESENTATION IN
     
     
     FIVE PARTS
     

     View Slide

106. View Slide

107. (1)
     
     
     THE (LONG) PATH TO
     GENERATIVE A.I.
     

     View Slide

108. IN ORDER TO
     UNDERSTAND
     WHERE THIS MIGHT
     ALL BE GOING …
     

     View Slide

109. IT IS USEFUL TO
     UNDERSTAND THE
     ARC THAT HAS
     BROUGHT US TO
     THIS MOMENT …
     

     View Slide

110. SO LETS GO BACK TO
     THE EARLY DAYS …
     

     View Slide

111. BUT BEFORE WE
     TALK ABOUT
     ANYTHING TOO
     FANCY …
     

     View Slide

112. I WOULD LIKE TO
     REMIND EVERYONE …
     

     View Slide

113. BEFORE THERE
     WERE COMPUTERS
     

     View Slide

114. HUMANS ARE THE
     ORIGINAL COMPUTERS
     

     View Slide

115. HUMANS ONCE DID ALL
     OF THE COMPUTING
     

     View Slide

116. OVER THE COURSE
     OF MANY YEARS WE
     WERE ABLE TO
     LEVERAGE
     ALTERNATIVE FORMS
     OF COMPUTATION …
     

     View Slide

117. View Slide

118. THE FIELD OF
     
     
     ARTIFICIAL
     INTELLIGENCE
     
     
     IS ROUGHLY
     
     
     75 YEARS OLD …
     

     View Slide

119. “BEFORE 1949, ‘COMPUTERS LACKED A
     KEY PREREQUISITE FOR INTELLIGENCE:
     THEY COULDN’T STORE COMMANDS,
     ONLY EXECUTE THEM …”
     http://sitn.hms.harvard.edu/
     fl
     ash/2017/history-arti
     fi
     cial-intelligence/
     

     View Slide

120. SO WHAT IS
     
     
     ‘ARTIFICIAL
     INTELLIGENCE’ ?
     

     View Slide

121. View Slide

122. View Slide

123. BIG IDEA IN AI IS TO DEVELOP
     IN MACHINES SOME LEVEL OF
     SYNTHETIC (OR ARTIFICIAL)
     REPRESENTATION OF A
     PREVIOUSLY HUMAN
     CENTERED PROCESS
     

     View Slide

124. ARTIFICIAL INTELLIGENCE IS A BROAD FIELD
     

     View Slide

125. View Slide

126. EVEN IN THE
     FOUNDATIONAL DAYS
     THERE WAS A VIEW
     ABOUT THE ROLE OF A
     ‘CONVERSATIONAL
     AGENT’ AS A MEANS TO
     EVALUATE THE QUALITY
     OF AN AI SYSTEM
     

     View Slide

127. WHICH IS TO SAY THAT
     LANGUAGE HAS ALWAYS
     BEEN A CORE TOPIC FOR
     THE FIELD OF A.I.
     

     View Slide

128. ALAN TURING IS
     ARGUABLY THE
     FATHER OF THE
     FIELD OF
     COMPUTER
     SCIENCE
     

     View Slide

129. AND THIS IDEA OF
     CONVERSATIONAL AGENT
     COMES FROM HIM …
     

     View Slide

130. NOTE THE MOVIE
     WAS MOSTLY
     ABOUT CRACKING
     ENIGMA BUT HERE
     IS WHERE IT GETS
     ITS TITLE
     THE TURING TEST
     

     View Slide

131. View Slide

132. IT IS IMPORTANT TO
     REMEMBER WITH TOPICS
     SUCH AS ARTIFICIAL
     INTELLIGENCE AND
     NATURAL LANGUAGE
     PROCESSING …
     

     View Slide

133. THE FIELD OF A.I. HAS
     HAD MANY FALSE STARTS
     AND A.I. WINTERS
     

     View Slide

134. EVEN SOME OF THE
     WORLD’S LEADING
     EXPERTS HAVE
     GOTTEN THINGS
     WRONG …
     

     View Slide

135. “MACHINES WILL BE CAPABLE,
     WITHIN TWENTY YEARS, OF DOING
     ANY WORK THAT A MAN CAN DO.”
     – Herbert Simon
     in 1965
     

     View Slide

136. “MACHINES WILL BE CAPABLE,
     WITHIN TWENTY YEARS, OF DOING
     ANY WORK THAT A MAN CAN DO.”
     – Herbert Simon
     in 1965
     did not pan out by 1985 but
     did turn out okay for him …
     

     View Slide

137. SO I MENTION THIS ONLY
     SO THAT WE GROUND
     OURSELVES A BIT …
     

     View Slide

138. View Slide

139. A.I. BROADLY HAS MADE
     MAJOR PROGRESS ON
     MANY TOPICS …
     

     View Slide

140. View Slide

141. MACHINE LEARNING
     

     View Slide

142. View Slide

143. MACHINE LEARNING HAS
     BEEN USED TO UNDERTAKE
     VARIOUS FORMS OF
     PREDICTION IN LAW
     

     View Slide

144. EXAMPLE
     
     
     LITIGATION
     
     
     PREDICTION
     

     View Slide

145. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174698
     Katz DM, Bommarito MJ II, Blackman J (2017), A General
     Approach for Predicting the Behavior of the Supreme Court
     of the United States. PLoS ONE 12(4): e0174698.
     

     View Slide

146. http://www.sciencemag.org/news/
     2017/05/arti
     fi
     cial-intelligence-prevails-
     predicting-supreme-court-decisions
     Professor Katz noted
     that in the long term …
     “We believe the blend
     of experts, crowds, and
     algorithms is the secret
     sauce for the whole
     thing.”
     May 2nd 2017
     

     View Slide

147. https://www.wsj.com/articles/hidden-
     in-plain-sight-a-powerful-way-to-
     beat-the-market-1497367597
     FT Big Read
     
     
     Feb 7, 2019
     

     View Slide

148. http://arxiv.org/abs/1508.05751
     available at
     http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
     

     View Slide

149. View Slide

150. LITIGATION FUNDING
     

     View Slide

151. View Slide

152. View Slide

153. View Slide

154. “Lawyers say the real value in mediation and arbitration might in the future
     come from large-scale data analysis of arbitrators and mediators themselves, in
     an effort to predict outcomes and potentially affect the course of settlements
     … Matthew Saunders, partner at Ashurst, notes that data analytics “could be
     extended to predicting which way arbitrators or a mediator might go”.
     
     
     Such technology may yet be some way off. In mediation, “a skilled facilitator
     helps the parties to explore where common ground can be found as the basis
     for an amicable settlement,” says James Freeman, arbitration partner at Allen &
     Overy. “The mediation process”, he adds, “is inherently a human one”.
     

     View Slide

155. DATA —> VARIABLES
     MODELS (STATISTICAL)
     OUTPUTS WITH VALIDATION
     

     View Slide

156. MUCH MORE COULD BE SAID
     ON THE LEGAL PREDICTION
     SIDE OF THIS …
     

     View Slide

157. View Slide

158. TODAY I AM MOSTLY GOING
     TO FOCUS ON GENERATIVE
     LANGUAGE MODELS …
     

     View Slide

159. A.I. IS A VERY BROAD FIELD
     

     View Slide

160. AND OF COURSE THERE ARE
     OTHER GENERATIVE MODELS
     MY SON AND I RECENTLY CREATED ‘BROCCOLI MAN’
     GENERATIVE A.I. ART
     

     View Slide

161. View Slide

162. HISTORICALLY LANGUAGE
     HAS ALWAYS LAGGED
     OTHER ADVANCES IN A.I. …
     

     View Slide

163. BECAUSE OF THE
     UNDERLYING
     STATISTICAL
     DEPENDENCIES -
     LANGUAGE IS A
     *HARD* PROBLEM …
     

     View Slide

164. YET MAJOR ADVANCES
     ARE FINALLY COMING
     TO MARKET …
     

     View Slide

165. View Slide

166. HISTORICALLY NLP
     (LIKE THE REST OF A.I.)
     WAS RULES-BASED
     

     View Slide

167. RULES-BASED A.I. WAS
     ACTUALLY THE ONLY
     OPTION AT ONE POINT
     

     View Slide

168. Moore’s law
     !
     

     View Slide

169. Kryder’s law
     !
     

     View Slide

170. EVENTUALLY COMPUTER
     CHIPS WERE FAST
     ENOUGH AND DATA
     STORAGE WAS CHEAP
     ENOUGH …
     

     View Slide

171. THAT NLP COULD REALLY
     EMBRACE STATISTICAL
     MODELING …
     

     View Slide

172. NLP SLOWLY BECOMES A
     STATISTICAL FIELD IN THE
     1990’S AND 2000’S
     

     View Slide

173. MARK LIBERMAN, "THE TREND
     TOWARDS STATISTICAL MODELS IN
     NATURAL LANGUAGE
     PROCESSING." NATURAL
     LANGUAGE AND SPEECH:
     SYMPOSIUM PROCEEDINGS
     BRUSSELS, SPRINGER, (1991)
     

     View Slide

174. BUT THE PRIMARY EFFORT
     HERE WAS ON SYNTAX
     BASED APPROACHES …
     

     View Slide

175. “There have been a series of
     clever approaches to
     backdoor into semantics* …
     (*while also being scalable)
     Semantic Methods
     (Fairly Difficult)
     Syntax Methods
     (Fairly Easy)
     Historically, Big Divide between Semantics and Syntax
     Quasi-Semantic
     Methods
     

     View Slide

176. View Slide

177. SYNTAX
     

     View Slide

178. WORDS
     
     
     WORD FREQUENCY
     
     
     PART OF SPEECH FREQUENCY
     
     
     ETC.
     

     View Slide

179. CTRL + F IS EXACT
     ‘STRING’ MATCHING
     

     View Slide

180. REGULAR EXPRESSION
     
     
     (REGEX)
     RULES-BASED METHOD(S) THAT CAN BE
     USED TO LOOK FOR WORD PATTERNS AND
     RETURN RESULTS FOR THOSE PATTERNS
     

     View Slide

181. TF - IDF
     TERM
     FREQUENCY
     INVERSE
     DOCUMENT
     FREQUENCY
     EXPLOITS THE FREQUENCY OF
     WORDS IN DOCUMENTS IN
     ORDER TO PROFILE THEM
     

     View Slide

182. View Slide

183. SEMANTICS
     

     View Slide

184. SEMANTICS IS ABOUT THE
     MEANING OF INDIVIDUAL WORDS
     SEMANTICS IS RELATIONSHIP
     BETWEEN WORDS THAT
     INTERACT TO PRODUCE
     HIGHER ORDER MEANING
     

     View Slide

185. LOTS OF THE TRAINING FOR
     LAWYERS IS ACTUALLY ABOUT THE
     DEEP SEMANTIC INTERPRETATION
     OF LANGUAGE
     CONTRACTS
     
     
     STATUTES
     
     
     REGULATIONS
     
     
     JUDICIAL DECISIONS
     EXAMPLES —>
     

     View Slide

186. SEMANTICS IS
     HARD FOR
     MACHINES
     

     View Slide

187. View Slide

188. BUT THE PAST
     DECADE THERE
     HAS BEGUN TO BE
     PROGRESS ON
     THIS FRONT …
     

     View Slide

189. WE HAVE SEEN THE
     EMERGENCE OF
     
     
     QUASI-SEMANTIC
     METHODS
     

     View Slide

190. QUASI-SEMANTIC
     METHODS WHICH CAN
     INCREASINGLY TAKE
     CONTEXT INTO ACCOUNT
     

     View Slide

191. WHAT WE SEE TODAY ARE
     FOLKS USING DEEP
     NEURAL NETWORKS TO
     WORK ON NLP PROBLEMS
     

     View Slide

192. GENERATIVE A.I. IN THE
     LINGUISTIC CONTEXT IS A MASH
     UP OF DEEP LEARNING + NLP
     

     View Slide

193. CLASSIC VS NEURAL NLP
     

     View Slide

194. View Slide

195. BTW WE SEE THIS SAME
     PROGRESSION IN
     LEGAL NLP AS WELL …
     

     View Slide

196. HERE WE ANALYZE TRENDS
     WITHIN EVERY LEGAL NLP
     PAPER (MORE THAN 600+
     PAPERS) WRITTEN OVER
     THE PAST DECADE
     

     View Slide

197. GROWTH IN
     NUMBER OF
     PAPERS
     
     
     AND
     LANGUAGES
     

     View Slide

198. THE NEURAL
     AGENDA HAS COME
     TO LEGAL NLP
     

     View Slide

199. ALMOST NO TEXT GENERATION
     
     
     OR MACHINE SUMMARIZATION
     
     
     (BUT NOW THAT IS VERY LIKELY TO CHANGE)
     

     View Slide

200. View Slide

201. (2)
     
     
     WHAT IS GPT ?
     
     
     WHAT IS CHATGPT ?
     
     
     WHAT IS GPT-4 ?
     

     View Slide

202. GPT = GENERATIVE
     
     
     PRE-TRAINED
     
     
     TRANSFORMER
     (WE CAN TRY TO UNPACK ALL OF THIS IN A MINUTE)
     

     View Slide

203. GPT =
     IT IS A
     
     
     LARGE LANGUAGE MODEL (LLM)
     
     
     BUT FAR FROM THE ONLY ONE
     GENERATIVE
     
     
     PRE-TRAINED
     
     
     TRANSFORMER
     

     View Slide

204. TODAY I WILL MOSTLY
     LIMIT MY COMMENTS
     TO GPT BUT JUST
     WANTED TO FLAG
     THERE ARE OTHER
     MODELS OUT THERE
     

     View Slide

205. XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND
     CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730
     THE LEADING LLM MODELS
     

     View Slide

206. THE LEADING LLM MODELS
     XAVIER AMATRIAIN, TRANSFORMER MODELS: AN INTRODUCTION AND
     CATALOG, ARXIV:2302.07730 (2023) HTTPS://ARXIV.ORG/ABS/2302.07730
     

     View Slide

207. View Slide

208. THE OPEN AI GPT MODELS
     COMBINE SEVERAL IDEAS
     TOGETHER FROM THE
     VARIOUS PAPERS IN THIS
     PROGRESSION …
     

     View Slide

209. Word2Vec (2013)
     

     View Slide

210. View Slide

211. View Slide

212. ELMO (2018)
     

     View Slide

213. BERT (2019)
     

     View Slide

214. View Slide

215. View Slide

216. View Slide

217. View Slide

218. View Slide

219. View Slide

220. View Slide

221. BTW IT SHOULD BE
     NOTED THAT …
     

     View Slide

222. CHATGPT / GPT-3.5 / GPT-4 ARE
     PROPRIETARY MODELS SO WE
     DO NOT KNOW FOR CERTAIN
     ALL OF THE DEEP DETAILS
     

     View Slide

223. View Slide

224. WE WILL NOT GET INTO
     THE SUPER TECHNICAL
     DETAILS TODAY …
     

     View Slide

225. FOR A VERY APPROACHABLE
     TREATMENT GO HERE
     https://writings.stephenwolfram.com/2023/02/
     what-is-chatgpt-doing-and-why-does-it-work/
     

     View Slide

226. I MET STEPHEN
     WOLFRAM’S ROBOT
     PRESENCE IN
     HOUSTON IN 2011
     HE APPEARED AT A
     COMPUTATIONAL LAW
     CONFERENCE VIA ROBOT
     
     
     (A REAL BOSS MOVE)
     STEPHEN AND YOURS TRULY
     TALKING ABOUT RULE 90
     IT WAS A
     ‘BIG BANG’
     MOMENT
     

     View Slide

227. View Slide

228. BETTER
     HARDWARE
     PARALLEL
     
     
     COMPUTING
     ATTENTION
     
     
     MECHANISM
     THE PATH TO MEGASCALE
     
     
     LARGE LANGUAGE MODELS
     
     
     HAS BEEN DRIVEN BY A MIXTURE OF
     

     View Slide

229. THE SAME TECH THAT HAS
     POWERED HIGHLY IMMERSIVE
     GAMING HAS ALSO PUSHED
     SCIENCE FORWARD …
     https://www.thegamer.com/pc-games-best-intense-graphics/
     

     View Slide

230. APRIL 5, 2023
     NOT COMPARED DIRECTLY WITH THE H100 BUT THIS SHOWS
     YOU THE NATURE OF THE COMPETITION TAKING PLACE
     https://www.cnbc.com/2023/04/05/google-
     reveals-its-newest-ai-supercomputer-
     claims-it-beats-nvidia-.html
     

     View Slide

231. WOULD TAKE
     300 YEARS TO
     TRAIN EVEN
     GPT-3 ON A
     SINGLE GPU
     BUT WE
     TRANSFORMER
     
     
     ARCHITECTURE
     ALLOWS FOR
     SIGNIFICANT
     PARALLELIZATION
     

     View Slide

232. TRAINED MODEL ON
     A BILLION PAIRS OF
     WORDS IN JUST 3.5
     DAYS ON EIGHT
     NVIDIA GPUS …
     THIS APPROACH
     HIGHLIGHTED A
     PATH TO THE
     PRESENT WITH
     EVER LARGER
     FUTURE
     https://blogs.nvidia.com/blog/2022/03/25/
     what-is-a-transformer-model/
     

     View Slide

233. NOW PARALLEL COMPUTING IS
     BEING DONE ON A MEGASCALE
     

     View Slide

234. View Slide

235. GPT-3 (RELEASED IN 2020) IS THIRD
     GENERATION OF THE GPT FAMILY
     

     View Slide

236. GPT-3 (RELEASED IN 2020) IS THIRD
     GENERATION OF THE GPT FAMILY
     GPT-3.5 IS AN INTERMEDIATE
     IMPROVEMENT ON ORIGINAL GPT-3
     

     View Slide

237. GPT-3 (RELEASED IN 2020) IS THIRD
     GENERATION OF THE GPT FAMILY
     GPT-4 IS THE MOST RECENT
     RELEASE AND IT IS LIKELY TO
     BE A ‘FAMILY OF MODELS’
     GPT-3.5 IS AN INTERMEDIATE
     IMPROVEMENT ON ORIGINAL GPT-3
     

     View Slide

238. JUST LIKE GPT-3.5 IS NOW
     A FAMILY OF MODELS
     

     View Slide

239. SOME INPUTS TO GPT
     COMMON CRAWL
     WEBTEXT2 BOOKS1/2 WIKIPEDIA
     OPENAI LIKELY DID SOME SIGNIFICANT
     CLEANING / PREPROCESSING OF THESE
     SOURCES PRIOR TO TRAINING
     GPT-3 IS 175B PARAMETERS
     
     
     TRAINED WITH 499B TOKENS
     

     View Slide

240. TRANSFORMER
     
     
     ARCHITECTURE
     NEURAL
     
     
     NETWORK
     MODEL SCALE
     REINFORCEMENT
     
     
     LEARNING
     PRETRAINING
     JUST SOME KEY
     TERMINOLOGY
     
     
     YOU SHOULD
     LEARN
     CONTEXT WINDOW
     ATTENTION
     MECHANISM
     MODEL TUNING
     GRADIENT
     
     
     DESCENT
     

     View Slide

241. View Slide

242. TAKE ORIGINAL
     LLM GPT-3
     https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-
     Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
     CHATGPT IS AN
     OFFSHOOT OF GPT-3
     

     View Slide

243. COMBINE
     SPECIFIC
     TRAINING ON
     INSTRUCTIONS
     TAKE ORIGINAL
     LLM GPT-3
     https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-
     Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
     CHATGPT IS AN
     OFFSHOOT OF GPT-3
     

     View Slide

244. FURTHER
     
     
     TUNE ON
     INSTRUCTIONS
     COMBINE
     SPECIFIC
     TRAINING ON
     INSTRUCTIONS
     TAKE ORIGINAL
     LLM GPT-3
     https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-
     Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
     CHATGPT IS AN
     OFFSHOOT OF GPT-3
     

     View Slide

245. FURTHER
     
     
     TUNE ON
     INSTRUCTIONS
     SUPPORT WITH
     SPECIFIC
     
     
     REINFORCEMENT
     
     
     LEARNING
     FEEDBACK LOOP
     COMBINE
     SPECIFIC
     TRAINING ON
     INSTRUCTIONS
     TAKE ORIGINAL
     LLM GPT-3
     https://yaofu.notion.site/How-does-GPT-Obtain-its-Ability-Tracing-Emergent-
     Abilities-of-Language-Models-to-their-Sources-b9a57ac0fcf74f30a1ab9e3e36fa1dc1
     CHATGPT IS AN
     OFFSHOOT OF GPT-3
     

     View Slide

246. THE GPT-4
     TECHNICAL
     REPORT
     OFFERS
     SOME BUT
     NOT MOST
     OF THE KEY
     DETAILS
     

     View Slide

247. RLHF WAS
     CLEARLY
     PART OF THE
     BREW HERE
     BUT
     GENERALLY
     DOES NOT
     SEEM TO BE
     MOVING THE
     NEEDLE ALL
     THAT MUCH
     AT THIS
     POINT
     

     View Slide

248. SO THE POINT HERE
     IS TO MAKE CLEAR
     THAT THERE ARE
     MANY STEPS AND
     MOVING PARTS …
     

     View Slide

249. THERE ARE OTHER PLAYERS
     BESIDES OPENAI AND THEY
     WILL LIKELY PURSUE OTHER
     STEPS / MOVING PARTS …
     AND MANY MORE …
     

     View Slide

250. View Slide

251. (3)
     
     
     BAR EXAM AS WINDOW
     INTO INCREASED
     CAPABILITIES
     

     View Slide

252. AGAIN WE HAD BEEN
     TRYING TO DISCUSS /
     HIGHLIGHT THE INCREASING
     CAPABILITIES OF LANGUAGE
     MODELS FOR SOME TIME
     

     View Slide

253. WE THOUGHT THAT SOME
     SORT OF A PUBLIC
     DEMONSTRATION WOULD
     HELP MAKE THE POINT …
     

     View Slide

254. REMEMBER THIS IS A
     TASK THAT MANY
     WOULD THINK IS
     IMPOSSIBLE
     (LAST YEAR I WOULD HAVE SAID IT
     WOULD NOT OCCUR FOR MANY YEARS)
     

     View Slide

255. SOME TASKS LAWYERS
     REGULARLY UNDERTAKE ARE
     ACTUALLY WAY *EASIER*
     THAN THE BAR EXAM
     
     
     (AND OTHERS ARE HARDER)
     LET’S BE CLEAR …
     

     View Slide

256. TO HAVE A CHANCE
     ON THE BAR EXAM …
     

     View Slide

257. EXAMINEE MUST POSSESS A
     THRESHOLD AMOUNT OF
     
     
     LEGAL KNOWLEDGE AND
     READING COMPREHENSION
     SKILLS AND SEMANTIC AND
     SYNTACTIC COMMAND OF
     THE ENGLISH LANGUAGE
     

     View Slide

258. View Slide

259. RIGHT
     BEFORE
     THE
     HOLIDAYS
     ROUGHLY DEC 22, 2022
     

     View Slide

260. I CALLED MY FRIEND
     AND FREQUENT
     COLLABORATOR
     MIKE BOMMARITO
     

     View Slide

261. AND SAID LET’S
     EVALUATE GPT-3.5
     (TEXT-DAVINCI-003) ON
     THE MULTIPLE CHOICE
     PART OF THE BAR EXAM
     

     View Slide

262. AND IT DID
     PRETTY WELL …
     

     View Slide

263. V1.01 - December 29, 2022 V2.01 - January 3, 2023
     

     View Slide

264. View Slide

265. View Slide

266. LITTLE DID WE
     KNOW AT THE
     TIME THAT THIS
     PAPER PUT US
     ON A PATH …
     

     View Slide

267. TO BE INVOLVED
     IN THE LAUNCH
     OF GPT-4*
     *OBVIOUSLY WE ARE JUST PLAYING A
     VERY SMALL ROLE IN THE BIG PICTURE
     (BUT WE WILL TAKE IT!)
     

     View Slide

268. GPT-4 BAR EXAM
     https://openai.com/research/gpt-4
     March 14, 2023
     

     View Slide

269. CHATGPT BAR
     EXAM PASSES
     Paper Now Available on SSRN!
     March 15, 2023 - Version 1.01
     https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233
     

     View Slide

270. View Slide

271. View Slide

272. https://www.law.com/legaltechnews/
     2023/03/17/how-gpt-4-mastered-the-
     entire-bar-exam-and-why-that-matters/
     abajournal.com/web/article/latest-
     version-of-chatgpt-aces-the-bar-exam-
     with-score-in-90th-percentile
     

     View Slide

273. https://www.lawnext.com/
     2023/03/gpt-takes-the-bar-exam-
     again-this-time-it-score-among-
     top-10-of-test-takers.html
     

     View Slide

274. https://www.youtube.com/watch?v=4pGG79VdNmU
     

     View Slide

275. View Slide

276. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4389233
     

     View Slide

277. < THE UBE >
     UNIFORM BAR EXAM
     

     View Slide

278. View Slide

279. View Slide

280. < THE MBE >
     

     View Slide

281. MULTISTATE BAR EXAM (MBE)
     SUBJECTS
     
     
     TESTED
     TORTS
     
     
     CONTRACTS
     
     
     EVIDENCE
     
     
     REAL PROPERTY
     
     
     CIVIL PROCEDURE
     
     
     CONSTITUTIONAL LAW
     
     
     CRIMINAL LAW AND PROCEDURE
     

     View Slide

282. View Slide

283. View Slide

284. View Slide

285. View Slide

286. < THE MEE >
     

     View Slide

287. MULTISTATE ESSAY
     EXAMINATION (MEE)
     SUBJECTS
     
     
     TESTED
     TORTS
     
     
     EVIDENCE
     
     
     CONTRACTS
     
     
     FAMILY LAW
     
     
     REAL PROPERTY
     
     
     CONFLICT OF LAWS
     
     
     TRUSTS AND ESTATES
     
     
     CONSTITUTIONAL LAW
     
     
     BUSINESS ASSOCIATIONS
     
     
     FEDERAL CIVIL PROCEDURE
     
     
     UNIFORM COMMERCIAL CODE
     
     
     CRIMINAL LAW AND PROCEDURE
     

     View Slide

288. MULTISTATE ESSAY
     EXAMINATION (MEE)
     https://github.com/mjbommar/gpt4-passes-the-bar/blob/main/data/MEE.md
     

     View Slide

289. ‘REPRESENTATIVE
     GOOD ANSWERS’
     https://mdcourts.gov/sites/default/
     fi
     les/import/ble/
     examanswers/2022/202207uberepgoodanswers.pdf
     SEVERAL STATE BARS RELEASE
     REPRESENTATIVE GOOD ANSWERS
     
     
     THESE ARE VERY HELPFUL FOR
     EVALUATION PURPOSES AS THEY
     ARE ACTUAL ANSWERS WHICH ARE
     ABOVE MERELY PASSING ANSWERS
     (BUT NOT NECESSARILY PERFECT)
     

     View Slide

290. View Slide

291. (GPT-3.0 was released in 2020)
     

     View Slide

292. (GPT-3.0 was released in 2020)
     This ain’t passing any bar exam
     

     View Slide

293. View Slide

294. View Slide

295. < THE MPT >
     

     View Slide

296. MULTISTATE
     
     
     PERFORMANCE TEST (MPT)
     SKILLS
     
     
     TESTED
     EXAMINEE MUST COMPLETE A
     PRACTICAL LAWYERING TASK SUCH AS
     LEGAL ANALYSIS, FACT ANALYSIS,
     PROBLEM SOLVING, ORGANIZATION
     AND MANAGEMENT OF INFORMATION,
     AND CLIENT COMMUNICATION
     

     View Slide

297. MULTISTATE
     PERFORMANCE EXAM
     (MPT)
     
     
     10-15 PAGES OF
     MATERIALS
     
     
     THE FILE = THE FACTS
     
     
     THE LIBRARY = THE LAW
     
     
     ~5000 TOKEN INPUTS
     
     
     (ACCESS HERE)
     
     
     BIT.LY/40F3FQ2
     

     View Slide

298. View Slide

299. View Slide

300. OVERALL
     
     
     PERFORMANCE
     

     View Slide

301. SOME KEY SHORTCOMINGS OF
     GPT-4 ON THE BAR EXAM
     FAILED TO PROPERLY CALCULATE THE DISTRIBUTION OF
     ASSETS FROM A TESTAMENTARY TRUST WHICH HAS BEEN
     DEEMED TO BE INVALID
     
     
     PROVIDED AN INCORRECT ANSWER ON A CIVIL PROCEDURE
     QUESTION REGARDING DIVERSITY JURISDICTION AFTER
     THE JOINDER OF A NECESSARY PARTY
     
     
     PROVIDED IMPROPER ANALYSIS ON A REAL PROPERTY
     (REAL ESTATE) SUBQUESTION REGARDING BOTH THE
     PROPER DESIGNATION OF A FUTURE INTEREST AND THE
     APPLICATION OF THE RULE AGAINST PERPETUITIES
     

     View Slide

302. View Slide

303. View Slide

304. https://github.com/mjbommar/gpt4-passes-the-bar
     ACCESS THE FULL OUTPUT AND
     OTHER SUPPORTING MATERIALS HERE
     

     View Slide

305. View Slide

306. OUR BAR EXAM ANALYSIS
     IS WHAT IS CALLED A
     ‘ZERO SHOT’ ANALYSIS
     

     View Slide

307. ZERO SHOT ANALYSIS
     REFLECTS THE FLOOR AND
     NOT THE CEILING OF
     CURRENT CAPABILITIES
     

     View Slide

308. View Slide

309. ZERO SHOT
     ENTER
     PROMPT
     RECEIVE
     ANSWER
     ’PROMPT ENGINEERING’ IS ABOUT
     TUNING / REFINING PROMPTS TO
     OBTAIN BETTER ANSWERS
     

     View Slide

310. View Slide

311. THIS ZERO SHOT AND NOT EVEN
     THE LATEST MODEL (I.E. NOT GPT-4)
     

     View Slide

312. *NOT* A VERY
     SOPHISTICATED TAKE
     ON THIS SITUATION
     BUT REFLECTIVE OF THE
     MODAL PERSPECTIVE OF
     LAWYERS / LAW PROFS
     NOTE: ALL COMMERICAL RESEARCH
     TOOLS ALREADY HAVE ‘A.I.’ IN THEM
     

     View Slide

313. View Slide

314. ONE WAY TO DRAMATICALLY
     REDUCE THE LIKELIHOOD OF
     A HALLUCINATION IS TO
     MOVE OUT OF THE ‘ZERO
     SHOT’ PARADIGM
     

     View Slide

315. AND ANCHOR INITIAL
     OUTPUT AGAINST SOME
     ‘GROUND TRUTH’ …
     

     View Slide

316. ONE SHOT
     GET
     
     
     RESULT
     QUERY THAT
     RESULT AGAINST
     SOMETHING ELSE
     REFINE
     
     
     RESULT
     ENTER
     PROMPT
     OUTPUT
     
     
     FINAL
     ANSWER
     (RETRIEVAL AUGMENTATION)
     (IF NEEDED)
     EXAMPLES
     

     View Slide

317. View Slide

318. THE GENERALIZATION OF ALL OF
     THIS IS AN ORCHESTRATION LAYER
     TO BRING MULTIPLE STREAMS /
     TECHNOLOGIES TOGETHER
     

     View Slide

319. FEW SHOT
     (CHAIN OF THOUGHT PROMPTING)
     

     View Slide

320. HERE DAMIEN RIEHL FROM FASTCASE
     WILL WALK YOU THROUGH HOW TO
     BUILD MORE OF A BRIEF THROUGH
     SEQUENTIAL PROMPTING …
     HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT-
     LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/
     

     View Slide

321. HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT-
     LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/
     

     View Slide

322. FEW SHOT
     (LANGCHAIN / AUTOGPT PROMPTING)
     

     View Slide

323. FEW SHOT (LEGAL DOMAIN)
     (LANGCHAIN / AUTOGPT STYLE
     ORCHESTRATION LAYER)
     https://docs.kelvin.legal/docs/examples/due-dilligence/
     https://docs.kelvin.legal/docs/examples/litigation-automation/
     

     View Slide

324. View Slide

325. THERE ARE LOTS OF
     OPPORTUNITIES OUTSIDE
     THE ZERO-SHOT CONTEXT
     

     View Slide

326. OR EVEN OTHER PLUGINS …
     FOR EXAMPLE
     WOLFRAM COULD
     HELP SOLVE FOR
     ISSUES WITH
     QUANTITATIVE
     REASONING
     

     View Slide

327. https://arxiv.org/pdf/2303.17651.pdf
     https://selfre
     fi
     ne.info/
     https://github.com/madaan/self-re
     fi
     ne
     

     View Slide

328. GENERATIVE AGENTS
     

     View Slide

329. View Slide

330. View Slide

331. (4)
     
     
     LLMs IN THE DELIVERY
     OF LEGAL SERVICES
     

     View Slide

332. IN GENERAL …
     

     View Slide

333. DRAFTING + EDITING
     
     
     COMMUNICATIONS
     EMAILS
     
     
     REPORTS
     
     
     PRESENTATIONS
     
     
     MARKETING MATERIALS
     

     View Slide

334. DRAFTING + EDITING
     
     
     LEGAL DOCUMENTS
     CONTRACTS
     
     
     BRIEFS
     
     
     MEMOS
     
     
     INTERROGATORIES
     DEMAND LETTERS
     

     View Slide

335. CONDUCTING LEGAL RESEARCH
     EXTERNAL POINTS OF LAW
     
     
     INTERNAL KNOWLEDGE MGMT
     

     View Slide

336. SUMMARIZING LARGE BODIES
     OF TEXTUAL MATERIAL
     SUMMARIZING LEGAL DOCUMENTS
     

     View Slide

337. AND REMEMBER WE
     CAN COMBINE
     OUTPUTS WITH OTHER
     MACHINE LEARNING
     MODELS, ETC.
     

     View Slide

338. OR EVEN OTHER PLUGINS …
     FOR EXAMPLE
     WOLFRAM COULD
     HELP SOLVE FOR
     ISSUES WITH
     QUANTITATIVE
     REASONING
     

     View Slide

339. View Slide

340. BUT LETS LOOK AT SEVERAL
     CONCRETE EXAMPLES …
     

     View Slide

341. (1) DRAFT A MNDA
     

     View Slide

342. View Slide

343. View Slide

344. (2)
     
     
     DRAFT A REPLY BRIEF
     HERE DAMIEN RIEHL FROM FASTCASE
     WILL WALK YOU THROUGH HOW TO
     BUILD MORE OF A BRIEF THROUGH
     SEQUENTIAL PROMPTING …
     HTTPS://WWW.LINKEDIN.COM/PULSE/CHATGPT-
     LEGAL-BRIEFWRITING-TOOL-DAMIEN-RIEHL/
     

     View Slide

345. View Slide

346. (3)
     
     
     DRAFT REPORT OR PRESENTATION
     
     
     

     View Slide

347. https://youtu.be/S7xTBa93TX8
     

     View Slide

348. View Slide

349. DUE DILIGENCE /
     
     
     M&A SUPPORT
     (4)
     

     View Slide

350. View Slide

351. https://docs.kelvin.legal/docs/examples/due-dilligence/
     

     View Slide

352. https://docs.kelvin.legal/docs/examples/due-dilligence/
     DUE DILIGENCE /M&A SUPPORT
     

     View Slide

353. https://docs.kelvin.legal/docs/examples/due-dilligence/
     DUE DILIGENCE /M&A SUPPORT
     

     View Slide

354. https://docs.kelvin.legal/docs/examples/due-dilligence/
     

     View Slide

355. View Slide

356. LITIGATION SUPPORT
     & TRIAGE
     (5)
     

     View Slide

357. View Slide

358. View Slide

359. (6)
     
     
     BUILD A LEGAL KNOWLEDGE GRAPH
     https://www.ibm.com/topics/knowledge-graph
     

     View Slide

360. (6)
     
     
     BUILD A LEGAL KNOWLEDGE GRAPH
     HTTPS://TAX-GRAPH.273VENTURES.COM/
     

     View Slide

361. (6)
     
     
     BUILD A LEGAL KNOWLEDGE GRAPH
     

     View Slide

362. (6) BUILD A LEGAL KNOWLEDGE GRAPH
     

     View Slide

363. View Slide

364. (7)
     
     
     LANGUAGE MODELS ARE
     ZERO-SHOT TAGGERS
     

     View Slide

365. https://www.linkedin.com/posts/daniel-
     katz-3b001539_feedbackfriday-legaltech-
     legaldata-activity-7045070829428113409-fIZz
     

     View Slide

366. View Slide

367. https://www.law.com/
     legaltechnews/2023/03/20/sali-
     harnesses-most-advanced-gpt-
     models-to-help-legal-industry-
     speak-the-same-data-language/
     March 23, 2023
     Michael Bommarito -
     Gentleman Farmer
     

     View Slide

368. https://sali-search.kelvin.legal
     

     View Slide

369. View Slide

370. (8)
     
     
     REGULATORY MONITORING
     

     View Slide

371. https://www.linkedin.com/posts/
     bommarito_summarize-everything-published-in-
     the-federal-activity-7048621468422725632-CjuO
     April 3, 2023
     

     View Slide

372. View Slide

373. (9)
     
     
     SUBSTANTIVE LEGAL
     ANALYSIS
     (COPYRIGHT)
     

     View Slide

374. HTTPS://WWW.YOUTUBE.COM/WATCH?V=NQZCRHR8YPU&T=11S
     

     View Slide

375. View Slide

376. HERE ARE JUST A FEW MORE OF
     MANY OTHER POTENTIAL AREAS
     WHERE THIS CLASS OF TECH
     MIGHT BE USEFUL …
     E-DISCOVERY
     
     
     LEGAL BILLING
     
     
     DEALS DATABASES
     
     
     CONTRACT DRAFTING
     AND MANY MANY OTHER EXAMPLES
     

     View Slide

377. HERE A JUST A COUPLE OF
     RESOURCES THAT HAVE
     BEEN MADE AVAILABLE
     (THERE WILL BE MANY MORE)
     https://ssrn.com/abstract=4404017
     

     View Slide

378. https://thebrainyacts.beehiiv.com/
     

     View Slide

379. View Slide

380. NOW A QUICK
     
     
     WORD OF CAUTION
     

     View Slide

381. BECAUSE OF THE
     POTENTIAL FOR MODEL
     HALLUCINATION, ETC.
     

     View Slide

382. IT IS IMPORTANT TO USE
     THESE TOOLS (AT LEAST FOR
     NOW) AS PART OF A HUMAN
     IN THE LOOP PROCESS
     

     View Slide

383. IN OTHER WORDS,
     FOR MOST USE CASES HUMANS
     SHOULD ALWAYS BE PART OF THE
     
     
     ‘RETRIEVAL AUGMENTATION’ LAYER
     

     View Slide

384. AND OF COURSE
     OR MACHINES CAN AID IN
     
     
     RETRIEVAL AUGMENTATION /
     
     
     QUALITY ASSURANCE
     

     View Slide

385. CENTAUR CHESS
     HUMAN
     
     
     +
     
     
     MACHINE
     HUMAN
     OR
     MACHINE
     >
     

     View Slide

386. View Slide

387. (5)
     
     
     STRATEGIC CONSIDERATIONS
     AND THE ROAD AHEAD
     

     View Slide

388. FUD
     VS
     FEAR,
     
     
     UNCERTAINTY,
     
     
     & DOUBT
     FEAR OF
     MISSING
     OUT
     FOMO
     

     View Slide

389. FUD
     FEAR,
     
     
     UNCERTAINTY,
     
     
     & DOUBT
     HALLUCINATIONS
     
     
     CONFIDENTIALITY
     
     
     INFORMATION SECURITY
     
     
     

     View Slide

390. AMAZING CAPABILITIES
     
     
     KEEPING UP WITH THE MARKET
     
     
     NOT WANTING TO LOOK OUT OF
     STEP TO CLIENTS
     FEAR OF
     MISSING
     OUT
     FOMO
     

     View Slide

391. FUD
     VS
     FEAR,
     
     
     UNCERTAINTY,
     
     
     & DOUBT
     FEAR OF
     MISSING
     OUT
     FOMO
     ORGANIZATIONS / INDIVIDUALS TOGGLE
     BETWEEN THESE TWO POINTS OF VIEW
     (SOMETIME WITHIN THE SAME CONVERSATION)
     

     View Slide

392. View Slide

393. SEVERAL EARLY
     TOOLS IN THE
     MARKET …
     

     View Slide

394. View Slide

395. MAY 4, 2023
     https://www.lexisnexis.com/en-
     us/products/lexis-plus-ai.page
     

     View Slide

396. MAY 23, 2023
     https://www.lawnext.com/2023/05/
     thomson-reuters-previews-its-plans-for-
     generative-ai-announces-integration-with-
     microsoft-365-copilot.html
     

     View Slide

397. View Slide

398. View Slide

399. OBVIOUSLY WE HAVE A COMPANY
     AND WE WORKED WITH CASETEXT
     ON THE BAR PAPER — SO PERHAPS
     WE ARE NOT 100% NEUTRAL …
     SHANG
     
     
     GAO
     PABLO
     ARREDONDO
     

     View Slide

400. BUT OF COURSE
     THESE WILL NOT BE
     THE LAST TOOLS AND
     IT REMAINS TO BE
     SEEN HOW IT WILL
     ALL SHAKE OUT …
     

     View Slide

401. I THINK IT IS
     CLEAR THAT THIS
     IS GOING TO BE
     A LONG RACE …
     

     View Slide

402. View Slide

403. WHAT IS THE DATA AND
     TECHNOLOGY STRATEGY THAT
     ORGANIZATIONS CAN EMPLOY IN
     LIGHT OF THESE CHANGE IN THE
     TECHNOLOGY LANDSCAPE ?
     

     View Slide

404. BUILD VS ASSEMBLE VS BUY
     BUILD IS LIKELY OFF THE
     TABLE FOR THE SHORT TO
     MEDIUM TERM
     
     
     ASSEMBLE HOWEVER IS A
     VIABLE OPTION
     
     
     SO CAREFUL PROCUREMENT
     IS GOING TO BE THE
     STRATEGIC PATH
     

     View Slide

405. DATA STRATEGY
     HOW TO COLLECT / REGULARIZE
     DATA FOR USE INSIDE THESE
     SYSTEMS OR AS A LAYER ON
     TOP OF THESE SYSTEMS
     

     View Slide

406. View Slide

407. SEVERAL OTHER STRATEGIC
     CONSIDERATIONS
     WHAT IS THE ROLE OF DOMAIN SPECIFIC TRAINING?
     HOW COULD I CUSTOMIZE THESE MODELS FOR USE
     WITHIN MY OWN ORGANIZATION ?
     WHICH MODELS / LLMS DO I LEVERAGE ?
     
     
     HOW DO I CHOOSE ? CAN I MIX AND MATCH ?
     WHAT IS THE DIFFERENTIAL IMPACT OF THESE
     MODELS BY LEGAL ORGANIZATION ?
     HOW DO I THINK ABOUT QUESTIONS OF PRIVACY AND
     INFORMATION SECURITY ? COPYRIGHT?
     

     View Slide

408. WHAT IS THE DIFFERENTIAL
     IMPACT OF THESE MODELS
     BY LEGAL ORGANIZATION ?
     (1)
     

     View Slide

409. I THINK THAT THERE
     ARE GOING TO BE
     DIFFERENTIAL
     IMPACTS BY TYPE OF
     LEGAL ORGANIZATION
     

     View Slide

410. MANAGERS SHOULD
     REORGANIZE JOBS SO
     ORGANIZATION CAN
     FURTHER CONCENTRATE
     TASKS WHICH CAN BE
     ENABLED BY A.I.
     MICRO LEVEL
     

     View Slide

411. LOOK TO PLACES WHERE THERE
     IS A CONCENTRATION OF LOW
     TO MEDIUM COMPLEXITY TASKS
     MACRO LEVEL
     ALSP, LPO, LEGAL OPS,
     SPECIALIZE LAW FIRMS, ETC.
     

     View Slide

412. TRAINING FOLKS TO USE
     THESE TOOLS TO THE
     MAXIMUM EXTENT POSSIBLE
     HUMAN CAPITAL
     (THIS WILL BE AN IMPORANT LIMITATION
     ON ORGANIZATIONAL SUCCESS)
     

     View Slide

413. View Slide

414. HOW SHOULD I THINK ABOUT
     QUESTIONS OF PRIVACY AND
     INFORMATION SECURITY ?
     COPYRIGHT?
     (2)
     

     View Slide

415. SHOULD / CAN I PUT
     CONFIDENTIAL CLIENT
     DATA IN THESE SYSTEMS ?
     MANY FOLKS HAVE ASKED ME —
     

     View Slide

416. PROCEED WITH CAUTION —
     
     
     I WOULD TELL YOU TO AUDIT
     PRECISELY WHAT IS HAPPENING
     AND WHERE IT IS HAPPENING
     
     
     SHOULD / CAN I PUT
     CONFIDENTIAL CLIENT
     DATA IN THESE SYSTEMS ?
     (CLIENT CONSENT + AUDITED SAFEGUARDS)
     MANY FOLKS HAVE ASKED ME —
     

     View Slide

417. THE ROGUE ASSOCIATE
     (OR PARTNER) PASTING
     STUFF INTO GPT OR
     ANOTHER LLM MODEL
     ONE VERSION OF THE CONCERN
     

     View Slide

418. https://mashable.com/article/samsung-chatgpt-leak-details
     

     View Slide

419. WE HAVE BEEN WORKING
     WITH A FEW FIRMS ON
     HOW TO NAVIGATE THE
     THORNY INFO SEC AND
     LEGAL ETHICS ISSUES
     

     View Slide

420. IF NOT, DOES VENDOR HAVE SOC2 / ISO2700 ?
     HOW MANY OTHER PARTIES ARE IN THE FLOW
     OF YOUR INFORMATION ? CONTRACTUAL
     REPRESENTATIONS FROM THOSE 3RD PARTIES?
     IS THE SOLUTION AVAILABLE ON PREM ?
     SOME VENDOR QUESTIONS
     WHAT CONTROLS DOES THE VENDOR
     APPLY TO THE TRANSITING OF YOUR DATA ?
     

     View Slide

421. LEVERAGING THESE TOOLS
     ON INTERNAL DATA IS
     REALLY THE HOLY GRAIL ….
     INTERNAL DATA
     WE ARE GOING TO GET THERE
     BUT IT REQUIRES A REAL
     THOUGHT OUT APPROACH HERE
     

     View Slide

422. WHAT IS GOING TO
     HAPPEN WITH COPYRIGHT ?
     MANY FOLKS HAVE ASKED ME —
     

     View Slide

423. WHAT IS GOING TO
     HAPPEN WITH COPYRIGHT ?
     I THINK THIS IS GOING TO
     LEAD TO A MAJOR DECISION
     IN COPYRIGHT AND/OR
     ACTION BY CONGRESS, ETC.
     MANY FOLKS HAVE ASKED ME —
     

     View Slide

424. https://arxiv.org/pdf/2303.15715.pdf
     THIS IS A VERY GOOD
     PAPER ON THE TOPIC
     

     View Slide

425. View Slide

426. WHAT IS THE ROLE OF
     DOMAIN SPECIFIC TRAINING?
     (3)
     

     View Slide

427. WILL A GENERAL MODEL BEAT A
     DOMAIN SPECIFIC MODEL ?
     SURE THAT IS POSSIBLE / LIKELY —
     IF THE GENERAL MODEL IS LARGE
     AND DOMAIN MODEL IS SMALL
     

     View Slide

428. AGAIN HERE IS AN EXAMPLE
     WITH ACTUAL RESULTS
     ATTACHED TO IT …
     

     View Slide

429. https://www.bloomberg.com/company/press/
     bloomberggpt-50-billion-parameter-llm-tuned-
     fi
     nance/
     https://arxiv.org/pdf/2303.17564.pdf
     

     View Slide

430. https://twitter.com/rasbt/status/1642880757566676992
     

     View Slide

431. SO HOW COULD THIS BE
     IMPROVED EVEN FURTHER ?
     

     View Slide

432. SOURCES OF LLM MODEL IMPROVEMENT
     USE DIFFERENT TRAINING DATA
     NEURAL NET ACCESS / IMPLEMENTATION
     INSTRUCTION MODULE
     FINE TUNING
     RETRIEVAL AUGMENTATION (RAG)
     PROMPT ENGINEERING
     CHAIN OF THOUGHT PROMPTING
     RLHF
     AGENT KNOWLEDGE GRAPH TRAVERSAL
     MODEL
     CREATOR
     MODEL
     USER
     (REINFORCEMENT LEARNING FROM HUMAN FEEDBACK)
     

     View Slide

433. SO WILL THERE BE A ‘LAW GPT’?
     PROBABLY SOON
     IT IS AN OPEN QUESTION
     WILL IT BE BETTER THAN
     GPT-4 ON LAW TASKS?
     

     View Slide

434. View Slide

435. HOW COULD I
     CUSTOMIZE THESE
     MODELS FOR USE
     WITHIN MY OWN
     ORGANIZATION ?
     (4)
     

     View Slide

436. HOW IS ALL OF YOUR
     DATA GOING TO GET
     INSIDE OF THESE
     SYSTEMS … ?
     

     View Slide

437. WE ARE LIKELY ON A
     PATH TO INTERNAL
     GPT STYLE OFFERINGS
     

     View Slide

438. FOLKS ARE EVENTUALLY
     GOING BUILD / TRAIN /
     TUNE THEIR OWN GPT
     STYLE MODELS…
     

     View Slide

439. TO DO SO FOLKS
     ARE GOING TO
     NEED TO
     STRUCTURE /
     PREPROCESS
     THEIR DATA
     

     View Slide

440. https://nationalmagazine.ca/en-ca/articles/legal-market/legal-tech/2023/chatting-with-gpt
     

     View Slide

441. CONNECTING THE THREE LAYERS
     https://nationalmagazine.ca/en-ca/articles/
     legal-market/legal-tech/2023/chatting-with-gpt
     

     View Slide

442. View Slide

443. WHICH MODELS / LLMS
     DO I LEVERAGE ?
     
     
     HOW DO I CHOOSE ?
     
     
     CAN I MIX AND MATCH ?
     (5)
     

     View Slide

444. SO WE HAVE
     SEEN QUITE A
     BIT OF THIS
     SINCE THE
     LAUNCH OF
     CHATGPT
     NOW
     

     View Slide

445. THIS IS NOT
     INHERENTLY BAD
     BUT I THINK
     ORGANIZATIONS
     NEED TO THINK
     ABOUT THEIR
     PRECISE
     STRATEGY HERE NOW
     

     View Slide

446. SOURCES OF LLM MODEL IMPROVEMENT
     USE DIFFERENT TRAINING DATA
     NEURAL NET ACCESS / IMPLEMENTATION (ALPACA, ETC.)
     INSTRUCTION MODULE
     FINE TUNING
     RETRIEVAL AUGMENTATION (RAG)
     PROMPT ENGINEERING
     CHAIN OF THOUGHT PROMPTING (LANGCHAIN, ETC.)
     RLHF
     (E.G. BLOOMBERG GPT)
     (DOLLY, ETC.)
     (PLUGINS / UI / GROUNDING)
     MODEL
     CREATOR
     MODEL
     USER
     (VARIOUS FORMS)
     (VARIOUS FORMS)
     (REINFORCEMENT LEARNING FROM HUMAN FEEDBACK)
     (AGENTS BUILD / TRAVERSE GRAPHS)
     (MANY APPROACHES)
     AGENT KNOWLEDGE GRAPH TRAVERSAL
     

     View Slide

447. REMEMBER YOU ARE GOING
     TO HAVE DIRECT ACCESS …
     

     View Slide

448. SO WHAT IS THE VENDOR VALUE
     ADD OVER BASE LLM MODEL?
     TRUST BUT VERIFY
     THERE IS GOING TO BE
     SIGNIFICANT PRESSURE ON
     VENDORS TO OVERSTATE
     THE ACTUAL CONTRIBUTION
     OF THEIR OFFERING ABOVE
     JUST AN LLM API CALL
     I SUGGEST YOU
     
     
     TRUST BUT VERIFY
     
     
     ALL CLAIMS
     

     View Slide

449. CAN YOU DETERMINE WHETHER A
     VENDOR IS SIMPLY SELLING YOU A UI/
     UX WRAPPER ON TOP OF AN LLM ?
     AGAIN THIS MIGHT BE OKAY BUT DO
     NOT PAY AN EXCESSIVE PREMIUM …
     

     View Slide

450. IT WOULD BE A BIG MISTAKE TO
     GO ALL IN ON ANY VENDOR OR
     SPECIFIC LLM AT THIS POINT
     PLACE A SMALL TO
     MEDIUM BET
     

     View Slide

451. AS THE YEAR PROGRESSES,
     
     
     I BELIEVE THIS WILL
     
     
     *NOT* JUST BE A GPT STORY …
     

     View Slide

452. IT IS A HIGHLY COMPETITVE
     LANDSCAPE AND OTHER FOLKS
     WILL BE ENTERING THE MARKET
     WITH ADDITIONAL OFFERINGS
     

     View Slide

453. THERE ARE OTHER PLAYERS
     BESIDES OPENAI AND THEY
     WILL LIKELY PURSUE OTHER
     STEPS / MOVING PARTS …
     AND MANY MORE …
     

     View Slide

454. REMEMBER THAT GOOGLE
     ACTUALLY INVENTED MUCH OF THE
     TECH THAT BROUGHT YOU GPT
     https://www.forbes.com/sites/richardnieva/
     2023/02/08/google-openai-chatgpt-microsoft-
     bing-ai/
     https://www.nytimes.com/2023/01/20/
     technology/google-chatgpt-arti
     fi
     cial-
     intelligence.html
     

     View Slide

455. CAN YOU BOTH RAPIDLY AND
     RIGOROUSLY EVALUATE THE
     QUALITY OF MODELS OR
     FIGURE OUT HOW TO MIX
     AND MATCH THEM … ?
     (ENSEMBLE LEARNING ANYONE?)
     

     View Slide

456. View Slide

457. FINAL
     
     
     THOUGHTS
     

     View Slide

458. THIS IS SOMETHING OF
     A DIFFERENT FLAVOR
     FROM THE AVERAGE
     TECHNOLOGICAL
     INNOVATION …
     

     View Slide

459. View Slide

460. NOTE THIS IS A CHATGPT PAPER NOT
     EVEN CONSIDERING POTENTIAL
     ADDITIONAL GAINS IN GPT-4
     https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4375283
     “TIME TAKEN DECREASES BY
     0.8 SDS AND OUTPUT
     QUALITY RISES BY 0.4 SDS.”
     

     View Slide

461. OVER 50%
     PRODUCTIVITY GAINS
     FROM GPT-3.5 VERSION
     OF COPILOT (USED FOR
     PROGRAMMING)
     https://arxiv.org/pdf/2302.06590
     COPILOT X RELEASED
     
     
     MARCH 23RD 2023
     https://github.blog/2023-03-22-github-copilot-
     x-the-ai-powered-developer-experience/
     FEBRUARY 13 2023
     

     View Slide

462. https://arxiv.org/abs/2303.10130
     “OUR FINDINGS REVEAL THAT
     AROUND 80% OF THE U.S.
     WORKFORCE COULD HAVE AT
     LEAST 10% OF THEIR WORK TASKS
     AFFECTED BY THE INTRODUCTION
     OF LLMS, WHILE APPROXIMATELY
     19% OF WORKERS MAY SEE AT
     LEAST 50% OF THEIR TASKS
     IMPACTED. WE DO NOT MAKE
     PREDICTIONS ABOUT THE
     DEVELOPMENT OR ADOPTION
     TIMELINE OF SUCH LLMS”
     

     View Slide

463. View Slide

464. TASKS VS JOBS
     WHAT IS BEING MADE POSSIBLE?
     WE RACE WITH THE MACHINE
     LABOR MARKET IMPLICATIONS
     
     
     (IT IS COMPLICATED)
     (IN A WORLD WHERE TEXT GENERATION IS FAR CHEAPER)
     

     View Slide

465. View Slide

466. OVERALL, THERE SEEMS TO BE
     NON-LINEAR PERFORMANCE
     PERHAPS AKIN TO BROADER LANGUAGE ACQUISTION ?
     

     View Slide

467. IS THERE IS
     SOMETHING MORE
     FUNDAMENTAL
     GOING ON
     EMERGENT
     
     
     BEHAVIOR ?
     

     View Slide

468. CUT+PASTE
     SAFETY FIRST
     THIS IS ALL KINDA
     BACK TO THE
     FUTURE FOR ME
     

     View Slide

469. https://research.google/pubs/pub52065/
     

     View Slide

470. https://futurism.com/gpt-4-sparks-of-agi
     https://www.youtube.com/watch?v=qbIk7-JPB2c
     

     View Slide

471. THIS IS AN ACTIVE
     DEBATE IN THE
     ACADEMIC
     LITERATURE
     
     
     I WOULD SAY THE
     JURY IS STILL OUT
     

     View Slide

472. AND WHILE THERE ARE LIMITS
     AND THINGS DO NOT GROW
     EXPONENTIALLY FOREVER
     

     View Slide

473. I THINK THIS IS
     GOING TO BE THE
     MOST EXCITING YEAR
     IN TECHNOLOGY IN A
     VERY LONG TIME !
     

     View Slide

474. View Slide

475. View Slide

476. View Slide

477. 273VENTURES.COM
     KELVIN.LEGAL
     

     View Slide

478. https://www.linkedin.com/in/daniel-katz-3b001539/
     

     View Slide

479. @COMPUTATIONAL
     

     View Slide

480. G E N E R AT I V E A . I . + L AW
     @ computational
     professor daniel martin katz danielmartinkatz.com
     Illinois tech - chicago kent law 273Ventures.com
     B AC KG R O U N D, A P P L I CAT I O N S A N D U S E CA S E S
     I N C LU D I N G G P T- 4 PA S S E S T H E B A R E X A M
     https://bit.ly/3yxhY2e
     

     View Slide