Towards Structured Data: LLMs from Prototype to Production

Slide 1

Slide 1 text

Ines Montani Explosion TOWARDS STRUCTURED LARGE LANGUAGE MODELS ✨ CHATGPT 🤖 ARTIFICIAL INTELLIGENCE 🧠 MACHINE LEARNING ✨ PROTOTYPE TO PRODUCTION LLAMA 🦙 NATURAL LANGUAGE PROCESSING 💬 ✨ OPEN SOURCE 🌎 PYTHON 🐍 PROMPT ENGINEERING ⚙ ZERO-SHOT LEARNING 🎯 GPT-4 EVALUATION 📈 COPILOT 🚀 GENERATIVE AI 👾 DATA LLMS FROM Ines Montani 💥 Explosion

Slide 2

Slide 2 text

SPACY SPACY.IO 🌎 GITHUB.COM/EXPLOSION/SPACY Open-source library for industrial-strength Natural Language Processing 225m+ downloads

Slide 3

Slide 3 text

SPACY SPACY.IO 🌎 GITHUB.COM/EXPLOSION/SPACY Open-source library for industrial-strength Natural Language Processing 225m+ downloads ChatGPT can write spaCy code!

Slide 4

Slide 4 text

PRODIGY Modern scriptable annotation tool for machine learning developers PRODIGY.AI 10k+ users 900+ companies

Slide 5

Slide 5 text

PRODIGY Modern scriptable annotation tool for machine learning developers PRODIGY.AI 10k+ users 900+ companies

Slide 6

Slide 6 text

SOFTWARE IN INDUSTRY

Slide 7

Slide 7 text

SOFTWARE IN INDUSTRY modular 🧩

Slide 8

Slide 8 text

SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎

Slide 9

Slide 9 text

SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮

Slide 10

Slide 10 text

SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private

Slide 11

Slide 11 text

SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private ✅ reliable

Slide 12

Slide 12 text

SOFTWARE IN INDUSTRY modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private ✅ reliable 💸 a ff ordable

Slide 13

Slide 13 text

SOFTWARE IN INDUSTRY black-box models modular 🧩 transparent 🔎 explainable 🔮 🔒 data-private ✅ reliable 💸 a ff ordable

Slide 14

Slide 14 text

SOFTWARE IN INDUSTRY black-box models modular 🧩 transparent 🔎 explainable 🔮 third-party APIs 🔒 data-private ✅ reliable 💸 a ff ordable

Slide 15

Slide 15 text

📖 single/multi-doc summarization ✅ problem solving ✍ paraphrasing 🧮 reasoning 🖼 style transfer Generative ❓question answering 📚 text classification 🏷 entity recognition 🔗 relation extraction 🧬 grammar & morphology 🎯 semantic parsing 👫 coreference resolution 💬 discourse structure Predictive UNDERSTANDING NLP TASKS

Slide 16

Slide 16 text

Slide 17

Slide 17 text

🔮 large generative model

Slide 18

Slide 18 text

🔮 large generative model 📦 distilled task-specific model

Slide 19

Slide 19 text

🔮 large generative model 📦 distilled task-specific model in-context learning Falcon MIXTRAL GPT-4

Slide 20

Slide 20 text

🔮 large generative model 📦 distilled task-specific model in-context learning Falcon MIXTRAL GPT-4 transfer learning ELECTRA T5

Slide 21

Slide 21 text

🔮 large generative model 📦 distilled task-specific model in-context learning Falcon MIXTRAL GPT-4 transfer learning ELECTRA T5 BERT-base still very competitive!

Slide 22

Slide 22 text

GITHUB.COM/EXPLOSION/SPACY-LLM TOWARDS STRUCTURED DATA Prompt Template 🔮 LLM London is bigger than Berlin LOCATION: London, Berlin LOCATION

Slide 23

Slide 23 text

GITHUB.COM/EXPLOSION/SPACY-LLM TOWARDS STRUCTURED DATA Prompt Template 🔮 LLM London is bigger than Berlin LOCATION: London, Berlin LOCATION

Slide 24

Slide 24 text

GITHUB.COM/EXPLOSION/SPACY-LLM 💬 unstructured text input 📊 structured Doc object

Slide 25

Slide 25 text

GITHUB.COM/EXPLOSION/SPACY-LLM Named Entity Recognition Text Classification Relation Extraction Lemma- tization 💬 unstructured text input 📊 structured Doc object

Slide 26

Slide 26 text

GITHUB.COM/EXPLOSION/SPACY-LLM Named Entity Recognition Text Classification Relation Extraction Lemma- tization 💬 unstructured text input 📊 structured Doc object 🔮 LLM ⚙ Supervised Model ✍ Rules mix, match and replace techniques

Slide 27

Slide 27 text

CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION

Slide 28

Slide 28 text

CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs

Slide 29

Slide 29 text

CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation

Slide 30

Slide 30 text

CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation EXPLOSION.AI/BLOG/APPLIED-NLP-THINKING 🎯 assess utility, not just accuracy

Slide 31

Slide 31 text

CLOSE THE GAP BETWEEN PROTOTYPE AND PRODUCTION 🔗 standardize inputs and outputs 📈 start with evaluation EXPLOSION.AI/BLOG/APPLIED-NLP-THINKING 🎯 assess utility, not just accuracy 🔁 work on data iteratively

Slide 32

Slide 32 text

Slide 33

Slide 33 text

processing pipeline prototype 🔮 📦 GITHUB.COM/EXPLOSION/SPACY-LLM processing pipeline in production 📦 📦 📦 📦 📊 structured Doc object 📊 structured Doc object PROTOTYPE TO PRODUCTION

Slide 34

Slide 34 text

processing pipeline prototype 🔮 📦 prompt model & transform output to structured data GITHUB.COM/EXPLOSION/SPACY-LLM processing pipeline in production 📦 📦 📦 📦 📊 structured Doc object 📊 structured Doc object PROTOTYPE TO PRODUCTION

Slide 35

Slide 35 text

🔮 HUMAN IN THE LOOP

Slide 36

Slide 36 text

continuous evaluation baseline 🔮 HUMAN IN THE LOOP

Slide 37

Slide 37 text

continuous evaluation baseline prompting 🔮 HUMAN IN THE LOOP

Slide 38

Slide 38 text

continuous evaluation baseline prompting PRODIGY.AI 🔮 HUMAN IN THE LOOP

Slide 39

Slide 39 text

continuous evaluation baseline prompting 📦 transfer learning PRODIGY.AI 🔮 HUMAN IN THE LOOP

Slide 40

Slide 40 text

continuous evaluation baseline prompting 📦 transfer learning PRODIGY.AI distilled model 🔮 HUMAN IN THE LOOP

Slide 41

Slide 41 text

task-specific distillation workflow continuous evaluation baseline prompting 📦 transfer learning PRODIGY.AI distilled model 🔮 HUMAN IN THE LOOP

Slide 42

Slide 42 text

■ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND

Slide 43

Slide 43 text

■ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts ■ used LLM during annotation SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND

Slide 44

Slide 44 text

■ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts ■ used LLM during annotation ■ beat few-shot LLM baseline of 0.74 with task-specific model SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND

Slide 45

Slide 45 text

■ PyData NYC 2023 workshop: extracting dishes, ingredients and equipment from r/cooking Reddit posts ■ used LLM during annotation ■ beat few-shot LLM baseline of 0.74 with task-specific model ■ 20× inference time speedup SPACY.FYI/PYDATA-NYC CASE STUDY 🕓 8 hours DATA DEV TIME 📦 400mb MODEL SIZE 🔥 2000+ WORDS / SECOND

Slide 46

Slide 46 text

CONCLUSION

Slide 47

Slide 47 text

■ LLMs can be one part of a product or process, and swapped for di ff erent approaches. CONCLUSION

Slide 48

Slide 48 text

■ LLMs can be one part of a product or process, and swapped for di ff erent approaches. ■ Iteration and the right tooling can get you past the prototype plateau. CONCLUSION

Slide 49

Slide 49 text

■ LLMs can be one part of a product or process, and swapped for di ff erent approaches. ■ Iteration and the right tooling can get you past the prototype plateau. ■ There’s no need to compromise on development best practices or privacy. CONCLUSION

Slide 50

Slide 50 text

THANK YOU! 💥 Explosion 💫 spaCy ✨ Prodigy 🐦 Twitter 🐘 Mastodon 🦋 Bluesky 💼 LinkedIn explosion.ai spacy.io prodigy.ai @_inesmontani @[email protected] @inesmontani.bsky.social