Applied NLP with LLMs: Beyond Black-Box Monoliths

Large Language Models (LLMs) have enormous potential, but also challenge existing workflows in industry that require modularity, transparency and data privacy. In this talk, I'll show some practical solutions for using the latest state-of-the-art models in real-world applications and distilling their knowledge into smaller and faster components that you can run and maintain in-house.

_________________________________________________

▪️ Case Study #1: https://speakerdeck.com/inesmontani/workshop-half-hour-of-labeling-power-can-we-beat-gpt
▪️ Case Study #2: https://explosion.ai/blog/sp-global-commodities
▪️ Case Study #3: https://explosion.ai/blog/gitlab-support-insights

Transcript

1. Ines Montani Explosion LLM
2. 270m+ 270m+ spaC y Open-source library for industrial- strength natural language processing spacy.io downloads
3. 270m+ 270m+ spaC y ChatGPT can write spaCy code! Open-source library for industrial- strength natural language processing spacy.io downloads
4. 900+ 10k+ Prodi g y Modern scriptable annotation tool for machine learning developers prodigy.ai 900+ companies 10k+ users
5. 900+ 10k+ Prodi g y Modern scriptable annotation tool for machine learning developers prodigy.ai Alex Smith Developer Kim Miller Analyst GPT-4 API 900+ companies 10k+ users
6. Falcon MIXTRAL GPT-4 LLM
7. Falcon MIXTRAL GPT-4 good contextual results LLM
8. Falcon MIXTRAL GPT-4 good contextual results easy to use & configure LLM
9. Falcon MIXTRAL GPT-4 good contextual results easy to use & configure fast prototyping LLM
10. Falcon MIXTRAL GPT-4 good contextual results ⚠ transparency easy to use & configure fast prototyping LLM
11. Falcon MIXTRAL GPT-4 good contextual results ⚠ transparency ⚠ e iciency easy to use & configure fast prototyping LLM
12. Falcon MIXTRAL GPT-4 good contextual results ⚠ data privacy ⚠ transparency ⚠ e iciency easy to use & configure fast prototyping LLM
13. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN
14. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN How to avoid the prototype plateau?
15. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN 📝 standardize inputs and outputs How to avoid the prototype plateau?
16. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN 📝 standardize inputs and outputs 📈 start with evaluation How to avoid the prototype plateau?
17. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN 📝 standardize inputs and outputs 📈 start with evaluation 🔮 assess utility, not just accuracy explosion.ai/blog/applied-nlp-thinking How to avoid the prototype plateau?
18. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN 📝 standardize inputs and outputs 📈 start with evaluation 🔮 assess utility, not just accuracy explosion.ai/blog/applied-nlp-thinking 🛠 work on data iteratively How to avoid the prototype plateau?
19. Pro t ot y pe & Productio n CLOSE THE GAP BETWEEN CLOSE THE GAP BETWEEN 📝 standardize inputs and outputs 📈 start with evaluation 🔮 assess utility, not just accuracy explosion.ai/blog/applied-nlp-thinking 💬 consider structure and ambiguity of natural language 🛠 work on data iteratively How to avoid the prototype plateau?
20. P rototype task-specific output 💬 prompt 📖 text LLM GPT-4 API
21. P rototype task-specific output 💬 prompt 📖 text LLM prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
22. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
23. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm GPT-4 API
24. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ modular GPT-4 API
25. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ small & fast ✅ modular GPT-4 API
26. 📖 text task-specific output P roduction P rototype task-specific output 💬 prompt 📖 text LLM distilled task-specific components prompt model & transform output to structured data github.com/explosion/spacy-llm ✅ data-private ✅ small & fast ✅ modular GPT-4 API
27. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation LLM
28. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM
29. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting
30. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting
31. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting transfer learning CO M PO N EN T
32. in the loop H uma n explosion.ai/blog/human-in-the-loop-distillation continuous evaluation baseline LLM prompting transfer learning CO M PO N EN T distilled model
33. Case Stud y : PyData NYC 8hr 400mb 2k+ 8hr 400mb 2k+ • extracting dishes, ingredients and equipment from r/cooking Reddit posts model size words/second data dev time spacy.fyi/pydata-nyc
34. Case Stud y : PyData NYC 8hr 400mb 2k+ 8hr 400mb 2k+ • extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation model size words/second data dev time spacy.fyi/pydata-nyc
35. Case Stud y : PyData NYC 8hr 400mb 2k+ 8hr 400mb 2k+ • extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation • 20× inference time speedup model size words/second data dev time spacy.fyi/pydata-nyc
36. Case Stud y : PyData NYC 8hr 400mb 2k+ 8hr 400mb 2k+ • extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation • 20× inference time speedup • beat few-shot LLM baseline of 0.74 with task-specific model model size words/second data dev time spacy.fyi/pydata-nyc
37. Case Stud y : PyData NYC 8hr 400mb 2k+ 8hr 400mb 2k+ • extracting dishes, ingredients and equipment from r/cooking Reddit posts • used LLM during annotation • 20× inference time speedup • beat few-shot LLM baseline of 0.74 with task-specific model model size words/second data dev time spacy.fyi/pydata-nyc
38. Case Stud y : S&P Global 99% 6mb 16k+ 99% 6mb 16k+ • real-time commodities trading insights by extracting structured attributes model size words/second F-score explosion.ai/blog/sp-global-commodities
39. Case Stud y : S&P Global 99% 6mb 16k+ 99% 6mb 16k+ • real-time commodities trading insights by extracting structured attributes • high-security environment model size words/second F-score explosion.ai/blog/sp-global-commodities
40. Case Stud y : S&P Global 99% 6mb 16k+ 99% 6mb 16k+ • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation model size words/second F-score explosion.ai/blog/sp-global-commodities
41. Case Stud y : S&P Global 99% 6mb 16k+ 99% 6mb 16k+ • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop model size words/second F-score explosion.ai/blog/sp-global-commodities
42. Case Stud y : S&P Global 99% 6mb 16k+ 99% 6mb 16k+ • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production model size words/second F-score explosion.ai/blog/sp-global-commodities
43. Case Stud y : S&P Global 99% 6mb 16k+ 99% 6mb 16k+ • real-time commodities trading insights by extracting structured attributes • high-security environment • used LLM during annotation • 10× data development speedup with humans and model in the loop • 8 market pipelines in production model size words/second F-score explosion.ai/blog/sp-global-commodities
45. break down larger problems
46. break down larger problems make problem easier
47. break down larger problems make problem easier reassess dependencies
48. break down larger problems make problem easier reassess dependencies choose the best techniques
49. break down larger problems make problem easier reassess dependencies choose the best techniques iterate on code and data
50. break down larger problems make problem easier factor out business logic reassess dependencies choose the best techniques iterate on code and data
51. Case Stud y : GitLab 1 year 6× 1 year 6× • extract actionable insights from support tickets and usage questions speedup of support tickets explosion.ai/blog/gitlab-support-insights
52. Case Stud y : GitLab 1 year 6× 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment speedup of support tickets explosion.ai/blog/gitlab-support-insights
53. Case Stud y : GitLab 1 year 6× 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions speedup of support tickets explosion.ai/blog/gitlab-support-insights
54. Case Stud y : GitLab 1 year 6× 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions • separated general-purpose features from product-specific logic speedup of support tickets explosion.ai/blog/gitlab-support-insights
55. Case Stud y : GitLab 1 year 6× 1 year 6× • extract actionable insights from support tickets and usage questions • high-security environment • easy to adapt to new scenarios and business questions • separated general-purpose features from product-specific logic speedup of support tickets explosion.ai/blog/gitlab-support-insights
56. Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI
57. Reason and refactor. The key to success lies in your data and may surprise you! Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI
58. Reason and refactor. The key to success lies in your data and may surprise you! Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI Iterate. The right tooling and mindset gets you past the “prototype plateau”.
59. Reason and refactor. The key to success lies in your data and may surprise you! LLM Stay ambitious. Don’t compromise on best practices, e iciency and privacy. Summar y APPLIED NLP & GEN AI APPLIED NLP & GEN AI Iterate. The right tooling and mindset gets you past the “prototype plateau”.
60. Explosion spaCy Prodigy Twitter Mastodon Bluesky explosion.ai spacy.io prodigy.ai @_inesmontani @ines@sigmoid.social @inesmontani.bsky.social LinkedIn