Incorporating LLMs into practical NLP workflows

In this talk, I'll show how large language models such as GPT-3 complement rather than replace existing machine learning workflows. Initial annotations are gathered from the OpenAI API via zero- or few-shot learning, and then corrected by a human decision maker using an annotation tool. The resulting annotations can then be used to train and evaluate models as normal. This process results in higher accuracy than can be achieved from the OpenAI API alone, with the added benefit that you'll own and control the model for runtime.

Video: https://youtu.be/Bd2ciwinFUE

Transcript

1. Ines Montani Explosion incorporating 
 llms into practical 
 nlp workflows
2. spaCy Open-source library for industrial-strength Natural Language Processing 100k+ USERS 130m+ DOWNLOADS → spacy.io
3. → spacy.io
4. prodigy Annotation tool for creating training data for machine learning models 8000+ USERS → prodigy.ai
5. → prodigy.ai
6. incorporating 
 llms* into practical 
 nlp workflows * large language models
7. practical 
 workflows
8. • supervised learning practical 
 workflows
9. • supervised learning • tell computers exactly what to do practical 
 workflows
10. • supervised learning • tell computers exactly what to do • needs enough good data practical 
 workflows
11. • supervised learning • tell computers exactly what to do • needs enough good data • ML + business logic practical 
 workflows
12. LLMs as a tool #1 specific is better
13. faster is better LLMs as a tool #2
14. private is better LLMs as a tool #3
15. better is better LLMs as a tool #4
16. problems
17. problems • prompt engineering
18. problems • prompt engineering • inconsistent results
19. problems • prompt engineering • inconsistent results • unstructured responses
20. working 
 with llms
21. working 
 with llms • iterative (prompting, parsing)
22. working 
 with llms • iterative (prompting, parsing) • evaluation is extremely important
23. working 
 with llms • iterative (prompting, parsing) • evaluation is extremely important • improve, not replace task-specific models
24. working 
 with llms • iterative (prompting, parsing) • evaluation is extremely important • improve, not replace task-specific models scriptable workflows
25. working 
 with llms • iterative (prompting, parsing) • evaluation is extremely important • improve, not replace task-specific models scriptable workflows human in the loop
26. working 
 with llms • iterative (prompting, parsing) • evaluation is extremely important • improve, not replace task-specific models scriptable workflows human in the loop business logic
27. → github.com/explosion/prodigy-openai-recipes
28. → prodigy.ai
29. → prodigy.ai query LLM and parse response
30. → prodigy.ai query LLM and parse response tune prompt if needed
31. → prodigy.ai
32. → prodigy.ai
33. → prodigy.ai correct mistakes
34. → prodigy.ai correct mistakes
35. → prodigy.ai correct mistakes add correct answer to prompt to tune it
36. → prodigy.ai
37. → prodigy.ai generate and display reason
38. → prodigy.ai
39. reality is not 
 an end-to-end 
 prediction problem
40. “Microsoft acquires software development platform GitHub for $7.5 billion”
41. “Microsoft acquires software development platform GitHub for $7.5 billion”
42. TEXT CLASSIFIER “Microsoft acquires software development platform GitHub for $7.5 billion”
43. TEXT CLASSIFIER ENTITY RECOGNIZER “Microsoft acquires software development platform GitHub for $7.5 billion”
44. TEXT CLASSIFIER ENTITY RECOGNIZER ENTITY LINKER “Microsoft acquires software development platform GitHub for $7.5 billion”
45. TEXT CLASSIFIER ENTITY RECOGNIZER ENTITY LINKER ATTRIBUTE LOOKUP “Microsoft acquires software development platform GitHub for $7.5 billion”
46. TEXT CLASSIFIER ENTITY RECOGNIZER ENTITY LINKER ATTRIBUTE LOOKUP CURRENCY NORMALIZER “Microsoft acquires software development platform GitHub for $7.5 billion”
47. TEXT CLASSIFIER ENTITY RECOGNIZER ENTITY LINKER ATTRIBUTE LOOKUP CURRENCY NORMALIZER “Microsoft acquires software development platform GitHub for $7.5 billion” * *
48. → github.com/explosion/prodigy-openai-recipes summary
49. → github.com/explosion/prodigy-openai-recipes summary • LLMs are a great tool for creating better data 
 faster and iteratively
50. → github.com/explosion/prodigy-openai-recipes summary • LLMs are a great tool for creating better data 
 faster and iteratively • you’ll always need task-specific data
51. → github.com/explosion/prodigy-openai-recipes summary • LLMs are a great tool for creating better data 
 faster and iteratively • you’ll always need task-specific data • many new applications in the future
52. future 
 work
53. future 
 work • data structures for result parsing
54. future 
 work • data structures for result parsing • workflows for robust evaluation
55. future 
 work • data structures for result parsing • workflows for robust evaluation • interactive prompt testing
56. future 
 work • data structures for result parsing • workflows for robust evaluation • interactive prompt testing • support for open-source models
57. 💥 Explosion 
 explosion.ai 📲 Twitter 
 @_inesmontani 📲 Mastodon 
 @ines@sigmoid.social thank you!