Workshop: Half hour of labeling power: Can we beat GPT?

Video: https://www.youtube.com/watch?v=Ta45SfbZNcM

Large Language Models (LLMs) offer a lot of value for modern NLP and can typically achieve surprisingly good accuracy on predictive NLP tasks with a reasonably structured prompt and pretty much no labelled examples. But can we do even better than that? It’s much more effective to use LLMs to create classifiers, instead of using them as classifiers. By using LLMs to assist with annotation, we can quickly create labelled data and systems that are much faster and much more accurate than using LLM prompts alone. In this workshop, we'll show you how to use LLMs at development time to create high-quality datasets and train specific, smaller, private and more accurate fine-tuned models for your business problems.

Transcript

1. Ines Montani & Ryan Wesslen Explosion Half hour of labeling power Can we beat GPT?
2. spacy.io
3. spacy.io Open-source library for industrial-strength natural language processing 170m+ downloads
4. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+ downloads
5. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+ downloads Modern scriptable annotation tool for machine learning developers 9k+ users 800+ companies
6. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+ downloads Modern scriptable annotation tool for machine learning developers 9k+ users 800+ companies prodigy.ai/teams
7. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+ downloads Modern scriptable annotation tool for machine learning developers 9k+ users 800+ companies prodigy.ai/teams Collaborative data development platform GPT-4 API Alex Smith Developer
8. Generative ! single/multi-doc summarization " reasoning ✅ problem solving ✍ paraphrasing % style transfer ❓question answering Predictive ' text classification ( relation extraction ) coreference * grammar & morphology + entity recognition , semantic parsing - discourse structure
9. SST2 AG News Banking77 GPT-3 65 70 75 80 85 90 95 100 1% 5% 10% 20% 50% 100% Text Classification
10. SST2 AG News Banking77 GPT-3 65 70 75 80 85 90 95 100 1% 5% 10% 20% 50% 100% Text Classification 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 FabNER Claude 2 Entity Recognition
11. SST2 AG News Banking77 GPT-3 65 70 75 80 85 90 95 100 1% 5% 10% 20% 50% 100% Text Classification 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 FabNER Claude 2 Entity Recognition
13. 1 2
14. Correct LLM few-shot results
15. Correct LLM few-shot results
16. Annotation Guidelines DISH known food dishes, e.g. lobster ravioli, garlic bread INGREDIENT EQUIPMENT individual parts of a food dish, including herbs and spices any kind of cooking equipment, e.g. oven, cooking pot, grill
17. annotate evaluate update
18. annotate evaluate update 1
19. annotate evaluate update 1 resolve disagreements retrospective meetings assess if more data is needed 2
20. annotate evaluate update 1 resolve disagreements retrospective meetings assess if more data is needed 2 update annotation guidelines add more examples expand label definitions 3
21. spacy-llm config prompt template spacy.io/usage/large-language-models
22. Evaluation Results 0 20 40 60 80 100 Zero-shot Chain-of-thought Few-shot Task-specific ? F DISH (F) INGREDIENT (F) EQUIPMENT (F)
23. Evaluation Results 0 20 40 60 80 100 Zero-shot Chain-of-thought Few-shot Task-specific ? F DISH (F) INGREDIENT (F) EQUIPMENT (F)
24. Annotation Guidelines DISH known food dishes, e.g. lobster ravioli, garlic bread INGREDIENT EQUIPMENT individual parts of a food dish, including herbs and spices any kind of cooking equipment, e.g. oven, cooking pot, grill
25. Evaluation Results 0 20 40 60 80 100 Zero-shot Chain-of-thought Few-shot Task-specific F DISH (F) INGREDIENT (F) EQUIPMENT (F) 2000 words/second
26. prodigy.ai/features/task-routing Use task routing to distribute workloads and determine inter-annotator agreement. pro tip:
27. koaning.io/posts/large-disagreement-models Focus on examples where models disagree, similar to active learning. pro tip:
28. ChatGPT Use generative models to create spaCy rule sets! pro tip: spacy.io/usage/rule-based-matching
29. Takeaways Generative complements predictive, it doesn't replace it.
30. Takeaways Generative complements predictive, it doesn't replace it. Use generative models to create better, more accurate, faster, smaller and private task-specific models.
31. Takeaways Generative complements predictive, it doesn't replace it. Use generative models to create better, more accurate, faster, smaller and private task-specific models. With good tooling, you can make human input more e icient.
32. thank you! Explosion spaCy Prodigy explosion.ai spacy.io prodigy.ai Twitter Mastodon Bluesky @explosion_ai @explosion@sigmoid.social @explosion-ai.bsky.social LinkedIn