Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Workshop: Half hour of labeling power: Can we b...

Workshop: Half hour of labeling power: Can we beat GPT?

Video: https://www.youtube.com/watch?v=Ta45SfbZNcM

Large Language Models (LLMs) offer a lot of value for modern NLP and can typically achieve surprisingly good accuracy on predictive NLP tasks with a reasonably structured prompt and pretty much no labelled examples. But can we do even better than that? It’s much more effective to use LLMs to create classifiers, instead of using them as classifiers. By using LLMs to assist with annotation, we can quickly create labelled data and systems that are much faster and much more accurate than using LLM prompts alone. In this workshop, we'll show you how to use LLMs at development time to create high-quality datasets and train specific, smaller, private and more accurate fine-tuned models for your business problems.

Ines Montani

November 01, 2023
Tweet

Video

More Decks by Ines Montani

Other Decks in Programming

Transcript

  1. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+

    downloads Modern scriptable annotation tool for machine learning developers 9k+ users 800+ companies
  2. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+

    downloads Modern scriptable annotation tool for machine learning developers 9k+ users 800+ companies prodigy.ai/teams
  3. spacy.io prodigy.ai Open-source library for industrial-strength natural language processing 170m+

    downloads Modern scriptable annotation tool for machine learning developers 9k+ users 800+ companies prodigy.ai/teams Collaborative data development platform GPT-4 API Alex Smith Developer
  4. Generative ! single/multi-doc summarization " reasoning ✅ problem solving ✍

    paraphrasing % style transfer ❓question answering Predictive ' text classification ( relation extraction ) coreference * grammar & morphology + entity recognition , semantic parsing - discourse structure
  5. SST2 AG News Banking77 GPT-3 65 70 75 80 85

    90 95 100 1% 5% 10% 20% 50% 100% Text Classification
  6. SST2 AG News Banking77 GPT-3 65 70 75 80 85

    90 95 100 1% 5% 10% 20% 50% 100% Text Classification 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 FabNER Claude 2 Entity Recognition
  7. SST2 AG News Banking77 GPT-3 65 70 75 80 85

    90 95 100 1% 5% 10% 20% 50% 100% Text Classification 10 20 30 40 50 60 70 80 90 100 0 100 200 300 400 500 FabNER Claude 2 Entity Recognition
  8. 1

  9. 1 2

  10. Annotation Guidelines DISH known food dishes, e.g. lobster ravioli, garlic

    bread INGREDIENT EQUIPMENT individual parts of a food dish, including herbs and spices any kind of cooking equipment, e.g. oven, cooking pot, grill
  11. annotate evaluate update 1 resolve disagreements retrospective meetings assess if

    more data is needed 2 update annotation guidelines add more examples expand label definitions 3
  12. Evaluation Results 0 20 40 60 80 100 Zero-shot Chain-of-thought

    Few-shot Task-specific ? F DISH (F) INGREDIENT (F) EQUIPMENT (F)
  13. Evaluation Results 0 20 40 60 80 100 Zero-shot Chain-of-thought

    Few-shot Task-specific ? F DISH (F) INGREDIENT (F) EQUIPMENT (F)
  14. Annotation Guidelines DISH known food dishes, e.g. lobster ravioli, garlic

    bread INGREDIENT EQUIPMENT individual parts of a food dish, including herbs and spices any kind of cooking equipment, e.g. oven, cooking pot, grill
  15. Evaluation Results 0 20 40 60 80 100 Zero-shot Chain-of-thought

    Few-shot Task-specific F DISH (F) INGREDIENT (F) EQUIPMENT (F) 2000 words/second
  16. ChatGPT Use generative models to create spaCy rule sets! pro

    tip: spacy.io/usage/rule-based-matching
  17. Takeaways Generative complements predictive, it doesn't replace it. Use generative

    models to create better, more accurate, faster, smaller and private task-specific models.
  18. Takeaways Generative complements predictive, it doesn't replace it. Use generative

    models to create better, more accurate, faster, smaller and private task-specific models. With good tooling, you can make human input more e icient.