Efficient data development: Optimizing annotation workflows

Tips and advice for how to build efficient human-in-the-loop data development workflows, break down business problems into actionable annotation steps and make the most of automation and model assistance. All examples are inspired by real use cases.

Blog post: https://explosion.ai/blog/optimizing-annotation-workflows

Transcript

1. Ines Montani Explosion Optimizing annotation workflows
2. 520m+ downloads Open-source library for industrial-strength natural language processing spacy.io SPACY Modern scriptable annotation tool for machine learning developers prodigy.ai PRODIgy 12,000+ users
8. math
9. F actor out business logic: result = business_logic(classification(text)) math
10. Annotation guidelines: Annotation Meetings: explosion.ai/blog/guardian
12. DO THIS for annotation_type in annotation_types: for example in examples: annotate(example, annotation_type) ✅ NOT THIS for example in examples: for annotation_type in annotation_types: annotate(example, annotation_type) ❌ Humans have a cache, too!
13. DO THIS for annotation_type in annotation_types: for example in examples: annotate(example, annotation_type) ✅ NOT THIS for example in examples: for annotation_type in annotation_types: annotate(example, annotation_type) ❌ Humans have a cache, too! 1. labels, classification, spans etc. 2. individual examples
14. explosion.ai/blog/human-in-the-loop-distillation
15. explosion.ai/blog/human-in-the-loop-distillation
16. explosion.ai/blog/sp-global-commodities
17. explosion.ai/blog/sp-global-commodities
18. explosion.ai/blog/sp-global-commodities 10× faster overall:
20. variate ana prodigy.ai/docs selection “snaps” to token boundaries Automate what machines are better at!
21. variate ana prodigy.ai/docs selection “snaps” to token boundaries Automate what machines are better at!
22. variate ana prodigy.ai/docs selection “snaps” to token boundaries GPT-4 API model pre-annotates Automate what machines are better at!
23. Reframe the problem:
24. Reframe the problem: generated by model
26. prodigy.ai/docs/recipes Will more data improve the model?
27. =============== Train curve diagnostic =============== Training 4 times with 25%, 50%, 75%, 100% of the data % Score ner ---- ------ ------ 0% 0.00 0.00 25% 0.31 ▲ 0.31 ▲ 50% 0.44 ▲ 0.44 ▲ 75% 0.43 ▼ 0.43 ▼ 100% 0.56 ▲ 0.56 ▲ Prodigy train curve experiment: prodigy.ai/docs/recipes Will more data improve the model?
28. =============== Train curve diagnostic =============== Training 4 times with 25%, 50%, 75%, 100% of the data % Score ner ---- ------ ------ 0% 0.00 0.00 25% 0.31 ▲ 0.31 ▲ 50% 0.44 ▲ 0.44 ▲ 75% 0.43 ▼ 0.43 ▼ 100% 0.56 ▲ 0.56 ▲ Prodigy train curve experiment: Accuracy 0 25 50 75 100 % of examples 25 50 75 100 125 150 projection prodigy.ai/docs/recipes Will more data improve the model?
29. explosion.ai/blog/sp-global-commodities Workflow:
30. 🧑💻 human experts in the loop explosion.ai/blog/sp-global-commodities Workflow:
31. 🧑💻 human experts in the loop 🔮 suggestions from LLM explosion.ai/blog/sp-global-commodities Workflow:
32. 🧑💻 human experts in the loop 📚 task-specific data ➕ 🔮 suggestions from LLM explosion.ai/blog/sp-global-commodities Workflow:
33. 🧑💻 human experts in the loop 📦 model package 📚 task-specific data ➕ 🔮 suggestions from LLM explosion.ai/blog/sp-global-commodities Workflow:
34. 🧑💻 human experts in the loop 🚀 structured data 📦 model package 📚 task-specific data ➕ 🔮 suggestions from LLM explosion.ai/blog/sp-global-commodities Workflow:
35. Explosion spaCy Prodigy Bluesky Mastodon explosion.ai spacy.io prodigy.ai @inesmontani.bsky.social @ines@sigmoid.social LinkedIn