eLUCID8 NEXT talk

Transcript

1. NEXT: Crowdsourcing, machine learning and cartoons Scott Sievert LUCID trainee UW–Madison ECE
2. Data collection with crowdsourcing can be expensive Problem
3. comic by P. C. Vey Cardinal Bandits Select the street that looks safer Dueling Bandits Select face on the bottom most similar to the face on top Pool based triplets Crowdsourcing problems
4. Existing crowdsourcing systems are passive Goal: adapt to previously collected responses Adapting to previous responses requires fewer data One solution
5. Adaptive sampling can have large benefits NEXT Mechanical Turk Benefits
6. unlabeled data Example of adaptive algorithm
7. Kevin Jamieson Rob Nowak Lalit Jain Daniel Ross nextml.org Homepage: http://nextml.org Source: https://github.com/nextml/NEXT Documentation: https://github.com/nextml/NEXT/wiki
8. UW Psychology uses NEXT to find the best algorithms for adaptive data collection in cognitive science. The New Yorker uses NEXT to crowd-source the weekly cartoon caption contest. Air Force Research Lab uses NEXT for active image classification. ML Researchers Experimentalists Practitioners Theory Practice NEXT users
9. Bob Mankoff Comic by P. C. Vey The New Yorker has to find the funniest caption from ~5,000 captions Example problem
10. http://www.newyorker.com/cartoons/vote http://nextml.org/captioncontest Comic by P. C. Vey Interface
11. Histogram of responses Histogram of time responses received Experiment Info Data from contests: https://github.com/nextml/caption-contest-data Dashboard
12. 0 not funny 1 somewhat funny 2 funny Data 4x times fewer ratings needed Experimentalist Benefits BR-lilUCB Random Software enhancements
13. Crowdsourcing Adaptive sampling algorithms fewer responses more accurate models real-world data participant fatigue, label quality, algorithm delays Goal: enable this feedback loop Enabling this feedback loop requires software that is easy to use by both parties NEXT
14. NEXT can also be used with REST API comic by P. C. Vey Cardinal Bandits Select the street that looks safer Dueling Bandits Select face on the bottom most similar to the face on top Pool based triplets By default, NEXT has adaptive algorithms for the 3 default question types Software uses
15. More detail on documentation: https://github.com/nextml/NEXT/wiki 0. Web browser 1. Amazon AWS account 2. ZIP of targets (e.g., images) 3. Experiment description (which has good documentation!) Result requirements After NEXT link sent to crowdsourcing service, results can be generated!
16. 1. Adaptive sampling reduces data collection cost. 2. NEXT is a crowdsourcing data collection tool that can use adaptive sampling techniques 3. NEXT is easy* to use by experimentalists, algorithm developers and practitioners, and a mathematical background is not required. 4. NEXT developers experimentalist engagement to aid research and to gain feedback to improve the software * NEXT has been created by an academic research group in collaboration with psychologists Key messages
17. https://xkcd.com/1543/