SciPy 2017: NEXT

Proceedings PDF: http://conference.scipy.org/proceedings/scipy2017/pdfs/scott_sievert.pdf
Homepage (http://nextml.org) has other links

Transcript

1. NEXT: Crowdsourcing, machine learning and cartoons @stsievert Link to slides and proceedings: tinyurl.com/scipy-next Scott Sievert
2. Data collection can be costly https://xkcd.com/1838/ (comic text hidden) tinyurl.com/scipy-next Problem
3. Data collection done with crowdsourcing can be expensive tinyurl.com/scipy-next Example Goal: achieve goal with minimal responses
4. Existing crowdsourcing systems are passive tinyurl.com/scipy-next One solution Goal: adapt to previously collected responses Adapting to previous responses requires fewer data
5. Adaptive sampling can have large benefits $$$ NEXT Mechanical Turk tinyurl.com/scipy-next Benefits
6. tinyurl.com/scipy-next Example solution Adapting to previous responses yields better results
7. Kevin Jamieson Rob Nowak Lalit Jain Daniel Ross nextml.org Homepage: http://nextml.org Source: https://github.com/nextml/NEXT Documentation: https://github.com/nextml/NEXT/wiki tinyurl.com/scipy-next
8. UW Psychology uses NEXT to find the best algorithms for adaptive data collection in cognitive science. The New Yorker uses NEXT to crowd-source the weekly cartoon caption contest. Air Force Research Lab uses NEXT for active image classification. ML Researchers Experimentalists Practitioners Theory Practice tinyurl.com/scipy-next NEXT users
9. Bob Mankoff Comic by P. C. Vey tinyurl.com/scipy-next Example problem The New Yorker has to find the funniest caption from ~5,000 captions
10. http://www.newyorker.com/cartoons/vote http://nextml.org/captioncontest Comic by P. C. Vey tinyurl.com/scipy-next Interface
11. Histogram of responses Histogram of time responses received Experiment Info Data from contests: https://github.com/nextml/caption-contest-data tinyurl.com/scipy-next Dashboard
12. Crowdsourcing Adaptive sampling algorithms fewer responses more accurate models real-world data participant fatigue algorithm delays participant label quality Goal: let both parties easily use NEXT tinyurl.com/scipy-next Goal
13. NEXT can also be used with REST API tinyurl.com/scipy-next comic by P. C. Vey Cardinal Bandits Select the street that looks safer Dueling Bandits Select face on the bottom most similar to the face on top Pool based triplets Software uses By default, NEXT can be applied to 3 problems
14. tinyurl.com/scipy-next Participants Monitoring Applications Algorithm developer NEXT NEXT use
15. Algorithm developer use getQuery processAnswer getModel initExp Algorithm developer and mathematician Pr (|y ˆ y| < ✏)  1 tinyurl.com/scipy-next
16. 1. Treat algorithms as black boxes • (for each function, inputs and outputs are documented and type-checked) 2. Use wrapper to allow easy access to experiment information and background jobs 3. Objects are abstracted to integers (i.e., object 42, not {‘filename’: foo.png, ‘url’: …}) (more detail in proceedings and on docs) tinyurl.com/scipy-next Algorithm design decisions
17. tinyurl.com/scipy-next Participants Monitoring Applications NEXT Algorithm developer Algorithm use
18. See https://github.com/nextml/NEXT/wiki for details and more launching options (more detail in proceedings and on docs) tinyurl.com/scipy-next Launching NEXT via Amazon EC2 AMI
19. tinyurl.com/scipy-next NEXT startup page
20. 1. Adaptive sampling reduces data collection cost. 2. NEXT is a crowdsourcing data collection tool that can use adaptive sampling techniques 3. NEXT is easy* to use by experimentalists, algorithm developers and practitioners, and a mathematical background is not required. 4. NEXT developers experimentalist engagement to aid research and to gain feedback to improve the software tinyurl.com/scipy-next Key messages * NEXT has been created by an academic research group for collaboration with psychologists
21. https://xkcd.com/1543/ tinyurl.com/scipy-next
22. Extras…
23. • Documented exactly in apps/[app-id]/algs/Algs.yaml • Function implementation Algorithm inputs and outputs Depends on a library we developed: https://github.com/daniel3735928559/pijemont
24. • Fundamentally requires 4 functions: • initExp: experiment initialization • getQuery: selects query to present to participant • processAnswer: process participants response • getModel: provides experiment monitoring Adaptive data flow