Syndrome A concept describing individuals who are marked by an inability to internalize their accomplishments and have a persistent fear of being exposed as a "fraud" Pauline R. Clance and Suzanne A. Imes; 1978
you are interested in personal growth and development, by definition you are always going to be pushing yourself into something which is new. And when things are new, of course we don’t feel as comfortable in our skin as when we are doing something which is deeply familiar to us, and which we’ve been doing for five or 10 years. Caroline Webb
did Mechanical Turk Start? Internal Amazon tool for categorizing products & identifying duplicates Publicly launched as a AWS product in 2005 Marketplace for online micro-jobs
Groups Of HITS (Questions) ~300,000 HITS (Assignments) ~15,000 Most # of HITS In A Single HIT Group $0.01 Lowest HIT Reward $150 Highest HIT Reward (2+ Hours Audio Transcription)
requirements for this talk ✓ Not simple for Artificial Intelligence* ✓ Social Media Content ✓ Pittsburgh-centric ✓ Make attendees excited for dinner/happy hour *Yes- AI can do a lot
I used Turkee by Jim Jones (aantix) Built on top of RTurk; DB Models; Easily converts forms; Launch/Import tasks https://github.com/aantix/turkee on top of: RTurk by Ryan Tate (ryantate) A simple wrapper and library for Amazon's Mechanical Turk https://github.com/ryantate/rturk
name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash)
name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash) batch = Batch.create( name: "Sandwich Wizard", title: "Look at a picture of a delicious sandwich and determine…”, description: "You You will be provided with an image of a sandwich. Please analyze…” instructions: "<p>Please look at the image of a sandwich provided and…”, output_field_name: "category", output_field_opts: { yes: "Yes- it has french fries inside", no: "No- it does not have french fries inside" } ) post_ids = [‘BJ8ohZEAjWQ’, ‘BOn4nJvFAf7’, ‘BQirlsNFAd5’… post_ids.each do |post_id| batch.batch_items.create(post_id: post_id, reward_in_cents: 15) end
process if Adjudicator.new(self).approved? update_attributes(status: “complete”, result: adjudicator.result, confidence: adjudicator.confidence) else queue_for_reprocessing end end end
process if Adjudicator.new(self).approved? update_attributes(status: “complete”, result: adjudicator.result, confidence: adjudicator.confidence) else queue_for_reprocessing end end end Send back to Turk for two more opinions
process if Adjudicator.new(self).approved? update_attributes(status: “complete”, result: adjudicator.result, confidence: adjudicator.confidence) else queue_for_reprocessing end end end
:import_and_process do Turkee::TurkeeTask.process_hits batch_items_to_process = BatchItem.incomplete.all_results_in batch_items_to_process.each(&:process) end
could we extend the app? ✓ Multiple batch items in a TurkeeTask ✓ Complex reprocessing flow ✓ Multiple inputs/outputs for a batch item ✓ Different output types
should we extend the app? ✓ Multiple batch items in a TurkeeTask ✓ Complex reprocessing flow ✓ Multiple inputs/outputs for a batch item ✓ Different output types
Inputs/Outputs BatchItem batch_id post_id reward_in_cents status result confidence Batch name title description instructions output_field_name output_field_opts (json)
Inputs/Outputs BatchItem batch_id post_id reward_in_cents status result confidence Batch name title description instructions output_field_name output_field_opts (json)
key: “has_fries”, format: “categories”, display_settings:{ category_opts: { yes: "Yes- it has french fries inside", no: "No- it does not have french fries inside” } }, adjudicator: ‘mode’, adjudicator_settings:{ acceptance: .5, } )
key: “has_fries”, format: “categories”, display_settings:{ category_opts: { yes: "Yes- it has french fries inside", no: "No- it does not have french fries inside” } }, adjudicator: ‘mode’, adjudicator_settings:{ acceptance: .5, } ) How we display the output How we confirm the output data
key: “fry_count”, format: “counter”, display_settings:{ instruct: “if yes, how many fries”, category_opts: { min: 0, max: 50 } } adjudicator: ‘number’, adjudicator_settings:{ variance: .20, } ) How we display the output How we confirm the output data
of mTurk Amazon Mechanical Turk: Gold Mine or Coal Mine? University of Colorado 2011 Amazon's Mechanical Turk workers protest: 'I am a human being, not an algorithm’ The Guardian December 3 2014
http://turkernation.com/ https://turkopticon.ucsd.edu/ /r/mturk “Turking isn't beer money for me, it's rent money and food money” “12 hours a day on here to make an average of $100 a day” “A higher-end worker on MTurk can expect to make $6-$12 per hour”
andrewglass1
seike460
ski
itsmedreamwalker
pospome
bkuhlmann
mizdra
kasacchiful
youkidearitai
gussan0223
izumin5210
xiombatsg
1wedge_one
kneath
eileencodes
swwweet
bluesmoon
jonyablonski
chrislema
aleyda
lara
ufuk
reverentgeek
csswizardry
sferik
