Tweet
Tweet

Slide 1

Slide 1 text

Human Powered Rails Automated Crowdsourcing In Your Rails App Andy Glass RailsConf 2018 Unusual Rails Apps

Slide 2

Slide 2 text

Andy Glass | RailsConf 2018 | Human Powered Rails Why am I talking about mTurk?

Slide 3

Slide 3 text

Andy Glass | RailsConf 2018 | Human Powered Rails But first: Thanks, Rails

Slide 4

Slide 4 text

Andy Glass | RailsConf 2018 | Human Powered Rails So, I owe you unusual…

Slide 5

Slide 5 text

Andy Glass | RailsConf 2018 | Human Powered Rails This is not a talk about Mechanical Turk

Slide 6

Slide 6 text

Andy Glass | RailsConf 2018 | Human Powered Rails This is not a talk about Mechanical Turk (yet)

Slide 7

Slide 7 text

Andy Glass | RailsConf 2018 | Human Powered Rails This is not a talk about Crowdsourcing

Slide 8

Slide 8 text

Andy Glass | RailsConf 2018 | Human Powered Rails This is a talk about being an imposter

Slide 9

Slide 9 text

Andy Glass | RailsConf 2018 | Human Powered Rails Imposter Syndrome A concept describing individuals who are marked by an inability to internalize their accomplishments and have a persistent fear of being exposed as a "fraud" Pauline R. Clance and Suzanne A. Imes; 1978

Slide 10

Slide 10 text

Andy Glass | RailsConf 2018 | Human Powered Rails We’re in good company… Maya Angelou Tom Hanks Neil Gaiman Sheryl Sandberg Sonia Sotomayor

Slide 11

Slide 11 text

Andy Glass | RailsConf 2018 | Human Powered Rails Is Imposter Syndrome bad?

Slide 12

Slide 12 text

Andy Glass | RailsConf 2018 | Human Powered Rails Is Imposter Syndrome good?

Slide 13

Slide 13 text

Andy Glass | RailsConf 2018 | Human Powered Rails Is Imposter Syndrome good? Yeah.

Slide 14

Slide 14 text

Andy Glass | RailsConf 2018 | Human Powered Rails Why Imposter Syndrome is Good for you Shane Ferro Huffington Post January 15, 2016

Slide 15

Slide 15 text

Andy Glass | RailsConf 2018 | Human Powered Rails If you are interested in personal growth and development, by definition you are always going to be pushing yourself into something which is new. And when things are new, of course we don’t feel as comfortable in our skin as when we are doing something which is deeply familiar to us, and which we’ve been doing for five or 10 years. Caroline Webb

Slide 16

Slide 16 text

Andy Glass | RailsConf 2018 | Human Powered Rails Imposter Syndrome Experience

Slide 17

Slide 17 text

Andy Glass | RailsConf 2018 | Human Powered Rails This is not a talk about Mechanical Turk

Slide 18

Slide 18 text

Andy Glass | RailsConf 2018 | Human Powered Rails This is a talk about learning to fake shit

Slide 19

Slide 19 text

Andy Glass | RailsConf 2018 | Human Powered Rails Amazon Mechanical Turk is about faking shit

Slide 20

Slide 20 text

Andy Glass | RailsConf 2018 | Human Powered Rails Amazon Mechanical Turk is about faking shit (ok, now lets talk about turk)

Slide 21

Slide 21 text

Andy Glass | RailsConf 2018 | Human Powered Rails How did Mechanical Turk Start? Internal Amazon tool for categorizing products & identifying duplicates Publicly launched as a AWS product in 2005 Marketplace for online micro-jobs

Slide 22

Slide 22 text

Andy Glass | RailsConf 2018 | Human Powered Rails The Turk

Slide 23

Slide 23 text

Andy Glass | RailsConf 2018 | Human Powered Rails I told you it was about faking shit…

Slide 24

Slide 24 text

Andy Glass | RailsConf 2018 | Human Powered Rails What can we use mTurk for? • Image/Video Processing • Data verification & Clean up • Information Gathering • Data processing

Slide 25

Slide 25 text

Andy Glass | RailsConf 2018 | Human Powered Rails Lets make some (as a turk worker)

Slide 26

Slide 26 text

Andy Glass | RailsConf 2018 | Human Powered Rails ~1,500 Groups Of HITS (Questions) ~300,000 HITS (Assignments) ~15,000 Most # of HITS In A Single HIT Group $0.01 Lowest HIT Reward $150 Highest HIT Reward (2+ Hours Audio Transcription)

Slide 27

Slide 27 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 28

Slide 28 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 29

Slide 29 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 30

Slide 30 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 31

Slide 31 text

Andy Glass | RailsConf 2018 | Human Powered Rails What should we use mTurk for today?

Slide 32

Slide 32 text

Andy Glass | RailsConf 2018 | Human Powered Rails My requirements for this talk ✓ Not simple for Artificial Intelligence* ✓ Social Media Content ✓ Pittsburgh-centric ✓ Make attendees excited for dinner/happy hour *Yes- AI can do a lot

Slide 33

Slide 33 text

Andy Glass | RailsConf 2018 | Human Powered Rails Does this sandwich have french fries in it?

Slide 34

Slide 34 text

Andy Glass | RailsConf 2018 | Human Powered Rails Manual mTurk process ✓ Assemble sample data (IG Posts) ✓ Create new mTurk project ✓ Load mTurk batch ✓ Review Results

Slide 35

Slide 35 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 36

Slide 36 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 37

Slide 37 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 38

Slide 38 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 39

Slide 39 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 40

Slide 40 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 41

Slide 41 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 42

Slide 42 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 43

Slide 43 text

Andy Glass | RailsConf 2018 | Human Powered Rails **plays Cardi B album for 23 minutes**

Slide 44

Slide 44 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 45

Slide 45 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 46

Slide 46 text

Andy Glass | RailsConf 2018 | Human Powered Rails 4 seconds Min time 702 seconds Max time 42 seconds Median Time 239 seconds Avg Time

Slide 47

Slide 47 text

Andy Glass | RailsConf 2018 | Human Powered Rails 1 Min assignments/worker 10 Max assignments/worker 3 Avg assignments/worker

Slide 48

Slide 48 text

Andy Glass | RailsConf 2018 | Human Powered Rails 100% Consensus (both workers agreed) 95.2% Accuracy (20/21 sandwiches)

Slide 49

Slide 49 text

Andy Glass | RailsConf 2018 | Human Powered Rails Correctly identified: fries

Slide 50

Slide 50 text

Andy Glass | RailsConf 2018 | Human Powered Rails Correctly identified: no fries

Slide 51

Slide 51 text

Andy Glass | RailsConf 2018 | Human Powered Rails The edge-case

Slide 52

Slide 52 text

Andy Glass | RailsConf 2018 | Human Powered Rails The edge-case

Slide 53

Slide 53 text

Andy Glass | RailsConf 2018 | Human Powered Rails The edge-case

Slide 54

Slide 54 text

Andy Glass | RailsConf 2018 | Human Powered Rails There are tips + tricks to getting accurate results

Slide 55

Slide 55 text

Andy Glass | RailsConf 2018 | Human Powered Rails We aren’t here to use .csv

Slide 56

Slide 56 text

Andy Glass | RailsConf 2018 | Human Powered Rails Lets Automate It

Slide 57

Slide 57 text

Andy Glass | RailsConf 2018 | Human Powered Rails Automation ✓ Create mTurk RoR service ✓ Create processes for loading tasks ✓ Create processes for approving results/re-inputting tasks ✓ Serve results via API

Slide 58

Slide 58 text

Andy Glass | RailsConf 2018 | Human Powered Rails Gems I used Turkee by Jim Jones (aantix) Built on top of RTurk; DB Models; Easily converts forms; Launch/Import tasks https://github.com/aantix/turkee on top of: 
 RTurk by Ryan Tate (ryantate) A simple wrapper and library for Amazon's Mechanical Turk https://github.com/ryantate/rturk

Slide 59

Slide 59 text

Andy Glass | RailsConf 2018 | Human Powered Rails Batch name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash)

Slide 60

Slide 60 text

Andy Glass | RailsConf 2018 | Human Powered Rails Batch name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash) batch = Batch.create( name: "Sandwich Wizard", title: "Look at a picture of a delicious sandwich and determine…”, description: "You You will be provided with an image of a sandwich. Please analyze…” instructions: "

Please look at the image of a sandwich provided and…”, output_field_name: "category", output_field_opts: { yes: "Yes- it has french fries inside", no: "No- it does not have french fries inside" } ) post_ids = [‘BJ8ohZEAjWQ’, ‘BOn4nJvFAf7’, ‘BQirlsNFAd5’… post_ids.each do |post_id| batch.batch_items.create(post_id: post_id, reward_in_cents: 15) end

Slide 61

Slide 61 text

rails g turkee 1. Creates config file (fill with AWS credentials) 2. Creates 2 DB models

Slide 62

Slide 62 text

Andy Glass | RailsConf 2018 | Human Powered Rails Batch name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash) Turkee::TurkeeTask Turkee::TurkeeImportedAssignment turkee_task_id assignment_id (from mturk) worker_id (from mturk) result_id hit_url (from mturk) task_type hit_id complete (boolean) number_completed_assignments (integer)

Slide 63

Slide 63 text

batch.batch_items.each(&:launch)

Slide 64

Slide 64 text

Andy Glass | RailsConf 2018 | Human Powered Rails host = “https://sandwichspotter.herokuapp.com” hit_title = batch.title hit_description = batch.description model_name = :BatchItemResult num_assignments = 2 reward = batch_item.price_in_cents/100.0 hit_lifetime = 5 #days duration = 1 #hours qualifications = {approval_rate: {gt: 95}} params = {} opts = {form_url: "https://sandwichspotter.herokuapp.com/ batches/#{batch.id}/batch_items/#{id}/batch_item_results/new"}

Slide 65

Slide 65 text

Andy Glass | RailsConf 2018 | Human Powered Rails host = “https://sandwichspotter.herokuapp.com” hit_title = batch.title hit_description = batch.description model_name = :BatchItemResult num_assignments = 2 reward = batch_item.price_in_cents/100.0 hit_lifetime = 5 #days duration = 1 #hours qualifications = {approval_rate: {gt: 95}} params = {} opts = {form_url: "https://sandwichspotter.herokuapp.com/ batches/#{batch.id}/batch_items/#{id}/batch_item_results/new"}

Slide 66

Slide 66 text

Andy Glass | RailsConf 2018 | Human Powered Rails host = “https://sandwichspotter.herokuapp.com” hit_title = batch.title hit_description = batch.description model_name = :BatchItemResult num_assignments = 2 reward = batch_item.price_in_cents/100.0 hit_lifetime = 5 #days duration = 1 #hours qualifications = {approval_rate: {gt: 95}} params = {} opts = {form_url: "https://sandwichspotter.herokuapp.com/ batches/#{batch.id}/batch_items/#{id}/batch_item_results/new"}

Slide 67

Slide 67 text

Andy Glass | RailsConf 2018 | Human Powered Rails host = “https://sandwichspotter.herokuapp.com” hit_title = batch.title hit_description = batch.description model_name = :BatchItemResult num_assignments = 2 reward = batch_item.price_in_cents/100.0 hit_lifetime = 5 #days duration = 1 #hours qualifications = {approval_rate: {gt: 95}} params = {} opts = {form_url: "https://sandwichspotter.herokuapp.com/ batches/#{batch.id}/batch_items/#{id}/batch_item_results/new"}

Slide 68

Slide 68 text

Andy Glass | RailsConf 2018 | Human Powered Rails host = “https://sandwichspotter.herokuapp.com” hit_title = batch.title hit_description = batch.description model_name = :BatchItemResult num_assignments = 2 reward = batch_item.price_in_cents/100.0 hit_lifetime = 5 #days duration = 1 #hours qualifications = {approval_rate: {gt: 95}} params = {} opts = {form_url: "https://sandwichspotter.herokuapp.com/ batches/#{batch.id}/batch_items/#{id}/batch_item_results/new"}

Slide 69

Slide 69 text

Andy Glass | RailsConf 2018 | Human Powered Rails Turkee::TurkeeTask.create_hit(host, hit_title, hit_description, model_name, num_assignments, reward, hit_lifetime, duration, qualifications, params, opts)

Slide 70

Slide 70 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch_item_results/new.html.erb

<%= @batch.title %>

<%= @batch.description %>

<%= @batch.instructions.html_safe %>

<%= turkee_form_for @batch_item_result, params %> <%= @batch.output_field_opts.each do |label, instruction| %>

<%= instruction %>

]” type=“radio" value="<%= label %>" /> <% end %> <%= f.submit “submit" %> <% end %>

Slide 71

Slide 71 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch_item_results/new.html.erb

<%= @batch.title %>

<%= @batch.description %>

<%= @batch.instructions.html_safe %>

<%= turkee_form_for @batch_item_result, params %> <%= @batch.output_field_opts.each do |label, instruction| %>

<%= instruction %>

]” type=“radio" value="<%= label %>" /> <% end %> <%= f.submit “submit" %> <% end %> -Sets form url to https://www.mturk.com/mturk/externalSubmit -Adds hidden fields for capture (worker id, assignment ids, etc) Lets your form mimic a regular rails form

Slide 72

Slide 72 text

Andy Glass | RailsConf 2018 | Human Powered Rails Importing result data

Slide 73

Slide 73 text

Andy Glass | RailsConf 2018 | Human Powered Rails Importing result data Turkee::TurkeeTask.process_hits ✓ Creates Turkee::TurkeeImportedAssignments records ✓ Creates BatchItemResponse records

Slide 74

Slide 74 text

Andy Glass | RailsConf 2018 | Human Powered Rails Importing result data # #’No’} >

Slide 75

Slide 75 text

Andy Glass | RailsConf 2018 | Human Powered Rails Batch name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash) Turkee::TurkeeTask Turkee::TurkeeImportedAssignment turkee_task_id assignment_id (from mturk) worker_id (from mturk) result_id hit_url (from mturk) task_type hit_id complete (boolean) number_completed_assignments (integer)

Slide 76

Slide 76 text

Andy Glass | RailsConf 2018 | Human Powered Rails Batch name title description instructions output_field_name output_field_opts (hash) BatchItem batch_id post_id reward_in_cents status result confidence BatchItemResult batch_item_id result (hash) Turkee::TurkeeTask Turkee::TurkeeImportedAssignment turkee_task_id assignment_id (from mturk) worker_id (from mturk) result_id hit_url (from mturk) task_type hit_id complete (boolean) number_completed_assignments (integer)

Slide 77

Slide 77 text

Andy Glass | RailsConf 2018 | Human Powered Rails We still need to process and confirm the data

Slide 78

Slide 78 text

Andy Glass | RailsConf 2018 | Human Powered Rails Batch Item (With results completed) Adjudicator Batch Item Complete Reprocess In Turk " #

Slide 79

Slide 79 text

Andy Glass | RailsConf 2018 | Human Powered Rails def process if Adjudicator.new(self).approved? update_attributes(status: “complete”, result: adjudicator.result, confidence: adjudicator.confidence) else queue_for_reprocessing end end end

Slide 80

Slide 80 text

Andy Glass | RailsConf 2018 | Human Powered Rails def process if Adjudicator.new(self).approved? update_attributes(status: “complete”, result: adjudicator.result, confidence: adjudicator.confidence) else queue_for_reprocessing end end end Send back to Turk for two more opinions

Slide 81

Slide 81 text

Andy Glass | RailsConf 2018 | Human Powered Rails class Adjudicator def initialize(batch_item) @results_array = batch_item.bundle_results #flat array results accumulated from the BatchItem’s BatchItemResponse records (e.g. [‘yes’, ‘no’] @best_response_with_frequency = best_response_with_frequency end def confidence @best_response_with_frequency[1] == 1 ? “high” : “low” end def approved? (@best_response_with_frequency[1] / @results_array.count).to_f >.5 end def result @best_response_with_frequency[0] unless !approved? end private def best_response_with_frequency histo = result_array.to_histogram.sort{|resp, freq| -freq} histo[0] end end

Slide 82

Slide 82 text

Andy Glass | RailsConf 2018 | Human Powered Rails def process if Adjudicator.new(self).approved? update_attributes(status: “complete”, result: adjudicator.result, confidence: adjudicator.confidence) else queue_for_reprocessing end end end

Slide 83

Slide 83 text

Andy Glass | RailsConf 2018 | Human Powered Rails task :import_and_process do Turkee::TurkeeTask.process_hits batch_items_to_process = BatchItem.incomplete.all_results_in batch_items_to_process.each(&:process) end

Slide 84

Slide 84 text

Andy Glass | RailsConf 2018 | Human Powered Rails rake import_and_process

Slide 85

Slide 85 text

Andy Glass | RailsConf 2018 | Human Powered Rails rake import_and_process (repeat)

Slide 86

Slide 86 text

Andy Glass | RailsConf 2018 | Human Powered Rails Serve completed batch_items via API

Slide 87

Slide 87 text

Andy Glass | RailsConf 2018 | Human Powered Rails Its

Slide 88

Slide 88 text

Andy Glass | RailsConf 2018 | Human Powered Rails Its lit

Slide 89

Slide 89 text

Andy Glass | RailsConf 2018 | Human Powered Rails Its lit AF

Slide 90

Slide 90 text

Andy Glass | RailsConf 2018 | Human Powered Rails Its lit AF

Slide 91

Slide 91 text

Andy Glass | RailsConf 2018 | Human Powered Rails Basic Use Case (sandwich spotter) Extensible (any use case)

Slide 92

Slide 92 text

Andy Glass | RailsConf 2018 | Human Powered Rails How could we extend the app? ✓ Multiple batch items in a TurkeeTask ✓ Complex reprocessing flow ✓ Multiple inputs/outputs for a batch item ✓ Different output types

Slide 93

Slide 93 text

Andy Glass | RailsConf 2018 | Human Powered Rails How should we extend the app? ✓ Multiple batch items in a TurkeeTask ✓ Complex reprocessing flow ✓ Multiple inputs/outputs for a batch item ✓ Different output types

Slide 94

Slide 94 text

Andy Glass | RailsConf 2018 | Human Powered Rails Multiple Inputs/Outputs BatchItem batch_id post_id reward_in_cents status result confidence Batch name title description instructions output_field_name output_field_opts (json)

Slide 95

Slide 95 text

Andy Glass | RailsConf 2018 | Human Powered Rails Multiple Inputs/Outputs BatchItem batch_id post_id reward_in_cents status result confidence Batch name title description instructions output_field_name output_field_opts (json)

Slide 96

Slide 96 text

Andy Glass | RailsConf 2018 | Human Powered Rails BatchItem batch_id post_id reward_in_cents status result confidence input_data (json) results (json) confidence_levels (json) Batch name title description instructions output_field_name output_field_opts (hash) BatchInput batch_id key format settings (json) BatchOutput batch_id key format display_settings (json) adjudicator adjudicator_settings (json)

Slide 97

Slide 97 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch = Batch.create( name: "Sandwich Wizard", ) batch.batch_inputs.create( key: “sandwich_ig”, format: “instagram_post” )

Slide 98

Slide 98 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch = Batch.create( name: "Sandwich Wizard", ) batch.batch_inputs.create( key: “sandwich_ig”, format: “instagram_post” ) batch.batch_items.create( input_data:{ sandwich_ig: “BJ8ohZEAjWQ” } )

Slide 99

Slide 99 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch.batch_outputs.create( key: “has_fries”, format: “categories”, display_settings:{ category_opts: { yes: "Yes- it has french fries inside", no: "No- it does not have french fries inside” } }, adjudicator: ‘mode’, adjudicator_settings:{ acceptance: .5, } )

Slide 100

Slide 100 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch.batch_outputs.create( key: “has_fries”, format: “categories”, display_settings:{ category_opts: { yes: "Yes- it has french fries inside", no: "No- it does not have french fries inside” } }, adjudicator: ‘mode’, adjudicator_settings:{ acceptance: .5, } ) How we display the output How we confirm the output data

Slide 101

Slide 101 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch.batch_outputs.create( key: “fry_count”, format: “counter”, display_settings:{ instruct: “if yes, how many fries”, category_opts: { min: 0, max: 50 } } adjudicator: ‘number’, adjudicator_settings:{ variance: .20, } )

Slide 102

Slide 102 text

Andy Glass | RailsConf 2018 | Human Powered Rails batch.batch_outputs.create( key: “fry_count”, format: “counter”, display_settings:{ instruct: “if yes, how many fries”, category_opts: { min: 0, max: 50 } } adjudicator: ‘number’, adjudicator_settings:{ variance: .20, } ) How we display the output How we confirm the output data

Slide 103

Slide 103 text

Andy Glass | RailsConf 2018 | Human Powered Rails Possible input formats ✓ Business listing information ✓ Image ✓ Social posts (tweets, grams) ✓ Twilio embed (phone calls) ✓ Video (with JS helpers for splicing, etc)

Slide 104

Slide 104 text

Andy Glass | RailsConf 2018 | Human Powered Rails Possible output formats ✓ Text (phone, email, website, etc) ✓ Radio buttons ✓ Multi-select categories ✓ Number ✓ Video data

Slide 105

Slide 105 text

Andy Glass | RailsConf 2018 | Human Powered Rails Adjudicator Types ✓ Single text output ✓ Multiple text output ✓ Email ✓ URL ✓ Number

Slide 106

Slide 106 text

Andy Glass | RailsConf 2018 | Human Powered Rails Tips for Turk Accuracy ✓ Explicit instructions ✓ UX ✓ Simple and straightforward tasks ✓ “Gold” Data ✓ Approved worker pools/worker qualifications

Slide 107

Slide 107 text

Andy Glass | RailsConf 2018 | Human Powered Rails Task Speed Rate = e * r * n Ease of Task reward number of tasks

Slide 108

Slide 108 text

Andy Glass | RailsConf 2018 | Human Powered Rails Task Speed Rate = e * r * n Ease of Task reward number of tasks Andy’s Theorem™

Slide 109

Slide 109 text

Andy Glass | RailsConf 2018 | Human Powered Rails Ethics of mTurk Amazon Mechanical Turk: Gold Mine or Coal Mine? University of Colorado 2011 Amazon's Mechanical Turk workers protest: 'I am a human being, not an algorithm’ The Guardian December 3 2014

Slide 110

Slide 110 text

Andy Glass | RailsConf 2018 | Human Powered Rails Culture http://turkernation.com/ https://turkopticon.ucsd.edu/ /r/mturk “Turking isn't beer money for me, it's rent money and food money” “12 hours a day on here to make an average of $100 a day” “A higher-end worker on MTurk can expect to make $6-$12 per hour”

Slide 111

Slide 111 text

Andy Glass | RailsConf 2018 | Human Powered Rails Cambridge Analytica + mTurk How Amazon Helped Cambridge Analytica Harvest Americans’ Facebook Data Fast Company March 27 2018

Slide 112

Slide 112 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 113

Slide 113 text

Andy Glass | RailsConf 2018 | Human Powered Rails

Slide 114

Slide 114 text

Andy Glass | RailsConf 2018 | Human Powered Rails So, why did you listen to me talk about mTurk?

Slide 115

Slide 115 text

Andy Glass | RailsConf 2018 | Human Powered Rails Thanks for listening to me talk about mTurk.