Tasks  Finding  Op-miza-on     in  Crowdsourcing  Pla8orms   Pavel  Kucherbaev   PhD  student  at  ICT  Doctoral  School  and  EIT  ICT  Labs   [email protected]   

“Crowdsourcing  is  the  prac-ce  of   outsourcing  work  to  an  unknown   group  of  people  via  the   internet”  [13]   

Requestors   Tasks   Workers   Publish  on  pla=orms   Find  and  execute   

Amazon  Mechanical  Turk   CrowdFlower   

Task  1   Task  2   ...   Task  N   Task  List   Task  to  Work  on   All  available   tasks   Searching  Tasks   Tasks  Selec-on   Workers  find  tasks   

The  Goal   To  improve  the  searching   and  selec-on  of  tasks  in   crowdsourcing  pla=orms   

Op-miza-on   reward   Dme  spent   interest   

Model   OptimizedValue = max( f (T,W,t)) T =< Action,Object,Skills,Reward > W =< Skills, Experience,Preferences > Experience = {< Task,result,t >} t = t searching +t selection +t execution 

Pla8orms  scope   •  Micro  tasks  requiring  5  sec  –  10  min  for  execuDon;   •  Tasks  not  assigned  to  specific  workers;   •  Micro  rewards  of  $0.01-­‐5$.   

State  of  the  art   Related  work  in  Crowdsourcing   Tools  for  workers  on   Crowdsourcing  pla=orms   Recommender     systems   Recommender  systems  in   Crowdsourcing   

SOA  –  Related  work   Task  1   Task  2   ...   Task  N   Task  List   Task  to  Work  on   All  available   tasks   Searching  Tasks   Tasks  Selec-on   Prefer  to  select  tasks  similar  to  the   ones  a  worker  performed  before   [35]   Filter  by  recently  posted  or  max   amount  of  instances  and  watch  the   1st  2  pages  [4]   Limited  by  the  current  tasks  lis=ng  page   func=onality  [15]   

SOA  –  Tools  for  workers   [17]   

SOA  –  Recommender  systems   [10,24,29]   People  who  bought  this  also   bought  that   Movies  similar  to  ones  you  watched   Songs  similar  to  songs  of  this  arDst   

SOA  –  Recommender    systems  in   crowdsourcing   At  present  none  of  these  algorithms  was  tested  on  a  real   crowdsourcing  pla=orm.   Similarity   [1,34,36]     [18]     bag-­‐of-­‐words  [1],  Task  Rank  [34],  transform  worker’s   behavior  into  raDngs  [36]     Predicts  worker’s  accuracy  for  a  given  task,  based  on  the   past  accuracy  of  a  worker  [18]   Accuracy  predicDon   

SOA  –  Missing   •  How  effecDve  are  current  crowdsourcing   pla=orms  in  describing  tasks  to  workers?   •  How  should  tasks  be  recommended  to   workers  from  the  UI  and  UX  perspecDve?   •  The  comparaDve  analysis  of  different   recommender  algorithms.   

•  Design  and  implement  a  recommendaDon  system;   •  Validate  the  recommendaDon  system;     Plan   •  InvesDgate  the  crowdsourcing  domain  –  [20,8];   •  Validate  the  problem  –  2  surveys  done;   •  Create  a  test  bed  for  experiments  –  UI  prototype  for  CrowdFlower.   •  Experiment  with  different  ways  of  presenDng  tasks;   •  Validate  the  pla=orm  design  and  UX  for  workers;   •  Experiment  with  different  recommendaDon  systems;   2013   2014   2015   

Preliminary  work   

All  Dme  on  a  pla=orm   Finding   -me   27%   Execu-on  -me   Survey  as  a  task  on  CrowdFlower.   Self  Reported  data.   500  parDcipants.  All  from  the  USA.   hbp://kucherbaev.com/research/CF-­‐survey/   Survey  1   

USA   Europe   Asia   Finding  is  a  criDcal   problem,  %   35   31   31   Finding  is  a  problem,  not   criDcal,  %   29   42   38   Finding  is  not  a  problem,   %   36   27   31   Survey  2  –  For  how  many  workers  finding  a   task  to  work  on  is  a  problem   Survey  as  a  task  on  CrowdFlower.   Self  Reported  data.   750  parDcipants.  250  from  each  region.   hbps://github.com/pavelk2/CrowdFlower_internship   

Finding  tasks  is  a  problem   2/3   Finding  tasks   is  not  a   problem   All  workers   

CrowdLab  –  a  community  of  about  50  trus=ul  workers   across  different  channels.     CrowdLab  members:   “It  is  hard  to  understand  a  task  before   actually  star5ng  working  on  it”   UI  Design  –  how  to  display  the  tasks   informaDon  to  make  a  decision  faster   hbp://codesign.io/ubcswp/   hbp://anvil.crowdflower.com   

Next  steps   

Tasks  Lis-ng  page  UI   Experiments  with  the  UI:   •  To  display  average  execuDon  Dme  and  approximate   wage;   •  To  display  number  of  support  requests;   •  To  apply  fuzzy  logic  for  amount  of  instances;   •  To  display  the  percentage  of  completed  tasks  instances   to  all  started;   •  To  present  tasks  as  a  grid,  graph  or  a  cloud;   

Recommender  system   Sugges-on  Box   Preferences   Radio-­‐Mode   Collabora-ve  Filtering   Input   Implicit  raDngs:   •  Searching  history  (saw  or  not  the  task)   •  Clicked/Started/Completed   Explicit  raDngs:   •  Exit  survey  raDng   Output   Top-­‐k  relaDve  tasks     

Contribu-ons   •  A  recommender  algorithm  for   suggesDng  tasks  to  workers  -­‐  RecSys   •  A  user  interface  of  the  tasks  lisDng   page  –  CHI,  CSCW   •  A  recommender  systems  built  on  top   of  the  tasks  lisDng  page  –  HCOMP,   CrowdConf,  RecSys   

Collabora-on  with   CrowdFlower   <  2  000  000  workers  on  channels   CrowdLab  community  of  workers   >  1  BLN  tasks  completed   Ability  to  test  in  produc-on     Internship  at  CrowdFlower  by  “PhD  on  the  Move”,  Trento  Rise   

Valida-on:   Implement  the  recommender  system  and  the  user  interface  at  CrowdFlower.   Run  experiments  at  CrowdFlower  with  10%  of  all  traffic  based  on  the  whole   tasks  history.   User  Interface   Recommender  System   Metrics   •  amount  of  money  earned,     •  amount  of  tasks  completed,   •  amount  of  tasks  started  but  not  completed   •  propor5on  of  5me  spent  on  execu5on  to  all  5me  on  a  pla=orm,   •  average  normalized  sa5sfac5on  level  from  exit  surveys.   To  split  the  log  data  set  into  training  and  test  sets.  To   validate  algorithm  using  these  sets.     To  run  real  tests  in  producDon  by  comparing  the   metrics  for  workers  with  and  without  recommender   systems.  To  compare  different  recommender   algorithms  with  each  other.     To  calculate  metrics  with  and  without  displaying   extra  informaDon  about  tasks.  In  addiDon  to  log  data   analysis  –  to  conduct  surveys  and  interviews  to   validate  and  improve  the  user  interface.   

Thank  you   •  Design  and  implement  a  recommendaDon  system;   •  Validate  the  recommendaDon  system;     •  Experiment  with  different  ways  of  presenDng  tasks;   •  Validate  the  pla=orm  design  and  UX  for  workers;   •  Experiment  with  different  recommendaDon  systems;   2014   2015   Pavel  Kucherbaev   PhD  student  at  ICT  Doctoral  School  and  EIT  ICT  Labs   [email protected]   

[1]  V.  AmbaD,  S.  Vogel,  and  J.  G.  Carbonell.  Towards  task  recommendaDon  in  micro-­‐task  markets.  In  Human  ComputaDon,   volume  WS-­‐11-­‐11  of  AAAI  Workshops.  AAAI,  2011.     [2]  E.  E.  Arolas  and  F.  G.-­‐L.  de  Guevara.  Towards  an  integrated  crowdsourcing  definiDon.  J.  InformaDon  Science,  38(2):189– 200,  2012.     [3]  E.  Blabberg.  Crowdsourcing  industry  landscape.  hbp://bit.ly/vuNxyb.     [4]  L.  B.  Chilton,  J.  J.  Horton,  R.  C.  Miller,  and  S.  Azenkot.  Task  search  in  a  human  computaDon  market.  In  Proceedings  of  the   ACM  SIGKDD  Workshop  on  Human  ComputaDon,  HCOMP  ’10,  pages  1–9,  New  York,  NY,  USA,  2010.  ACM.     [5]  D.  Cosley,  D.  Frankowski,  L.  Terveen,  and  J.  Riedl.     Suggestbot:  using  intelligent  task  rouDng  to  help  people  find  work  in  wikipedia.  In  Proceedings  of  the  12th  internaDonal   conference  on  Intelligent  user  interfaces,  IUI  ’07,  pages  32–41,  New  York,  NY,  USA,  2007.  ACM.     [6]    A.  Doan,  R.  Ramakrishnan,  and  A.  Y.  Halevy.  Crowdsourcing  systems  on  the  world-­‐wide  web.  Commun.  ACM,  54(4):86– 96,  Apr.  2011.     [7]    T.  Finin,  W.  Murnane,  A.  Karandikar,  N.  Keller,  J.  MarDneau,  and  M.  Dredze.  AnnotaDng  named  enDDes  in  twiber  data   with  crowdsourcing.  In  Proceedings  of  the  NAACL  HLT  2010  Workshop  on  CreaDng  Speech  and  Language  Data  with   Amazon’s  Mechanical  Turk,  CSLDAMT  ’10,  pages  80–88,  Stroudsburg,  PA,  USA,  2010.  AssociaDon  for  ComputaDonal   LinguisDcs.     [8]    A.  Finnerty,  P.  Kucherbaev,  S.  Tranquillini,  and  G.  ConverDno.  Keep  it  simple:  Reward  and  task  design  in  crowdsourcing.   In  Proceedings  of  the  Biannual  Conference  of  the  Italian  Chapter  of  SIGCHI,  CHItaly  ’13,  pages  14:1–14:4,  New  York,  NY,   USA,  2013.  ACM.     [9]    D.  Geiger,  S.  Seedorf,  T.  Schulze,  R.  C.  Nickerson,  and  M.  Schader.  Managing  the  crowd:  Towards  a  taxonomy  of   crowdsourcing  processes.  In  V.  Sambamurthy  and  M.  Tanniru,  editors,  AMCIS.  AssociaDon  for  InformaDon  Systems,  2011.     [10]    J.  L.  Herlocker,  J.  A.  Konstan,  and  J.  Riedl.  Explaining  collaboraDve  filtering  recommendaDons.  In  Proceedings  of  the   2000  ACM  Conference  on  Computer  Supported  CooperaDve  Work,  CSCW  ’00,  pages  241–250,  New  York,  NY,  USA,  2000.   ACM.     [11]    J.  L.  Herlocker,  J.  A.  Konstan,  L.  G.  Terveen,  and  J.  T.  Riedl.  EvaluaDng  collaboraDve  filtering  recommender  systems.   ACM  Trans.  Inf.  Syst.,  22(1):5–53,  Jan.  2004.     [12]    L.  Hetmank.  Components  and  funcDons  of  crowdsourcing  systems  -­‐  a  systemaDc  literature  review.  In   WirtschaysinformaDk,  page  4,  2013.       

[13]    J.  Howe.  The  rise  of  crowdsourcing.  Wired,  14(14):1–7,  October  2006.     [14]    P.  G.  IpeiroDs.  A  plea  to  amazon:  Fix  mechanical  turk!  hbp://bit.ly/GRvrAn.     [15]    P.  G.  IpeiroDs.  Analyzing  the  amazon  mechanical  turk  marketplace.  XRDS,  17(2):16–21,  Dec.  2010.     [16]    P.  G.  IpeiroDs.  Demographics  of  mechanical  turk.  volume  CeDER-­‐10-­‐01  of  CeDER  Working  Papers.   CeDER,  2010.     [17]    L.  C.  Irani  and  M.  S.  Silberman.  TurkopDcon:  InterrupDng  worker  invisibility  in  amazon  mechanical  turk.   In  Proceedings  of  the  SIGCHI  Conference  on  Human  Factors  in  CompuDng  Systems,  CHI  ’13,  pages  611–620,   New  York,  NY,  USA,  2013.  ACM.     [18]    H.  J.  Jung  and  M.  Lease.  Crowdsourced  task  rouDng  via  matrix  factorizaDon.  CoRR,  abs/1310.5142,   2013.     [19]    N.  Kaufmann,  T.  Schulze,  and  D.  Veit.  More  than  fun  and  money.  worker  moDvaDon  in  crowdsourcing  -­‐   a  study  on  mechanical  turk.  In  AMCIS,  2011.     [20]    P.  Kucherbaev,  S.  Tranquillini,  F.  Daniel,  F.  CasaD,  M.  Marchese,  M.  Brambilla,  and  P.  Fraternali.   Business  processes  for  the  crowd  computer.  In  M.  Rosa  and  P.  Soffer,  editors,  Business  Process   Management  Workshops,  volume  132  of  Lecture  Notes  in  Business  InformaDon  Processing,  pages  256–267.   Springer  Berlin  Heidelberg,  2013.     [21]  A.  Kulkarni,  P.  Gutheim,  P.  Narula,  D.  Rolnitzky,  T.  S.  Parikh,  and  B.  Hartmann.  Mobileworks:  Designing   for  quality  in  a  managed  crowdsourcing  architecture.  IEEE  Internet  CompuDng,  16(5):28–35,  2012.     [22]  M.  Lease.  On  quality  control  and  machine  learning  in  crowdsourcing.  In  Human  ComputaDon,  volume   WS-­‐11-­‐11  of  AAAI  Workshops.  AAAI,  2011.     [23]  M.  Lease  and  E.  Yilmaz.  Crowdsourcing  for  informaDon  retrieval.  SIGIR  Forum,  45(2):66–75,  Jan.  2012.     [24]  G.  Linden,  B.  Smith,  and  J.  York.  Amazon.com  recommendaDons:  Item-­‐to-­‐item  collaboraDve  filtering.   IEEE  Internet  CompuDng,  7(1):76–80,  Jan.  2003.     

[25]  D.  Oleson,  A.  Sorokin,  G.  P.  Laughlin,  V.  Hester,   J.  Le,  and  L.  Biewald.  ProgrammaDc  gold:  Targeted  and  scalable  quality  assurance  in  crowdsourcing.  In   Human  ComputaDon,  volume  WS-­‐11-­‐11  of  AAAI  Workshops.  AAAI,  2011.     [26]  G.  Paolacci,  J.  Chandler,  and  P.  G.  IpeiroDs.  Running  experiments  on  amazon  mechanical  turk.  Judgment   and  Decision  Making,  5(5):411–419,  August  2010.     [27]  A.  C.  Rouse.  A  preliminary  taxonomy  of  crowdsourcing.  In  ACIS.  AssociaDon  for  InformaDon  Systems,   2010.     [28]  G.  Salton  and  M.  J.  Mcgill.  IntroducDon  to  Modern  InformaDon  Retrieval.  McGraw-­‐Hill,  Inc.,  New  York,   NY,  USA,  1986.     [29]  B.  Sarwar,  G.  Karypis,  J.  Konstan,  and  J.  Riedl.  Item-­‐based  collaboraDve  filtering  recommendaDon   algorithms.  In  Proceedings  of  the  10th  InternaDonal  Conference  on  World  Wide  Web,  WWW  ’01,  pages   285–295,  New  York,  NY,  USA,  2001.  ACM.     [30]  J.  Surowiecki.  The  Wisdom  of  Crowds.  Random  House,  New  York,  2004.     [31]  M.  Vukovic.  Crowdsourcing  for  enterprises.  In  SERVICES  I,  pages  686–692,  2009.     [32]  S.  Winchester.  The  strange  case  of  the  surgeon  at  crowthorne.  j-­‐SMITHSONIAN,  29(6),  Sept.  1998.     [33]  M.-­‐C.  Yuen,  I.  King,  and  K.-­‐S.  Leung.  A  survey  of  crowdsourcing  systems.  In  SocialCom/PASSAT,  pages   766–773,  2011.     [34]  M.-­‐C.  Yuen,  I.  King,  and  K.-­‐S.  Leung.  Task  matching  in  crowdsourcing.  In  Proceedings  of  the  2011   InternaDonal  Conference  on  Internet  of  Things  and  4th  InternaDonal  Conference  on  Cyber,  Physical  and   Social  CompuDng,  ITHINGSCPSCOM  ’11,  pages  409–412,  Washington,  DC,  USA,  2011.  IEEE  Computer  Society.     [35]  M.-­‐C.  Yuen,  I.  King,  and  K.-­‐S.  Leung.  Task  recommendaDon  in  crowdsourcing  systems.  In  Proceedings  of   the  First  InternaDonal  Workshop  on  Crowdsourcing  and  Data  Mining,  CrowdKDD  ’12,  pages  22–26,  New   York,  NY,  USA,  2012.  ACM.     [36]  M.-­‐C.  Yuen,  I.  King,  and  K.-­‐S.  Leung.  Taskrec:  ProbabilisDc  matrix  factorizaDon  in  task  recommendaDon   in  crowdsourcing  systems.  In  ICONIP  (2),  pages  516–525,  2012.