ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽SF$"15$)" 俑㶵钠陎ءأذيח➂穈鴥 ˑNPSPJEH˒ ˑNPSSJOH˒ ˑNPSOJOH˒ 4UFQ剅硂⚥ך俑㶵אך0$3ءأذيח钠陎ׇׁ 4UFQ穠卓ָ♶♧荜ךהֹ➂ח㉏ְさׇ L. von Ahn et al.: reCAPTCHA: Human-based character recognition via web security measures, In Science, 2008. 4/58
-tap to take a photo. -tap to begin recording your question and again to stop.
side,
User
? Database - al Client mote Services and Worker Interface ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽7J[8J[ 鋔鋙ꥺְָ罏佄䴂ءأذيח➂穈鴥 Ύءأذيⰻ鿇 ך➂ָ㔐瘶 J. Bigham et al.: VizWiz: Nearly real-time answers to visual questions, In UIST, 2010. ِ٦ؠָ颵㉏䫎珲 ⢽չ؝٦ָٝⰅ 綸כוպ 5/58
ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽'PME*U S. Cooper et al.: Predicting protein structures with a multiplayer online game, In Nature, 2010. فٖ؎َ٦כ 넝أ؝،杆ְ 圓鸡㢌⻉ׇׁ ةٝػؙ颵ך圓鸡✮庠 ،ىظꃐꂁַ 甧⡤圓鸡✮庠ׅ㉏겗 ةٝػؙ颵圓鸡✮庠٦ي⻉׃➂ח鍑ַׇ 6/58
㔐瘶罏ך腉⸂罋䣁 ! 㔐瘶罏ָ醱侧㉏ח㔐瘶ֿׅהⵃ欽׃ג 㔐瘶罏ך腉⸂䱿㹀׃姻鍑✮庠ח欽ְ YES YES YES NO NO YES YES YES NO YES NO YES 㔐瘶罏 ? ? ? ㉏겗 腉⸂䱿㹀 A. P. Dawid and A. M. Skene: Maximum likelihood estimation of observer error-rates using the EM algorithm, Journal of the Royal Statistical Society. Series C (Applied Statistics), 1979. 㔐瘶罏ך腉⸂䱿㹀׃姻鍑✮庠ח欽ְ 姻鍑 17/58
㔐瘶罏ך腉⸂罋䣁 ! 姻鍑悵㖈㢌侧ה׃&.،ٕ؞ٔؤيד 姻鍑ה腉⸂❛✼ח䱿㹀ׅ o 4UFQ腉⸂㔿㹀׃ג姻鍑䱿㹀 o 4UFQ姻鍑㔿㹀׃ג腉⸂䱿㹀 ㉏겗ך姻鍑ה㔐瘶罏ך腉⸂❛✼ח䱿㹀 qi = Pr [ti = 1 | {yij }] / p Y j ✓yij j (1 ✓j)(1 yij ) 姻鍑ָ:&4ך㉏겗דך姻瘶桦ך״ֲזך ✓j = P i qiyij P i qi , j = P i (1 qi) yij P i (1 qi) p = Pr [ti = 1] 19/58 姻鍑
㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝罋䣁 ! չꨇ׃ְ㉏겗ד姻瘶ׅ㔐瘶罏ך倯ָ腉⸂ָ넝ֲպ ̔㉏겗ךꨇ僒䏝罋䣁ׅ YES YES YES NO NO YES YES YES NO YES NO YES ? ? ? ㉏겗 J. Whitehill et al.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise, In NIPS, 2009. 㔐瘶罏 㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝䱿㹀׃姻鍑✮庠 ꨇ僒䏝䱿㹀 姻鍑 20/58
湫㉏さׇ ! ➭➂ך瘶ִ⢪ג鍑ַׇֿהד姻׃ְ瘶ִ䒷ֹ⳿ׅ o ⢽ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״俑剅吤姻 4PZMFOUכծ'JOE'JY7FSJGZך媮ꥡד吤姻㹋倵 ⎼ 'JOE㉏겗ָ֮皘䨽ך嗚⳿ ⎼ 'JY吤姻 ⎼ 7FSJGZ 吤姻铎ך嗚⳿ ➭➂ך瘶ִ⢪ג鍑ַׇ Figure 2. Crowdproof is a human-augmented proofreader. J. Bigham et al.: Soylent: A word processor with a crowd inside, In UIST, 2010. 穠卓 穠卓 剑穄穠卓 29/58
荈䊹鎍姻 ! 4UFQ⡦鋅ׇ׆ח㔐瘶ׇׁ ! 4UFQ➭ך㔐瘶罏ך窟さ鍑䲿爙׃ծⱄ䏝㔐瘶ׇׁ N. B. Shah and D. Zhou: No oops, you won’t do it again: Mechanisms for self-correction in crowdsourcing, In ICML, 2016. ➭➂ך瘶ִ䲿爙׃荈䊹鎍姻⤛ׅ չ؟ٝؿٓٝءأ؝ךⱖ溪鼅דְֻׁպ չ֮זך㔐瘶ָ麩גְה䙼ֲ㜥さכ⥜姻׃גְֻׁպ ֮זך㔐瘶 ➭ך➂ך㔐瘶 30/58
⥜姻ה鐰⣣ך鷵如㹋遤 ! 5VS,POUSPM䧭卓暟ך⥜姻ה鐰⣣粸鵤ׅ،ٕ؞ٔؤي o ⥜姻٥鐰⣣ך銲♶銲 ㅷ颵朐䡾הׅ鿇ⴓ錁庠وٕ؝ؿ寸㹀麓玎ד寸㹀 o ⼧ⴓח⥜姻דֹהֿד穄✪ 䧭卓暟 ⥜姻ָ 䗳銲 ⥜姻 㹋倵 鐰⣣ָ 䗳銲 ⥜姻䖓ך 嫰鯰 穄✪ YES NO NO YES P. Dai et al.: Decision-theoretic control of crowd-sourced workflows, In AAAI, 2010. ،ٕ؞ٔؤيⵖ䖴ח״⥜姻ה鐰⣣粸鵤ׅ 31/58 ⥜姻䖓ךוַ䱰欽
鐰⣣罏腉⸂罋䣁♧㼎嫰鯰 ! 鐰⣣ָ♧㼎嫰鯰ד遤㜥さח 鐰⣣罏ך腉⸂䱿㹀׃䧭卓暟ךㅷ颵✮庠 ♧㼎嫰鯰ך穠卓ַ䧭卓暟ךㅷ颵✮庠ׅ ? "ךㅷ颵 ⡲䧭罏 T. Sunahase, Y. Baba and H. Kashima, Pairwise HITS: Quality estimation from pairwise comparisons in creator-evaluator crowdsourcing process, In AAAI, 2017 䧭卓暟 " 䧭卓暟 # " " # # # "
嗚稊ؙؒٔח״㼔㹺涪鋅 ! 嗚稊鸬⹛㘗䎢デַ㉏겗פ钷㼪׃ 㼔㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ涪鋅 Figure 3: Example ad to attract users time a user clicks on the a we record a conversion ev the advertising system. T the system to optimize th mizing the number of con contribution yield, instead the number of clicks. Although optimizing fo Figure 1: Screenshot of the Quizz system. healthline 1*QFJSPUJT BOE&(BCSJMPWJDI2VJ[[UBSHFUFEDSPXETPVSDJOHXJUIBCJMMJPO QPUFOUJBM VTFST
*O888
㼔㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ䩛ַָהׅ 姻瘶桦؝ٝغ٦آّٝ桦 ה׃ג⢪欽 39/58
姻鍑劢濼ך㉏겗⢪㼔㹺涪鋅 ! 姻鍑䌢ח欽䠐דֹהכꣲזְ ! 㢳侧寸鍑ד➿欽׃ְָ 㼔㹺ָ㼰侧崢הז㉏겗דכ㢳侧寸כ㣟侁ׅ C A A E D E A D A C A B B C A A D B E A D E A C E D A A E E D E D C D E A E D E A D D B C D A A A A A A A A A A A A A A A A A A B B B B B B B B B B B B B B B B B B B B B B B C C C C C C C C C C C C C C C C C C C C C C D D D D D D D D D D D D D D E E E E E E E E E E E E E E E E E E Question Worker { Experts Question Question A D A B C C D MV ꬊ㼔㹺 㢳 侧 寸 ㉏겗 㼔㹺 䌢ח姻瘶 ٓٝتيח㔐瘶 ˟"։&ך䫛
Ⰻגך㉏겗ד"ָ姻瘶 1 2 3 4 铎瘶ָ涪欰 J. Li, Y. Baba and H. Kashima: Hyper Questions: Unsupervised Targeting of a Few Experts in Crowdsourcing, In CIKM, 2017. 姻鍑劢濼ך㉏겗ח㼎ׅ㔐瘶ַ㼔㹺鋅אֽ 㢳侧寸ָ㣟侁ׅ噰畭ז⢽ 40/58
姻鍑劢濼ך㉏겗⢪㼔㹺涪鋅 ! 㼔㹺ず㡦ך㔐瘶כ醱侧㉏ד♧荜׃װְָׅ ꬊ㼔㹺ず㡦ך㔐瘶כ♧荜׃חְֻ ! 醱侧㉏תהג♧אך㉏겗הז׃ծ )ZQFSRVFTUJPOח㼎ׅ㔐瘶ד㢳侧寸《 ㉏겗 1 2 3 4 (1, 2, 3) (1, 2, 4) (1, 3, 4) (2, 3, 4) B B B )ZQFS RVFTUJPO ח㢌䳔 A A A A A A A A A A A B B B B C D D E E ਪᕚ 㙢㖽㘞 {1, 2, 3} AAA AAA {1, 2, 4} AAA AAA {1, 3, 4} {2, 3, 4} AAA AAA AAA AAA ABD ABA ADA BDA EBC EBA ECA BCA EBB EBD EBD BBD ൴ਪᕚ 㙢㖽㘞 ൴ਪᕚ 㗸㗮Ϋഈʹ ม౬ 1 εӔ 2 A A A A A A A A A A A B B B B C D D E E ਪᕚ 㙢㖽㘞 {1, 2, 3} AAA AAA {1, 2, 4} AAA AAA {1, 3, 4} {2, 3, 4} AAA AAA AAA AAA ABD ABA ADA BDA EBC EBA ECA BCA EBB EBD EBD BBD ൴ਪᕚ 㙢㖽㘞 ൴ਪᕚ 㗸㗮Ϋഈʹ ม౬ 1 εӔ 2 㼔㹺ず㡦ך㔐瘶ָ♧荜ֿׅהⵃ欽׃㢳侧寸 㼔㹺ך㔐瘶כ♧荜׃ 㢳侧崢הז ꬊ㼔㹺ך㔐瘶 כלאֻ ˘ ˘ 41/58
䱿讂ח״㼔㹺涪鋅 ! %"31"/FUXPSL$IBMMFOHF o Ⰻ碛皘䨽ח饔ְ괏菹鏣縧 o 剑ⴱחⰋגך괏菹鋅אֽث٦يח颣ꆃ o ⮚ث٦ي չ괏菹鋅אֲֽז➂պ 〡؝ىד涪鋅 +5BOHFUBM3FGMFDUJOHPOUIF%"31"SFECBMMPPODIBMMFOHF
*O $PNNVOJDBUJPOTPGUIF"$.
contributed articles pla use pu nin ins et ing dis on Ma use cro wa cia on ba the tea in pro firs Figure 1. Locations in the DARPA Red Balloon Challenge. Figure 2. Example recursive incentive-structure process for the MIT team. ➭➂ַך䱿讂ח㛇בְג㼔㹺鋅אֽ 42/58
䱿讂ח״㼔㹺涪鋅 ! ⟰噟ⰻדךةأؙⶴ䔲穗騟ⴓ匿 o ةأؙⶴ䔲荈ⴓד鍑ֽזְةأؙ➭罏ח⣛걾 ! אךⶴ䔲ػة٦ٝ穈さׇגٌرٕ⻉ o 荈魦הך㼔䚍ך䊴ח㛇בֻⶴ䔲 ⎼ 㼔䚍ָ鵚ַ׆黅ַ׆ך➂ח ⣛걾ׅ⫘ぢָ֮ o 植㖈ךةأؙꆀח㛇בֻⶴ䔲 o ٓٝتيⶴ䔲 ! 㼔䚍כةأؙ⚥ך⽃铂ה麓ך㸣✪㾶娖ַ䱿㹀 ➂ָ㼔㹺䱱ׅ麓玎ٌرٕ⻉ׅ H. Sun et al.: Analyzing expert behaviors in collaborative networks, In KDD, 2014. customers. A task is posted e network from an expert to When an expert cannot solve ., where to transfer a task) is y affect the completion time empt to deduce the cognitive model the decision making of s where a routing decision is patterns. interesting phenomenon that ask to someone whose knowl- or too different from his own. xpertise difference based rout- e formalize multiple routing nt both rational and random a generative model to com- of tasks, our model not only ences very well, but also accu- n time. Under three different significantly outperforms al- han 75% accuracy gain. In In service businesses, a service provider of expert network where service agents coll problems reported by customers. Bugzilla ing system where software developers joint ed bugs in projects. In a classic collaborati receiving a task, an expert first tries to so the expert will route the task to another is completed until it reaches an expert w solution. ( $ % & ' ) * + ƚϭ ƚϮ Figure 1: A Sample Collaborativ Figure 1 shows a sample collaborative n 44/58
⼿锃㉏겗ⴓⶴ ! 5VSLPNBUJD㉏겗ⴓⶴ٥鍑寸꧊㔚ד遤ֲ➬穈 o 4UFQ㉏겗ⴓⶴׅ ⎼ ⼧ⴓ⽃秪ז鿇ⴓ㉏겗חזתדⴓⶴ粸鵤ׅ o 4UFQぐ鿇ⴓ㉏겗➂ָ鍑ֻ o 4UFQ鿇ⴓ㉏겗ך鍑꧊秈׃剑穄涸ז鍑⳿⸂ A. Kulkarni et al.: Collaboratively crowdsourcing workflows with Turkomatic, In CSCW, 2012. ㉏겗ⴓⶴ꧊㔚ד遤ְ㣐鋉垷ז㉏겗鍑寸ׅ ㉏겗⢽չջ㎳倯⤑ռכ姻׃ְַպָذ٦وך㼭锷俑㛁瘗 鿇ⴓ㉏겗չ㎳כ״ֻזְպהְֲ媮衅㛁瘗 鿇ⴓ㉏겗չ䗳銲ז㎳ָ֮պהְֲ媮衅㛁瘗 ˘ 47/58
⼿锃剑黝⻉ ! 鍑ָ侧⦼ד邌ׁ㉏겗חֶֽ⼿锃剑黝⻉ o ٓٝتيח㔐瘶罏鼅ע o 㔐瘶罏כ♧㹀眔㔲ⰻך㥨ך㜥䨽ח鍑⹛ַׅ o 眔㔲杞זָ鍑ך刿倜粸鵤ׅ ! 圫ղז؛٦أדך 勲䚍ָ 椚锷涸٥㹋꿀涸ח爙ׁגְ N. Garg et al.: Collaborative optimization for collective decision-making in continuous spaces, In WWW, 2017. 然桦涸⺟ꂁ꣬♴岀ח⦺ג꧊㔚ד鍑䱱稊ׅ ㉏겗⢽✮皾ך寸㹀 猰㷕䮶莆顤 爡⠓⥂ꥺ顤 植㖈ך鍑 鍑⹛ַׅ 48/58
،؎ر،ך䱱稊 ! ꧊㔚ד،؎ر،ך鼅䫙٥❛⿷粸鵤ׅ 鼋⠗涸،ٕ؞ٔؤيח⦺ג،؎ر،䱱稊ׅ ⢽㶨⣘ぢֽך喱㶨ךرؠ؎ٝ L. Yu and J. V. Nickerson: Cooks or cobblers? crowd creativity through combination, In CHI, 2011. 4UFQ痥⚅➿ך⦐⡤ぐ荈ָ欰䧭 4UFQ䫎牰ד⮚葺⦐⡤ 鼅䫙׃如⚅➿ך⦐⡤ ❛⿷ח״欰䧭 4UFQ鼅䫙٥❛⿷粸鵤ׅ 50/58
،؎ر،겲⡂䏝ך〳鋔⻉ ! ،؎ر،겲⡂䏝ך〳鋔⻉ ⡂،؎ر،鵚ֻחꂁ縧ֿׅהדⰋ⡤⫷䪾䳢佄䴂 ! ،؎ر،겲⡂䏝꧊㔚ָⴻ倖 o 湱㼎涸ז겲⡂䏝㼥י չ"ה#ךוך倯ָ $ח⡂גְպ P. Siangliulue et al.: Toward collaborative ideation at scale: Leveraging ideas from others to generate more creative and diverse ideas, In CSCW, 2015. ،؎ر،鋅ׇさֲֿהד✼ְך涪䟝⤛ׅ " # $ 51/58
،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉ ! 겲⡂䚍ֽדכזֻ⮚⸋䲿爙ֿׅהד ،؎ر،ך鼅䫙佄䴂 ! 珏겲ך㉏さׇח״겲⡂䏝٥⮚⸋꧊㔚דⴻ倖 o ،؎ر،ل،ך겲⡂䏝鐰⣣ o ،؎ر،ل،ך⮚⸋鐰⣣ J. Li, Y. Baba, and H. Kashima: Simultaneous clustering and ranking from pairwise comparisons, In IJCAI, 2018 (to appear). ،؎ر،겲⡂䏝٥⮚⸋꧊㔚דⴻ倖׃〳鋔⻉חⵃ欽 ㉏겗⢽չ㹀劍ذأزדךؕٝصּؚٝחכպ "չؕٝصؚٝכ⽯鷌㷕պ #չ㉏겗ךꂁ㢌ִպ "չ娄ֹ㔐ג湊鋔պ #չ欰䖝׀הח㉏겗㢌ִպ "ה#כ⡂גְ "ה#כ⡂גְזְ "ך倯ָ葺ְ #ך倯ָ葺ְ 53/58
➙䖓ך铬겗 ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝךؚٓٝسثٍٖٝآ o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״灇瑔涪 ⎼ 僥歗չٖٗٝخؓךؔ؎ٕպ 稆➂㣗㮑ָ俑柃锃叨ה㼔㹺הך陽锷ח״ 䜁㶨ך氺孡ך屚洽岀涪鋅 o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״鄩ⴻ㆞鄩ⴻ o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״佟瘻寸㹀 넝䏝ד醱꧟ז㹋爡⠓㉏겗פך黝欽 57/58
yukinobaba
dougneiner
lara
edds
sachag
samanthasiow
marcduiker
wjessup
lynnandtonic
jcasabona
denniskardys
panda_program