ؼُ٦وٝ ؝ٝؾُذ٦ءّٝה ؙٓؐسا٦ءؚٝ 꼛㜥ꨒ⛆瘰岚㣐㷕 䎃䏝➂䊨濼腉㷕⠓Ⰻ㕂㣐⠓ثُ٦زٔ،ٕ 䎃剢傈

劤ثُ٦زٔ،ٕ 㼔Ꟍ㹺涪鋅 ⼿锃佄䴂 ⚛⴨㉏さׇ 湫⴨㉏さׇ ➂꟦ך꧊㔚ה➂䊨濼腉ח״׷㉏겗鍑寸ך䩛岀׾稱➜ 2/58

ؼُ٦وٝ؝ٝؾُذ٦ءّٝ ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝ ➂꟦ה➂䊨濼腉׾穈׫さ׻ׇ׷ֿהד ו׍׵ַ♧倯׌ֽדכ鍑ֽזְ㉏겗׾鍑寸 ➂䊨濼腉ח״׷㉏겗鍑寸׾ ➂꟦ָ佄䴂 ➂꟦ח״׷㉏겗鍑寸׾ ➂䊨濼腉ָ佄䴂 ➂꟦ה➂䊨濼腉ך穈׫さ׻ׇדꨇ׃ְ㉏겗׾鍑寸 3/58

ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽SF$"15$)" 俑㶵钠陎ءأذيח➂꟦׾穈׫鴥׬ ˑNPSPJEH˒ ˑNPSSJOH˒ ˑNPSOJOH˒ 4UFQ剅硂⚥ך俑㶵׾אך0$3ءأذيח钠陎ׇׁ׷ 4UFQ穠卓ָ♶♧荜ךהֹ➂꟦ח㉏ְさ׻ׇ׷ L. von Ahn et al.: reCAPTCHA: Human-based character recognition via web security measures, In Science, 2008. 4/58

-tap to take a photo. -tap to begin recording your question and again to stop. side, User ? Database - al Client mote Services and Worker Interface ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽7J[8J[ 鋔鋙ꥺְָ罏佄䴂ءأذيח➂꟦׾穈׫鴥׬ Ύءأذيⰻ鿇 ך➂꟦ָ㔐瘶 J. Bigham et al.: VizWiz: Nearly real-time answers to visual questions, In UIST, 2010. ΍ِ٦ؠָ颵㉏׾䫎珲 ⢽չ؝٦ָٝⰅ׏ ׋綸כו׸պ 5/58

ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽'PME*U S. Cooper et al.: Predicting protein structures with a multiplayer online game, In Nature, 2010. فٖ؎َ٦כ 넝أ؝،׾杆ְ 圓鸡׾㢌⻉ׇׁ׷ ةٝػؙ颵ך圓鸡✮庠 ،ىظꃐꂁ⴨ַ׵ 甧⡤圓鸡׾✮庠ׅ׷㉏겗 ةٝػؙ颵圓鸡✮庠׾؜٦ي⻉׃➂꟦ח鍑ַׇ׷ 6/58

ؙٓؐسا٦ءؚٝ ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝדכ㢳ֻך⿫⸇罏ָ䗳銲 ! ؙٓؐسا٦ءؚٝ o ؎ٝة٦طحز׾鸐ׄג➂꟦ח⡲噟׾⣛걾ׅ׷➬穈׫ o ⢽"NB[PO.FDIBOJDBM5VSL ٓٝ؟٦ؤ o ♧鿇ך؟٦ؽأדכ"1*ח״׷⡲噟⣛걾ָ〳腉 㢳侧ך➂꟦ח،ؙإأׅ׷׋׭ך➬穈׫ ؙٓؐس ا٦ءؚٝ ⣛걾罏 ⡲噟罏 ΍ةأؙ涪遤 Ύةأؙⶴ䔲 Ώ穠卓䲿⳿ ΐ㜠ꂹ锜実 7/58

"NB[PO.FDIBOJDBM5VSLךⵃ欽⢽ 4UFQةأؙذٝفٖ٦ز׾鏣鎘׃ةأؙ׾涪遤 ةأؙ⢽ չⱖ溪ך➂暟ח♧殢䔲גכת׷ せ⵸׾鼅׿דֻ׌ְׁպ ⱖ溪63-瘝כر٦ةؿ؋؎ٕד䭷㹀 8/58

"NB[PO.FDIBOJDBM5VSLךⵃ欽⢽ 4UFQةأؙ♧鋮ח䲓鯹ׁ׸⡲噟罏ָ⡲噟Ꟛ㨣 9/58

"NB[PO.FDIBOJDBM5VSLךⵃ欽⢽ 4UFQ⡲噟穠卓׾然钠٥䪫钠ꬊ䪫钠׾寸׭׷ ⡲噟罏*% ⡲噟穠卓 10/58

ؼُ٦وٝ؝ٝؾُذ٦ءّٝך铬겗 ! ➂꟦ָչ䌢ח٥铩ד׮պ姻׃ְ瘶ִ׾鵤ׅהכꣲ׵זְ o ♶然㹋䚍➂כ圫ղז銲㔓דىأ׾ׅ׷ o 㢳圫䚍➂ח״׏ג腉⸂כ殯ז׷ ! ➂꟦ַ׵姻׃ְ瘶ִ׾䒷ֹ⳿ׅ׋׭ך䊨㣗ָ䗳銲 ➂꟦ַ׵וֲװ׏ג姻׃ְ瘶ִ׾䒷ֹ⳿ַׅ 11/58

SF$"15$)"ך،فٗ٦ث ! ΍ ⹛堣בֽ俑㶵钠陎⡲噟׾钠鏾ءأذيח穈׫鴥׬ o 㔐瘶罏כչ荈ⴓָ➂꟦׌պה爙ׅ׋׭溪⶛ח䮋׬ ! Ύ 姻鍑傀濼ך㉏겗ךⵃ欽אך俑㶵⴨׾䲿爙ׅ׷ o 姻鍑劢濼ך俑㶵⴨钠陎׃׋ְ俑㶵⴨ o 姻鍑傀濼ך俑㶵⴨➂꟦ַنحزַךⴻ㹀חⵃ欽 ! Ώ ⚛⴨⻉醱侧ךչ➂꟦պָずׄ瘶ִ׾鵤׃׋׵䱰欽 姻鍑劢濼 姻鍑傀濼 姻瘶ז׵➂꟦ 铎瘶ז׵نحز 钠鏾ءأذيח穈׫鴥׫姻׃ְ瘶ִ׾䒷ֹ⳿ׅ 12/58

劤ثُ٦زٔ،ٕךذ٦و꧊㔚׾崞ַ׃׋،فٗ٦ث ➂䊨濼腉ח״׷㉏겗鍑寸׾ ➂꟦ָ佄䴂 ➂꟦ח״׷㉏겗鍑寸׾ ➂䊨濼腉ָ佄䴂 ꧊㔚ך⸂ד姻׃ְ瘶ִ׾䒷ֹ⳿ׅ 㼔Ꟍ㹺涪鋅 ⼿锃佄䴂 ⚛⴨㉏さׇ 湫⴨㉏さׇ 知⽃٥⽃秪ז㉏겗 㔭ꨇ٥醱꧟ז㉏겗 ! ꧊㔚ד㉏겗׾鍑ַׇ׷٥鍑ֽ׷➂׾鋅אֽ׷ֿהד ➂꟦ַ׵姻׃ְ瘶ִ׾䒷ֹ⳿ׅ 13/58

΍⚛⴨㉏さׇ

⚛⴨㉏さׇ ! ㉏ְさ׻ׇ湱䩛ָ♧➂׌ה ׉ך➂ָىأ׾ׅ׷ה姻׃ְ瘶ִָ䖤׵׸זְ ! 醱侧➂ח㉏ְさ׻ׇג㔐瘶׾窟さׅ׷ֿהד 姻׃ְ瘶ִ׾䒷ֹ⳿ׅ o 知⽃ז䩛岀㢳侧寸 ⴽղח鍑ַׇג瘶ִ׾תה׭׷ ⱖ溪ח둷ָⱖ׏גְתַׅ :&4 /0 NO YES YES YES ׾ 瘶ִה׃ג 䱰欽 15/58

窟鎘涸ז㔐瘶窟さ ! 㢳侧寸״׶׮峤箺ׁ׸׋倯岀窟鎘涸ז㔐瘶窟さ ! 醱侧➂ך㔐瘶ַ׵姻鍑׾✮庠ׅ׷㉏겗ה׃ג㹀䒭⻉ o 欰䖝ך㔐瘶׌ַֽ׵鑐꿀ך姻鍑׾✮庠ׅ׷״ֲז׮ך YES YES YES NO NO YES YES YES NO YES NO YES ? ? ? ㉏겗 醱侧➂ך㔐瘶ַ׵窟鎘涸ח姻鍑׾✮庠ׅ׷ 姻鍑 16/58

㔐瘶罏ך腉⸂׾罋䣁 ! 㔐瘶罏ָ醱侧㉏ח㔐瘶ׅ׷ֿה׾ⵃ欽׃ג 㔐瘶罏ך腉⸂׾䱿㹀׃姻鍑✮庠ח欽ְ׷ YES YES YES NO NO YES YES YES NO YES NO YES 㔐瘶罏 ? ? ? ㉏겗 腉⸂׾䱿㹀 A. P. Dawid and A. M. Skene: Maximum likelihood estimation of observer error-rates using the EM algorithm, Journal of the Royal Statistical Society. Series C (Applied Statistics), 1979. 㔐瘶罏ך腉⸂׾䱿㹀׃姻鍑✮庠ח欽ְ׷ 姻鍑 17/58

㔐瘶罏ך腉⸂׾罋䣁 ! 㔐瘶罏ך腉⸂׾珏겲ך然桦ד邌植ׅ׷ o 姻鍑ָ:&4ךהֹח姻׃ֻ:&4ה瘶ִ׷然桦 o 姻鍑ָ/0ךהֹח姻׃ֻ/0ה瘶ִ׷然桦 ! ֿך然桦׾欽ְ׷ה如ך״ֲז㔐瘶欰䧭ٌرָٕ䖤׵׸׷ 腉⸂׾然桦ד邌植׃㔐瘶欰䧭麓玎׾ٌرٕ⻉ ✓j j 姻鍑錁庠דֹזְ 㔐瘶 /0:&4 Pr [yij | ti = 1] = ✓yij j (1 ✓j)(1 yij ) Pr [yij | ti = 0] = (1 yij ) j (1 j)yij 18/58

㔐瘶罏ך腉⸂׾罋䣁 ! 姻鍑׾悵㖈㢌侧ה׃׋&.،ٕ؞ٔؤيד 姻鍑ה腉⸂׾❛✼ח䱿㹀ׅ׷ o 4UFQ腉⸂׾㔿㹀׃ג姻鍑׾䱿㹀 o 4UFQ姻鍑׾㔿㹀׃ג腉⸂׾䱿㹀 ㉏겗ך姻鍑ה㔐瘶罏ך腉⸂׾❛✼ח䱿㹀 qi = Pr [ti = 1 | {yij }] / p Y j ✓yij j (1 ✓j)(1 yij ) 姻鍑ָ:&4ך㉏겗דך姻瘶桦ך״ֲז׮ך ✓j = P i qiyij P i qi , j = P i (1 qi) yij P i (1 qi) p = Pr [ti = 1] 19/58 姻鍑

㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝׾罋䣁 ! չꨇ׃ְ㉏겗ד姻瘶ׅ׷㔐瘶罏ך倯ָ腉⸂ָ넝׉ֲպ ̔㉏겗ךꨇ僒䏝׮罋䣁ׅ׷ YES YES YES NO NO YES YES YES NO YES NO YES ? ? ? ㉏겗 J. Whitehill et al.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise, In NIPS, 2009. 㔐瘶罏 㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝׾䱿㹀׃姻鍑✮庠 ꨇ僒䏝׾䱿㹀 姻鍑 20/58

㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝׾罋䣁 ! 㔐瘶罏ה㉏겗ךػًٓ٦ة׾㼪Ⰵ o 㔐瘶罏ך腉⸂ o ㉏겗ך知⽃ׁ ! 㔐瘶罏٥㉏겗ךػًٓ٦ة׾欽ְג姻瘶然桦׾邌植׃ 姻鍑הぐػًٓ٦ة׾䱿㹀 腉⸂ָPS知⽃ָׁ׌ה姻瘶然桦 腉⸂٥知⽃ָׁ㣐ְֹקו姻瘶然桦ָח鵚בֻ 姻鍑 㔐瘶 腉⸂ 知⽃ׁ Pr [ yij = ti] = 1 1 + exp ( ✓j⌫i) ✓j ⌫i 腉⸂٥ꨇ僒䏝ח㛇בֻ姻瘶然桦׾邌植 21/58

㔐瘶罏ך然⥋䏝׾罋䣁 ! 㔐瘶罏ח然⥋䏝׮耀ֻ ! ׋׌׃然⥋䏝׾姻׃ֻ瘶ִ׷⥂鏾כזְ o 铎瘶זךחչ荈⥋ָ֮׷պה瘶ִ׷➂ְָ׷ 荈⥋麓ⶱ o 姻瘶זךחչ荈⥋ָזְպה瘶ִ׷➂ְָ׷ 荈⥋麓㼭 ⱖ溪ח둷ָⱖ׏גְתַׅ 㔐瘶ח荈⥋ָ֮׶תַׅ כְ כְ ְְִ ְְִ 40ZBNB :#BCB :4BLVSBJ BOE) ,BTIJNB"DDVSBUFJOUFHSBUJPOPGDSPXETPVSDFEMBCFMT VTJOHXPSLFSTTFMGSFQPSUFEDPOGJEFODFTDPSFT *O *+$"* 㔐瘶罏ח然⥋䏝׮耀ְג姻鍑✮庠ח欽ְ׷ 22/58

㔐瘶罏ך然⥋䏝׾罋䣁 ! 然⥋䏝ך⫘ぢ׾然桦ד邌植 ! 然⥋䏝ך欰䧭ٌرٕ׾㼪Ⰵ׃ 㔐瘶欰䧭ٌرٕה穈׫さ׻ׇג姻鍑׾✮庠ׅ׷ ⇡(00) j ⇡(01) j ⇡(10) j ⇡(11) j 姻鍑ָ/0 姻鍑ָ:&4 㔐瘶ָ/0 㔐瘶ָ:&4 Pr [cij | ti = 0, yij = 1] = ⇣ ⇡(01) j ⌘cij ⇣ 1 ⇡(01) j ⌘(1 cij ) 㔐瘶 姻鍑 然⥋䏝 չ荈⥋ָ֮׷պה 瘶ִ׷然桦 然⥋䏝ך欰䧭麓玎׮ٌرٕ⻉ 23/58

♧㼎嫰鯰ך㔐瘶窟さ ! 醱侧➂ח♧㼎嫰鯰׾遤׻ׇ姻鍑ؚٓٝؗٝ׾✮庠ׅ׷ o ⢽ֶ׫װ־׾➂孡갫חؚٓٝؗٝ׃׋ְ > > > ♧㼎嫰鯰 㔐瘶窟さ穠卓ؚٓٝؗٝ 9$IFOFUBM1BJSXJTFSBOLJOHBHHSFHBUJPOJOBDSPXETPVSDFETFUUJOH *O84%. > > > ♧㼎嫰鯰ך穠卓ַ׵姻鍑ؚٓٝؗٝ׾✮庠ׅ׷ 24/58

♧㼎嫰鯰ך㔐瘶窟さ ! ،؎ذي׀החչ䓼ׁպָ֮׷הׅ׷ o ،؎ذيך䓼ׁ ! 㔐瘶罏ך腉⸂׾然桦ד邌植 o 䓼ְ،؎ذيח姻׃ֻ䫎牰ׅ׷然桦 ! 如䒭ך䫎牰ٌرٕח㛇בֹծ،؎ذيך䓼ׁ׾✮庠ׅ׷ ti ׁ׿ָ،؎ذيח䫎牰ׅ׷然桦 ،؎ذيָ״׶䓼ְ然桦 姻׃ֻח䫎牰ׅ׷然桦 铎׏גח䫎牰ׅ׷然桦 Pr [i j k] = ✓j eti eti + etk + (1 ✓j) etk eti + etk ✓j ،؎ذيך䓼ׁ٥㔐瘶罏腉⸂ח㛇בֻ䫎牰ٌرٕ 25/58

侧⦼㔐瘶ך窟さ ! 醱侧ך侧⦼㔐瘶ַ׵姻鍑׾✮庠ׅ׷ ! 䎂㖱׾《׷ה姻鍑הך铎䊴ָ㣐ֹֻז׷㜥さ׮֮׷ ! ✳⦼㔐瘶װ♧㼎嫰鯰ה殯ז׶㔐瘶⦪酡ָ搀ꣲח㶷㖈 43 215 215 860 860 860 860 860 215 215 姻鍑כ ㉏겗⢽չך秈侧ךֲ׍剑㣐ך㣼侧כְֻאַպ $-JOFUBM$SPXETPVSDJOHDPOUSPMNPWJOHCFZPOENVMUJQMFDIPJDF *O6"* 㔐瘶⦪酡ָ搀ꣲח֮׷侧⦼㉏겗ך姻鍑׾✮庠ׅ׷ 26/58

侧⦼㔐瘶ך窟さ ! ꟦麩ִ倯ך⨉׶׾⚥螟俱椚䏄麓玎ד邌植 ! 珏겲ך㔐瘶遤⹛׾׉׸׊׸ٌرٕ⻉ ΍ 姻瘶ׅ׷ Ύ 傀⳿ך铎瘶׾ׅ׷ Ώ 劢⳿ך铎瘶׾ׅ׷ 860 43 … ⚥螟俱椚䏄麓玎 չ➂ָ㢳ְذ٦ـٕח➂ָ꧊ת׶װְׅ⫘ぢպ׾邌植 ⚥螟俱椚䏄麓玎ח㛇בֻ㔐瘶遤⹛ٌرٕ׾㼪Ⰵ 27/58

Ύ湫⴨㉏さׇ

湫⴨㉏さׇ ! ➭➂ך瘶ִ׾⢪׏ג鍑ַׇ׷ֿהד姻׃ְ瘶ִ׾䒷ֹ⳿ׅ o ⢽ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷俑剅吤姻 4PZMFOUכծ'JOE'JY7FSJGZך媮ꥡד吤姻׾㹋倵 ⎼ 'JOE㉏겗ָ֮׷皘䨽ך嗚⳿ ⎼ 'JY吤姻 ⎼ 7FSJGZ 吤姻铎׶ך嗚⳿ ➭➂ך瘶ִ׾⢪׏ג鍑ַׇ׷ Figure 2. Crowdproof is a human-augmented proofreader. J. Bigham et al.: Soylent: A word processor with a crowd inside, In UIST, 2010. 穠卓 穠卓 剑穄穠卓 29/58

荈䊹鎍姻 ! 4UFQ⡦׮鋅ׇ׆ח㔐瘶ׇׁ׷ ! 4UFQ➭ך㔐瘶罏ך窟さ鍑׾䲿爙׃ծⱄ䏝㔐瘶ׇׁ׷ N. B. Shah and D. Zhou: No oops, you won’t do it again: Mechanisms for self-correction in crowdsourcing, In ICML, 2016. ➭➂ך瘶ִ׾䲿爙׃荈䊹鎍姻׾⤛ׅ չ؟ٝؿٓٝءأ؝ךⱖ溪׾鼅׿דֻ׌ְׁպ չ֮ז׋ך㔐瘶ָ꟦麩׏גְ׷ה䙼ֲ㜥さכ⥜姻׃גֻ׌ְׁպ ֮ז׋ך㔐瘶 ➭ך➂ך㔐瘶 30/58

⥜姻ה鐰⣣ך鷵如㹋遤 ! 5VS,POUSPM䧭卓暟ך⥜姻ה鐰⣣׾粸׶鵤ׅ،ٕ؞ٔؤي o ⥜姻٥鐰⣣ך銲♶銲׾ ㅷ颵׾朐䡾הׅ׷鿇ⴓ錁庠وٕ؝ؿ寸㹀麓玎ד寸㹀 o ⼧ⴓח⥜姻דֹ׋הֿ׹ד穄✪ 䧭卓暟 ⥜姻ָ 䗳銲 ⥜姻׾ 㹋倵 鐰⣣ָ 䗳銲 ⥜姻⵸䖓ך 嫰鯰 穄✪ YES NO NO YES P. Dai et al.: Decision-theoretic control of crowd-sourced workflows, In AAAI, 2010. ،ٕ؞ٔؤيⵖ䖴ח״׶⥜姻ה鐰⣣׾粸׶鵤ׅ 31/58 ⥜姻⵸䖓ךו׍׵ַ׾䱰欽

鐰⣣罏腉⸂׾罋䣁窫㼎鐰⣣ ! 鐰⣣ָ姻׃ְהכꣲ׵זְ ! 鐰⣣罏٥⡲䧭罏ך腉⸂׾䱿㹀׃׋♳ד䧭卓暟ךㅷ颵׾✮庠 ? 䧭卓暟 鐰⣣罏 ㅷ颵 鐰⣣罏ػًٓ٦ة ˖غ؎،أ % ˖⥋걾䚍 % ⡲䧭罏ػًٓ٦ة ˖腉⸂ ( ˖⥋걾䚍 ( ⡲䧭罏 鐰⣣罏腉⸂׾罋䣁׃䧭卓暟ךㅷ颵׾✮庠ׅ׷ Y. Baba and H. Kashima: Statistical quality estimation for general crowdsourcing tasks, In KDD, 2013. 32/58

鐰⣣罏腉⸂׾罋䣁窫㼎鐰⣣ ! 䧭卓暟ה鐰⣣ך欰䧭ٌرٕ׾㼪Ⰵ׃ 鐰⣣穠卓ַ׵䧭卓暟ךㅷ颵׾✮庠ׅ׷ o 4UFQ⡲䧭罏ָㅷ颵,,( ׾׮א䧭卓暟׾欰䧭 o 4UFQ鐰⣣罏ָ鐰⣣,,(,% ׾欰䧭ׅ׷ 䧭卓暟ה鐰⣣ך欰䧭麓玎׾ٌرٕ⻉ qt,a ⇠ N qt,a | µa, 1 a yt,a,r ⇠ N yt,a,r | qt,a + ⌘r,  1 r ⡲䧭罏ך腉⸂ 鐰⣣罏ךغ؎،أ ㅷ颵 ㅷ颵 33/58

鐰⣣罏腉⸂׾罋䣁♧㼎嫰鯰 ! 鐰⣣ָ♧㼎嫰鯰ד遤׻׸׷㜥さח 鐰⣣罏ך腉⸂׾䱿㹀׃䧭卓暟ךㅷ颵׾✮庠 ♧㼎嫰鯰ך穠卓ַ׵䧭卓暟ךㅷ颵׾✮庠ׅ׷ ? "ךㅷ颵 ⡲䧭罏 T. Sunahase, Y. Baba and H. Kashima, Pairwise HITS: Quality estimation from pairwise comparisons in creator-evaluator crowdsourcing process, In AAAI, 2017 䧭卓暟 " 䧭卓暟 # " " # # # " ? #ךㅷ颵 34/58

鐰⣣罏腉⸂׾罋䣁♧㼎嫰鯰 ! 1BJSXJTF)*54 )*54،ٕ؞ٔؤي׾䘔欽׃䧭卓暟ךㅷ颵׾✮庠 ! ⟃♴ך䩛竲ֹ׾粸׶鵤ׅ o 4UFQ鐰⣣罏腉⸂0 ׾㔿㹀׃ㅷ颵1 ׾刿倜 o 4UFQㅷ颵1 ׾㔿㹀׃鐰⣣罏腉⸂0 ׾刿倜 qj qj0 = X i2Vj j0 ri X i2Vj0 j ri ח䫎牰׃׋➂ ri = |{(j j0) 2 Vi | qj > qj0 }| |Vi | 姻׃ְ鐰⣣ ךⶴさ 鐰⣣罏腉⸂ה䧭卓暟ㅷ颵׾❛✼ח刿倜 35/58

Ώ㼔Ꟍ㹺涪鋅

㼔Ꟍ㹺涪鋅 ! 㼔Ꟍ䚍ָ䗳銲ז㉏겗דכ 鍑ֽ׷➂׾鋅אֽזְה姻׃ְ瘶ִָ䖤׵׸זְ o ⢽⼔洽חꟼׅ׷㉏겗 ! 知⽃ז倯岀姻鍑傀濼㉏겗דך姻瘶桦ד㼔Ꟍ㹺׾鋅אֽ׷ ! ذأز׾欽䠐׃Ⰻ㆞ח「ׇֽׁ׷ךכ撕꧟זךד ➭ך䩛ַָ׶׾ⵃ欽׃׋ְ 鍑ֽ׷➂׾꧊㔚ך⚥ַ׵鋅אֽ׷ 2過⚥慬䏝׾然钠ׅ׷䗳銲䚍ָ剑׮넝ְ⼔讒ㅷכו׸ַ " ،أؾٔٝ # ؿٗإىس $ ذؔؿ؍ٔٝ % ؎ٝسًةءٝ 㼔Ꟍ䚍ָ䗳銲ז㉏겗⢽ 37/58

㾩䚍ח״׷㼔Ꟍ㹺涪鋅 ! 㼔Ꟍ䚍ָ㔐瘶罏ך㾩䚍㷕娖ծ㷕鿇瘝ח ⣛㶷ׅ׷ֿהכ״ֻ֮׷ ! 㔐瘶罏ך㾩䚍׾ⵃ欽׃ג㼔Ꟍ㹺׾涪鋅 o 㔐瘶罏ך㾩䚍كؙزָٕ♷ִ׵׸׷ o 㔐瘶罏腉⸂׾㾩䚍كؙزٕך简䕎ㄤד邌植׃ ぐ㾩䚍ח㼎ׅ׷ꅾ׫׾䱿㹀 㼔Ꟍ㹺חⰟ鸐ׅ׷㾩䚍׾䩛ַָ׶הׅ׷ aj = (aj1, aj2, . . . , ajn) ⼔㷕鿇⳿魦 穗幥㷕鿇⳿魦 ✓j ⇠ N w>aj, 2 )-JFUBM5IF8JTEPNPGNJOPSJUZEJTDPWFSJOHBOEUBSHFUJOHUIFSJHIUHSPVQPGXPSLFSTGPS DSPXETPVSDJOH *O888 腉⸂ 㾩䚍ךꅾ׫ 38/58

嗚稊ؙؒٔח״׷㼔Ꟍ㹺涪鋅 ! 嗚稊鸬⹛㘗䎢デַ׵㉏겗פ钷㼪׃ 㼔Ꟍ㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ׾涪鋅 Figure 3: Example ad to attract users time a user clicks on the a we record a conversion ev the advertising system. T the system to optimize th mizing the number of con contribution yield, instead the number of clicks. Although optimizing fo Figure 1: Screenshot of the Quizz system. healthline 1*QFJSPUJT BOE&(BCSJMPWJDI2VJ[[UBSHFUFEDSPXETPVSDJOHXJUIBCJMMJPO QPUFOUJBM VTFST *O888 㼔Ꟍ㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ׾䩛ַָ׶הׅ׷ 姻瘶桦׾؝ٝغ٦آّٝ桦 ה׃ג⢪欽 39/58

姻鍑劢濼ך㉏겗׾⢪׏׋㼔Ꟍ㹺涪鋅 ! 姻鍑׾䌢ח欽䠐דֹ׷הכꣲ׵זְ ! 㢳侧寸鍑ד➿欽׃׋ְָ 㼔Ꟍ㹺ָ㼰侧崢הז׷㉏겗דכ㢳侧寸כ㣟侁ׅ׷ C A A E D E A D A C A B B C A A D B E A D E A C E D A A E E D E D C D E A E D E A D D B C D A A A A A A A A A A A A A A A A A A B B B B B B B B B B B B B B B B B B B B B B B C C C C C C C C C C C C C C C C C C C C C C D D D D D D D D D D D D D D E E E E E E E E E E E E E E E E E E Question Worker { Experts Question Question A D A B C C D MV ꬊ㼔Ꟍ㹺 㢳 侧 寸 ㉏겗 㼔Ꟍ㹺 䌢ח姻瘶 ٓٝتيח㔐瘶 ˟"։&ך䫛 Ⰻגך㉏겗ד"ָ姻瘶 1 2 3 4 铎瘶ָ涪欰 J. Li, Y. Baba and H. Kashima: Hyper Questions: Unsupervised Targeting of a Few Experts in Crowdsourcing, In CIKM, 2017. 姻鍑劢濼ך㉏겗ח㼎ׅ׷㔐瘶ַ׵㼔Ꟍ㹺׾鋅אֽ׷ 㢳侧寸ָ㣟侁ׅ׷噰畭ז⢽ 40/58

姻鍑劢濼ך㉏겗׾⢪׏׋㼔Ꟍ㹺涪鋅 ! 㼔Ꟍ㹺ず㡦ך㔐瘶כ醱侧㉏ד♧荜׃װְָׅ ꬊ㼔Ꟍ㹺ず㡦ך㔐瘶כ♧荜׃חְֻ ! 醱侧㉏׾תה׭ג♧אך㉏겗ה׫ז׃׋ծ )ZQFSRVFTUJPOח㼎ׅ׷㔐瘶ד㢳侧寸׾《׷ ㉏겗 1 2 3 4 (1, 2, 3) (1, 2, 4) (1, 3, 4) (2, 3, 4) B B B )ZQFS RVFTUJPO ח㢌䳔 A A A A A A A A A A A B B B B C D D E E ਪᕚ 㙢㖽㘞 {1, 2, 3} AAA AAA {1, 2, 4} AAA AAA {1, 3, 4} {2, 3, 4} AAA AAA AAA AAA ABD ABA ADA BDA EBC EBA ECA BCA EBB EBD EBD BBD ൴ਪᕚ 㙢㖽㘞 ൴ਪᕚ 㗸㗮Ϋഈʹ ม౬ 1 ε਺Ӕ 2 A A A A A A A A A A A B B B B C D D E E ਪᕚ 㙢㖽㘞 {1, 2, 3} AAA AAA {1, 2, 4} AAA AAA {1, 3, 4} {2, 3, 4} AAA AAA AAA AAA ABD ABA ADA BDA EBC EBA ECA BCA EBB EBD EBD BBD ൴ਪᕚ 㙢㖽㘞 ൴ਪᕚ 㗸㗮Ϋഈʹ ม౬ 1 ε਺Ӕ 2 㼔Ꟍ㹺ず㡦ך㔐瘶ָ♧荜ׅ׷ֿה׾ⵃ欽׃׋㢳侧寸 㼔Ꟍ㹺ך㔐瘶כ♧荜׃ 㢳侧崢הז׷ ꬊ㼔Ꟍ㹺ך㔐瘶 כל׵אֻ ˘ ˘ 41/58

䱿讂ח״׷㼔Ꟍ㹺涪鋅 ! %"31"/FUXPSL$IBMMFOHF o Ⰻ碛皘䨽ח饔ְ괏菹׾鏣縧 o 剑ⴱחⰋגך괏菹׾鋅אֽ׋ث٦يח颣ꆃ o ⮚⹧ث٦ي չ괏菹׾鋅אֽ׉ֲז➂պ ׾〡؝ىד涪鋅 +5BOHFUBM3FGMFDUJOHPOUIF%"31"SFECBMMPPODIBMMFOHF *O $PNNVOJDBUJPOTPGUIF"$. contributed articles pla use pu nin ins et ing dis on Ma use cro wa cia on ba the tea in pro firs Figure 1. Locations in the DARPA Red Balloon Challenge. Figure 2. Example recursive incentive-structure process for the MIT team. ➭➂ַ׵ך䱿讂ח㛇בְג㼔Ꟍ㹺׾鋅אֽ׷ 42/58

䱿讂ח״׷㼔Ꟍ㹺涪鋅 ! 鄃䱿讂罏ָ괏菹׾涪鋅׃׋㜥さծ 䱿讂罏ח׮㜠ꂹָⴓꂁׁ׸׷״ֲח鏣鎘 䱿讂 䱿讂 䱿讂 괏菹ך 㜠ꂹ 䱿讂ך 㜠ꂹ 䱿讂ך 㜠ꂹ 䱿讂ך 㜠ꂹ 괏菹ך 㜠ꂹ 剣渣ז䱿讂ָ䖤׵׸׷״ֲח㜠ꂹ׾鏣鎘 43/58 괏菹׾涪鋅 괏菹׾涪鋅

䱿讂ח״׷㼔Ꟍ㹺涪鋅 ! ⟰噟ⰻדךةأؙⶴ䔲穗騟׾ⴓ匿 o ةأؙⶴ䔲荈ⴓד鍑ֽזְةأؙ׾➭罏ח⣛걾 ! אךⶴ䔲ػة٦ٝ׾穈׫さ׻ׇגٌرٕ⻉ o 荈魦הך㼔Ꟍ䚍ך䊴ח㛇בֻⶴ䔲 ⎼ 㼔Ꟍ䚍ָ鵚ַ׵׆黅ַ׵׆ך➂ח ⣛걾ׅ׷⫘ぢָ֮׷ o 植㖈ךةأؙꆀח㛇בֻⶴ䔲 o ٓٝتيⶴ䔲 ! 㼔Ꟍ䚍כةأؙ⚥ך⽃铂ה麓⿠ך㸣✪㾶娖ַ׵䱿㹀 ➂꟦ָ㼔Ꟍ㹺׾䱱ׅ麓玎׾ٌرٕ⻉ׅ׷ H. Sun et al.: Analyzing expert behaviors in collaborative networks, In KDD, 2014. customers. A task is posted e network from an expert to When an expert cannot solve ., where to transfer a task) is y affect the completion time empt to deduce the cognitive model the decision making of s where a routing decision is patterns. interesting phenomenon that ask to someone whose knowl- or too different from his own. xpertise difference based rout- e formalize multiple routing nt both rational and random a generative model to com- of tasks, our model not only ences very well, but also accu- n time. Under three different significantly outperforms al- han 75% accuracy gain. In In service businesses, a service provider of expert network where service agents coll problems reported by customers. Bugzilla ing system where software developers joint ed bugs in projects. In a classic collaborati receiving a task, an expert first tries to so the expert will route the task to another is completed until it reaches an expert w solution. ( $ % & ' ) * + ƚϭ ƚϮ Figure 1: A Sample Collaborativ Figure 1 shows a sample collaborative n 44/58

ΐ⼿锃佄䴂

⼿锃佄䴂 ! 㔭ꨇ٥醱꧟ז㉏겗ך鍑寸חכ➂꟦ず㡦ך⼿锃ָ♶〳妀 o ㉏겗׾ⴓⶴ׃׫׿זד鍑ֻ o 葺ְ鍑׾׫׿זד鋅אֽ׷ ! 鎉铂ח״׷陽锷כأ؛٦ٕ׃זְ׋׭ ءأذوذ؍حؙז⼿锃佄䴂ָ実׭׵׸׷ ꧊㔚ח״׷㉏겗鍑寸׾佄䴂ׅ׷ 46/58

⼿锃㉏겗ⴓⶴ ! 5VSLPNBUJD㉏겗ⴓⶴ٥鍑寸׾꧊㔚ד遤ֲ➬穈׫ o 4UFQ㉏겗׾ⴓⶴׅ׷ ⎼ ⼧ⴓ⽃秪ז鿇ⴓ㉏겗חז׷תדⴓⶴ׾粸׶鵤ׅ o 4UFQぐ鿇ⴓ㉏겗׾➂꟦ָ鍑ֻ o 4UFQ鿇ⴓ㉏겗ך鍑׾꧊秈׃剑穄涸ז鍑׾⳿⸂ A. Kulkarni et al.: Collaboratively crowdsourcing workflows with Turkomatic, In CSCW, 2012. ㉏겗ⴓⶴ׾꧊㔚ד遤ְ㣐鋉垷ז㉏겗׾鍑寸ׅ׷ ㉏겗⢽չջ㎳׮倯⤑ռכ姻׃ְַպָذ٦وך㼭锷俑㛁瘗 鿇ⴓ㉏겗չ㎳כ״ֻזְպהְֲ媮衅׾㛁瘗 鿇ⴓ㉏겗չ䗳銲ז㎳ָ֮׷պהְֲ媮衅׾㛁瘗 ˘ 47/58

⼿锃剑黝⻉ ! 鍑ָ侧⦼ד邌ׁ׸׷㉏겗חֶֽ׷⼿锃剑黝⻉ o ٓٝتيח㔐瘶罏׾鼅ע o 㔐瘶罏כ♧㹀眔㔲ⰻך㥨׫ך㜥䨽ח鍑׾⹛ַׅ o 眔㔲׾杞׭זָ׵鍑ך刿倜׾粸׶鵤ׅ ! 圫ղז؛٦أדך　勲䚍ָ 椚锷涸٥㹋꿀涸ח爙ׁ׸גְ׷ N. Garg et al.: Collaborative optimization for collective decision-making in continuous spaces, In WWW, 2017. 然桦涸⺟ꂁ꣬♴岀ח⦺׏ג꧊㔚ד鍑׾䱱稊ׅ׷ ㉏겗⢽✮皾ך寸㹀 猰㷕䮶莆顤 爡⠓⥂ꥺ顤 植㖈ך鍑 鍑׾⹛ַׅ 48/58

،؎ر،ך　꧊ ! *OOPDFOUJWF 暴㹀ך㉏겗ך鍑،؎ر،׾⹫꧊ׅ׷فٓحزؿؓ٦ي o ㉏겗⢽ չؿح稆ػؐت٦׾瑞孡⚥ח䭁侔ׇׁ׆ח娎熊ֹ ثُ٦ـחⰅ׸׷倯岀כպ o ،؎ر،ָ䱰欽ׁ׸׷ה颣ꆃָ佄䩪׻׸׷ ꧊㔚ַ׵،؎ر،׾⹫꧊ׅ׷فٓحزؿؓ٦ي *OOPDFOUJWF 49/58

،؎ر،ך䱱稊 ! ꧊㔚ד،؎ر،ך鼅䫙٥❛⿷׾粸׶鵤ׅ 鼋⠗涸،ٕ؞ٔؤيח⦺׏ג،؎ر،׾䱱稊ׅ׷ ⢽㶨⣘ぢֽך喱㶨ךرؠ؎ٝ L. Yu and J. V. Nickerson: Cooks or cobblers? crowd creativity through combination, In CHI, 2011. 4UFQ痥⚅➿ך⦐⡤׾ぐ荈ָ欰䧭 4UFQ䫎牰ד⮚葺⦐⡤ ׾鼅䫙׃如⚅➿ך⦐⡤ ׾❛⿷ח״׶欰䧭 4UFQ鼅䫙٥❛⿷׾粸׶鵤ׅ 50/58

،؎ر،겲⡂䏝ך〳鋔⻉ ! ،؎ر،겲⡂䏝ך〳鋔⻉ ⡂׋،؎ر،׾鵚ֻחꂁ縧ׅ׷ֿהדⰋ⡤⫷䪾䳢׾佄䴂 ! ،؎ر،겲⡂䏝׮꧊㔚ָⴻ倖 o 湱㼎涸ז겲⡂䏝׾㼥י׷ չ"ה#ךו׍׵ך倯ָ $ח⡂גְ׷պ P. Siangliulue et al.: Toward collaborative ideation at scale: Leveraging ideas from others to generate more creative and diverse ideas, In CSCW, 2015. ،؎ر،׾鋅ׇさֲֿהד✼ְך涪䟝׾⤛ׅ " # $ 51/58

،؎ر،겲⡂䏝ך〳鋔⻉ ⢽娕ך嶊꣇㡦פך钰欰傈ًحإ٦آ 52/58

،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉ ! 겲⡂䚍׌ֽדכזֻ⮚⸋׮䲿爙ׅ׷ֿהד ،؎ر،ך鼅䫙׾佄䴂 ! 珏겲ך㉏さׇח״׶겲⡂䏝٥⮚⸋׾꧊㔚דⴻ倖 o ،؎ر،ل،ך겲⡂䏝鐰⣣ o ،؎ر،ل،ך⮚⸋鐰⣣ J. Li, Y. Baba, and H. Kashima: Simultaneous clustering and ranking from pairwise comparisons, In IJCAI, 2018 (to appear). ،؎ر،겲⡂䏝٥⮚⸋׾꧊㔚דⴻ倖׃〳鋔⻉חⵃ欽 ㉏겗⢽չ㹀劍ذأزדךؕٝصؚٝ׾꣇ּחכպ "չؕٝصؚٝכ⽯鷌㷕պ #չ㉏겗ךꂁ⴨׾㢌ִ׷պ "չ娄ֹ㔐׏ג湊鋔պ #չ欰䖝׀הח㉏겗׾㢌ִ׷պ "ה#כ⡂גְ׷ "ה#כ⡂גְזְ "ך倯ָ葺ְ #ך倯ָ葺ְ 53/58

،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉ ! 겲⡂䏝٥⮚⸋ך鐰⣣穠卓׾ꟼ鸬בֽ⿽倯ך䱿㹀׾⸬桦⻉ o չ⡂גְ׷،؎ر،ず㡦כずֻׄ׵ְ葺ְպ o չ葺ְ،؎ر،ה䝤ְ،؎ر،כ⡂גְזְպ ! ،؎ر،ך悵㖈暴䗙 ∈ ℝ5׾➜׃겲⡂䏝ה葺ׁ׾ꟼ鸬בֽ ! 겲⡂䏝٥⮚⸋ך鐰⣣穠卓ַ׵ 0 0 ה׾䱿㹀׃ 〳鋔⻉חⵃ欽 悵㖈暴䗙ך㼪Ⰵד겲⡂䏝٥⮚⸋ך䱿㹀׾⸬桦⻉ ⌧i = w > xi ،؎ر،Jך葺ׁ ،؎ر،JהKך 겲⡂䏝 54/58 ij = exp k xi xj k2 AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/ AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/ AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/ AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/

תה׭

תה׭ ➂䊨濼腉ח״׷㉏겗鍑寸׾ ➂꟦ָ佄䴂 ➂꟦ח״׷㉏겗鍑寸׾ ➂䊨濼腉ָ佄䴂 㼔Ꟍ㹺涪鋅 ⼿锃佄䴂 ⚛⴨㉏さׇ 湫⴨㉏さׇ 知⽃٥⽃秪ז㉏겗 㔭ꨇ٥醱꧟ז㉏겗 ➂꟦ך꧊㔚ה➂䊨濼腉ח״׷㉏겗鍑寸ך䩛岀׾稱➜ 56/58

➙䖓ך铬겗 ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝךؚٓٝسثٍٖٝآ o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷灇瑔Ꟛ涪 ⎼ 僥歗չٖٗٝخؓךؔ؎ٕպ 稆➂㣗㮑ָ俑柃锃叨ה㼔Ꟍ㹺הך陽锷ח״׶ 䜁㶨ך氺孡ך屚洽岀׾涪鋅 o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷鄩ⴻ㆞鄩ⴻ o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷佟瘻寸㹀 넝䏝ד醱꧟ז㹋爡⠓㉏겗פך黝欽 57/58

㹑⠗ 劤傈ך鑧겗ך㢳ֻכֿך剅硂דؕغ٦ׁ׸גְתׅ 58/58