ؼُ٦وٝ ؝ٝؾُذ٦ءّٝ ח״׷⼿锃㷕统佄䴂 꼛㜥ꨒ⛆瘰岚㣐㷕 /FX&EVDBUJPO&YQP 䎃剢傈

嚊銲 չؼُ٦وٝ؝ٝؾُذ٦ءّٝպך倯岀׾欽ְ 欰䖝ֶָ✼ְך㷕统׾⸔ֽさֲչ⼿锃㷕统պ׾佄䴂 湱✼鐰⣣ 湱✼⡲㉏ 湱✼幐⵴ Ⱏずⶼ⡲ 2/15

ؼُ٦وٝ؝ٝؾُذ٦ءّٝ ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝ ➂꟦ה➂䊨濼腉׾穈׫さ׻ׇ׷ֿהד չו׍׵ַ♧倯׌ֽדכ鍑ֽזְ㉏겗պך鍑寸׾湡䭷ׅծ ➂䊨濼腉ך♧ⴓꅿ ➂䊨濼腉ח״׷㉏겗鍑寸׾ ➂꟦ָ佄䴂 ➂꟦ח״׷㉏겗鍑寸׾ ➂䊨濼腉ָ佄䴂 ➂꟦ה➂䊨濼腉ך穈׫さ׻ׇדꨇ׃ְ㉏겗׾鍑寸 3/15

ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽SF$"15$)" ➂䊨濼腉ך➿׻׶ח➂꟦ָ俑㶵钠陎 ˑNPSPJEH˒ ˑNPSSJOH˒ ˑNPSOJOH˒ 4UFQ剅硂⚥ך俑㶵׾אך0$3ءأذيח钠陎ׇׁ׷ 4UFQ穠卓ָ♶♧荜ךהֹ➂꟦ח㉏ְさ׻ׇ׷ L. von Ahn et al.: reCAPTCHA: Human-based character recognition via web security measures, In Science, 2008. 4/15

-tap to take a photo. -tap to begin recording your question and again to stop. side, User ? Database - al Client mote Services and Worker Interface ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽7J[8J[ ➂䊨濼腉ך➿׻׶ח➂꟦ָ鋔鋙ꥺְָ罏׾佄䴂 Ύءأذيⰻ鿇 ך➂꟦ָ㔐瘶 J. Bigham et al.: VizWiz: Nearly real-time answers to visual questions, In UIST, 2010. ΍ِ٦ؠָ颵㉏׾䫎珲 ⢽չ؝٦ָٝⰅ׏ ׋綸כו׸պ 5/15

ؼُ٦وٝ؝ٝؾُذ٦ءّٝה侄肪 ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝך侄肪פך崞欽⢽䱰挿 o 侄䌌ח״׷ٖه٦ز瘝ך䱰挿׾ ➂䊨濼腉ח״׷荈⹛䱰挿ח縧ֹ䳔ִ׋ְָ劢׌㔭ꨇ ☞ ➂䊨濼腉ך➿׻׶ח欰䖝ず㡦ד䱰挿湱✼鐰⣣ ! 湱✼鐰⣣כ 欰䖝ず㡦ֶָ✼ְך㷕统׾⸔ֽさֲ⼿锃㷕统 ך⤛鹌ח׮אזָ׷ ➂䊨濼腉ך➿׻׶ח欰䖝ָ侄䌌׾佄䴂 6/15

湱✼鐰⣣ ! 欰䖝Ⰻ㆞ָ姻׃ֻ鐰⣣דֹ׷הכꣲ׵זְ ‛ ♧אך瘶周׾醱侧ך欰䖝ָ鐰⣣ծ 鐰⣣罏ך腉⸂׾罋䣁׃姻׃ְ挿侧׾✮庠 ? 瘶周 鐰⣣罏 挿侧 㔐瘶罏 湱✼鐰⣣穠卓ַ׵姻׃ְ挿侧׾✮庠 Y. Baba and H. Kashima: Statistical quality estimation for general crowdsourcing tasks, In KDD, 2013. 挿 挿 挿 ぐ鐰⣣罏ך䱰挿穠卓ַ׵ 鐰⣣罏ך腉⸂׾罋䣁׃ג挿侧׾✮庠 7/15

湱✼鐰⣣ٌرٕ ! 㔐瘶罏ָ瘶周׾⡲׷麓玎٥鐰⣣罏ָ瘶周׾䱰挿ׅ׷麓玎׾ ٌرٕ⻉ׅ׷ֿהד瘶周ך溪ך挿侧׾✮庠דֹ׷ o 4UFQ㔐瘶罏ָ㉏겗ח㼎׃$% 挿ך瘶周׾⡲䧭ׅ׷ծ 挿侧$% כ 䎂㖱% , ⴓ侔% *ח䖞ֲ 瘶周ך溪ך挿侧 㔐瘶罏ך挿侧ךⴓ侔 㔐瘶罏ך腉⸂ 瘶周 㔐瘶罏 qjk ⇠ N qjk | µk, 2 k $%挿 㔐瘶罏ָ瘶周׾⡲׷麓玎׾ٌرٕ⻉ 8/15

! 㔐瘶罏ָ瘶周׾⡲׷麓玎٥鐰⣣罏ָ瘶周׾䱰挿ׅ׷麓玎׾ ٌرٕ⻉ׅ׷ֿהד瘶周ך溪ך挿侧׾✮庠דֹ׷ o 4UFQ鐰⣣罏ָ瘶周ח㼎׃-$% 挿׾➰♷ׅ׷ծ 挿侧-$% כ䎂㖱$% + - , ⴓ侔- *ח䖞ֲ 湱✼鐰⣣ٌرٕ 鐰⣣罏ך挿侧ךⴓ侔 瘶周ך溪ך挿侧 鐰⣣罏ךغ؎،أ 瘶周 鐰⣣罏 -$%挿 sijk ⇠ N sijk | qjk + ⌘i, 2 i 鐰⣣罏ָ瘶周׾䱰挿ׅ׷麓玎׾ٌرٕ⻉ 9/15

♧㼎嫰鯰ח״׷湱✼鐰⣣ ! 瘶周ח挿侧׾➰ֽ׷ךָꨇ׃ְ㜥さ׮֮׷ ‛ 瘶周ل،׾嫰鯰ׇׁծ 嫰鯰穠卓ַ׵鐰⣣罏ך腉⸂׾罋䣁׃ג瘶周ך挿侧׾✮庠 ? "ך挿侧 㔐瘶罏 T. Sunahase, Y. Baba and H. Kashima, Pairwise HITS: Quality estimation from pairwise comparisons in creator-evaluator crowdsourcing process, In AAAI, 2017 瘶周" 瘶周# " " # # # " ? #ך挿侧 ぐ鐰⣣罏ך♧㼎嫰鯰穠卓ַ׵ 鐰⣣罏ך腉⸂׾罋䣁׃ג挿侧׾✮庠 ♧㼎嫰鯰穠卓ַ׵姻׃ְ挿侧׾✮庠 10/15

♧㼎嫰鯰ٌرٕ ! 溪ך挿侧ה鐰⣣罏ך腉⸂׾❛✼ח刿倜 o 4UFQ鐰⣣罏腉⸂- ׾㔿㹀׃溪ך挿侧$ ׾刿倜 o 4UFQ溪ך挿侧$ ׾㔿㹀׃鐰⣣罏腉⸂- ׾刿倜 qj qj0 = X i2Vj j0 ri X i2Vj0 j ri 瘶周ח䫎牰׃׋ ➂ך腉⸂ךㄤ ri = |{(j j0) 2 Vi | qj > qj0 }| |Vi | 鐰⣣罏ך 姻׃ְ䫎牰 ךⶴさ 挿侧ה鐰⣣罏腉⸂׾❛✼ח刿倜׃挿侧׾✮庠 11/15 瘶周ה’ך溪 ך挿侧ך䊴 瘶周ˏח䫎牰׃ ׋➂ך腉⸂ךㄤ 溪ך挿侧ָ넝ְ倯ך 瘶周פך鐰⣣罏ך 䫎牰㔐侧 鐰⣣罏ך 䫎牰㔐侧

湱✼幐⵴ ! ֶ✼ְח䱰挿ׅ׷׌ֽדכזֻ ֶ✼ְח幐⵴׮ׅ׷ֿהד瘶周פךؿ؍٦سغحؙ׾䲿⣘ ! 䱰挿ה幐⵴ך♧顐䚍׾ⵃ欽׃ג幐⵴罏腉⸂׾䱿㹀ծ 姻׃ְ挿侧ך✮庠ח崞欽 o 䱰挿ה幐⵴ָ♧顐׃זְ⢽ չ挿侧כ⡚ְךח幐⵴ָⰋ搫זְպ չ挿侧ָ넝ְךח幐⵴׾׋ֻׁ׿ׅ׷պ We observe that sometimes there is inconsistency between a grade and a correction; for example, a grader provides a high grade with a submission but she corrects many errors. We observed the occasional inconsistency between a grade and the correction; for example, a grader provides a high grade for a submission, but many errors were corrected. 挿 T. Sunahase, Y. Baba and H. Kashima, Statistical modeling of peer correction and peer assessment, submitted. 䱰挿ה幐⵴ך♧顐䚍׾ⵃ欽׃ג鐰⣣罏腉⸂׾䱿㹀 12/15

湱✼⡲㉏ ! 侄䌌ך➿׻׶ח欰䖝ָذأزך㉏겗׾⡲䧭 o ぐ欰䖝כ➭ך欰䖝ָ⡲׏׋㉏겗׾鍑ֻ ! 欰䖝ָ⡲׏׋㉏겗ח㼎ׅ׷㔐瘶ַ׵ぐ欰䖝ך统擾䏝׾✮庠 o 葺ְ㉏겗׾⡲׷欰䖝׮ְ׸ל׉ֲׄׯזְ欰䖝׮ְ׷ Ԃ欰䖝׀הך⡲㉏腉⸂׾罋䣁 A. Taniguchi and S. Inoue, A method for automatic assessment of user-generated tests and its evaluation, In UbiComp/ISWC Adjunct, 2015. ،وبٝך،ؚٓ٘ٔ屎 崧㚖ד饯ֿ׷˘ "ؾٗٗحؕ #ػٗٗحؕ $هٗٗحؕ %لٗٗحؕ 欰䖝ָ㉏겗׾⡲䧭 ➭ך欰䖝ָ㔐瘶 欰䖝ָ✼ְח㉏겗׾⳿׃さֲהֹך⡲㉏腉⸂׾䱿㹀 13/15

Ⱏずⶼ⡲ L. Yu and J. V. Nickerson: Cooks or cobblers? crowd creativity through combination, In CHI, 2011. ⢽喱㶨ךرؠ؎ٝ 4UFQ痥⚅➿ך⦐⡤׾ぐ荈ָ欰䧭 4UFQ䫎牰ד⮚葺⦐⡤ ׾鼅䫙׃如⚅➿ך⦐⡤ ׾❛⿷ח״׶欰䧭 4UFQ鼅䫙٥❛⿷׾粸׶鵤ׅ 鼋⠗涸،ٕ؞ٔؤيח⦺ְⰟずⶼ⡲׾佄䴂 ! ✼ְך涪䟝׾⤛׃׫׿זד⡲ㅷ׾峤箺ׇׁגְֻ 14/15

תה׭ ➂䊨濼腉ך♧ⴓꅿד֮׷ չؼُ٦وٝ؝ٝؾُذ٦ءّٝպ׾欽ְ 欰䖝ֶָ✼ְך㷕统׾⸔ֽさֲչ⼿锃㷕统պ׾佄䴂 湱✼鐰⣣ 湱✼⡲㉏ 湱✼幐⵴ Ⱏずⶼ⡲ 鐰⣣罏腉⸂׾䱿㹀 幐⵴罏腉⸂׾䱿㹀 ⡲㉏罏腉⸂׾䱿㹀 涪䟝׾佄䴂 15/15