ஜ೾େֶγεςϜ৘ใܥഅ৔ઇ೫ CBCB!DTUTVLVCBBDKQ !ZVLJOP )VNBOJOUIF-PPQ 
 ػցֶश

/35 )VNBOJOUIF-PPQػցֶशɿਓ͕ؒࢀՃ͢Δػցֶशϓϩηε 2 1BSU 
 ܈ऺʹΑΔ 
 ܇࿅σʔλ࡞੒ 1BSU 
 ܈ऺ͔Βͷֶश 1BSU 
 ػցֶशͷ༷ʑͳ৔໘Ͱͷ 
 ܈ऺ׆༻ Ϟσϧ adelie gentoo chinstrap ܇࿅σʔλ Ϟσϧ ໰߹ͤ ڭࢣϥϕϧ 
 ͷఏڙ ໰߹ͤ ༷ʑͳ஌ݟ 
 ͷఏڙ 2ΑΓྑ͍ϞσϧΛޮ཰తʹֶश͢ΔͨΊʹਓؒΛͲ͏׆༻͢Δ͔ʁ ڭࢣϥϕϧ 
 ͷఏڙ ෆಛఆଟ਺ͷਓʢ܈ऺʣΛ૝ఆ

/35 Ϋϥ΢υιʔγϯάɿෆಛఆଟ਺ͷਓʹগֹͰ୯७࡞ۀΛґཔ͢Δ࢓૊Έ 3 Ϩγʔτͷॻ͖ى͜͠ ʢʣ ֆըͷײ৘ϥϕϧ෇༩ ʢʣ ྫɿ"NB[PO.FDIBOJDBM5VSL ґཔҰཡ ࡞ۀը໘ https://www.mturk.com

1BSU 
 ܈ऺʹΑΔ܇࿅σʔλ࡞੒

/35 ˔ ܈ऺʹϥϕϧΛ໰͍߹Θͤ܇࿅σʔλΛ࡞੒ ˔ ෆಛఆଟ਺ͷਓ͕ࢀՃ͢ΔͨΊڭࢣϥϕϧͷ඼࣭͕՝୊ ৴པੑ͕௿͍ࢀՃऀ΋͍Δ ϥϕϧ෇͚ཁ͕݅े෼ʹ఻ΘΒͳ͍͜ͱ͕͋Δ େྔͷϥϕϧ෇͚݁ՌΛґཔऀ͕શͯݕূ͢Δͷ͸ࠔ೉ ඼࣭อূ ڭࢣϥϕϧͷ඼࣭͕՝୊ 5 Choose the correct category Adelie Chinstrap Gentoo “This was the best book I ever read!!! Thank you so much! :)” What emotion does this text convey? Anger Disgust Fear Happiness Sadness Surprise

/35 ࢀՃऀબൈ΍ฒྻԽɾ௚ྻԽͰ඼࣭อূ 6 ࢀՃऀબൈ ฒྻԽ ௚ྻԽ ࣄલςετͰબൈ ճ౴ ໰୊ Chinstrap ໰୊ Adelie ໰୊ Adelie ਖ਼ղط஌ͷ໰୊Λࠞͥͯબൈ ճ౴ ໰୊ Gentoo ໰୊ Adelie ໰୊ Gentoo ·ͨ͸ ໰୊ͷਖ਼ղ͸"EFMJF 
 ˠޡ౴ͨ͠ɹɹͷճ౴͸શͯআ֎ ඼࣭อূ Adelie Adelie Adelie Gentoo ಉ͡໰୊΁ͷෳ਺ਓͷճ౴Λ౷߹ Adelie Chinstrap Gentoo ଟ਺ܾ౳Ͱ 
 ౷߹ ͋Δਓͷճ౴Λଞऀ͕ݕূɾमਖ਼ ճ౴ ໰୊ Gentoo ໰୊ Adelie ໰୊ Gentoo ճ౴ ໰୊ Gentoo ໰୊ Adelie ໰୊ Gentoo ❌ ⭕ ❌ ݕূ ࣮૷͕༰қͳฒྻԽ͕ 
 ޿͘༻͍ΒΕ͍ͯΔ

/35 ฒྻԽʹΑΔ඼࣭อূ ଟ਺ܾͰ͸֤ࣗͷ৴པੑΛߟྀͰ͖ͳ͍ 7 NO YES YES YES YES YES NO YES YES YES YES NO YES YES YES ໰୊ ଟ਺ܾ͸֤ࣗͷճ౴ͷॏΈ͕౳͍͠ͱΈͳ͕͢ 
 ճ౴ऀ͝ͱʹ৴པੑ͸ҟͳΔ͸ͣ ଟ਺ܾͰ 
 ༧ଌͨ͠ਖ਼ղ “Is a bird in 
 the picture?" ❌ ⭕ ⭕

/35 ฒྻԽʹΑΔ඼࣭อূ ֤ࣗͷ৴པੑ͕ෆ໌ͳͨΊॏΈ෇͖ଟ਺ܾ͸࢖͑ͳ͍ 8 ໰୊ NO YES YES YES YES YES NO YES YES YES YES NO ɹɹɹ YES ɹɹɹ YES ɹɹɹ NO ॏΈ 
 ॏΈ 
 ॏΈ 
 ॏΈ 
 :&4ථ 
 /0ථ :&4ථ 
 /0ථ :&4ථ 
 /0ථ ॏΈ෇͖ଟ਺ܾͰ 
 ༧ଌͨ͠ਖ਼ղ ࣮ࡍʹ͸ॏΈ͸Θ͔Βͳ͍ “Is a bird in 
 the picture?" ⭕ ⭕ ⭕

/35 ˔ ճ౴͔Β֤ࣗͷ৴པੑͳͲΛਪఆ͠ਖ਼ղͷ༧ଌʹ༻͍Δ ˔ ੜెͷճ౴͚͔ͩΒࢼݧͷਖ਼ղΛ༧ଌ͢ΔΑ͏ͳ΋ͷ ਅ࣮ൃݟ ਅ࣮ൃݟ 5SVUIEJTDPWFSZ ɿෳ਺ਓͷճ౴͔Βͷਖ਼ղ༧ଌ໰୊ 9 ਖ਼ղ ໰୊ NO YES YES YES YES YES NO YES YES YES YES NO ? ? ? “Is a bird in 
 the picture?"

/35 ਅ࣮ൃݟ %BXJE4LFOFճ౴ऀͷ৴པੑΛࠞಉߦྻͰදݱ 10 A. P. Dawid and A. M. Skene: Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society, Series C (Applied Statistics), 1979. ɿճ౴ऀ ͕ਖ਼ղ͕YESͷ໰୊ʹYESͱ౴͑Δ֬཰ αj j ɿճ౴ऀ ͕ਖ਼ղ͕NOͷ໰୊ʹNOͱ౴͑Δ֬཰ βj j ճ౴ऀͷ৴པੑύϥϝʔλʢࠞಉߦྻʣ ti ਖ਼ղ YES ti = NO ti = yij βj ճ౴ ճ౴Ϟσϧʢ໰୊ ʹର͢Δճ౴ऀ ͷճ౴ʣ i j αj ճ౴ YES NO ਖ਼ 
 ղ YES NO αj βj 1 − αj 1 − βj ࠞಉߦྻ Pr[yij ∣ ti = 1] = αyij j (1 − αj )(1−yij ) Pr[yij ∣ ti = 0] = β(1−yij ) j (1 − βj )yij

/35 ਅ࣮ൃݟ %BXJE4LFOFճ౴ऀͷ৴པੑΛࠞಉߦྻͰදݱ 11 qi = Pr[ti = 1 ∣ {yij }] ∝ p∏ j αyij j (1 − αj )1−yij αj = ∑ i qi yij ∑ i qi , βj = ∑ i (1 − qi )yij ∑ i (1 − qi ) , p = ∑ i qi N ਖ਼ղͷࣄޙ֬཰ ˔ ΋ Λ࢖ͬͯܭࢉ͠ Λࢉग़ ˔ Pr[ti = 0 ∣ {yij }] βj qi p = Pr[ti = 1] ਖ਼ղ͕YESͷ໰୊Ͱͷਖ਼౴཰ͷΑ͏ͳ΋ͷ &TUFQճ౴ऀ৴པੑΛݻఆͯ͠ਖ਼ղΛਪఆ .TUFQਖ਼ղΛݻఆͯ͠ճ౴ऀ৴པੑΛਪఆ ˞ ͸໰୊਺ N &.ΞϧΰϦζϜΛ࢖͍ճ౴ऀ৴པੑͱਖ਼ղΛަޓʹਪఆ

/35 ਅ࣮ൃݟ %BXJE4LFOFʢͷվྑ൛ʣ͸"NB[PO.FDIBOJDBM5VSLͰར༻Մೳ 12 Amazon SageMaker GroundTruth 
 "NB[PO.FDIBOJDBM5VSLΛར༻ͨ͠܇࿅σʔλ࡞੒Λࢧԉ https://aws.amazon.com/sagemaker/groundtruth/ 
 https://aws.amazon.com/jp/blogs/news/use-the-wisdom-of-crowds-with-amazon-sagemaker-ground-truth-to-annotate-data-more-accurately/ ฒྻλεΫΛࣗಈൃߦɺ 
 %BXJE4LFOFͰ 
 ਖ਼ղΛ༧ଌͯ͠ग़ྗ ฒྻ਺Λࢦఆ ୯ՁΛࢦఆ λεΫͷछྨΛࢦఆ

/35 ਅ࣮ൃݟ $PNNVOJUZ#$$ࠞಉߦྻΛෳ਺ਓͰڞ௨Խ 13 M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi: Community-based bayesian aggregation models for crowdsourcing. WWW 2014. Truth Predicted Truth Predicted Truth Predicted 0 1 234 4 3 2 3 1 0 0 1 234 4 3 2 3 1 0 0 1 234 4 3 2 3 1 0 0.8 0.4 0 0.8 0.4 0 0.8 0.4 0 Decisive (5%) Conservative (4%) Calibrated (91%) ճ౴ऀ͕൑அ܏޲ʹج͍ͮͯ 
 ίϛϡχςΟΛܗ੒͍ͯ͠Δͱ͢Δ ίϛϡχςΟͷࠞಉߦྻ͔Β 
 ֤ࣗͷࠞಉߦྻ͕ੜ੒͞ΕΔ ࠞಉߦྻ ࠞಉߦྻͷਪఆྫɿίϛϡχςΟ͝ͱʹҟͳΔ൑அ܏޲Λଊ͍͑ͯΔ 0: Negative 1: Neutral 2: Positive 3: Not-related 4: Unknown ࠞಉߦྻ

/35 ˔ ྫɿ܈ऺʹΑΔ࠶൜ϦεΫ༧ଌ͸ಛఆͷਓछʹରͯ͠ෆެฏ ˔ ճ౴ऀͷόΠΞεΛਪఆɾআڈ͠ͳ͕Βެฏͳਖ਼ղΛਪఆ ਅ࣮ൃݟ 'BJS5%ෆެฏͳճ౴͔Βͷެฏͳਅ࣮ൃݟ 14 Y. Li, H. Sun, W. H. Wang: Towards fair truth discovery from biased crowdsourced answers. KDD 2020. yij = ti + 𝒩 (0,σ2 j ) + bA=a j “The defendant is a [RACE] [SEX] aged [AGE]. They have been charged with: [CRIME CHARGE]. This crime is classi fi ed as a [CRIMINAL DEGREE]. They have been convicted of [NON-JUVENILE PRIOR COUNT] prior crimes. They have [JUVENILE- FELONY COUNT] juvenile felony charges and [JUVENILE-MISDEMEANOR COUNT] juvenile misdemeanor charges on their record.” Do you think this person commit another crime within 2 years? J. Dressel and H. Farid: The accuracy, fairness, and limits of predicting recidivism. Science advances, 2018. ճ౴Ϟσϧ อޢάϧʔϓʹର͢ΔόΠΞε ΞϧΰϦζϜ ਪఆͨ͠ਖ਼ղͷόΠΞεΛࢉग़ɺ 
 ٯ޲͖ͷόΠΞεΛ࣋ͭճ౴ऀΛҰਓબ୒ બ୒ͨ͠ճ౴ऀҎ֎ͷόΠΞεΛআڈͯ͠ਖ਼ղΛਪఆ ਪఆͨ͠ਖ਼ղΛ༻͍ͯ Λߋ৽ bA=a j , σj

/35 අ༻ͷޮ཰Խ Ұఆਓ਺͕ಉ͡ճ౴Λͨ࣌͠఺Ͱ໰߹ͤΛதࢭ 15 L. von Ahn, B. Maurer, C. McMillen, D. Abraham, and M. Blum: reCAPTCHA: Human-based character recognition via web security measures. Science, 2008. morning morninq morning ̎ਓ͕ಉ͡౴͑Λฦͨ͠ͷͰ 
 morning 
 Λೝࣝ݁Ռͱͯ͠࠾༻ͯ͠ऴྃ Type the word Type the word SF$"15$)"ҹ࡮෺ͷจࣈΛਓؒʹೝࣝͤ͞ΔϓϩδΣΫτ

/35 ˔ ݱࡏ·Ͱͷ:&4/0ͷճ౴਺ʹԠͯ͡໰߹ͤͷ௥ՃΛܾఆ ˔ ޡ൑ఆͷ֬཰͕ҰఆҎ্ͷ৔߹ʹ໰߹ͤΛ௥Ճ͢Δ අ༻ͷޮ཰Խ ෆ࣮֬ੑʹԠͯ͡৽ͨͳ໰߹ͤΛ௥Ճ 16 V. S. Sheng, F. Provost, and P. G. Ipeirotis: Get another label? Improving data quality and data mining using multiple, noisy labelers. KDD 2008. NO YES YES YES NO :&4ͷਓ਺ /0ͷਓ਺ n = 3 ¯ n = 2 ผͷਓʹ໰͍߹ΘͤΔ͔ʁ ਖ਼ղ͕YESͰ͋Δࣄޙ֬཰ ͕ ʹै͏ͱ͢Δ q Beta(n + 1,¯ n + 1) I0.5 (n, ¯ n) NOͱ൑ఆ YESͱ൑ఆ ਖ਼ղ͕YESͳΒޡ൑ఆͷ֬཰͸I0.5 (n, ¯ n) ਖ਼ղ͕NO ͳΒޡ൑ఆͷ֬཰͸ 1 − I0.5 (n, ¯ n) ὎ ͷͱ͖໰߹ͤ௥Ճ min{I0.5 (n, ¯ n),1 − I0.5 (n, ¯ n)} > ϵ

/35 ଟ਺ܾͷ౴͑Λਖ਼ղͱݟͳͯ͠ 
 ֤ࣗͷਖ਼౴֬཰ͱͦͷ৴པ۠ؒΛਪఆ ৴པ۠ؒͷ্ݶ͕ߴ͍ճ౴ऀɹɹɹΛ࠾༻ ˔ ৴པੑͷߴ͍ճ౴ऀʹ໰͍߹ΘͤΔ͜ͱͰ༧ࢉΛޮ཰ར༻͍ͨ͠ ˔ ୭͕৴པͰ͖Δ͔͸ࣄલʹ͸Θ͔Βͳ͍ͷͰ 
 ଟ࿹όϯσΟοτ໰୊ʹ͓͚Δ6$#ઓུΛར༻ͯ͠ 
 ৴པͰ͖Δճ౴ऀ΁ͷ໰߹ͤʢ׆༻ʣͱಉ࣌ʹ৴པͰ͖Δճ౴ऀΛ୳ࡧ අ༻ͷޮ཰Խ *&5ISFTI৴པͰ͖Δճ౴ऀΛ୳ࡧɾ׆༻ 17 P. Donmez, J. G. Carbonell, and J. Schneider: E ff i ciently learning the accuracy of labeling sources for selective sampling. KDD 2009. ਖ਼౴֬཰ ਖ਼౴֬཰͕ߴ͍ˠ׆༻ ୳ࡧ͕ෆे෼ɹˠ୳ࡧ {

1BSU 
 ܈ऺ͔Βͷֶश

/35 ܈ऺ͔Βͷֶश ܈ऺͷճ౴͔ΒػցֶशϞσϧΛ௚઀ֶश 19 ௨ৗͷֶश ܈ऺ͕࡞੒ͨ͠ 
 ܇࿅σʔλ͔Βͷֶश ܈ऺ͔Βͷֶश adelie chinstrap chinstrap chinstrap gentoo adelie gentoo adelie chinstrap gentoo gentoo gentoo adelie gentoo chinstrap ਖ਼ղ ֶश adelie gentoo chinstrap ֶश ਖ਼ղΛਪఆ adelie chinstrap chinstrap chinstrap gentoo adelie gentoo adelie chinstrap gentoo gentoo gentoo ֶश Ϟσϧ

/35 ܈ऺ͔Βͷֶश ճ౴Ϟσϧ͔Βਪఆͨ͠ਖ਼ղΛར༻ͯ͠෼ྨϞσϧΛֶश 20 V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy: Learning from crowds. Journal of Machine Learning Research, 2010. ෼ྨϞσϧ fw (xi ) = Pr [yi = 1 ∣ xi] = 1 1 + exp(−w⊤xi ) ෼ྨϞσϧͷֶशखॱʢऩଋ͢Δ·Ͱ܁Γฦ͢ʣ &TUFQ 
 ճ౴ऀࠞಉߦྻΛݻఆͯ͠ਖ਼ղΛਪఆʢ%BXJE4LFOFͱಉ༷ʣ .TUFQ 
 ਪఆͨ͠ਖ਼ղΛ༻͍ͯճ౴ऀࠞಉߦྻͱ෼ྨϞσϧΛߋ৽ ௨ৗͷ܇࿅σʔλ {(xi , yi )} {(xi , yi1 , yi2 , …)} ܈ऺ͔Βͷֶशͷ܇࿅σʔλ ճ౴Ϟσϧ Pr[yij ∣ yi = 1] = αyij j (1 − αj )(1−yij ) Pr[yij ∣ yi = 0] = β(1−yij ) j (1 − βj )yij

/35 ܈ऺ͔Βͷֶश "HH/FUճ౴Ϟσϧ͔Βਪఆͨ͠ਖ਼ղΛར༻ͯ͠ਂ૚ֶशϞσϧΛֶश 21 S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, and N. Navab: AggNet: Deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Transactions on Medical Imaging, 2016. ਂ૚ֶशʹΑΔ෼ྨϞσϧ ճ౴Ϟσϧ &TUFQͰਪఆͨ͠ਖ਼ղЖΛ ༻͍ͯ෼ྨϞσϧΛߋ৽ ALBARQOUNI et al.: AGGNET: DEEP LEARNING FROM CROWDS FOR MITOSIS DET Fig. 2. AggNet architecture: The same CNN architecture is used for different w e t b a g f m

/35 ܈ऺ͔Βͷֶश ڞ௨ɾݸผͷࠞಉߦྻΛ༻ҙͯ͠ਪఆΛޮ཰Խ 22 Z. Chu, J. Ma, and H. Wang: Learning from crowds by modeling common confusions. AAAI 2021. model parameter estimation. We define cross-entropy loss on the observed annotations and use error back-propagation to update the classifier’s output and the network parameters simultaneously. e 1:R xi Classifier Aux.Net Wa 1:R i 1:R i (1 1:R i ) + = f i Wgf i W1:Rf i h1:R i input parallel noise adaptation layers backbone model predicted anno. dist. where (W terms fo moid fun fer the p the mag or small we norm before c Based modelin the netw ing the feature v ing the c and pred ڞ௨ࠞಉߦྻ ݸผࠞಉߦྻ ࣮ࡍͷճ౴ͱ 
 ൺֱ͠ଛࣦΛܭࢉ ෼ྨϞσϧ ೖྗ ճ౴ऀಛ௃ ڞ௨ͷࠞಉߦྻΛ࢖͏֬཰ ิॿϞσϧ

/35 ೳಈֶश ֶशʹ༗༻ͳαϯϓϧΛબͼϥϕϧ෇͚ճ਺ΛݮΒ͢ 23 ֶशʹ༗༻ͳαϯϓϧΛબΜͰϥϕϧ෇͚ 
 ˠ৽͍͠ϥϕϧͰ෼ྨϞσϧΛߋ৽ adelie ༗༻ͳ 
 αϯϓϧΛબ୒ ϞσϧΛߋ৽ ϥϕϧ෇͚ ೳಈֶश "DUJWFMFBSOJOH ༗༻ੑͷࢦඪ ༧ଌ෼෍Λ༻͍ͯෆ࣮֬ੑΛଌΔ 
 ʢΤϯτϩϐʔ౳Λར༻ʣ adelie chinstrap gentoo 0 0.25 0.5 0.75 1 adelie chinstrap gentoo 0 0.25 0.5 0.75 1 ༧ଌ෼෍ ༧ଌ͕࣮֬ ༧ଌ͕ෆ࣮֬

/35 ܈ऺ͔Βͷೳಈֶश ༗༻ͳαϯϓϧʹ৴པੑͷߴ͍ճ౴ऀΛׂΓ౰ͯΔ 24 Y. Yan, R. Rosales, G. Fung, and J. G. Dy: Active learning from crowds. ICML 2011. fw (x) = Pr [y = 1 ∣ x] = 1 1 + exp(−w⊤x) ෼ྨϞσϧ ηwj (x) = 1 1 + exp(−w⊤ j x) ճ౴Ϟσϧʢਖ਼౴֬཰ʣ ೳಈֶशͷखॱ ෼ྨϞσϧʹΑΔ༧ଌ͕࠷΋ෆ࣮֬ͳαϯϓϧ ΛબͿ ਖ਼౴֬཰ ͕࠷େͷճ౴ऀΛબͼճ౴Λऔಘ ಘΒΕͨճ౴Λ༻͍ͯ෼ྨϞσϧͱճ౴Ϟσϧ Λߋ৽ x* ηwj (x*) w, {wj }

/35 ܈ऺ͔Βͷೳಈֶश ਂ૚ֶशʹ༗༻ͳαϯϓϧʹ৴པੑͷߴ͍ճ౴ऀΛׂΓ౰ͯΔ 25 J. Yang, T. Drake, A. Damianou, and Y. Maarek: Leveraging crowdsourcing data for deep active Learning an application: Learning intents in Alexa. WWW 2018. ϞϯςΧϧϩυϩοϓΞ΢τʹΑΔෆ࣮֬ੑධՁ ˔ ਂ૚ֶशωοτϫʔΫͷҰ෦ΛϥϯμϜʹ 
 ܽམͤͨ͞΋ͷΛෳ਺༻ҙ ˔ ༧ଌ෼෍ͷฏۉΛ༻͍ͯෆ࣮֬ੑΛධՁ ճ౴Ϟσϧʢਖ਼౴֬཰ʣ ηwj (x) = 1 1 + exp(−w⊤ j Fx) , wj ∈ Rk, F ∈ Rk×d, x ∈ Rd Λ௿࣍ݩʹຒΊࠐΉ x ˞ೳಈֶशͷखॱ͸લεϥΠυͷख๏ͱಉ༷

1BSU 
 ػցֶशͷ༷ʑͳ৔໘Ͱͷ 
 ܈ऺ׆༻

/35 ಛ௃நग़ ܭࢉػͰ͸ଊ͑ΒΕͳ͍ಛ௃Λਓ͕ؒநग़ 27 S. Branson, C. Wah, F. Schro ff , B. Babenko, P. Welinder, P. Perona, S. Belongie: Visual recognition with humans in the loop. ECCV 2010. Visual Recognition with Humans in the Loop 3 mputer vision is helpful Computer vision is not helpful mputer vision is helpful Computer vision is not helpful The bird is a Black‐footed Albatross Is the belly white? yes Are the eyes white? yes Th bi d i Is the beak cone‐shaped? yes Is the upper‐tail brown? yes Is the breast solid colored? no Is the breast striped? yes I h h hi ? The bird is a Parakeet Auklet Is the throat white? yes The bird is a Henslow’s Sparrow 2. Examples of the visual 20 questions game on the 200 class Bird dataset. an responses (shown in red) to questions posed by the computer (shown in blue) sed to drive up recognition accuracy. In the left image, computer vision algorithms uess the bird species correctly without any user interaction. In the middle image, uter vision reduces the number of questions to 2. In the right image, computer n provides little help. 2ෲ͕ന͍ʁ 2໨͕ന͍ʁ 2ͪ͘͹͕͠ίʔϯܕʁ YES YES NO ಛ௃̍ ಛ௃ ಛ௃ 1 1 0 ػցֶशͰ͸ಛ௃ઃܭ͕ॏཁ 
 ˠࣄલʹ࣭໰Λ༻ҙ͓͖ͯ͠܈ऺʹճ౴ͤͯ͞ಛ௃Λநग़

/35 ಛ௃ઃܭ "EB'MPDLಛ௃நग़͚ͩͰ͸ͳ͘ಛ௃ઃܭ΋ਓ͕࣮ؒࢪ 28 R. Takahama, Y. Baba, N. Shimizu, S. Fujita, and H. Kashima: AdaFlock: Adaptive feature discovery for human-in-the-loop predictive modeling. AAAI 2018. B ਖ਼ྫɾෛྫΛ 
 ਓؒʹఏࣔ C ਖ਼ྫɾෛྫΛ 
 ۠ผ͢Δ࣭໰จΛ 
 ਓ͕ؒੜ੒ 
 ʢಛ௃ઃܭʣ D ਓ͕࣭ؒ໰ʹճ౴ 
 ʢಛ௃நग़ʣ E ෼ྨثΛߋ৽͠ 
 ಛ௃ઃܭʹ࢖͏ 
 αϯϓϧΛબ୒ ྫɿϞωͱγεϨʔͷֆͷ෼ྨ

/35 Ϟσϧݕࠪ ෆద੾ͳಛ௃Λਓؒʹআڈͤͯ͞ϞσϧΛվળ 29 M. T. Ribeiro, S. Singh, and C. Guestrin: "Why should I trust you?": Explaining the predictions of any classi fi er. KDD 2016. https://drive.google.com/ fi le/d/0ByblrZgHugfYZ0ZCSWNPWFNONEU/view ྫɿफڭؔ࿈ϝʔϧͷ൑ఆ Ϟσϧͷ 
 ൑அࠜڌΛఏࣔ )PTU 1PTUJOH //51౳ͷ 
 ແؔ܎ͳ୯ޠΛॏࢹ͢Δͷ͸ෆద੾

/35 ղऍੑ ਓؒʹΑΔ൑அ࣌ؒΛߟྀͯ͠ղऍੑͷߴ͍ϞσϧΛൃݟ 30 I. Lage, A. Ross, S. J. Gershman, B. Kim, and F. Doshi-Velez: Human-in-the-loop interpretability prior. NeurIPS 2018. ਫ਼౓ͷߴ͍ϞσϧΛީิͱͯ͠ྻڍ ਓؒʹධՁͤ͞ΔϞσϧ Λબ୒ M ਓ͕ؒϞσϧ ͷղऍੑ ΛධՁ M p(M) ࠷ྑͷϞσϧΛܾఆ p(M) ≈ 1 N ∑ x 𝖧 𝖨𝖲 (x, M) 𝖧𝖨𝖲 (x, M) = max{0, 𝗆𝖺 𝗑 𝖱𝖳 − 𝗆𝖾𝖺 𝗇𝖱𝖳 (x, M)} ฏۉ൑அ࣌ؒ (a) An example of our interface with a tree trained on Ϟσϧͷઆ໌ʹج͍ͮͯਓؒʹ༧ଌΛͤ͞Δ 
 ˠॴཁ͕࣌ؒ୹͍΄Ͳղऍੑͷߴ͍Ϟσϧ

/35 ఢରతੜ੒ωοτϫʔΫʢ("/ʣ )VNBO("/ਓؒΛ("/ͷࣝผثʹͯ͠ਓؒͷײੑΛऔΓࠐΉ 31 K. Fujii, Y. Saito, S. Takamichi, Y. Baba, and H. Saruwatari: HumanGAN: Generative adversarial network with human-based discriminator and its evaluation in speech perception modeling. ICASSP 2020. Pertur- bation Crowd- workers Backpropagation using approximated Worker’s answer to “to what degree are two samples different?” [times] 2 Fig. 2. Generator training procedure of proposed HumanGAN. Crowdworkers state a perceptual difference (i.e., difference of pos- terior probabilities) of two perturbed samples. Answer and perturba- tion are used for backpropagation to train generator. University of Tokyo, Japan. ty of Tsukuba, Japan. minator Natu- ral -based ata Distr. of human perception of basic GAN and proposed HumanGAN. rator by fooling DNN-based discriminator scriminator), and generator finally represents n. In comparison, HumanGAN trains gener- ࣝผثͱͯ͠ਓؒΛ༻͍Δ ਓؒΛ᱐͢ੜ੒ثΛֶश͢Δ͜ͱͰ ਓؒͷײੑΛऔΓࠐΉ

/35 ਓؒͱܭࢉػͷڠௐϞσϧ ਓؒ΁ͷ໰͍߹ΘͤΛؚΊͨϞσϧΛֶश 32 B. Wilder, E. Horvitz, and E. Kamar: Learning to complement humans. IJCAI 2020. ਓؒͱܭࢉػͷڠௐϞσϧ ೖྗ x ࣭໰ثq(x) ਓؒʹ໰͍߹Θͤͳ͍ ਓؒʹ໰͍߹ΘͤΔ h ෼ྨث ̂ y = f(x) ̂ y = f(x, h) ڠௐϞσϧͷֶश ܇࿅σʔλ Λ༻͍ͯ ͕࠷େʹͳΔΑ͏ʹ 
 ෼ྨثͱ࣭໰ثΛֶश {(x, y, h)} q(x){u(y, f(x, h)) − c} + (1 − q(x))u(y, f(x)) ༧ଌͷਖ਼͠͞ͷޮ༻ؔ਺ Λग़ྗ Λग़ྗ අ༻

·ͱΊ

/35 )VNBOJOUIF-PPQػցֶशɿਓ͕ؒࢀՃ͢Δػցֶशϓϩηε 34 1BSU 
 ܈ऺʹΑΔ 
 ܇࿅σʔλ࡞੒ 1BSU 
 ܈ऺ͔Βͷֶश 1BSU 
 ػցֶशͷ༷ʑͳ৔໘Ͱͷ 
 ܈ऺ׆༻ Ϟσϧ adelie gentoo chinstrap ܇࿅σʔλ Ϟσϧ ໰߹ͤ ڭࢣϥϕϧ 
 ͷఏڙ ໰߹ͤ ༷ʑͳ஌ݟ 
 ͷఏڙ ڭࢣϥϕϧ 
 ͷఏڙ ˔ ਅ࣮ൃݟ ˔ අ༻ͷޮ཰Խ ˔ ܈ऺ͔Βͷਂ૚ֶश ˔ ೳಈֶश ˔ ಛ௃நग़ɾઃܭ ˔ ղऍੑ޲্ ˔ ਓؒͱܭࢉػͷڠௐϞσϧ ˔ FUD

/35 35 ࣛౡٱ࢚ খࢁ૱ അ৔ઇ೫ 
 ώϡʔϚϯίϯϐϡςʔγϣϯͱΫϥ΢ιʔγϯά 
 ߨஊࣾ Robert (Munro) Monarch. 
 Human-in-the-Loop Machine Learning: 
 Active learning and annotation for human-centered AI. 
 Manning Publications, 2021. ࢀߟจݙ