Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ヒューマンコンピュテーションとクラウドソーシング / Human Computation and Crowdsourcing

Yukino Baba
June 08, 2018
2.8k

ヒューマンコンピュテーションとクラウドソーシング / Human Computation and Crowdsourcing

2018年度人工知能学会全国大会チュートリアル

Yukino Baba

June 08, 2018
Tweet

Transcript

  1. ؼُ٦وٝ
    ؝ٝؾُذ٦ءّٝה
    ؙٓؐسا٦ءؚٝ
    꼛㜥ꨒ⛆瘰岚㣐㷕
    䎃䏝➂䊨濼腉㷕⠓Ⰻ㕂㣐⠓ثُ٦زٔ،ٕ
    䎃剢傈

    View full-size slide

  2. 劤ثُ٦زٔ،ٕ
    㼔Ꟍ㹺涪鋅 ⼿锃佄䴂
    ⚛⴨㉏さׇ 湫⴨㉏さׇ
    ➂꟦ך꧊㔚ה➂䊨濼腉ח״׷㉏겗鍑寸ך䩛岀׾稱➜
    2/58

    View full-size slide

  3. ؼُ٦وٝ؝ٝؾُذ٦ءّٝ
    ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝ
    ➂꟦ה➂䊨濼腉׾穈׫さ׻ׇ׷ֿהד
    ו׍׵ַ♧倯׌ֽדכ鍑ֽזְ㉏겗׾鍑寸
    ➂䊨濼腉ח״׷㉏겗鍑寸׾
    ➂꟦ָ佄䴂
    ➂꟦ח״׷㉏겗鍑寸׾
    ➂䊨濼腉ָ佄䴂
    ➂꟦ה➂䊨濼腉ך穈׫さ׻ׇדꨇ׃ְ㉏겗׾鍑寸
    3/58

    View full-size slide

  4. ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽SF$"15$)"
    俑㶵钠陎ءأذيח➂꟦׾穈׫鴥׬
    ˑNPSPJEH˒
    ˑNPSSJOH˒
    ˑNPSOJOH˒
    4UFQ剅硂⚥ך俑㶵׾אך0$3ءأذيח钠陎ׇׁ׷
    4UFQ穠卓ָ♶♧荜ךהֹ➂꟦ח㉏ְさ׻ׇ׷
    L. von Ahn et al.: reCAPTCHA: Human-based character recognition via web security measures, In
    Science, 2008.
    4/58

    View full-size slide

  5. -tap to take a photo.
    -tap to begin recording
    your question and again to stop.


    side,
    User ?
    Database -
    al Client
    mote Services and Worker Interface
    ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽7J[8J[
    鋔鋙ꥺְָ罏佄䴂ءأذيח➂꟦׾穈׫鴥׬
    Ύءأذيⰻ鿇
    ך➂꟦ָ㔐瘶
    J. Bigham et al.: VizWiz: Nearly real-time answers to visual questions, In UIST, 2010.
    ΍ِ٦ؠָ颵㉏׾䫎珲
    ⢽չ؝٦ָٝⰅ׏
    ׋綸כו׸պ
    5/58

    View full-size slide

  6. ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽'PME*U
    S. Cooper et al.: Predicting protein structures with a multiplayer online game, In Nature, 2010.
    فٖ؎َ٦כ
    넝أ؝،׾杆ְ
    圓鸡׾㢌⻉ׇׁ׷
    ةٝػؙ颵ך圓鸡✮庠
    ،ىظꃐꂁ⴨ַ׵
    甧⡤圓鸡׾✮庠ׅ׷㉏겗
    ةٝػؙ颵圓鸡✮庠׾؜٦ي⻉׃➂꟦ח鍑ַׇ׷
    6/58

    View full-size slide

  7. ؙٓؐسا٦ءؚٝ
    ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝדכ㢳ֻך⿫⸇罏ָ䗳銲
    ! ؙٓؐسا٦ءؚٝ
    o ؎ٝة٦طحز׾鸐ׄג➂꟦ח⡲噟׾⣛걾ׅ׷➬穈׫
    o ⢽"NB[PO.FDIBOJDBM5VSL ٓٝ؟٦ؤ
    o ♧鿇ך؟٦ؽأדכ"1*ח״׷⡲噟⣛걾ָ〳腉
    㢳侧ך➂꟦ח،ؙإأׅ׷׋׭ך➬穈׫
    ؙٓؐس
    ا٦ءؚٝ
    ⣛걾罏 ⡲噟罏
    ΍ةأؙ涪遤 Ύةأؙⶴ䔲
    Ώ穠卓䲿⳿
    ΐ㜠ꂹ锜実
    7/58

    View full-size slide

  8. "NB[PO.FDIBOJDBM5VSLךⵃ欽⢽
    4UFQةأؙذٝفٖ٦ز׾鏣鎘׃ةأؙ׾涪遤
    ةأؙ⢽
    չⱖ溪ך➂暟ח♧殢䔲גכת׷
    せ⵸׾鼅׿דֻ׌ְׁպ
    ⱖ溪63-瘝כر٦ةؿ؋؎ٕד䭷㹀
    8/58

    View full-size slide

  9. "NB[PO.FDIBOJDBM5VSLךⵃ欽⢽
    4UFQةأؙ♧鋮ח䲓鯹ׁ׸⡲噟罏ָ⡲噟Ꟛ㨣
    9/58

    View full-size slide

  10. "NB[PO.FDIBOJDBM5VSLךⵃ欽⢽
    4UFQ⡲噟穠卓׾然钠٥䪫钠ꬊ䪫钠׾寸׭׷
    ⡲噟罏*% ⡲噟穠卓
    10/58

    View full-size slide

  11. ؼُ٦وٝ؝ٝؾُذ٦ءّٝך铬겗
    ! ➂꟦ָչ䌢ח٥铩ד׮պ姻׃ְ瘶ִ׾鵤ׅהכꣲ׵זְ
    o ♶然㹋䚍➂כ圫ղז銲㔓דىأ׾ׅ׷
    o 㢳圫䚍➂ח״׏ג腉⸂כ殯ז׷
    ! ➂꟦ַ׵姻׃ְ瘶ִ׾䒷ֹ⳿ׅ׋׭ך䊨㣗ָ䗳銲
    ➂꟦ַ׵וֲװ׏ג姻׃ְ瘶ִ׾䒷ֹ⳿ַׅ
    11/58

    View full-size slide

  12. SF$"15$)"ך،فٗ٦ث
    ! ΍ ⹛堣בֽ俑㶵钠陎⡲噟׾钠鏾ءأذيח穈׫鴥׬
    o 㔐瘶罏כչ荈ⴓָ➂꟦׌պה爙ׅ׋׭溪⶛ח䮋׬
    ! Ύ 姻鍑傀濼ך㉏겗ךⵃ欽אך俑㶵⴨׾䲿爙ׅ׷
    o 姻鍑劢濼ך俑㶵⴨钠陎׃׋ְ俑㶵⴨
    o 姻鍑傀濼ך俑㶵⴨➂꟦ַنحزַךⴻ㹀חⵃ欽
    ! Ώ ⚛⴨⻉醱侧ךչ➂꟦պָずׄ瘶ִ׾鵤׃׋׵䱰欽
    姻鍑劢濼 姻鍑傀濼
    姻瘶ז׵➂꟦
    铎瘶ז׵نحز
    钠鏾ءأذيח穈׫鴥׫姻׃ְ瘶ִ׾䒷ֹ⳿ׅ
    12/58

    View full-size slide

  13. 劤ثُ٦زٔ،ٕךذ٦و꧊㔚׾崞ַ׃׋،فٗ٦ث
    ➂䊨濼腉ח״׷㉏겗鍑寸׾
    ➂꟦ָ佄䴂
    ➂꟦ח״׷㉏겗鍑寸׾
    ➂䊨濼腉ָ佄䴂
    ꧊㔚ך⸂ד姻׃ְ瘶ִ׾䒷ֹ⳿ׅ
    㼔Ꟍ㹺涪鋅 ⼿锃佄䴂
    ⚛⴨㉏さׇ 湫⴨㉏さׇ
    知⽃٥⽃秪ז㉏겗 㔭ꨇ٥醱꧟ז㉏겗
    ! ꧊㔚ד㉏겗׾鍑ַׇ׷٥鍑ֽ׷➂׾鋅אֽ׷ֿהד
    ➂꟦ַ׵姻׃ְ瘶ִ׾䒷ֹ⳿ׅ
    13/58

    View full-size slide

  14. ΍⚛⴨㉏さׇ

    View full-size slide

  15. ⚛⴨㉏さׇ
    ! ㉏ְさ׻ׇ湱䩛ָ♧➂׌ה
    ׉ך➂ָىأ׾ׅ׷ה姻׃ְ瘶ִָ䖤׵׸זְ
    ! 醱侧➂ח㉏ְさ׻ׇג㔐瘶׾窟さׅ׷ֿהד
    姻׃ְ瘶ִ׾䒷ֹ⳿ׅ
    o 知⽃ז䩛岀㢳侧寸
    ⴽղח鍑ַׇג瘶ִ׾תה׭׷
    ⱖ溪ח둷ָⱖ׏גְתַׅ
    :&4 /0
    NO YES YES
    YES ׾
    瘶ִה׃ג
    䱰欽
    15/58

    View full-size slide

  16. 窟鎘涸ז㔐瘶窟さ
    ! 㢳侧寸״׶׮峤箺ׁ׸׋倯岀窟鎘涸ז㔐瘶窟さ
    ! 醱侧➂ך㔐瘶ַ׵姻鍑׾✮庠ׅ׷㉏겗ה׃ג㹀䒭⻉
    o 欰䖝ך㔐瘶׌ַֽ׵鑐꿀ך姻鍑׾✮庠ׅ׷״ֲז׮ך
    YES YES YES NO
    NO YES YES YES
    NO YES NO YES
    ?
    ?
    ?
    ㉏겗
    醱侧➂ך㔐瘶ַ׵窟鎘涸ח姻鍑׾✮庠ׅ׷
    姻鍑
    16/58

    View full-size slide

  17. 㔐瘶罏ך腉⸂׾罋䣁
    ! 㔐瘶罏ָ醱侧㉏ח㔐瘶ׅ׷ֿה׾ⵃ欽׃ג
    㔐瘶罏ך腉⸂׾䱿㹀׃姻鍑✮庠ח欽ְ׷
    YES YES YES NO
    NO YES YES YES
    NO YES NO YES
    㔐瘶罏
    ?
    ?
    ?
    ㉏겗
    腉⸂׾䱿㹀
    A. P. Dawid and A. M. Skene: Maximum likelihood estimation of observer error-rates using
    the EM algorithm, Journal of the Royal Statistical Society. Series C (Applied Statistics), 1979.
    㔐瘶罏ך腉⸂׾䱿㹀׃姻鍑✮庠ח欽ְ׷
    姻鍑
    17/58

    View full-size slide

  18. 㔐瘶罏ך腉⸂׾罋䣁
    ! 㔐瘶罏ך腉⸂׾珏겲ך然桦ד邌植ׅ׷
    o 姻鍑ָ:&4ךהֹח姻׃ֻ:&4ה瘶ִ׷然桦
    o 姻鍑ָ/0ךהֹח姻׃ֻ/0ה瘶ִ׷然桦
    ! ֿך然桦׾欽ְ׷ה如ך״ֲז㔐瘶欰䧭ٌرָٕ䖤׵׸׷
    腉⸂׾然桦ד邌植׃㔐瘶欰䧭麓玎׾ٌرٕ⻉
    ✓j
    j
    姻鍑錁庠דֹזְ
    㔐瘶 /0:&4

    Pr [yij
    | ti = 1] = ✓yij
    j
    (1 ✓j)(1 yij )
    Pr [yij
    | ti = 0] = (1 yij )
    j
    (1 j)yij
    18/58

    View full-size slide

  19. 㔐瘶罏ך腉⸂׾罋䣁
    ! 姻鍑׾悵㖈㢌侧ה׃׋&.،ٕ؞ٔؤيד
    姻鍑ה腉⸂׾❛✼ח䱿㹀ׅ׷
    o 4UFQ腉⸂׾㔿㹀׃ג姻鍑׾䱿㹀
    o 4UFQ姻鍑׾㔿㹀׃ג腉⸂׾䱿㹀
    ㉏겗ך姻鍑ה㔐瘶罏ך腉⸂׾❛✼ח䱿㹀
    qi = Pr [ti = 1 | {yij
    }] / p
    Y
    j
    ✓yij
    j
    (1 ✓j)(1 yij )
    姻鍑ָ:&4ך㉏겗דך姻瘶桦ך״ֲז׮ך
    ✓j =
    P
    i
    qiyij
    P
    i
    qi
    , j =
    P
    i
    (1 qi) yij
    P
    i
    (1 qi)
    p = Pr [ti = 1]
    19/58
    姻鍑

    View full-size slide

  20. 㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝׾罋䣁
    ! չꨇ׃ְ㉏겗ד姻瘶ׅ׷㔐瘶罏ך倯ָ腉⸂ָ넝׉ֲպ
    ̔㉏겗ךꨇ僒䏝׮罋䣁ׅ׷
    YES YES YES NO
    NO YES YES YES
    NO YES NO YES
    ?
    ?
    ?
    ㉏겗
    J. Whitehill et al.: Whose vote should count more: optimal integration of labels from labelers
    of unknown expertise, In NIPS, 2009.
    㔐瘶罏
    㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝׾䱿㹀׃姻鍑✮庠
    ꨇ僒䏝׾䱿㹀
    姻鍑
    20/58

    View full-size slide

  21. 㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝׾罋䣁
    ! 㔐瘶罏ה㉏겗ךػًٓ٦ة׾㼪Ⰵ
    o 㔐瘶罏ך腉⸂
    o ㉏겗ך知⽃ׁ
    ! 㔐瘶罏٥㉏겗ךػًٓ٦ة׾欽ְג姻瘶然桦׾邌植׃
    姻鍑הぐػًٓ٦ة׾䱿㹀
    腉⸂ָPS知⽃ָׁ׌ה姻瘶然桦
    腉⸂٥知⽃ָׁ㣐ְֹקו姻瘶然桦ָח鵚בֻ
    姻鍑
    㔐瘶
    腉⸂ 知⽃ׁ
    Pr [
    yij =
    ti] =
    1
    1 + exp (
    ✓j⌫i)
    ✓j
    ⌫i
    腉⸂٥ꨇ僒䏝ח㛇בֻ姻瘶然桦׾邌植
    21/58

    View full-size slide

  22. 㔐瘶罏ך然⥋䏝׾罋䣁
    ! 㔐瘶罏ח然⥋䏝׮耀ֻ
    ! ׋׌׃然⥋䏝׾姻׃ֻ瘶ִ׷⥂鏾כזְ
    o 铎瘶זךחչ荈⥋ָ֮׷պה瘶ִ׷➂ְָ׷
    荈⥋麓ⶱ
    o 姻瘶זךחչ荈⥋ָזְպה瘶ִ׷➂ְָ׷
    荈⥋麓㼭

    ⱖ溪ח둷ָⱖ׏גְתַׅ

    㔐瘶ח荈⥋ָ֮׶תַׅ
    כְ
    כְ
    ְְִ
    ְְִ
    40ZBNB :#BCB :4BLVSBJ BOE) ,BTIJNB"DDVSBUFJOUFHSBUJPOPGDSPXETPVSDFEMBCFMT
    VTJOHXPSLFSTTFMGSFQPSUFEDPOGJEFODFTDPSFT *O *+$"*
    㔐瘶罏ח然⥋䏝׮耀ְג姻鍑✮庠ח欽ְ׷
    22/58

    View full-size slide

  23. 㔐瘶罏ך然⥋䏝׾罋䣁
    ! 然⥋䏝ך⫘ぢ׾然桦ד邌植
    ! 然⥋䏝ך欰䧭ٌرٕ׾㼪Ⰵ׃
    㔐瘶欰䧭ٌرٕה穈׫さ׻ׇג姻鍑׾✮庠ׅ׷
    ⇡(00)
    j
    ⇡(01)
    j
    ⇡(10)
    j
    ⇡(11)
    j
    姻鍑ָ/0
    姻鍑ָ:&4
    㔐瘶ָ/0 㔐瘶ָ:&4
    Pr [cij
    | ti = 0, yij = 1] =

    ⇡(01)
    j
    ⌘cij

    1 ⇡(01)
    j
    ⌘(1 cij )
    㔐瘶
    姻鍑
    然⥋䏝
    չ荈⥋ָ֮׷պה
    瘶ִ׷然桦
    然⥋䏝ך欰䧭麓玎׮ٌرٕ⻉
    23/58

    View full-size slide

  24. ♧㼎嫰鯰ך㔐瘶窟さ
    ! 醱侧➂ח♧㼎嫰鯰׾遤׻ׇ姻鍑ؚٓٝؗٝ׾✮庠ׅ׷
    o ⢽ֶ׫װ־׾➂孡갫חؚٓٝؗٝ׃׋ְ
    > > >
    ♧㼎嫰鯰
    㔐瘶窟さ穠卓ؚٓٝؗٝ
    9$IFOFUBM1BJSXJTFSBOLJOHBHHSFHBUJPOJOBDSPXETPVSDFETFUUJOH *O84%.
    > > >
    ♧㼎嫰鯰ך穠卓ַ׵姻鍑ؚٓٝؗٝ׾✮庠ׅ׷
    24/58

    View full-size slide

  25. ♧㼎嫰鯰ך㔐瘶窟さ
    ! ،؎ذي׀החչ䓼ׁպָ֮׷הׅ׷
    o ،؎ذيך䓼ׁ
    ! 㔐瘶罏ך腉⸂׾然桦ד邌植
    o 䓼ְ،؎ذيח姻׃ֻ䫎牰ׅ׷然桦
    ! 如䒭ך䫎牰ٌرٕח㛇בֹծ،؎ذيך䓼ׁ׾✮庠ׅ׷
    ti
    ׁ׿ָ،؎ذيח䫎牰ׅ׷然桦 ،؎ذيָ״׶䓼ְ然桦
    姻׃ֻח䫎牰ׅ׷然桦 铎׏גח䫎牰ׅ׷然桦
    Pr [i j k] = ✓j
    eti
    eti + etk
    + (1 ✓j)
    etk
    eti + etk
    ✓j
    ،؎ذيך䓼ׁ٥㔐瘶罏腉⸂ח㛇בֻ䫎牰ٌرٕ
    25/58

    View full-size slide

  26. 侧⦼㔐瘶ך窟さ
    ! 醱侧ך侧⦼㔐瘶ַ׵姻鍑׾✮庠ׅ׷
    ! 䎂㖱׾《׷ה姻鍑הך铎䊴ָ㣐ֹֻז׷㜥さ׮֮׷
    ! ✳⦼㔐瘶װ♧㼎嫰鯰ה殯ז׶㔐瘶⦪酡ָ搀ꣲח㶷㖈
    43 215 215 860 860
    860 860 860 215 215
    姻鍑כ
    ㉏겗⢽չך秈侧ךֲ׍剑㣐ך㣼侧כְֻאַպ
    $-JOFUBM$SPXETPVSDJOHDPOUSPMNPWJOHCFZPOENVMUJQMFDIPJDF *O6"*
    㔐瘶⦪酡ָ搀ꣲח֮׷侧⦼㉏겗ך姻鍑׾✮庠ׅ׷
    26/58

    View full-size slide

  27. 侧⦼㔐瘶ך窟さ
    ! ꟦麩ִ倯ך⨉׶׾⚥螟俱椚䏄麓玎ד邌植
    ! 珏겲ך㔐瘶遤⹛׾׉׸׊׸ٌرٕ⻉
    ΍ 姻瘶ׅ׷
    Ύ 傀⳿ך铎瘶׾ׅ׷
    Ώ 劢⳿ך铎瘶׾ׅ׷
    860 43 …
    ⚥螟俱椚䏄麓玎
    չ➂ָ㢳ְذ٦ـٕח➂ָ꧊ת׶װְׅ⫘ぢպ׾邌植
    ⚥螟俱椚䏄麓玎ח㛇בֻ㔐瘶遤⹛ٌرٕ׾㼪Ⰵ
    27/58

    View full-size slide

  28. Ύ湫⴨㉏さׇ

    View full-size slide

  29. 湫⴨㉏さׇ
    ! ➭➂ך瘶ִ׾⢪׏ג鍑ַׇ׷ֿהד姻׃ְ瘶ִ׾䒷ֹ⳿ׅ
    o ⢽ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷俑剅吤姻
    4PZMFOUכծ'JOE'JY7FSJGZך媮ꥡד吤姻׾㹋倵
    ⎼ 'JOE㉏겗ָ֮׷皘䨽ך嗚⳿
    ⎼ 'JY吤姻
    ⎼ 7FSJGZ
    吤姻铎׶ך嗚⳿
    ➭➂ך瘶ִ׾⢪׏ג鍑ַׇ׷
    Figure 2. Crowdproof is a human-augmented proofreader.
    J. Bigham et al.: Soylent: A word processor with a crowd inside, In UIST, 2010.
    穠卓 穠卓 剑穄穠卓
    29/58

    View full-size slide

  30. 荈䊹鎍姻
    ! 4UFQ⡦׮鋅ׇ׆ח㔐瘶ׇׁ׷
    ! 4UFQ➭ך㔐瘶罏ך窟さ鍑׾䲿爙׃ծⱄ䏝㔐瘶ׇׁ׷
    N. B. Shah and D. Zhou: No oops, you won’t do it again: Mechanisms for self-correction in
    crowdsourcing, In ICML, 2016.
    ➭➂ך瘶ִ׾䲿爙׃荈䊹鎍姻׾⤛ׅ
    չ؟ٝؿٓٝءأ؝ךⱖ溪׾鼅׿דֻ׌ְׁպ
    չ֮ז׋ך㔐瘶ָ꟦麩׏גְ׷ה䙼ֲ㜥さכ⥜姻׃גֻ׌ְׁպ
    ֮ז׋ך㔐瘶
    ➭ך➂ך㔐瘶
    30/58

    View full-size slide

  31. ⥜姻ה鐰⣣ך鷵如㹋遤
    ! 5VS,POUSPM䧭卓暟ך⥜姻ה鐰⣣׾粸׶鵤ׅ،ٕ؞ٔؤي
    o ⥜姻٥鐰⣣ך銲♶銲׾
    ㅷ颵׾朐䡾הׅ׷鿇ⴓ錁庠وٕ؝ؿ寸㹀麓玎ד寸㹀
    o ⼧ⴓח⥜姻דֹ׋הֿ׹ד穄✪
    䧭卓暟 ⥜姻ָ
    䗳銲
    ⥜姻׾
    㹋倵
    鐰⣣ָ
    䗳銲
    ⥜姻⵸䖓ך
    嫰鯰
    穄✪
    YES
    NO
    NO
    YES
    P. Dai et al.: Decision-theoretic control of crowd-sourced workflows, In AAAI, 2010.
    ،ٕ؞ٔؤيⵖ䖴ח״׶⥜姻ה鐰⣣׾粸׶鵤ׅ
    31/58
    ⥜姻⵸䖓ךו׍׵ַ׾䱰欽

    View full-size slide

  32. 鐰⣣罏腉⸂׾罋䣁窫㼎鐰⣣
    ! 鐰⣣ָ姻׃ְהכꣲ׵זְ
    ! 鐰⣣罏٥⡲䧭罏ך腉⸂׾䱿㹀׃׋♳ד䧭卓暟ךㅷ颵׾✮庠
    ?
    䧭卓暟
    鐰⣣罏 ㅷ颵
    鐰⣣罏ػًٓ٦ة
    ˖غ؎،أ %
    ˖⥋걾䚍 %
    ⡲䧭罏ػًٓ٦ة
    ˖腉⸂ (
    ˖⥋걾䚍 (
    ⡲䧭罏
    鐰⣣罏腉⸂׾罋䣁׃䧭卓暟ךㅷ颵׾✮庠ׅ׷
    Y. Baba and H. Kashima: Statistical quality estimation for general crowdsourcing tasks, In KDD,
    2013.
    32/58

    View full-size slide

  33. 鐰⣣罏腉⸂׾罋䣁窫㼎鐰⣣
    ! 䧭卓暟ה鐰⣣ך欰䧭ٌرٕ׾㼪Ⰵ׃
    鐰⣣穠卓ַ׵䧭卓暟ךㅷ颵׾✮庠ׅ׷
    o 4UFQ⡲䧭罏ָㅷ颵,,(
    ׾׮א䧭卓暟׾欰䧭
    o 4UFQ鐰⣣罏ָ鐰⣣,,(,%
    ׾欰䧭ׅ׷
    䧭卓暟ה鐰⣣ך欰䧭麓玎׾ٌرٕ⻉
    qt,a
    ⇠ N qt,a
    | µa, 1
    a
    yt,a,r
    ⇠ N yt,a,r
    | qt,a + ⌘r,  1
    r
    ⡲䧭罏ך腉⸂
    鐰⣣罏ךغ؎،أ
    ㅷ颵
    ㅷ颵
    33/58

    View full-size slide

  34. 鐰⣣罏腉⸂׾罋䣁♧㼎嫰鯰
    ! 鐰⣣ָ♧㼎嫰鯰ד遤׻׸׷㜥さח
    鐰⣣罏ך腉⸂׾䱿㹀׃䧭卓暟ךㅷ颵׾✮庠
    ♧㼎嫰鯰ך穠卓ַ׵䧭卓暟ךㅷ颵׾✮庠ׅ׷
    ?
    "ךㅷ颵
    ⡲䧭罏
    T. Sunahase, Y. Baba and H. Kashima, Pairwise HITS: Quality estimation from pairwise
    comparisons in creator-evaluator crowdsourcing process, In AAAI, 2017
    䧭卓暟
    "
    䧭卓暟
    #
    "
    "
    #
    #
    #
    "



    ?
    #ךㅷ颵
    34/58

    View full-size slide

  35. 鐰⣣罏腉⸂׾罋䣁♧㼎嫰鯰
    ! 1BJSXJTF)*54
    )*54،ٕ؞ٔؤي׾䘔欽׃䧭卓暟ךㅷ颵׾✮庠
    ! ⟃♴ך䩛竲ֹ׾粸׶鵤ׅ
    o 4UFQ鐰⣣罏腉⸂0
    ׾㔿㹀׃ㅷ颵1
    ׾刿倜
    o 4UFQㅷ颵1
    ׾㔿㹀׃鐰⣣罏腉⸂0
    ׾刿倜
    qj qj0
    =
    X
    i2Vj j0
    ri
    X
    i2Vj0 j
    ri
    ח䫎牰׃׋➂
    ri =
    |{(j j0) 2 Vi
    | qj > qj0
    }|
    |Vi
    |
    姻׃ְ鐰⣣
    ךⶴさ
    鐰⣣罏腉⸂ה䧭卓暟ㅷ颵׾❛✼ח刿倜
    35/58

    View full-size slide

  36. Ώ㼔Ꟍ㹺涪鋅

    View full-size slide

  37. 㼔Ꟍ㹺涪鋅
    ! 㼔Ꟍ䚍ָ䗳銲ז㉏겗דכ
    鍑ֽ׷➂׾鋅אֽזְה姻׃ְ瘶ִָ䖤׵׸זְ
    o ⢽⼔洽חꟼׅ׷㉏겗
    ! 知⽃ז倯岀姻鍑傀濼㉏겗דך姻瘶桦ד㼔Ꟍ㹺׾鋅אֽ׷
    ! ذأز׾欽䠐׃Ⰻ㆞ח「ׇֽׁ׷ךכ撕꧟זךד
    ➭ך䩛ַָ׶׾ⵃ欽׃׋ְ
    鍑ֽ׷➂׾꧊㔚ך⚥ַ׵鋅אֽ׷
    2過⚥慬䏝׾然钠ׅ׷䗳銲䚍ָ剑׮넝ְ⼔讒ㅷכו׸ַ
    "
    ،أؾٔٝ #
    ؿٗإىس
    $
    ذؔؿ؍ٔٝ %
    ؎ٝسًةءٝ
    㼔Ꟍ䚍ָ䗳銲ז㉏겗⢽
    37/58

    View full-size slide

  38. 㾩䚍ח״׷㼔Ꟍ㹺涪鋅
    ! 㼔Ꟍ䚍ָ㔐瘶罏ך㾩䚍㷕娖ծ㷕鿇瘝ח
    ⣛㶷ׅ׷ֿהכ״ֻ֮׷
    ! 㔐瘶罏ך㾩䚍׾ⵃ欽׃ג㼔Ꟍ㹺׾涪鋅
    o 㔐瘶罏ך㾩䚍كؙزָٕ♷ִ׵׸׷
    o 㔐瘶罏腉⸂׾㾩䚍كؙزٕך简䕎ㄤד邌植׃
    ぐ㾩䚍ח㼎ׅ׷ꅾ׫׾䱿㹀
    㼔Ꟍ㹺חⰟ鸐ׅ׷㾩䚍׾䩛ַָ׶הׅ׷
    aj = (aj1, aj2, . . . , ajn)
    ⼔㷕鿇⳿魦 穗幥㷕鿇⳿魦
    ✓j
    ⇠ N w>aj, 2
    )-JFUBM5IF8JTEPNPGNJOPSJUZEJTDPWFSJOHBOEUBSHFUJOHUIFSJHIUHSPVQPGXPSLFSTGPS
    DSPXETPVSDJOH *O888
    腉⸂
    㾩䚍ךꅾ׫
    38/58

    View full-size slide

  39. 嗚稊ؙؒٔח״׷㼔Ꟍ㹺涪鋅
    ! 嗚稊鸬⹛㘗䎢デַ׵㉏겗פ钷㼪׃
    㼔Ꟍ㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ׾涪鋅
    Figure 3: Example ad to attract users
    time a user clicks on the a
    we record a conversion ev
    the advertising system. T
    the system to optimize th
    mizing the number of con
    contribution yield, instead
    the number of clicks.
    Although optimizing fo
    Figure 1: Screenshot of the Quizz system.
    healthline
    1*QFJSPUJT BOE&(BCSJMPWJDI2VJ[[UBSHFUFEDSPXETPVSDJOHXJUIBCJMMJPO QPUFOUJBM
    VTFST
    *O888
    㼔Ꟍ㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ׾䩛ַָ׶הׅ׷
    姻瘶桦׾؝ٝغ٦آّٝ桦
    ה׃ג⢪欽
    39/58

    View full-size slide

  40. 姻鍑劢濼ך㉏겗׾⢪׏׋㼔Ꟍ㹺涪鋅
    ! 姻鍑׾䌢ח欽䠐דֹ׷הכꣲ׵זְ
    ! 㢳侧寸鍑ד➿欽׃׋ְָ
    㼔Ꟍ㹺ָ㼰侧崢הז׷㉏겗דכ㢳侧寸כ㣟侁ׅ׷
    C
    A A E D E
    A D A C
    A B B C A A D B E
    A D E A C E D
    A A E E D E D C D E
    A E D E
    A D D B C D
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    A
    B B
    B
    B
    B
    B B
    B B
    B
    B
    B
    B B
    B
    B B
    B
    B
    B
    B
    B
    B
    C
    C
    C
    C
    C
    C
    C
    C
    C
    C
    C
    C
    C
    C
    C C
    C
    C
    C
    C
    C
    C
    D
    D
    D
    D
    D
    D
    D
    D
    D
    D
    D
    D D
    D
    E E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    E
    Question
    Worker
    {
    Experts
    Question
    Question
    A
    D
    A
    B
    C
    C
    D
    MV
    ꬊ㼔Ꟍ㹺



    ㉏겗
    㼔Ꟍ㹺 䌢ח姻瘶 ٓٝتيח㔐瘶
    ˟"։&ך䫛 Ⰻגך㉏겗ד"ָ姻瘶
    1
    2
    3
    4
    铎瘶ָ涪欰
    J. Li, Y. Baba and H. Kashima: Hyper Questions: Unsupervised Targeting of a Few Experts in
    Crowdsourcing, In CIKM, 2017.
    姻鍑劢濼ך㉏겗ח㼎ׅ׷㔐瘶ַ׵㼔Ꟍ㹺׾鋅אֽ׷
    㢳侧寸ָ㣟侁ׅ׷噰畭ז⢽
    40/58

    View full-size slide

  41. 姻鍑劢濼ך㉏겗׾⢪׏׋㼔Ꟍ㹺涪鋅
    ! 㼔Ꟍ㹺ず㡦ך㔐瘶כ醱侧㉏ד♧荜׃װְָׅ
    ꬊ㼔Ꟍ㹺ず㡦ך㔐瘶כ♧荜׃חְֻ
    ! 醱侧㉏׾תה׭ג♧אך㉏겗ה׫ז׃׋ծ
    )ZQFSRVFTUJPOח㼎ׅ׷㔐瘶ד㢳侧寸׾《׷
    ㉏겗
    1
    2
    3
    4
    (1, 2, 3)
    (1, 2, 4)
    (1, 3, 4)
    (2, 3, 4)
    B
    B
    B
    )ZQFS
    RVFTUJPO
    ח㢌䳔
    A
    A A
    A
    A
    A
    A
    A
    A
    A
    A
    B B
    B
    B
    C
    D
    D
    E E
    ਪᕚ
    㙢㖽㘞
    {1, 2, 3} AAA
    AAA
    {1, 2, 4}
    AAA
    AAA
    {1, 3, 4}
    {2, 3, 4}
    AAA
    AAA
    AAA
    AAA
    ABD
    ABA
    ADA
    BDA
    EBC
    EBA
    ECA
    BCA
    EBB
    EBD
    EBD
    BBD
    ൴ਪᕚ
    㙢㖽㘞
    ൴ਪᕚ
    㗸㗮Ϋഈʹ
    ม౬
    1
    ε਺Ӕ
    2
    A
    A A
    A
    A
    A
    A
    A
    A
    A
    A
    B B
    B
    B
    C
    D
    D
    E E
    ਪᕚ
    㙢㖽㘞
    {1, 2, 3} AAA
    AAA
    {1, 2, 4}
    AAA
    AAA
    {1, 3, 4}
    {2, 3, 4}
    AAA
    AAA
    AAA
    AAA
    ABD
    ABA
    ADA
    BDA
    EBC
    EBA
    ECA
    BCA
    EBB
    EBD
    EBD
    BBD
    ൴ਪᕚ
    㙢㖽㘞
    ൴ਪᕚ
    㗸㗮Ϋഈʹ
    ม౬
    1
    ε਺Ӕ
    2
    㼔Ꟍ㹺ず㡦ך㔐瘶ָ♧荜ׅ׷ֿה׾ⵃ欽׃׋㢳侧寸
    㼔Ꟍ㹺ך㔐瘶כ♧荜׃
    㢳侧崢הז׷
    ꬊ㼔Ꟍ㹺ך㔐瘶
    כל׵אֻ
    ˘ ˘
    41/58

    View full-size slide

  42. 䱿讂ח״׷㼔Ꟍ㹺涪鋅
    ! %"31"/FUXPSL$IBMMFOHF
    o Ⰻ碛皘䨽ח饔ְ괏菹׾鏣縧
    o 剑ⴱחⰋגך괏菹׾鋅אֽ׋ث٦يח颣ꆃ
    o ⮚⹧ث٦ي
    չ괏菹׾鋅אֽ׉ֲז➂պ
    ׾〡؝ىד涪鋅
    +5BOHFUBM3FGMFDUJOHPOUIF%"31"SFECBMMPPODIBMMFOHF *O $PNNVOJDBUJPOTPGUIF"$.

    contributed articles
    pla
    use
    pu
    nin
    ins
    et
    ing
    dis
    on
    Ma
    use
    cro
    wa
    cia
    on
    ba
    the
    tea
    in
    pro
    firs
    Figure 1. Locations in the DARPA Red Balloon Challenge.
    Figure 2. Example recursive incentive-structure process for the MIT team.
    ➭➂ַ׵ך䱿讂ח㛇בְג㼔Ꟍ㹺׾鋅אֽ׷
    42/58

    View full-size slide

  43. 䱿讂ח״׷㼔Ꟍ㹺涪鋅
    ! 鄃䱿讂罏ָ괏菹׾涪鋅׃׋㜥さծ
    䱿讂罏ח׮㜠ꂹָⴓꂁׁ׸׷״ֲח鏣鎘
    䱿讂
    䱿讂
    䱿讂
    괏菹ך
    㜠ꂹ
    䱿讂ך
    㜠ꂹ
    䱿讂ך
    㜠ꂹ
    䱿讂ך
    㜠ꂹ
    괏菹ך
    㜠ꂹ
    剣渣ז䱿讂ָ䖤׵׸׷״ֲח㜠ꂹ׾鏣鎘
    43/58





    괏菹׾涪鋅
    괏菹׾涪鋅

    View full-size slide

  44. 䱿讂ח״׷㼔Ꟍ㹺涪鋅
    ! ⟰噟ⰻדךةأؙⶴ䔲穗騟׾ⴓ匿
    o ةأؙⶴ䔲荈ⴓד鍑ֽזְةأؙ׾➭罏ח⣛걾
    ! אךⶴ䔲ػة٦ٝ׾穈׫さ׻ׇגٌرٕ⻉
    o 荈魦הך㼔Ꟍ䚍ך䊴ח㛇בֻⶴ䔲
    ⎼ 㼔Ꟍ䚍ָ鵚ַ׵׆黅ַ׵׆ך➂ח
    ⣛걾ׅ׷⫘ぢָ֮׷
    o 植㖈ךةأؙꆀח㛇בֻⶴ䔲
    o ٓٝتيⶴ䔲
    ! 㼔Ꟍ䚍כةأؙ⚥ך⽃铂ה麓⿠ך㸣✪㾶娖ַ׵䱿㹀
    ➂꟦ָ㼔Ꟍ㹺׾䱱ׅ麓玎׾ٌرٕ⻉ׅ׷
    H. Sun et al.: Analyzing expert behaviors in collaborative networks, In KDD, 2014.
    customers. A task is posted
    e network from an expert to
    When an expert cannot solve
    ., where to transfer a task) is
    y affect the completion time
    empt to deduce the cognitive
    model the decision making of
    s where a routing decision is
    patterns.
    interesting phenomenon that
    ask to someone whose knowl-
    or too different from his own.
    xpertise difference based rout-
    e formalize multiple routing
    nt both rational and random
    a generative model to com-
    of tasks, our model not only
    ences very well, but also accu-
    n time. Under three different
    significantly outperforms al-
    han 75% accuracy gain. In
    In service businesses, a service provider of
    expert network where service agents coll
    problems reported by customers. Bugzilla
    ing system where software developers joint
    ed bugs in projects. In a classic collaborati
    receiving a task, an expert first tries to so
    the expert will route the task to another
    is completed until it reaches an expert w
    solution.
    (
    $
    %
    &
    '
    )
    *
    +
    ƚϭ
    ƚϮ
    Figure 1: A Sample Collaborativ
    Figure 1 shows a sample collaborative n
    44/58

    View full-size slide

  45. ΐ⼿锃佄䴂

    View full-size slide

  46. ⼿锃佄䴂
    ! 㔭ꨇ٥醱꧟ז㉏겗ך鍑寸חכ➂꟦ず㡦ך⼿锃ָ♶〳妀
    o ㉏겗׾ⴓⶴ׃׫׿זד鍑ֻ
    o 葺ְ鍑׾׫׿זד鋅אֽ׷
    ! 鎉铂ח״׷陽锷כأ؛٦ٕ׃זְ׋׭
    ءأذوذ؍حؙז⼿锃佄䴂ָ実׭׵׸׷
    ꧊㔚ח״׷㉏겗鍑寸׾佄䴂ׅ׷
    46/58

    View full-size slide

  47. ⼿锃㉏겗ⴓⶴ
    ! 5VSLPNBUJD㉏겗ⴓⶴ٥鍑寸׾꧊㔚ד遤ֲ➬穈׫
    o 4UFQ㉏겗׾ⴓⶴׅ׷
    ⎼ ⼧ⴓ⽃秪ז鿇ⴓ㉏겗חז׷תדⴓⶴ׾粸׶鵤ׅ
    o 4UFQぐ鿇ⴓ㉏겗׾➂꟦ָ鍑ֻ
    o 4UFQ鿇ⴓ㉏겗ך鍑׾꧊秈׃剑穄涸ז鍑׾⳿⸂
    A. Kulkarni et al.: Collaboratively crowdsourcing workflows with Turkomatic, In CSCW, 2012.
    ㉏겗ⴓⶴ׾꧊㔚ד遤ְ㣐鋉垷ז㉏겗׾鍑寸ׅ׷
    ㉏겗⢽չջ㎳׮倯⤑ռכ姻׃ְַպָذ٦وך㼭锷俑㛁瘗
    鿇ⴓ㉏겗չ㎳כ״ֻזְպהְֲ媮衅׾㛁瘗
    鿇ⴓ㉏겗չ䗳銲ז㎳ָ֮׷պהְֲ媮衅׾㛁瘗
    ˘
    47/58

    View full-size slide

  48. ⼿锃剑黝⻉
    ! 鍑ָ侧⦼ד邌ׁ׸׷㉏겗חֶֽ׷⼿锃剑黝⻉
    o ٓٝتيח㔐瘶罏׾鼅ע
    o 㔐瘶罏כ♧㹀眔㔲ⰻך㥨׫ך㜥䨽ח鍑׾⹛ַׅ
    o 眔㔲׾杞׭זָ׵鍑ך刿倜׾粸׶鵤ׅ
    ! 圫ղז؛٦أדך 勲䚍ָ
    椚锷涸٥㹋꿀涸ח爙ׁ׸גְ׷
    N. Garg et al.: Collaborative optimization for collective decision-making in continuous spaces,
    In WWW, 2017.
    然桦涸⺟ꂁ꣬♴岀ח⦺׏ג꧊㔚ד鍑׾䱱稊ׅ׷
    ㉏겗⢽✮皾ך寸㹀
    猰㷕䮶莆顤
    爡⠓⥂ꥺ顤
    植㖈ך鍑
    鍑׾⹛ַׅ
    48/58

    View full-size slide

  49. ،؎ر،ך ꧊
    ! *OOPDFOUJWF
    暴㹀ך㉏겗ך鍑،؎ر،׾⹫꧊ׅ׷فٓحزؿؓ٦ي
    o ㉏겗⢽
    չؿح稆ػؐت٦׾瑞孡⚥ח䭁侔ׇׁ׆ח娎熊ֹ
    ثُ٦ـחⰅ׸׷倯岀כպ
    o ،؎ر،ָ䱰欽ׁ׸׷ה颣ꆃָ佄䩪׻׸׷
    ꧊㔚ַ׵،؎ر،׾⹫꧊ׅ׷فٓحزؿؓ٦ي
    *OOPDFOUJWF
    49/58

    View full-size slide

  50. ،؎ر،ך䱱稊
    ! ꧊㔚ד،؎ر،ך鼅䫙٥❛⿷׾粸׶鵤ׅ
    鼋⠗涸،ٕ؞ٔؤيח⦺׏ג،؎ر،׾䱱稊ׅ׷
    ⢽㶨⣘ぢֽך喱㶨ךرؠ؎ٝ
    L. Yu and J. V. Nickerson: Cooks or cobblers? crowd creativity through combination, In CHI, 2011.
    4UFQ痥⚅➿ך⦐⡤׾ぐ荈ָ欰䧭
    4UFQ䫎牰ד⮚葺⦐⡤
    ׾鼅䫙׃如⚅➿ך⦐⡤
    ׾❛⿷ח״׶欰䧭
    4UFQ鼅䫙٥❛⿷׾粸׶鵤ׅ
    50/58

    View full-size slide

  51. ،؎ر،겲⡂䏝ך〳鋔⻉
    ! ،؎ر،겲⡂䏝ך〳鋔⻉
    ⡂׋،؎ر،׾鵚ֻחꂁ縧ׅ׷ֿהדⰋ⡤⫷䪾䳢׾佄䴂
    ! ،؎ر،겲⡂䏝׮꧊㔚ָⴻ倖
    o 湱㼎涸ז겲⡂䏝׾㼥י׷
    չ"ה#ךו׍׵ך倯ָ
    $ח⡂גְ׷պ
    P. Siangliulue et al.: Toward collaborative ideation at scale: Leveraging ideas from others to
    generate more creative and diverse ideas, In CSCW, 2015.
    ،؎ر،׾鋅ׇさֲֿהד✼ְך涪䟝׾⤛ׅ
    " #
    $
    51/58

    View full-size slide

  52. ،؎ر،겲⡂䏝ך〳鋔⻉
    ⢽娕ך嶊꣇㡦פך钰欰傈ًحإ٦آ
    52/58

    View full-size slide

  53. ،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉
    ! 겲⡂䚍׌ֽדכזֻ⮚⸋׮䲿爙ׅ׷ֿהד
    ،؎ر،ך鼅䫙׾佄䴂
    ! 珏겲ך㉏さׇח״׶겲⡂䏝٥⮚⸋׾꧊㔚דⴻ倖
    o ،؎ر،ل،ך겲⡂䏝鐰⣣
    o ،؎ر،ل،ך⮚⸋鐰⣣
    J. Li, Y. Baba, and H. Kashima: Simultaneous clustering and ranking from pairwise
    comparisons, In IJCAI, 2018 (to appear).
    ،؎ر،겲⡂䏝٥⮚⸋׾꧊㔚דⴻ倖׃〳鋔⻉חⵃ欽
    ㉏겗⢽չ㹀劍ذأزדךؕٝصؚٝ׾꣇ּחכպ
    "չؕٝصؚٝכ⽯鷌㷕պ
    #չ㉏겗ךꂁ⴨׾㢌ִ׷պ
    "չ娄ֹ㔐׏ג湊鋔պ
    #չ欰䖝׀הח㉏겗׾㢌ִ׷պ
    "ה#כ⡂גְ׷
    "ה#כ⡂גְזְ
    "ך倯ָ葺ְ
    #ך倯ָ葺ְ
    53/58

    View full-size slide

  54. ،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉
    ! 겲⡂䏝٥⮚⸋ך鐰⣣穠卓׾ꟼ鸬בֽ⿽倯ך䱿㹀׾⸬桦⻉
    o չ⡂גְ׷،؎ر،ず㡦כずֻׄ׵ְ葺ְպ
    o չ葺ְ،؎ر،ה䝤ְ،؎ر،כ⡂גְזְպ
    ! ،؎ر،ך悵㖈暴䗙 ∈ ℝ5׾➜׃겲⡂䏝ה葺ׁ׾ꟼ鸬בֽ
    ! 겲⡂䏝٥⮚⸋ך鐰⣣穠卓ַ׵ 0 0
    ה׾䱿㹀׃
    〳鋔⻉חⵃ欽
    悵㖈暴䗙ך㼪Ⰵד겲⡂䏝٥⮚⸋ך䱿㹀׾⸬桦⻉
    ⌧i =
    w
    >
    xi
    ،؎ر،Jך葺ׁ
    ،؎ر،JהKך
    겲⡂䏝
    54/58
    ij = exp
    k
    xi xj
    k2
    AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/
    AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/
    AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/
    AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/

    View full-size slide

  55. תה׭
    ➂䊨濼腉ח״׷㉏겗鍑寸׾
    ➂꟦ָ佄䴂
    ➂꟦ח״׷㉏겗鍑寸׾
    ➂䊨濼腉ָ佄䴂
    㼔Ꟍ㹺涪鋅 ⼿锃佄䴂
    ⚛⴨㉏さׇ 湫⴨㉏さׇ
    知⽃٥⽃秪ז㉏겗 㔭ꨇ٥醱꧟ז㉏겗
    ➂꟦ך꧊㔚ה➂䊨濼腉ח״׷㉏겗鍑寸ך䩛岀׾稱➜
    56/58

    View full-size slide

  56. ➙䖓ך铬겗
    ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝךؚٓٝسثٍٖٝآ
    o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷灇瑔Ꟛ涪
    ⎼ 僥歗չٖٗٝخؓךؔ؎ٕպ
    稆➂㣗㮑ָ俑柃锃叨ה㼔Ꟍ㹺הך陽锷ח״׶
    䜁㶨ך氺孡ך屚洽岀׾涪鋅
    o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷鄩ⴻ㆞鄩ⴻ
    o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״׷佟瘻寸㹀
    넝䏝ד醱꧟ז㹋爡⠓㉏겗פך黝欽
    57/58

    View full-size slide

  57. 㹑⠗
    劤傈ך鑧겗ך㢳ֻכֿך剅硂דؕغ٦ׁ׸גְתׅ
    58/58

    View full-size slide