Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CrowDEA: Multi-view Idea Prioritization with Crowds

CrowDEA: Multi-view Idea Prioritization with Crowds

Yukino Baba
PRO

October 12, 2020
Tweet

More Decks by Yukino Baba

Other Decks in Research

Transcript

  1. HCOMP 2020
    CrowDEA: Multi-view Idea
    Prioritization with Crowds
    Yukino Baba (University of Tsukuba)
    Jiyi Li (University of Yamanashi )
    Hisashi Kashima (Kyoto University)

    View Slide

  2. We can collect a large number of ideas from crowds
    What’s the best emblem
    design for the next Olympic
    Games?
    How can we prevent
    cheating in exams?
    Impose severe penalties …
    Have proctors watch students …
    Prepare di erent examination …

    View Slide

  3. How can we prioritize the collected ideas?
    • Criteria for ideas are usually multi-faced
    • There rarely exists the single winner in terms of all criteria
    Modern
    Traditional
    Low-cost
    E ective
    Impose severe penalties …
    Prepare di erent examination …

    View Slide

  4. CrowDEA finds promising ideas in diverse viewpoints
    Target ideas
    INPUT:
    Pairwise comparison
    ≻ ≻

    View Slide

  5. CrowDEA finds promising ideas in diverse viewpoints
    Target ideas OUTPUT: Priority map
    INPUT:
    Pairwise comparison
    ≻ ≻

    View Slide

  6. CrowDEA finds promising ideas in diverse viewpoints
    Target ideas OUTPUT: Priority map
    Priority
    INPUT:
    Pairwise comparison
    ≻ ≻

    View Slide

  7. CrowDEA finds promising ideas in diverse viewpoints
    Target ideas OUTPUT: Priority map
    Viewpoints
    INPUT:
    Pairwise comparison
    ≻ ≻

    View Slide

  8. CrowDEA finds promising ideas in diverse viewpoints
    Target ideas OUTPUT: Priority map
    Size
    Color
    INPUT:
    Pairwise comparison
    ≻ ≻

    View Slide

  9. CrowDEA finds promising ideas in diverse viewpoints
    Target ideas OUTPUT: Priority map
    Size
    Color
    Promising ideas
    INPUT:
    Pairwise comparison
    ≻ ≻

    View Slide

  10. CrowDEA estimates embeddings and best viewpoints
    [Constraint 1] from the best viewpoint of idea , it should be most
    valuable among all ideas:
    i
    vi
    ⊤xi
    > vi
    ⊤xj
    xi
    vi Best viewpoint
    Embedding
    Idea parameters
    wk Personal viewpoint
    Evaluator parameter

    View Slide

  11. CrowDEA estimates embeddings and best viewpoints
    [Constraint 2] if evaluator prefers idea over idea :
    k i j
    wk
    ⊤xi
    > wk
    ⊤xj
    [Constraint 1] from the best viewpoint of idea , it should be most
    valuable among all ideas:
    i
    vi
    ⊤xi
    > vi
    ⊤xj
    xi
    vi Best viewpoint
    Embedding
    Idea parameters
    wk Personal viewpoint
    Evaluator parameter

    View Slide

  12. Case1 : Olympic emblem designs
    Top-5 ideas ranked by BT
    Promising ideas detected by CrowDEA
    Modern
    Traditional

    View Slide

  13. Case 2: “How to prevent cheating in exams?”
    Impose severe penalties for cheating, such as cancellation of modules for an
    entire year.
    Prepare two types of examination sheets with di erently ordered items, and
    distribute one to every student such that neighboring students have di erent
    exam sheets.
    Have proctors watch students from the back of an examination room.
    Instead of multiple-choice questions or short answer questions, use essay
    questions to make it di cult to copy the answers of other students.
    Promising ideas detected by CrowDEA

    View Slide

  14. Result: Ranking accuracy
    BT Crowd BT Blade-Chest BPR CrowDEA
    Bike 0.772 0.779 0.757 0.827 0.833
    Cheat 0.768 0.767 0.813 0.789 0.893
    Meeting 0.817 0.815 0.829 0.800 0.877
    Night 0.790 0.790 0.903 0.853 0.917
    Visitor 0.818 0.825 0.868 0.933 0.938
    Character 0.902 0.912 0.866 0.929 0.930
    Olympics 0.926 0.926 0.920 0.940 0.936
    Dataset
    Method
    Average of nDCG@5 among various viewpoints

    View Slide