CrowDEA: Multi-view Idea Prioritization with Crowds

Transcript

1. HCOMP 2020
   CrowDEA: Multi-view Idea
   Prioritization with Crowds
   Yukino Baba (University of Tsukuba)
   Jiyi Li (University of Yamanashi )
   Hisashi Kashima (Kyoto University)
   

   View Slide

2. We can collect a large number of ideas from crowds
   What’s the best emblem
   design for the next Olympic
   Games?
   How can we prevent
   cheating in exams?
   Impose severe penalties …
   Have proctors watch students …
   Prepare di erent examination …
   

   View Slide

3. How can we prioritize the collected ideas?
   • Criteria for ideas are usually multi-faced
   • There rarely exists the single winner in terms of all criteria
   Modern
   Traditional
   Low-cost
   E ective
   Impose severe penalties …
   Prepare di erent examination …
   

   View Slide

4. CrowDEA finds promising ideas in diverse viewpoints
   Target ideas
   INPUT:
   Pairwise comparison
   ≻ ≻
   

   View Slide

5. CrowDEA finds promising ideas in diverse viewpoints
   Target ideas OUTPUT: Priority map
   INPUT:
   Pairwise comparison
   ≻ ≻
   

   View Slide

6. CrowDEA finds promising ideas in diverse viewpoints
   Target ideas OUTPUT: Priority map
   Priority
   INPUT:
   Pairwise comparison
   ≻ ≻
   

   View Slide

7. CrowDEA finds promising ideas in diverse viewpoints
   Target ideas OUTPUT: Priority map
   Viewpoints
   INPUT:
   Pairwise comparison
   ≻ ≻
   

   View Slide

8. CrowDEA finds promising ideas in diverse viewpoints
   Target ideas OUTPUT: Priority map
   Size
   Color
   INPUT:
   Pairwise comparison
   ≻ ≻
   

   View Slide

9. CrowDEA finds promising ideas in diverse viewpoints
   Target ideas OUTPUT: Priority map
   Size
   Color
   Promising ideas
   INPUT:
   Pairwise comparison
   ≻ ≻
   

   View Slide

10. CrowDEA estimates embeddings and best viewpoints
    [Constraint 1] from the best viewpoint of idea , it should be most
    valuable among all ideas:
    i
    vi
    ⊤xi
    > vi
    ⊤xj
    xi
    vi Best viewpoint
    Embedding
    Idea parameters
    wk Personal viewpoint
    Evaluator parameter
    

    View Slide

11. CrowDEA estimates embeddings and best viewpoints
    [Constraint 2] if evaluator prefers idea over idea :
    k i j
    wk
    ⊤xi
    > wk
    ⊤xj
    [Constraint 1] from the best viewpoint of idea , it should be most
    valuable among all ideas:
    i
    vi
    ⊤xi
    > vi
    ⊤xj
    xi
    vi Best viewpoint
    Embedding
    Idea parameters
    wk Personal viewpoint
    Evaluator parameter
    

    View Slide

12. Case1 : Olympic emblem designs
    Top-5 ideas ranked by BT
    Promising ideas detected by CrowDEA
    Modern
    Traditional
    

    View Slide

13. Case 2: “How to prevent cheating in exams?”
    Impose severe penalties for cheating, such as cancellation of modules for an
    entire year.
    Prepare two types of examination sheets with di erently ordered items, and
    distribute one to every student such that neighboring students have di erent
    exam sheets.
    Have proctors watch students from the back of an examination room.
    Instead of multiple-choice questions or short answer questions, use essay
    questions to make it di cult to copy the answers of other students.
    Promising ideas detected by CrowDEA
    

    View Slide

14. Result: Ranking accuracy
    BT Crowd BT Blade-Chest BPR CrowDEA
    Bike 0.772 0.779 0.757 0.827 0.833
    Cheat 0.768 0.767 0.813 0.789 0.893
    Meeting 0.817 0.815 0.829 0.800 0.877
    Night 0.790 0.790 0.903 0.853 0.917
    Visitor 0.818 0.825 0.868 0.933 0.938
    Character 0.902 0.912 0.866 0.929 0.930
    Olympics 0.926 0.926 0.920 0.940 0.936
    Dataset
    Method
    Average of nDCG@5 among various viewpoints
    

    View Slide