HCOMP 2020 CrowDEA: Multi-view Idea Prioritization with Crowds Yukino Baba (University of Tsukuba) Jiyi Li (University of Yamanashi ) Hisashi Kashima (Kyoto University)

We can collect a large number of ideas from crowds What’s the best emblem design for the next Olympic Games? How can we prevent cheating in exams? Impose severe penalties … Have proctors watch students … Prepare di erent examination …

How can we prioritize the collected ideas? • Criteria for ideas are usually multi-faced • There rarely exists the single winner in terms of all criteria Modern Traditional Low-cost E ective Impose severe penalties … Prepare di erent examination …

CrowDEA finds promising ideas in diverse viewpoints Target ideas INPUT: Pairwise comparison ≻ ≻

CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT: Priority map INPUT: Pairwise comparison ≻ ≻

CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT: Priority map Priority INPUT: Pairwise comparison ≻ ≻

CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT: Priority map Viewpoints INPUT: Pairwise comparison ≻ ≻

CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT: Priority map Size Color INPUT: Pairwise comparison ≻ ≻

CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT: Priority map Size Color Promising ideas INPUT: Pairwise comparison ≻ ≻

CrowDEA estimates embeddings and best viewpoints [Constraint 1] from the best viewpoint of idea , it should be most valuable among all ideas: i vi ⊤xi > vi ⊤xj xi vi Best viewpoint Embedding Idea parameters wk Personal viewpoint Evaluator parameter

CrowDEA estimates embeddings and best viewpoints [Constraint 2] if evaluator prefers idea over idea : k i j wk ⊤xi > wk ⊤xj [Constraint 1] from the best viewpoint of idea , it should be most valuable among all ideas: i vi ⊤xi > vi ⊤xj xi vi Best viewpoint Embedding Idea parameters wk Personal viewpoint Evaluator parameter

Case1 : Olympic emblem designs Top-5 ideas ranked by BT Promising ideas detected by CrowDEA Modern Traditional

Case 2: “How to prevent cheating in exams?” Impose severe penalties for cheating, such as cancellation of modules for an entire year. Prepare two types of examination sheets with di erently ordered items, and distribute one to every student such that neighboring students have di erent exam sheets. Have proctors watch students from the back of an examination room. Instead of multiple-choice questions or short answer questions, use essay questions to make it di cult to copy the answers of other students. Promising ideas detected by CrowDEA

Result: Ranking accuracy BT Crowd BT Blade-Chest BPR CrowDEA Bike 0.772 0.779 0.757 0.827 0.833 Cheat 0.768 0.767 0.813 0.789 0.893 Meeting 0.817 0.815 0.829 0.800 0.877 Night 0.790 0.790 0.903 0.853 0.917 Visitor 0.818 0.825 0.868 0.933 0.938 Character 0.902 0.912 0.866 0.929 0.930 Olympics 0.926 0.926 0.920 0.940 0.936 Dataset Method Average of nDCG@5 among various viewpoints