CrowDEA: Multi-view Idea Prioritization with Crowds

CrowDEA: Multi-view Idea Prioritization with Crowds

Ae9fcaea4c7d6d24a3a4ba97efd1d6da?s=128

Yukino Baba

October 12, 2020
Tweet

Transcript

  1. HCOMP 2020 CrowDEA: Multi-view Idea Prioritization with Crowds Yukino Baba

    (University of Tsukuba) Jiyi Li (University of Yamanashi ) Hisashi Kashima (Kyoto University)
  2. We can collect a large number of ideas from crowds

    What’s the best emblem design for the next Olympic Games? How can we prevent cheating in exams? Impose severe penalties … Have proctors watch students … Prepare di erent examination …
  3. How can we prioritize the collected ideas? • Criteria for

    ideas are usually multi-faced • There rarely exists the single winner in terms of all criteria Modern Traditional Low-cost E ective Impose severe penalties … Prepare di erent examination …
  4. CrowDEA finds promising ideas in diverse viewpoints Target ideas INPUT:

    Pairwise comparison ≻ ≻
  5. CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT:

    Priority map INPUT: Pairwise comparison ≻ ≻
  6. CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT:

    Priority map Priority INPUT: Pairwise comparison ≻ ≻
  7. CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT:

    Priority map Viewpoints INPUT: Pairwise comparison ≻ ≻
  8. CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT:

    Priority map Size Color INPUT: Pairwise comparison ≻ ≻
  9. CrowDEA finds promising ideas in diverse viewpoints Target ideas OUTPUT:

    Priority map Size Color Promising ideas INPUT: Pairwise comparison ≻ ≻
  10. CrowDEA estimates embeddings and best viewpoints [Constraint 1] from the

    best viewpoint of idea , it should be most valuable among all ideas: i vi ⊤xi > vi ⊤xj xi vi Best viewpoint Embedding Idea parameters wk Personal viewpoint Evaluator parameter
  11. CrowDEA estimates embeddings and best viewpoints [Constraint 2] if evaluator

    prefers idea over idea : k i j wk ⊤xi > wk ⊤xj [Constraint 1] from the best viewpoint of idea , it should be most valuable among all ideas: i vi ⊤xi > vi ⊤xj xi vi Best viewpoint Embedding Idea parameters wk Personal viewpoint Evaluator parameter
  12. Case1 : Olympic emblem designs Top-5 ideas ranked by BT

    Promising ideas detected by CrowDEA Modern Traditional
  13. Case 2: “How to prevent cheating in exams?” Impose severe

    penalties for cheating, such as cancellation of modules for an entire year. Prepare two types of examination sheets with di erently ordered items, and distribute one to every student such that neighboring students have di erent exam sheets. Have proctors watch students from the back of an examination room. Instead of multiple-choice questions or short answer questions, use essay questions to make it di cult to copy the answers of other students. Promising ideas detected by CrowDEA
  14. Result: Ranking accuracy BT Crowd BT Blade-Chest BPR CrowDEA Bike

    0.772 0.779 0.757 0.827 0.833 Cheat 0.768 0.767 0.813 0.789 0.893 Meeting 0.817 0.815 0.829 0.800 0.877 Night 0.790 0.790 0.903 0.853 0.917 Visitor 0.818 0.825 0.868 0.933 0.938 Character 0.902 0.912 0.866 0.929 0.930 Olympics 0.926 0.926 0.920 0.940 0.936 Dataset Method Average of nDCG@5 among various viewpoints