A Learner Sourcing Technique which helps instructors create practice questions quickly and @ scale

Invited Talk at Association for Computing Machinery, Special Interest Group in Information Retrieval, TOKYO (ACM-SIGIR TOKYO '20) in Apr 25, 2020.
(* It won't be held 'cause of COVID-19 as planned.)

Transcript

1. A Learner sourcing technique that helps instructors create practice questions quickly and at scale Takahiro Oda ACM SIGIR TOKYO JAPAN ʼ20 Workshop on Data Driven Education(DDE) April 25th, 2020 (Sat.)
2. About Me 3 2 Takahiro Oda •Born and raised in Osaka • 2nd Undergraduate Student at School of Economics, Keio University (Tokyo, Japan) •Research Internship at atama plus.inc •Research Internship at LearnLab at Carnegie Mellon University (Pittsburgh, PA) •My Personal Website is HERE !
3. Introduction & Problem Setting
4. Introduction 3 4 Open-ended assignments are frequently given
5. Cons of Open-ended Assignments 5 Effort in grading & providing feedback
6. Cons of Open-ended Assignments 6 Effort in grading & providing feedback Delayed feedback No repeated practice Lack of scaffolding
7. Upgrade Help Instructors 2 7 7 UpGrade : help instructors quickly create multiple-choice questions at scale using existing data Current state Preferred state Repeated practice Immediate feedback Scaffolding Auto-grading Scalable
8. Learner-sourcing 3 8 Sources of input: 1. Video watching traces (Kim et al., 14) 2. Video annotations (Kim et al., 14; Liu et al., 18; Weir et al., 15) 3. Explanations (Williams et al., 16)
9. Learner-sourcing 3 9 Sources of input: 1. Video watching traces (Kim et al., 14) 2. Video annotations (Kim et al., 14; Liu et al., 18; Weir et al., 15) 3. Explanations (Williams et al., 16) New source of input Student written assignment; instructor feedback
10. OUTLINE
11. OUTLINE Learning benefit UpGrade: a learnersourcing technique Quality control 11
12. UpGrade: a learnersourcing technique
13. Heuristic Evaluation 5 13
14. Workflow of UpGrade 5 14 Step ①: Log
15. Workflow of UpGrade 5 15 Reason Description of the scenario (with image) Rule violation Step ②: Segment
16. Workflow of UpGrade 5 16 Reason Description of the scenario (with image) Rule violation Step ③: Create Question Question Answer Explanation
17. Step3: Create Question 5 17 17
18. Step3: Create Question 5 18 18
19. Step3: Create Question 5 19 19 Description of the scenario (with image) Question
20. Step3: Create Question 5 20 20 Description of the scenario (with image) Question Rule Violation Answer
21. Step3: Create Question 5 21 Description of the scenario (with image) Question Rule Violation Answer Explanation Reason
22. Step3: Create Question 5 22 Schema 2: Question-Answer-Explanation (Revision) Schema 3: Answer-Feedback Schema 4: Question-Answer Schema 1: Question-Answer-Explanation
23. UpGrade: a learnersourcing technique 5 23 Hundreds of questions were auto-created after the instructor specified the schema
24. UpGrade: a learnersourcing technique 5 24 Hundreds of questions were auto-created after the instructor specified the schema Rubric Item # of Questions Created Identify Heuristic Rule Violation 478 Problem Severity 478 Problem Remedy 478 Problem Remedy Tradeoff 91
25. Learning benefit
26. Study Design Autumn offering of a user experience methods course @CMU (survey, interview, heuristic evaluation, think alouds, etc.) 5 26 28 Students
27. Class Group A Group B Open-ended Assignment UpGrade MCQ Assignment Quiz Survey Design Heuristics Evaluation 1 week 1 week Quiz UpGrade MCQ Assignment Quiz Open-ended Assignment Quiz Study Design 5 27
28. Specifics about UpGrade Assignment 5 28
29. ✔ Scaffolding 5 29
30. ✔ Repeated Practice 5 30
31. ✔ Immediate Feedback 5 31
32. We do not see a difference in studentsʼ learning 5 32 UpGrade reduces 30% learning time, without sacrificing learning outcomes !
33. We do not see a difference in studentsʼ learning 5 33 UpGrade reduces 30% learning time without sacrificing learning outcomes Assignment Completion Time Quiz Score
34. Instructor Feedback Liked this approach “Students grades are computed automatically saving substantial efforts of grading and offering feedback” 5 34
35. Instructor Feedback Liked this approach “Students grades are computed automatically saving substantial efforts of grading and offering feedback Concerns of open-ended assignment “Students are asked to design a survey when they didn’t actually know how to design a survey. Many assignments turned in were in very bad shape and I had to tell the students to go back and redo the assignment.” 5 35
36. Instructor Feedback 5 36 Future work could get students practice with UpGrade first and then go off to generate new contents Concerns of open-ended assignment “Students are asked to design a survey when they didn’t actually know how to design a survey. Many assignments turned in were in very bad shape and I had to tell the students to go back and redo the assignment.”
37. Quality control
38. example) 3 students working on a set of questions Q1 Q2 Q3 Total Student 1 1 1 0 ... 20 Student 2 1 0 0 ... 18 Student 3 0 0 1 ... 5 Q3 fails high performing students but favors lower performing students ↓ Thus can be unreliable Cronbachʼs Alpha to Identify Unreliable Items 5 38
39. MTurk(amazon Mechanical Turk) Study • 30/478 randomly selected questions, 70 participants • 21 reliable questions (Cronbachʼs alpha = 0.74) Usually 0.7 is good enough for classroom use Cronbachʼs alpha can be used to prune out unreliable questions !!
40. Summary
41. Summary 5 41 UpGrade reduces 30% learning time without sacrificing learning outcomes; no manual grading Multiple-choice questions can exercise evaluation UpGrade: a technique to create effective multiple- choice questions at scale A psychometric approach can be applied to prune out unreliable questions
42. Stakeholder Map for Learning Support Design 5 42 Learner
43. Stakeholder Map for Learning Support Design 5 43 Learner Instructor Peers Machine Support Past Students
44. Stakeholder Map for Learning Support Design 5 44 Learner Instructor Peers Machine Support Past Students Written content and feedback as new input
45. Stakeholder Map for Learning Support Design: Conclusion 5 45 Learner Instructor Peers Machine Support Past Students Written content and feedback as new input Intelligent content creation
46. Thank You for Listening !! * This work was funded in part by NSF grant ACI-1443068.
47. References (Aleven et al., IJAIED’09) : Vincent Aleven, Bruce M Mclaren, Jonathan Sewall, and Kenneth R Koedinger. 2009. A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education 19, 2 (2009), 105–154. (Ambrose et al., ‘10) : Susan A Ambrose, Michael W Bridges, Michele DiPietro, Marsha C Lovett, and Marie K Norman. 2010. How learning works: Seven research-based principles for smart teaching. John Wiley & Sons. (Chaiklin et al., ‘03) : The zone of proximal development in Vygotsky’s analysis of learning and instruction. Vygotsky’s educational theory in cultural context 1 (2003), 39–64. (Cronbuch, 1951) : Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika 16, 3 (1951), 297‒334. (Eriksson et al., 1993) : K Anders Ericsson, Ralf T Krampe, and Clemens Tesch-Römer. 1993. The role of deliberate practice in the acquisition of expert performance. Psychological review 100, 3 (1993), 363. (ETS) : Education Testing Services. Reliability and Comparability of TOEFL iBT Scores. Technical Report. (Harris, 1989) : Deborah Harris. 1989. Comparison of 1-, 2-, and 3-parameter IRT models. Educational Measurement: Issues and Practice 8, 1 (1989), 35–41. 5 47
48. References 5 48 (Hattie et al., RER’07) : John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research 77, 1 (2007), 81–112. (Heffernan et al., IJAIED’14) : Neil T Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments Ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24, 4 (2014), 470–497. (Kim, ‘15) : Juho Kim. 2015. Learnersourcing : improving learning with collective learner activity. Ph.D. Dissertation. Cambridge, MA, USA. (Kim et al., CHI’15) : Juho Kim, Elena L. Glassman, Andrés Monroy-Hernández, and Meredith Ringel Morris. 2015. RIMES: Embedding Interactive Multimedia Exercises in Lecture Videos. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI ’15). ACM, New York, NY, USA, 1535–1544. (Kim et al., UIST’14) : Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen (Daniel) Li, Krzysztof Z. Gajos, and Robert C. Miller. 2014a. Data-driven Interaction Techniques for Improving Navigation of Educational Videos. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST ’14). ACM, New York, NY, USA, 563–572.
49. References 5 49 (Kim et al., CHI’14) : Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, and Krzysztof Z. Gajos. 2014b. Crowdsourcing Step-by-step Information Extraction to Enhance Existing How-to Videos. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 4017–4026. (Kluger et al., 1996) : Avraham N Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological bulletin 119, 2 (1996), 254 (Kulkarni et al., TOCHI’13) : Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and Self Assessment in Massive Online Classes. ACM Trans. Comput.- Hum. Interact. 20, 6, Article 33 (Dec. 2013), 31 pages. (Liu et al., CHI’18) : Ching Liu, Juho Kim, and Hao-Chuan Wang. 2018. ConceptScape: Collaborative Concept Mapping for Video Learning. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA, Article 387, 12 pages. (Paas et al., ‘03) : Fred Paas, Alexander Renkl, and John Sweller. 2003. Cognitive load theory and instructional design: Recent developments. Educational psychologist 38, 1 (2003), 1–4.
50. References 5 50 (Schwartz et al., ‘16) : Daniel L. Schwartz, Jessica M. Tsang, and Kristen P. Blair. 2016. The ABCs of How We Learn. W. W. Norton & Company, New York, NY, USA. (Sweller, ‘06) : John Sweller. 2006. The worked example effect and human cognition. Learning and instruction (2006). (Tavacol et al., IJME’11) : Mohsen Tavakol and Reg Dennick. 2011. Making sense of Cronbachʼs alpha. International journal of medical education 2 (2011), 53. (Weir et al., CSCW’15) : Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ʼ15). ACM, New York, NY, USA, 405‒416. (Williams et al., L@S’16) : Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z. Gajos, Walter S. Lasecki, and Neil Heffernan. 2016. AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (L@S ʼ16). ACM, New York, NY, USA, 379‒388. (Wiison et al., ‘04) : Mark Wilson and Paul De Boeck. 2004. Descriptive and explanatory item response models. In Explanatory item response models. Springer, 43‒74.