Learning Instructor Intervention from MOOC Forums: Early Results and Issues

I presented this work at International Conference on Education Data Mining, 2015 at Madrid, Spain. #WING-NUS #EDM

Abstract:
With large student enrollment, MOOC instructors face the unique challenge in deciding when to intervene in forum discussions with their limited bandwidth. We study this problem of instructor intervention.
Using a large sample of forum data culled from 61 courses, we design a binary classifier to predict whether an instructor should intervene in a discussion thread or not. By incorporating novel information about a forum’s type into the classification process, we improve significantly over the previous state-of-the-art. We show how difficult this decision problem is in the real world by validating against indicative human judgment, and empirically show the problem’s sensitivity to instructors’ intervention preferences. We conclude this paper with our take on the future research issues in intervention.

Transcript

1. Muthu Kumar Chandrasekaran, Min-Yen Kan, Bernard C.Y. Tan, Kiruthika Ragupathi muthu.chandra@comp.nus.edu.sg Education Data Mining 2015, Madrid, Spain Learning instructor intervention from MOOC forums: Early results and issues
2. MOOCs have scaled classes in quantity Does their quality also scale? 27 June 2015 2 In particular, is MOOC intra- class communication effective?
3. Learning by discussion Student-student talk Instructor intervenes in student discussion Collaborative learning Learning via teachable moments, scaffolding Education Data Mining 2015, Madrid, Spain 27 June 2015 3
4. Scaling instructor intervention? • Pedagogical reasons to intervene exist – But how much and when to intervene? • Instructors cannot reply or even read every post on a MOOC forum • In this work, we propose a system to identify threads that needs instructor’s attention! Education Data Mining 2015, Madrid, Spain 27 June 2015 4
5. Related Work Education Data Mining 2015, Madrid, Spain 27 June 2015 5
6. Related work: Moderating Discussion Forums • Backstrom et al., 2013 – Distillation of social media message threads to enable moderation • Artzi et al, 2012 – Predicting responses and re-tweets • Wang et al, 2012 – Thread Solvedness using discourse cues Education Data Mining 2015, Madrid, Spain 27 June 2015 6
7. Related work: MOOC Discussion Forums 27 June 2015 Education Data Mining 2015, Madrid, Spain 7 • Stump et al., 2013 – Multidimensional categorisation of forum posts • Ezen-Can and Boyer – Dialog act based categories • Yang et al, 2014 – Question recommendation for peer learners on MOOC forums
8. 27 June 2015 Education Data Mining 2015, Madrid, Spain Closest related work for comparison 8 Details
9. Method Education Data Mining 2015, Madrid, Spain 27 June 2015 9
10. Intervention prediction cast as binary classification To classify if a given thread is intervened or not. • + ve instances: All threads replied to at least once by the instructor, TA or a community TA. • - ve instances: All threads never answered by instructor TA or a community TA. Education Data Mining 2015, Madrid, Spain 27 June 2015 10
11. Freely Annotated Data! MOOC Research Summit 2015 24 June 2015 11 Listing of forum threads on a Coursera MOOC
12. Corpus Education Data Mining 2015, Madrid, Spain 27 June 2015 12
13. Corpus Forum type All Intervened # threads #posts # threads #posts Total 7,408 51,770 2,932 11,561 Education Data Mining 2015, Madrid, Spain 27 June 2015 13 D14 Forum type All Intervened # threads #posts # threads #posts Total 26,643 205,835 7,740 31,779 From 61 courses D61 From 14 courses Details
14. Classifier • Logit classifier with bag of words + other features • Plus a simple technique to address class imbalance Education Data Mining 2015, Madrid, Spain 27 June 2015 14
15. Features that correlate with interventions Ratio of intervened to non intervened threads over D14 across 4 forum types Forum type encodes intervention priority as perceived by the instructor. MOOC Research Summit 2015 24 June 2015 15 • Discourse cues such as agreements, affirmations, appreciations to original post • Length of thread • # of posts, comments • # sentences, words •Threads with deep discussions indicated by ratio of comments to posts Details
16. Experiments and Results Education Data Mining 2015, Madrid, Spain 27 June 2015 16
17. Performance varies widely across courses Course Intervention Ratio F1 Individual (20% test set) F1 D14 (full course is test set) ml-005 0.45 64.96 56.56 rprog-003 0.32 49.62 48.70 calc1-003 0.60 51.29 68.91 smac-001 0.17 25.00 33.26 compilers-004 0.02 14.28 4.91 maththink-004 0.49 63.56 63.29 medicalneuro-002 0.76 75.36 81.94 musicproduction-006 0.01 0.00 1.03 gametheory2-001 0.19 28.57 30.16 Average 0.36 41.59 45.54 Weighted Macro Avg 0.40 49.04 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 17
18. Forum type and other features improve significantly over unigrams # Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 18
19. Forum type and other features improve significantly over unigrams # Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 8 Ablating course references 45.96 79.12 54.79 Education Data Mining 2015, Madrid, Spain 27 June 2015 19
20. Does scaling help? Corpus P R F1 D14 45.96 79.12 54.79* D61 42.80 76.29 50.96* *Uses the best performing feature set from the previous experiment, that is, all except courseref Education Data Mining 2015, Madrid, Spain 27 June 2015 20
21. Limitations and Future Work Education Data Mining 2015, Madrid, Spain 27 June 2015 21
22. Variation in # of threads The # of threads and their intervention ratios in forums over 14s courses Diversity across different courses in volumes of threads as well as interventions Education Data Mining 2015, Madrid, Spain 27 June 2015 22
23. Is intervention subjective? Further, indicated by weak human annotator agreement among instructors (k=0.53) Education Data Mining 2015, Madrid, Spain 27 June 2015 23
24. Simple baselines work better Course F_1 Individual courses (20% test set) F_1 @100%R F_1 on D14 (full course is test set) F_1 @100%R ml-005 64.96 63.79 72.35 61.83 rprog-003 49.62 47.39 48.55 49.31 calc1-003 51.29 74.83 70.63 75.33 smac-001 25.00 34.67 34.15 29.28 compilers-004 14.28 3.28 4.82 4.75 maththink-004 63.56 63.08 61.11 65.49 medicalneuro-002 75.36 88.66 78.06 85.67 musicproduction-006 0.00 4.35 1.09 1.72 gametheory2-001 28.57 45.16 27.12 30.56 Average 41.59 46.43 45.18 47.09 Weighted Macro Avg 49.04 51.51 54.79 53.22 Education Data Mining 2015, Madrid, Spain 27 June 2015 24
25. Intervention framework roadmap Real-time Re-intervention Role-based Thread Ranking Mitigates intervention subjectivity Makes intervention decision at post-level Optimises recommendations for instructor / TA Education Data Mining 2015, Madrid, Spain 27 June 2015 25 Via online-learning framework Details
26. Conclusion Education Data Mining 2015, Madrid, Spain 27 June 2015 26
27. • Large variance in quantum of intervention leads us to a ranking framework. • Scaling the MOOC corpus by combining forum data from various courses is non- trivial. • Non-reproducibility of results - a key issue stalling progress in MOOC research. Conclusions Education Data Mining 2015, Madrid, Spain 27 June 2015 27
28. Email : consortium@groups.nus.edu.sg Website: wing.comp.nus.edu.sg/downloads/moocdata The MOOC Data Consortium: Enabling reproducible large-scale research Muthu Kumar Chandrasekaran, Min-Yen Kan, Kiruthika Ragupathi, Bernard Tan Enabling replicable MOOC forum research among Coursera partners Coursera ’s Statement of Support “As a platform for delivering world-class education and advancing the frontiers of online pedagogy, Coursera encourages the use of its platform to facilitate novel research across a broad range of disciplines, while concurrently protecting the privacy of learners. We support the described research focusing on forum activity and the proposal that this research span courses from across our partner institutions.” Education Data Mining 2015, Madrid, Spain 27 June 2015 28 Thank you! Questions?
29. BACKUP Education Data Mining 2015, Madrid, Spain 27 June 2015 29
30. Forum type All Intervened # threads #posts # threads #posts Homework 3,868 31,255 1,385 6,120 Lecture 2,392 13,185 1,008 3,514 Errata 326 1,045 134 206 Exam 822 6,285 405 1,721 Total 7,408 51,770 2,932 11,561 D14 Corpus Data from 14 MOOCs ( D14) from diverse subject areas with different numbers of threads and interventions. Feature Study done using this corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 30 Back
31. Limitations • Variation in # of threads, interventions • Intervention decision may be subjective • Simple baselines outperform learned models • Previous results are not replicable Education Data Mining 2015, Madrid, Spain 27 June 2015 31
32. Intervention framework: thread ranking • Thread priority – Errata vs. Lecture vs. Exam – Due tomorrow vs. no due date – Rank threads in order of urgency Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 32 Back
33. Intervention framework: Reintervention • Topical diversity within a thread • Topic shifts in long discussions • How about new posts to previously intervened threads? Real-time Re- intervention Role-based Thread Ranking Thread Level Post Level Reintervention Education Data Mining 2015, Madrid, Spain 27 June 2015 33 Back
34. Intervention framework: Role based MOOC Instructor team has – Instructor(s) – Teaching Assistants (TA) – Community Teaching Assistants (CTA) • Different bandwidths, priorities, authorities • Could they intervening differently too? • To customise recommendation to the role of the staff Real-time Re- intervention Role- based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 34 Back
35. Intervention framework: thread ranking • Courses in real-time produce posts over time • Intervention priority may change over time • Rank posts by priority at any time t of a MOOC Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 35 Back
36. Experiments Set up details Experiments run on two different set up 1. Each individual course in D14 is treated as a corpus – Training and test set are form the same course. 20% of a course data is randomly sample as training 2. Treat D14 as a single corpus – Leave-one –course-out cross validation Education Data Mining 2015, Madrid, Spain 27 June 2015 36 Back
37. Chatruvedi et al, 2014 Models where , r is intervention HV1 is the Post Category OBS is observed post HV1 HV1 OBS OBS HV1 HV1 OBS OBS r HV1 HV1 OBS OBS HV1 HV1 OBS OBS Linear chain Markov model (LCMM) Global chain model (GCM) Education Data Mining 2015, Madrid, Spain 27 June 2015 37 Chaturvedi et al, 2014 • Infers 4 post categories that maximizes likelihood of instructor intervention • Post categories act as hidden variable in their sequence models that predict instructor intervention Back
38. Chaturvedi et al, 2014 Results Education Data Mining 2015, Madrid, Spain 27 June 2015 38 Back
39. Classifier • Logistic regression classifier • We use class Weights W, to counter balance inherent class imbalance in this data – Biases prediction towards majority class instances • Class weights are learnt from the training set by greedily optimising for maximum F1 score Education Data Mining 2015, Madrid, Spain 27 June 2015 39 Back
40. L1 Regularised Logistic Regression Source: http://cs.nyu.edu/~rostami/presentations/L1_vs_L2.pdf Education Data Mining 2015, Madrid, Spain 27 June 2015 40 Back
41. L1 Regularised Logistic Regression Ng, Andrew Y. "Feature selection, L 1 vs. L 2 regularization, and rotational invariance." Proceedings of the twenty-first international conference on Machine learning. ACM, 2004. Education Data Mining 2015, Madrid, Spain 27 June 2015 41 Back
42. Novel Features: Forum Type Ratio of intervened to non intervened threads over D14 across 4 forum types Encodes intervention priority as perceived by the instructor Education Data Mining 2015, Madrid, Spain 27 June 2015 42 back
43. Novel Features: Lexical References to course materials Education Data Mining 2015, Madrid, Spain 27 June 2015 43 back
44. Novel Features: Non-lexical References to course materials URLs Timestamps from videos Education Data Mining 2015, Madrid, Spain 27 June 2015 44 back
45. Other Features • Unigrams (~98,000 unique terms) • Thread properties – Length: as #posts, comment, total; as # sentences – Structure as average #comments / post • Affirmation of the original post by fellow students Education Data Mining 2015, Madrid, Spain 27 June 2015 45 back
46. FEATURES: QUOTING EXTERNAL SOURCES Terms ‘wikipedia’ and ‘wiki’ occur in 148 and 79 docs respectively in the D14 corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 46 back
47. QUOTING INSTRUCTORS / SLIDES Education Data Mining 2015, Madrid, Spain 27 June 2015 47 back
48. AFFIRMATIONS: COMMON IN BUG REPORTS Education Data Mining 2015, Madrid, Spain 27 June 2015 48 back