Muthu Kumar Chandrasekaran, Min-Yen Kan, Bernard C.Y. Tan, Kiruthika Ragupathi muthu.chandra@comp.nus.edu.sg Education Data Mining 2015, Madrid, Spain Learning instructor intervention from MOOC forums: Early results and issues

MOOCs have scaled classes in quantity Does their quality also scale? 27 June 2015 2 In particular, is MOOC intra- class communication effective?

Learning by discussion Student-student talk Instructor intervenes in student discussion Collaborative learning Learning via teachable moments, scaffolding Education Data Mining 2015, Madrid, Spain 27 June 2015 3

Scaling instructor intervention? • Pedagogical reasons to intervene exist – But how much and when to intervene? • Instructors cannot reply or even read every post on a MOOC forum • In this work, we propose a system to identify threads that needs instructor’s attention! Education Data Mining 2015, Madrid, Spain 27 June 2015 4

Related Work Education Data Mining 2015, Madrid, Spain 27 June 2015 5

Related work: Moderating Discussion Forums • Backstrom et al., 2013 – Distillation of social media message threads to enable moderation • Artzi et al, 2012 – Predicting responses and re-tweets • Wang et al, 2012 – Thread Solvedness using discourse cues Education Data Mining 2015, Madrid, Spain 27 June 2015 6

Related work: MOOC Discussion Forums 27 June 2015 Education Data Mining 2015, Madrid, Spain 7 • Stump et al., 2013 – Multidimensional categorisation of forum posts • Ezen-Can and Boyer – Dialog act based categories • Yang et al, 2014 – Question recommendation for peer learners on MOOC forums

27 June 2015 Education Data Mining 2015, Madrid, Spain Closest related work for comparison 8 Details

Method Education Data Mining 2015, Madrid, Spain 27 June 2015 9

Intervention prediction cast as binary classification To classify if a given thread is intervened or not. • + ve instances: All threads replied to at least once by the instructor, TA or a community TA. • - ve instances: All threads never answered by instructor TA or a community TA. Education Data Mining 2015, Madrid, Spain 27 June 2015 10

Freely Annotated Data! MOOC Research Summit 2015 24 June 2015 11 Listing of forum threads on a Coursera MOOC

Corpus Education Data Mining 2015, Madrid, Spain 27 June 2015 12

Corpus Forum type All Intervened # threads #posts # threads #posts Total 7,408 51,770 2,932 11,561 Education Data Mining 2015, Madrid, Spain 27 June 2015 13 D14 Forum type All Intervened # threads #posts # threads #posts Total 26,643 205,835 7,740 31,779 From 61 courses D61 From 14 courses Details

Classifier • Logit classifier with bag of words + other features • Plus a simple technique to address class imbalance Education Data Mining 2015, Madrid, Spain 27 June 2015 14

Features that correlate with interventions Ratio of intervened to non intervened threads over D14 across 4 forum types Forum type encodes intervention priority as perceived by the instructor. MOOC Research Summit 2015 24 June 2015 15 • Discourse cues such as agreements, affirmations, appreciations to original post • Length of thread • # of posts, comments • # sentences, words •Threads with deep discussions indicated by ratio of comments to posts Details

Experiments and Results Education Data Mining 2015, Madrid, Spain 27 June 2015 16

Performance varies widely across courses Course Intervention Ratio F1 Individual (20% test set) F1 D14 (full course is test set) ml-005 0.45 64.96 56.56 rprog-003 0.32 49.62 48.70 calc1-003 0.60 51.29 68.91 smac-001 0.17 25.00 33.26 compilers-004 0.02 14.28 4.91 maththink-004 0.49 63.56 63.29 medicalneuro-002 0.76 75.36 81.94 musicproduction-006 0.01 0.00 1.03 gametheory2-001 0.19 28.57 30.16 Average 0.36 41.59 45.54 Weighted Macro Avg 0.40 49.04 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 17

Forum type and other features improve significantly over unigrams # Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 18

Forum type and other features improve significantly over unigrams # Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 8 Ablating course references 45.96 79.12 54.79 Education Data Mining 2015, Madrid, Spain 27 June 2015 19

Does scaling help? Corpus P R F1 D14 45.96 79.12 54.79* D61 42.80 76.29 50.96* *Uses the best performing feature set from the previous experiment, that is, all except courseref Education Data Mining 2015, Madrid, Spain 27 June 2015 20

Limitations and Future Work Education Data Mining 2015, Madrid, Spain 27 June 2015 21

Variation in # of threads The # of threads and their intervention ratios in forums over 14s courses Diversity across different courses in volumes of threads as well as interventions Education Data Mining 2015, Madrid, Spain 27 June 2015 22

Is intervention subjective? Further, indicated by weak human annotator agreement among instructors (k=0.53) Education Data Mining 2015, Madrid, Spain 27 June 2015 23

Simple baselines work better Course F_1 Individual courses (20% test set) F_1 @100%R F_1 on D14 (full course is test set) F_1 @100%R ml-005 64.96 63.79 72.35 61.83 rprog-003 49.62 47.39 48.55 49.31 calc1-003 51.29 74.83 70.63 75.33 smac-001 25.00 34.67 34.15 29.28 compilers-004 14.28 3.28 4.82 4.75 maththink-004 63.56 63.08 61.11 65.49 medicalneuro-002 75.36 88.66 78.06 85.67 musicproduction-006 0.00 4.35 1.09 1.72 gametheory2-001 28.57 45.16 27.12 30.56 Average 41.59 46.43 45.18 47.09 Weighted Macro Avg 49.04 51.51 54.79 53.22 Education Data Mining 2015, Madrid, Spain 27 June 2015 24

Intervention framework roadmap Real-time Re-intervention Role-based Thread Ranking Mitigates intervention subjectivity Makes intervention decision at post-level Optimises recommendations for instructor / TA Education Data Mining 2015, Madrid, Spain 27 June 2015 25 Via online-learning framework Details

Conclusion Education Data Mining 2015, Madrid, Spain 27 June 2015 26

• Large variance in quantum of intervention leads us to a ranking framework. • Scaling the MOOC corpus by combining forum data from various courses is non- trivial. • Non-reproducibility of results - a key issue stalling progress in MOOC research. Conclusions Education Data Mining 2015, Madrid, Spain 27 June 2015 27

Email : consortium@groups.nus.edu.sg Website: wing.comp.nus.edu.sg/downloads/moocdata The MOOC Data Consortium: Enabling reproducible large-scale research Muthu Kumar Chandrasekaran, Min-Yen Kan, Kiruthika Ragupathi, Bernard Tan Enabling replicable MOOC forum research among Coursera partners Coursera ’s Statement of Support “As a platform for delivering world-class education and advancing the frontiers of online pedagogy, Coursera encourages the use of its platform to facilitate novel research across a broad range of disciplines, while concurrently protecting the privacy of learners. We support the described research focusing on forum activity and the proposal that this research span courses from across our partner institutions.” Education Data Mining 2015, Madrid, Spain 27 June 2015 28 Thank you! Questions?

BACKUP Education Data Mining 2015, Madrid, Spain 27 June 2015 29

Forum type All Intervened # threads #posts # threads #posts Homework 3,868 31,255 1,385 6,120 Lecture 2,392 13,185 1,008 3,514 Errata 326 1,045 134 206 Exam 822 6,285 405 1,721 Total 7,408 51,770 2,932 11,561 D14 Corpus Data from 14 MOOCs ( D14) from diverse subject areas with different numbers of threads and interventions. Feature Study done using this corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 30 Back

Limitations • Variation in # of threads, interventions • Intervention decision may be subjective • Simple baselines outperform learned models • Previous results are not replicable Education Data Mining 2015, Madrid, Spain 27 June 2015 31

Intervention framework: thread ranking • Thread priority – Errata vs. Lecture vs. Exam – Due tomorrow vs. no due date – Rank threads in order of urgency Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 32 Back

Intervention framework: Reintervention • Topical diversity within a thread • Topic shifts in long discussions • How about new posts to previously intervened threads? Real-time Re- intervention Role-based Thread Ranking Thread Level Post Level Reintervention Education Data Mining 2015, Madrid, Spain 27 June 2015 33 Back

Intervention framework: Role based MOOC Instructor team has – Instructor(s) – Teaching Assistants (TA) – Community Teaching Assistants (CTA) • Different bandwidths, priorities, authorities • Could they intervening differently too? • To customise recommendation to the role of the staff Real-time Re- intervention Role- based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 34 Back

Intervention framework: thread ranking • Courses in real-time produce posts over time • Intervention priority may change over time • Rank posts by priority at any time t of a MOOC Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 35 Back

Experiments Set up details Experiments run on two different set up 1. Each individual course in D14 is treated as a corpus – Training and test set are form the same course. 20% of a course data is randomly sample as training 2. Treat D14 as a single corpus – Leave-one –course-out cross validation Education Data Mining 2015, Madrid, Spain 27 June 2015 36 Back

Chatruvedi et al, 2014 Models where , r is intervention HV1 is the Post Category OBS is observed post HV1 HV1 OBS OBS HV1 HV1 OBS OBS r HV1 HV1 OBS OBS HV1 HV1 OBS OBS Linear chain Markov model (LCMM) Global chain model (GCM) Education Data Mining 2015, Madrid, Spain 27 June 2015 37 Chaturvedi et al, 2014 • Infers 4 post categories that maximizes likelihood of instructor intervention • Post categories act as hidden variable in their sequence models that predict instructor intervention Back

Chaturvedi et al, 2014 Results Education Data Mining 2015, Madrid, Spain 27 June 2015 38 Back

Classifier • Logistic regression classifier • We use class Weights W, to counter balance inherent class imbalance in this data – Biases prediction towards majority class instances • Class weights are learnt from the training set by greedily optimising for maximum F1 score Education Data Mining 2015, Madrid, Spain 27 June 2015 39 Back

L1 Regularised Logistic Regression Source: http://cs.nyu.edu/~rostami/presentations/L1_vs_L2.pdf Education Data Mining 2015, Madrid, Spain 27 June 2015 40 Back

L1 Regularised Logistic Regression Ng, Andrew Y. "Feature selection, L 1 vs. L 2 regularization, and rotational invariance." Proceedings of the twenty-first international conference on Machine learning. ACM, 2004. Education Data Mining 2015, Madrid, Spain 27 June 2015 41 Back

Novel Features: Forum Type Ratio of intervened to non intervened threads over D14 across 4 forum types Encodes intervention priority as perceived by the instructor Education Data Mining 2015, Madrid, Spain 27 June 2015 42 back

Novel Features: Lexical References to course materials Education Data Mining 2015, Madrid, Spain 27 June 2015 43 back

Novel Features: Non-lexical References to course materials URLs Timestamps from videos Education Data Mining 2015, Madrid, Spain 27 June 2015 44 back

Other Features • Unigrams (~98,000 unique terms) • Thread properties – Length: as #posts, comment, total; as # sentences – Structure as average #comments / post • Affirmation of the original post by fellow students Education Data Mining 2015, Madrid, Spain 27 June 2015 45 back

FEATURES: QUOTING EXTERNAL SOURCES Terms ‘wikipedia’ and ‘wiki’ occur in 148 and 79 docs respectively in the D14 corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 46 back

QUOTING INSTRUCTORS / SLIDES Education Data Mining 2015, Madrid, Spain 27 June 2015 47 back

AFFIRMATIONS: COMMON IN BUG REPORTS Education Data Mining 2015, Madrid, Spain 27 June 2015 48 back