Share
Learning Instructor Intervention from MOOC Forums: Early Results and Issues
I presented this work at International Conference on Education Data Mining, 2015 at Madrid, Spain. #WING-NUS #EDM
Abstract:
With large student enrollment, MOOC instructors face the unique challenge in deciding when to intervene in forum discussions with their limited bandwidth. We study this problem of instructor intervention.
Using a large sample of forum data culled from 61 courses, we design a binary classifier to predict whether an instructor should intervene in a discussion thread or not. By incorporating novel information about a forum’s type into the classification process, we improve significantly over the previous state-of-the-art. We show how difficult this decision problem is in the real world by validating against indicative human judgment, and empirically show the problem’s sensitivity to instructors’ intervention preferences. We conclude this paper with our take on the future research issues in intervention.
cmkumar87
Transcript
48 slides
-
Muthu Kumar Chandrasekaran, Min-Yen Kan, Bernard C.Y. Tan, Kiruthika Ragupathi
[email protected] Education Data Mining 2015, Madrid, Spain Learning instructor intervention from MOOC forums: Early results and issues -
MOOCs have scaled classes in quantity Does their quality also
scale? 27 June 2015 2 In particular, is MOOC intra- class communication effective? -
Learning by discussion Student-student talk Instructor intervenes in student discussion
Collaborative learning Learning via teachable moments, scaffolding Education Data Mining 2015, Madrid, Spain 27 June 2015 3 -
Scaling instructor intervention? • Pedagogical reasons to intervene exist –
But how much and when to intervene? • Instructors cannot reply or even read every post on a MOOC forum • In this work, we propose a system to identify threads that needs instructor’s attention! Education Data Mining 2015, Madrid, Spain 27 June 2015 4 -
Related work: Moderating Discussion Forums • Backstrom et al., 2013
– Distillation of social media message threads to enable moderation • Artzi et al, 2012 – Predicting responses and re-tweets • Wang et al, 2012 – Thread Solvedness using discourse cues Education Data Mining 2015, Madrid, Spain 27 June 2015 6 -
Related work: MOOC Discussion Forums 27 June 2015 Education Data
Mining 2015, Madrid, Spain 7 • Stump et al., 2013 – Multidimensional categorisation of forum posts • Ezen-Can and Boyer – Dialog act based categories • Yang et al, 2014 – Question recommendation for peer learners on MOOC forums -
27 June 2015 Education Data Mining 2015, Madrid, Spain Closest
related work for comparison 8 Details -
Intervention prediction cast as binary classification To classify if a
given thread is intervened or not. • + ve instances: All threads replied to at least once by the instructor, TA or a community TA. • - ve instances: All threads never answered by instructor TA or a community TA. Education Data Mining 2015, Madrid, Spain 27 June 2015 10 -
Freely Annotated Data! MOOC Research Summit 2015 24 June 2015
11 Listing of forum threads on a Coursera MOOC -
Corpus Forum type All Intervened # threads #posts # threads
#posts Total 7,408 51,770 2,932 11,561 Education Data Mining 2015, Madrid, Spain 27 June 2015 13 D14 Forum type All Intervened # threads #posts # threads #posts Total 26,643 205,835 7,740 31,779 From 61 courses D61 From 14 courses Details -
Classifier • Logit classifier with bag of words + other
features • Plus a simple technique to address class imbalance Education Data Mining 2015, Madrid, Spain 27 June 2015 14 -
Features that correlate with interventions Ratio of intervened to non
intervened threads over D14 across 4 forum types Forum type encodes intervention priority as perceived by the instructor. MOOC Research Summit 2015 24 June 2015 15 • Discourse cues such as agreements, affirmations, appreciations to original post • Length of thread • # of posts, comments • # sentences, words •Threads with deep discussions indicated by ratio of comments to posts Details -
Performance varies widely across courses Course Intervention Ratio F1 Individual
(20% test set) F1 D14 (full course is test set) ml-005 0.45 64.96 56.56 rprog-003 0.32 49.62 48.70 calc1-003 0.60 51.29 68.91 smac-001 0.17 25.00 33.26 compilers-004 0.02 14.28 4.91 maththink-004 0.49 63.56 63.29 medicalneuro-002 0.76 75.36 81.94 musicproduction-006 0.01 0.00 1.03 gametheory2-001 0.19 28.57 30.16 Average 0.36 41.59 45.54 Weighted Macro Avg 0.40 49.04 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 17 -
Forum type and other features improve significantly over unigrams #
Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 18 -
Forum type and other features improve significantly over unigrams #
Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 8 Ablating course references 45.96 79.12 54.79 Education Data Mining 2015, Madrid, Spain 27 June 2015 19 -
Does scaling help? Corpus P R F1 D14 45.96 79.12
54.79* D61 42.80 76.29 50.96* *Uses the best performing feature set from the previous experiment, that is, all except courseref Education Data Mining 2015, Madrid, Spain 27 June 2015 20 -
Variation in # of threads The # of threads and
their intervention ratios in forums over 14s courses Diversity across different courses in volumes of threads as well as interventions Education Data Mining 2015, Madrid, Spain 27 June 2015 22 -
Is intervention subjective? Further, indicated by weak human annotator agreement
among instructors (k=0.53) Education Data Mining 2015, Madrid, Spain 27 June 2015 23 -
Simple baselines work better Course F_1 Individual courses (20% test
set) F_1 @100%R F_1 on D14 (full course is test set) F_1 @100%R ml-005 64.96 63.79 72.35 61.83 rprog-003 49.62 47.39 48.55 49.31 calc1-003 51.29 74.83 70.63 75.33 smac-001 25.00 34.67 34.15 29.28 compilers-004 14.28 3.28 4.82 4.75 maththink-004 63.56 63.08 61.11 65.49 medicalneuro-002 75.36 88.66 78.06 85.67 musicproduction-006 0.00 4.35 1.09 1.72 gametheory2-001 28.57 45.16 27.12 30.56 Average 41.59 46.43 45.18 47.09 Weighted Macro Avg 49.04 51.51 54.79 53.22 Education Data Mining 2015, Madrid, Spain 27 June 2015 24 -
Intervention framework roadmap Real-time Re-intervention Role-based Thread Ranking Mitigates intervention
subjectivity Makes intervention decision at post-level Optimises recommendations for instructor / TA Education Data Mining 2015, Madrid, Spain 27 June 2015 25 Via online-learning framework Details -
• Large variance in quantum of intervention leads us to
a ranking framework. • Scaling the MOOC corpus by combining forum data from various courses is non- trivial. • Non-reproducibility of results - a key issue stalling progress in MOOC research. Conclusions Education Data Mining 2015, Madrid, Spain 27 June 2015 27 -
Email : [email protected] Website: wing.comp.nus.edu.sg/downloads/moocdata The MOOC Data Consortium: Enabling
reproducible large-scale research Muthu Kumar Chandrasekaran, Min-Yen Kan, Kiruthika Ragupathi, Bernard Tan Enabling replicable MOOC forum research among Coursera partners Coursera ’s Statement of Support “As a platform for delivering world-class education and advancing the frontiers of online pedagogy, Coursera encourages the use of its platform to facilitate novel research across a broad range of disciplines, while concurrently protecting the privacy of learners. We support the described research focusing on forum activity and the proposal that this research span courses from across our partner institutions.” Education Data Mining 2015, Madrid, Spain 27 June 2015 28 Thank you! Questions? -
Forum type All Intervened # threads #posts # threads #posts
Homework 3,868 31,255 1,385 6,120 Lecture 2,392 13,185 1,008 3,514 Errata 326 1,045 134 206 Exam 822 6,285 405 1,721 Total 7,408 51,770 2,932 11,561 D14 Corpus Data from 14 MOOCs ( D14) from diverse subject areas with different numbers of threads and interventions. Feature Study done using this corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 30 Back -
Limitations • Variation in # of threads, interventions • Intervention
decision may be subjective • Simple baselines outperform learned models • Previous results are not replicable Education Data Mining 2015, Madrid, Spain 27 June 2015 31 -
Intervention framework: thread ranking • Thread priority – Errata vs.
Lecture vs. Exam – Due tomorrow vs. no due date – Rank threads in order of urgency Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 32 Back -
Intervention framework: Reintervention • Topical diversity within a thread •
Topic shifts in long discussions • How about new posts to previously intervened threads? Real-time Re- intervention Role-based Thread Ranking Thread Level Post Level Reintervention Education Data Mining 2015, Madrid, Spain 27 June 2015 33 Back -
Intervention framework: Role based MOOC Instructor team has – Instructor(s)
– Teaching Assistants (TA) – Community Teaching Assistants (CTA) • Different bandwidths, priorities, authorities • Could they intervening differently too? • To customise recommendation to the role of the staff Real-time Re- intervention Role- based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 34 Back -
Intervention framework: thread ranking • Courses in real-time produce posts
over time • Intervention priority may change over time • Rank posts by priority at any time t of a MOOC Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 35 Back -
Experiments Set up details Experiments run on two different set
up 1. Each individual course in D14 is treated as a corpus – Training and test set are form the same course. 20% of a course data is randomly sample as training 2. Treat D14 as a single corpus – Leave-one –course-out cross validation Education Data Mining 2015, Madrid, Spain 27 June 2015 36 Back -
Chatruvedi et al, 2014 Models where , r is intervention
HV1 is the Post Category OBS is observed post HV1 HV1 OBS OBS HV1 HV1 OBS OBS r HV1 HV1 OBS OBS HV1 HV1 OBS OBS Linear chain Markov model (LCMM) Global chain model (GCM) Education Data Mining 2015, Madrid, Spain 27 June 2015 37 Chaturvedi et al, 2014 • Infers 4 post categories that maximizes likelihood of instructor intervention • Post categories act as hidden variable in their sequence models that predict instructor intervention Back -
Chaturvedi et al, 2014 Results Education Data Mining 2015, Madrid,
Spain 27 June 2015 38 Back -
Classifier • Logistic regression classifier • We use class Weights
W, to counter balance inherent class imbalance in this data – Biases prediction towards majority class instances • Class weights are learnt from the training set by greedily optimising for maximum F1 score Education Data Mining 2015, Madrid, Spain 27 June 2015 39 Back -
L1 Regularised Logistic Regression Source: http://cs.nyu.edu/~rostami/presentations/L1_vs_L2.pdf Education Data Mining 2015,
Madrid, Spain 27 June 2015 40 Back -
L1 Regularised Logistic Regression Ng, Andrew Y. "Feature selection, L
1 vs. L 2 regularization, and rotational invariance." Proceedings of the twenty-first international conference on Machine learning. ACM, 2004. Education Data Mining 2015, Madrid, Spain 27 June 2015 41 Back -
Novel Features: Forum Type Ratio of intervened to non intervened
threads over D14 across 4 forum types Encodes intervention priority as perceived by the instructor Education Data Mining 2015, Madrid, Spain 27 June 2015 42 back -
Novel Features: Lexical References to course materials Education Data Mining
2015, Madrid, Spain 27 June 2015 43 back -
Novel Features: Non-lexical References to course materials URLs Timestamps from
videos Education Data Mining 2015, Madrid, Spain 27 June 2015 44 back -
Other Features • Unigrams (~98,000 unique terms) • Thread properties
– Length: as #posts, comment, total; as # sentences – Structure as average #comments / post • Affirmation of the original post by fellow students Education Data Mining 2015, Madrid, Spain 27 June 2015 45 back -
FEATURES: QUOTING EXTERNAL SOURCES Terms ‘wikipedia’ and ‘wiki’ occur in
148 and 79 docs respectively in the D14 corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 46 back -
QUOTING INSTRUCTORS / SLIDES Education Data Mining 2015, Madrid, Spain
27 June 2015 47 back -
AFFIRMATIONS: COMMON IN BUG REPORTS Education Data Mining 2015, Madrid,
Spain 27 June 2015 48 back