Learning Instructor Intervention from MOOC Forums: Early Results and Issues

5234858ca6ead56b191eef00389e5189?s=47 cmkumar87
June 27, 2015

Learning Instructor Intervention from MOOC Forums: Early Results and Issues

I presented this work at International Conference on Education Data Mining, 2015 at Madrid, Spain. #WING-NUS #EDM

Abstract:
With large student enrollment, MOOC instructors face the unique challenge in deciding when to intervene in forum discussions with their limited bandwidth. We study this problem of instructor intervention.
Using a large sample of forum data culled from 61 courses, we design a binary classifier to predict whether an instructor should intervene in a discussion thread or not. By incorporating novel information about a forum’s type into the classification process, we improve significantly over the previous state-of-the-art. We show how difficult this decision problem is in the real world by validating against indicative human judgment, and empirically show the problem’s sensitivity to instructors’ intervention preferences. We conclude this paper with our take on the future research issues in intervention.

5234858ca6ead56b191eef00389e5189?s=128

cmkumar87

June 27, 2015
Tweet

Transcript

  1. 1.

    Muthu Kumar Chandrasekaran, Min-Yen Kan, Bernard C.Y. Tan, Kiruthika Ragupathi

    muthu.chandra@comp.nus.edu.sg Education Data Mining 2015, Madrid, Spain Learning instructor intervention from MOOC forums: Early results and issues
  2. 2.

    MOOCs have scaled classes in quantity Does their quality also

    scale? 27 June 2015 2 In particular, is MOOC intra- class communication effective?
  3. 3.

    Learning by discussion Student-student talk Instructor intervenes in student discussion

    Collaborative learning Learning via teachable moments, scaffolding Education Data Mining 2015, Madrid, Spain 27 June 2015 3
  4. 4.

    Scaling instructor intervention? • Pedagogical reasons to intervene exist –

    But how much and when to intervene? • Instructors cannot reply or even read every post on a MOOC forum • In this work, we propose a system to identify threads that needs instructor’s attention! Education Data Mining 2015, Madrid, Spain 27 June 2015 4
  5. 6.

    Related work: Moderating Discussion Forums • Backstrom et al., 2013

    – Distillation of social media message threads to enable moderation • Artzi et al, 2012 – Predicting responses and re-tweets • Wang et al, 2012 – Thread Solvedness using discourse cues Education Data Mining 2015, Madrid, Spain 27 June 2015 6
  6. 7.

    Related work: MOOC Discussion Forums 27 June 2015 Education Data

    Mining 2015, Madrid, Spain 7 • Stump et al., 2013 – Multidimensional categorisation of forum posts • Ezen-Can and Boyer – Dialog act based categories • Yang et al, 2014 – Question recommendation for peer learners on MOOC forums
  7. 8.
  8. 10.

    Intervention prediction cast as binary classification To classify if a

    given thread is intervened or not. • + ve instances: All threads replied to at least once by the instructor, TA or a community TA. • - ve instances: All threads never answered by instructor TA or a community TA. Education Data Mining 2015, Madrid, Spain 27 June 2015 10
  9. 11.

    Freely Annotated Data! MOOC Research Summit 2015 24 June 2015

    11 Listing of forum threads on a Coursera MOOC
  10. 13.

    Corpus Forum type All Intervened # threads #posts # threads

    #posts Total 7,408 51,770 2,932 11,561 Education Data Mining 2015, Madrid, Spain 27 June 2015 13 D14 Forum type All Intervened # threads #posts # threads #posts Total 26,643 205,835 7,740 31,779 From 61 courses D61 From 14 courses Details
  11. 14.

    Classifier • Logit classifier with bag of words + other

    features • Plus a simple technique to address class imbalance Education Data Mining 2015, Madrid, Spain 27 June 2015 14
  12. 15.

    Features that correlate with interventions Ratio of intervened to non

    intervened threads over D14 across 4 forum types Forum type encodes intervention priority as perceived by the instructor. MOOC Research Summit 2015 24 June 2015 15 • Discourse cues such as agreements, affirmations, appreciations to original post • Length of thread • # of posts, comments • # sentences, words •Threads with deep discussions indicated by ratio of comments to posts Details
  13. 17.

    Performance varies widely across courses Course Intervention Ratio F1 Individual

    (20% test set) F1 D14 (full course is test set) ml-005 0.45 64.96 56.56 rprog-003 0.32 49.62 48.70 calc1-003 0.60 51.29 68.91 smac-001 0.17 25.00 33.26 compilers-004 0.02 14.28 4.91 maththink-004 0.49 63.56 63.29 medicalneuro-002 0.76 75.36 81.94 musicproduction-006 0.01 0.00 1.03 gametheory2-001 0.19 28.57 30.16 Average 0.36 41.59 45.54 Weighted Macro Avg 0.40 49.04 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 17
  14. 18.

    Forum type and other features improve significantly over unigrams #

    Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 Education Data Mining 2015, Madrid, Spain 27 June 2015 18
  15. 19.

    Forum type and other features improve significantly over unigrams #

    Features Precision Recall F1 1 Uni 41.98 61.39 45.58 2 1+forum type 41.36 69.13 48.01 3 2+lexical course references 41.09 66.57 47.22 4 3+affirmations 41.20 68.94 47.68 5 4+thread_properties 42.99 70.54 48.86 6 5+# of sentences 43.08 69.88 49.77 7 6+non-lexical course references 42.37 74.11 50.56 8 Ablating course references 45.96 79.12 54.79 Education Data Mining 2015, Madrid, Spain 27 June 2015 19
  16. 20.

    Does scaling help? Corpus P R F1 D14 45.96 79.12

    54.79* D61 42.80 76.29 50.96* *Uses the best performing feature set from the previous experiment, that is, all except courseref Education Data Mining 2015, Madrid, Spain 27 June 2015 20
  17. 22.

    Variation in # of threads The # of threads and

    their intervention ratios in forums over 14s courses Diversity across different courses in volumes of threads as well as interventions Education Data Mining 2015, Madrid, Spain 27 June 2015 22
  18. 23.

    Is intervention subjective? Further, indicated by weak human annotator agreement

    among instructors (k=0.53) Education Data Mining 2015, Madrid, Spain 27 June 2015 23
  19. 24.

    Simple baselines work better Course F_1 Individual courses (20% test

    set) F_1 @100%R F_1 on D14 (full course is test set) F_1 @100%R ml-005 64.96 63.79 72.35 61.83 rprog-003 49.62 47.39 48.55 49.31 calc1-003 51.29 74.83 70.63 75.33 smac-001 25.00 34.67 34.15 29.28 compilers-004 14.28 3.28 4.82 4.75 maththink-004 63.56 63.08 61.11 65.49 medicalneuro-002 75.36 88.66 78.06 85.67 musicproduction-006 0.00 4.35 1.09 1.72 gametheory2-001 28.57 45.16 27.12 30.56 Average 41.59 46.43 45.18 47.09 Weighted Macro Avg 49.04 51.51 54.79 53.22 Education Data Mining 2015, Madrid, Spain 27 June 2015 24
  20. 25.

    Intervention framework roadmap Real-time Re-intervention Role-based Thread Ranking Mitigates intervention

    subjectivity Makes intervention decision at post-level Optimises recommendations for instructor / TA Education Data Mining 2015, Madrid, Spain 27 June 2015 25 Via online-learning framework Details
  21. 27.

    • Large variance in quantum of intervention leads us to

    a ranking framework. • Scaling the MOOC corpus by combining forum data from various courses is non- trivial. • Non-reproducibility of results - a key issue stalling progress in MOOC research. Conclusions Education Data Mining 2015, Madrid, Spain 27 June 2015 27
  22. 28.

    Email : consortium@groups.nus.edu.sg Website: wing.comp.nus.edu.sg/downloads/moocdata The MOOC Data Consortium: Enabling

    reproducible large-scale research Muthu Kumar Chandrasekaran, Min-Yen Kan, Kiruthika Ragupathi, Bernard Tan Enabling replicable MOOC forum research among Coursera partners Coursera ’s Statement of Support “As a platform for delivering world-class education and advancing the frontiers of online pedagogy, Coursera encourages the use of its platform to facilitate novel research across a broad range of disciplines, while concurrently protecting the privacy of learners. We support the described research focusing on forum activity and the proposal that this research span courses from across our partner institutions.” Education Data Mining 2015, Madrid, Spain 27 June 2015 28 Thank you! Questions?
  23. 30.

    Forum type All Intervened # threads #posts # threads #posts

    Homework 3,868 31,255 1,385 6,120 Lecture 2,392 13,185 1,008 3,514 Errata 326 1,045 134 206 Exam 822 6,285 405 1,721 Total 7,408 51,770 2,932 11,561 D14 Corpus Data from 14 MOOCs ( D14) from diverse subject areas with different numbers of threads and interventions. Feature Study done using this corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 30 Back
  24. 31.

    Limitations • Variation in # of threads, interventions • Intervention

    decision may be subjective • Simple baselines outperform learned models • Previous results are not replicable Education Data Mining 2015, Madrid, Spain 27 June 2015 31
  25. 32.

    Intervention framework: thread ranking • Thread priority – Errata vs.

    Lecture vs. Exam – Due tomorrow vs. no due date – Rank threads in order of urgency Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 32 Back
  26. 33.

    Intervention framework: Reintervention • Topical diversity within a thread •

    Topic shifts in long discussions • How about new posts to previously intervened threads? Real-time Re- intervention Role-based Thread Ranking Thread Level Post Level Reintervention Education Data Mining 2015, Madrid, Spain 27 June 2015 33 Back
  27. 34.

    Intervention framework: Role based MOOC Instructor team has – Instructor(s)

    – Teaching Assistants (TA) – Community Teaching Assistants (CTA) • Different bandwidths, priorities, authorities • Could they intervening differently too? • To customise recommendation to the role of the staff Real-time Re- intervention Role- based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 34 Back
  28. 35.

    Intervention framework: thread ranking • Courses in real-time produce posts

    over time • Intervention priority may change over time • Rank posts by priority at any time t of a MOOC Real-time Re- intervention Role-based Thread Ranking Education Data Mining 2015, Madrid, Spain 27 June 2015 35 Back
  29. 36.

    Experiments Set up details Experiments run on two different set

    up 1. Each individual course in D14 is treated as a corpus – Training and test set are form the same course. 20% of a course data is randomly sample as training 2. Treat D14 as a single corpus – Leave-one –course-out cross validation Education Data Mining 2015, Madrid, Spain 27 June 2015 36 Back
  30. 37.

    Chatruvedi et al, 2014 Models where , r is intervention

    HV1 is the Post Category OBS is observed post HV1 HV1 OBS OBS HV1 HV1 OBS OBS r HV1 HV1 OBS OBS HV1 HV1 OBS OBS Linear chain Markov model (LCMM) Global chain model (GCM) Education Data Mining 2015, Madrid, Spain 27 June 2015 37 Chaturvedi et al, 2014 • Infers 4 post categories that maximizes likelihood of instructor intervention • Post categories act as hidden variable in their sequence models that predict instructor intervention Back
  31. 39.

    Classifier • Logistic regression classifier • We use class Weights

    W, to counter balance inherent class imbalance in this data – Biases prediction towards majority class instances • Class weights are learnt from the training set by greedily optimising for maximum F1 score Education Data Mining 2015, Madrid, Spain 27 June 2015 39 Back
  32. 41.

    L1 Regularised Logistic Regression Ng, Andrew Y. "Feature selection, L

    1 vs. L 2 regularization, and rotational invariance." Proceedings of the twenty-first international conference on Machine learning. ACM, 2004. Education Data Mining 2015, Madrid, Spain 27 June 2015 41 Back
  33. 42.

    Novel Features: Forum Type Ratio of intervened to non intervened

    threads over D14 across 4 forum types Encodes intervention priority as perceived by the instructor Education Data Mining 2015, Madrid, Spain 27 June 2015 42 back
  34. 44.

    Novel Features: Non-lexical References to course materials URLs Timestamps from

    videos Education Data Mining 2015, Madrid, Spain 27 June 2015 44 back
  35. 45.

    Other Features • Unigrams (~98,000 unique terms) • Thread properties

    – Length: as #posts, comment, total; as # sentences – Structure as average #comments / post • Affirmation of the original post by fellow students Education Data Mining 2015, Madrid, Spain 27 June 2015 45 back
  36. 46.

    FEATURES: QUOTING EXTERNAL SOURCES Terms ‘wikipedia’ and ‘wiki’ occur in

    148 and 79 docs respectively in the D14 corpus. Education Data Mining 2015, Madrid, Spain 27 June 2015 46 back