Learning Instructor Intervention from MOOC Forums: Early Results and Issues

I presented this work at International Conference on Education Data Mining, 2015 at Madrid, Spain. #WING-NUS #EDM

Abstract:
With large student enrollment, MOOC instructors face the unique challenge in deciding when to intervene in forum discussions with their limited bandwidth. We study this problem of instructor intervention.
Using a large sample of forum data culled from 61 courses, we design a binary classifier to predict whether an instructor should intervene in a discussion thread or not. By incorporating novel information about a forum’s type into the classification process, we improve significantly over the previous state-of-the-art. We show how difficult this decision problem is in the real world by validating against indicative human judgment, and empirically show the problem’s sensitivity to instructors’ intervention preferences. We conclude this paper with our take on the future research issues in intervention.

Transcript

1. Muthu Kumar Chandrasekaran, Min-Yen Kan,
   Bernard C.Y. Tan, Kiruthika Ragupathi
   muthu.chandra@comp.nus.edu.sg
   Education Data Mining 2015, Madrid, Spain
   Learning instructor intervention
   from MOOC forums:
   Early results and issues
   

   View Slide

2. MOOCs have scaled classes in quantity
   Does their quality also
   scale?
   27 June 2015 2
   In particular, is MOOC intra-
   class communication effective?
   

   View Slide

3. Learning by discussion
   Student-student
   talk
   Instructor
   intervenes in
   student discussion
   Collaborative
   learning
   Learning via
   teachable moments,
   scaffolding
   Education Data Mining 2015, Madrid, Spain
   27 June 2015 3
   

   View Slide

4. Scaling instructor intervention?
   • Pedagogical reasons to intervene exist
   – But how much and when to intervene?
   • Instructors cannot reply or even read
   every post on a MOOC forum
   • In this work, we propose a system to
   identify threads that needs instructor’s
   attention!
   Education Data Mining 2015, Madrid, Spain
   27 June 2015 4
   

   View Slide

5. Related Work
   Education Data Mining 2015, Madrid, Spain
   27 June 2015 5
   

   View Slide

6. Related work:
   Moderating Discussion Forums
   • Backstrom et al., 2013
   – Distillation of social media message threads
   to enable moderation
   • Artzi et al, 2012
   – Predicting responses and re-tweets
   • Wang et al, 2012
   – Thread Solvedness using discourse cues
   Education Data Mining 2015, Madrid, Spain
   27 June 2015 6
   

   View Slide

7. Related work:
   MOOC Discussion Forums
   27 June 2015 Education Data Mining 2015, Madrid, Spain 7
   • Stump et al., 2013
   – Multidimensional categorisation of forum
   posts
   • Ezen-Can and Boyer
   – Dialog act based categories
   • Yang et al, 2014
   – Question recommendation for peer learners
   on MOOC forums
   

   View Slide

8. 27 June 2015 Education Data Mining 2015, Madrid, Spain
   Closest related work for comparison
   8
   Details
   

   View Slide

9. Method
   Education Data Mining 2015, Madrid, Spain
   27 June 2015 9
   

   View Slide

10. Intervention prediction
    cast as binary classification
    To classify if a given thread is intervened or not.
    • + ve instances: All threads replied to at least
    once by the instructor, TA or a community TA.
    • - ve instances: All threads never answered by
    instructor TA or a community TA.
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 10
    

    View Slide

11. Freely Annotated Data!
    MOOC Research Summit 2015
    24 June 2015 11
    Listing of forum threads on a Coursera MOOC
    

    View Slide

12. Corpus
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 12
    

    View Slide

13. Corpus
    Forum type
    All Intervened
    # threads #posts # threads #posts
    Total 7,408 51,770 2,932 11,561
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 13
    D14
    Forum type
    All Intervened
    # threads #posts # threads #posts
    Total 26,643 205,835 7,740 31,779
    From 61 courses
    D61
    From 14 courses
    Details
    

    View Slide

14. Classifier
    • Logit classifier with bag of words + other
    features
    • Plus a simple technique to address class
    imbalance
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 14
    

    View Slide

15. Features that correlate
    with interventions
    Ratio of intervened to non intervened
    threads over D14 across 4 forum types
    Forum type encodes intervention
    priority as perceived by the instructor.
    MOOC Research Summit 2015
    24 June 2015 15
    • Discourse cues such as
    agreements, affirmations,
    appreciations to original
    post
    • Length of thread
    • # of posts, comments
    • # sentences, words
    •Threads with deep
    discussions indicated by
    ratio of comments to posts
    Details
    

    View Slide

16. Experiments and Results
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 16
    

    View Slide

17. Performance varies widely across courses
    Course
    Intervention
    Ratio
    F1
    Individual
    (20% test set)
    F1
    D14
    (full course is test set)
    ml-005 0.45 64.96 56.56
    rprog-003 0.32 49.62 48.70
    calc1-003 0.60 51.29 68.91
    smac-001 0.17 25.00 33.26
    compilers-004 0.02 14.28 4.91
    maththink-004 0.49 63.56 63.29
    medicalneuro-002 0.76 75.36 81.94
    musicproduction-006 0.01 0.00 1.03
    gametheory2-001 0.19 28.57 30.16
    Average 0.36 41.59 45.54
    Weighted Macro Avg 0.40 49.04 50.56
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 17
    

    View Slide

18. Forum type and other features
    improve significantly over unigrams
    # Features Precision Recall F1
    1 Uni 41.98 61.39 45.58
    2 1+forum type 41.36 69.13 48.01
    3 2+lexical course references 41.09 66.57 47.22
    4 3+affirmations 41.20 68.94 47.68
    5 4+thread_properties 42.99 70.54 48.86
    6 5+# of sentences 43.08 69.88 49.77
    7 6+non-lexical course
    references
    42.37 74.11 50.56
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 18
    

    View Slide

19. Forum type and other features
    improve significantly over unigrams
    # Features Precision Recall F1
    1 Uni 41.98 61.39 45.58
    2 1+forum type 41.36 69.13 48.01
    3 2+lexical course references 41.09 66.57 47.22
    4 3+affirmations 41.20 68.94 47.68
    5 4+thread_properties 42.99 70.54 48.86
    6 5+# of sentences 43.08 69.88 49.77
    7 6+non-lexical course
    references
    42.37 74.11 50.56
    8 Ablating course references 45.96 79.12 54.79
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 19
    

    View Slide

20. Does scaling help?
    Corpus P R F1
    D14 45.96 79.12 54.79*
    D61 42.80 76.29 50.96*
    *Uses the best performing feature set from the previous
    experiment, that is, all except courseref
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 20
    

    View Slide

21. Limitations and Future Work
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 21
    

    View Slide

22. Variation in # of threads
    The # of threads and their intervention ratios in forums over 14s courses
    Diversity across different courses in volumes of threads
    as well as interventions
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 22
    

    View Slide

23. Is intervention subjective?
    Further, indicated by weak human annotator agreement among
    instructors (k=0.53)
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 23
    

    View Slide

24. Simple baselines work better
    Course
    F_1
    Individual courses
    (20% test set)
    F_1
    @100%R
    F_1 on D14
    (full course is
    test set)
    F_1
    @100%R
    ml-005 64.96 63.79 72.35 61.83
    rprog-003 49.62 47.39 48.55 49.31
    calc1-003 51.29 74.83 70.63 75.33
    smac-001 25.00 34.67 34.15 29.28
    compilers-004 14.28 3.28 4.82 4.75
    maththink-004 63.56 63.08 61.11 65.49
    medicalneuro-002 75.36 88.66 78.06 85.67
    musicproduction-006 0.00 4.35 1.09 1.72
    gametheory2-001 28.57 45.16 27.12 30.56
    Average 41.59 46.43 45.18 47.09
    Weighted Macro Avg 49.04 51.51 54.79 53.22
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 24
    

    View Slide

25. Intervention framework roadmap
    Real-time
    Re-intervention Role-based
    Thread Ranking
    Mitigates
    intervention
    subjectivity
    Makes intervention
    decision at post-level
    Optimises
    recommendations
    for instructor / TA
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 25
    Via online-learning
    framework
    Details
    

    View Slide

26. Conclusion
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 26
    

    View Slide

27. • Large variance in quantum of intervention
    leads us to a ranking framework.
    • Scaling the MOOC corpus by combining
    forum data from various courses is non-
    trivial.
    • Non-reproducibility of results - a key issue
    stalling progress in MOOC research.
    Conclusions
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 27
    

    View Slide

28. Email : consortium@groups.nus.edu.sg
    Website:
    wing.comp.nus.edu.sg/downloads/moocdata
    The MOOC Data Consortium:
    Enabling reproducible large-scale
    research
    Muthu Kumar Chandrasekaran,
    Min-Yen Kan,
    Kiruthika Ragupathi, Bernard Tan
    Enabling replicable
    MOOC forum research
    among Coursera partners
    Coursera ’s Statement of Support
    “As a platform for delivering world-class education and advancing
    the frontiers of online pedagogy, Coursera encourages the use of
    its platform to facilitate novel research across a broad range of
    disciplines, while concurrently protecting the privacy of learners.
    We support the described research focusing on forum activity and
    the proposal that this research span courses from across our
    partner institutions.”
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 28
    Thank you! Questions?
    

    View Slide

29. BACKUP
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 29
    

    View Slide

30. Forum type
    All Intervened
    # threads #posts # threads #posts
    Homework
    3,868 31,255 1,385 6,120
    Lecture 2,392 13,185 1,008 3,514
    Errata 326 1,045 134 206
    Exam 822 6,285 405 1,721
    Total 7,408 51,770 2,932 11,561
    D14 Corpus
    Data from 14 MOOCs ( D14) from diverse subject areas
    with different numbers of threads and interventions.
    Feature Study done using this corpus.
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 30
    Back
    

    View Slide

31. Limitations
    • Variation in # of threads, interventions
    • Intervention decision may be subjective
    • Simple baselines outperform learned
    models
    • Previous results are not replicable
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 31
    

    View Slide

32. Intervention framework: thread ranking
    • Thread priority
    – Errata vs. Lecture vs. Exam
    – Due tomorrow vs. no due date
    – Rank threads in order of urgency
    Real-time
    Re-
    intervention
    Role-based
    Thread
    Ranking
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 32
    Back
    

    View Slide

33. Intervention framework: Reintervention
    • Topical diversity within a thread
    • Topic shifts in long discussions
    • How about new posts to
    previously intervened threads?
    Real-time
    Re-
    intervention
    Role-based
    Thread
    Ranking
    Thread Level Post Level
    Reintervention
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 33
    Back
    

    View Slide

34. Intervention framework: Role based
    MOOC Instructor team has
    – Instructor(s)
    – Teaching Assistants (TA)
    – Community Teaching Assistants (CTA)
    • Different bandwidths, priorities, authorities
    • Could they intervening differently too?
    • To customise recommendation to the role
    of the staff
    Real-time
    Re-
    intervention
    Role-
    based
    Thread
    Ranking
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 34
    Back
    

    View Slide

35. Intervention framework: thread ranking
    • Courses in real-time produce
    posts over time
    • Intervention priority may change
    over time
    • Rank posts by priority at any time t of a
    MOOC
    Real-time
    Re-
    intervention
    Role-based
    Thread
    Ranking
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 35
    Back
    

    View Slide

36. Experiments Set up details
    Experiments run on two different set up
    1. Each individual course in D14 is treated
    as a corpus
    – Training and test set are form the same
    course. 20% of a course data is randomly
    sample as training
    2. Treat D14 as a single corpus
    – Leave-one –course-out cross validation
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 36
    Back
    

    View Slide

37. Chatruvedi et al, 2014 Models
    where , r is intervention
    HV1 is the Post Category
    OBS is observed post
    HV1 HV1
    OBS OBS
    HV1 HV1
    OBS OBS
    r
    HV1 HV1
    OBS OBS
    HV1 HV1
    OBS OBS
    Linear chain Markov model
    (LCMM)
    Global chain model (GCM)
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 37
    Chaturvedi et al, 2014
    • Infers 4 post categories that
    maximizes likelihood of instructor
    intervention
    • Post categories act as hidden
    variable in their sequence models
    that predict instructor intervention
    Back
    

    View Slide

38. Chaturvedi et al, 2014 Results
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 38
    Back
    

    View Slide

39. Classifier
    • Logistic regression classifier
    • We use class Weights W, to counter
    balance inherent class imbalance in this
    data
    – Biases prediction towards majority class
    instances
    • Class weights are learnt from the training
    set by greedily optimising for maximum F1
    score
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 39
    Back
    

    View Slide

40. L1 Regularised Logistic
    Regression
    Source: http://cs.nyu.edu/~rostami/presentations/L1_vs_L2.pdf
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 40
    Back
    

    View Slide

41. L1 Regularised Logistic
    Regression
    Ng, Andrew Y. "Feature selection, L 1 vs. L 2 regularization, and rotational
    invariance." Proceedings of the twenty-first international conference on Machine learning.
    ACM, 2004.
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 41
    Back
    

    View Slide

42. Novel Features: Forum Type
    Ratio of intervened to non intervened threads
    over D14 across 4 forum types
    Encodes intervention priority as perceived by the
    instructor
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 42
    back
    

    View Slide

43. Novel Features: Lexical
    References to course materials
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 43
    back
    

    View Slide

44. Novel Features: Non-lexical
    References to course materials
    URLs
    Timestamps
    from videos
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 44
    back
    

    View Slide

45. Other Features
    • Unigrams (~98,000 unique terms)
    • Thread properties
    – Length: as #posts, comment, total; as #
    sentences
    – Structure as average #comments / post
    • Affirmation of the original post by fellow
    students
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 45
    back
    

    View Slide

46. FEATURES: QUOTING EXTERNAL
    SOURCES
    Terms ‘wikipedia’ and ‘wiki’ occur in 148 and 79 docs respectively
    in the D14 corpus.
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 46
    back
    

    View Slide

47. QUOTING INSTRUCTORS / SLIDES
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 47
    back
    

    View Slide

48. AFFIRMATIONS: COMMON IN BUG
    REPORTS
    Education Data Mining 2015, Madrid, Spain
    27 June 2015 48
    back
    

    View Slide