Interventions in the Online Classroom

Presented at HCI Lunch in 2012.

Transcript

1. interventions
   NLINE
   CLASSROOM
   in the
   JUSTIN CHENG SCOTT KLEMMER
   CHINMAY KULKARNI
   1
   

   View Slide

2. How do we increase student
   retention in online classes?
   (Assignment Completion)
   2
   

   View Slide

3. Retention is important
   Interested students may need an
   additional push to complete a course
   Important metric in measuring how
   engaging a class is
   3
   

   View Slide

4. Sociality
   Feedback
   · Peer Assessment
   Individual / Group
   · Grades
   · Action Plans
   · Contextual Hints
   · Commitment Contracts
   · Office Hours
   · Generic Hints
   · Personalized Email
   · 1-on-1 tutors
   Passive / Active
   · Co-located group work
   4
   

   View Slide

5. Social Effects
   People are consistent in their actions
   [Cialdini ‘01]
   People are influenced by what their
   friends do [Cialdini ‘07]
   – “Scott and 4 other friends completed this
   assignment”
   5
   

   View Slide

6. View Slide

7. Commitment
   Sign a commitment contract [Werner, et. al ‘95]
   A small commitment followed by a larger
   one [Taylor & Booth-Butterfield ‘93]
   Bet on your success [Giné, et. Al ‘08]
   7
   

   View Slide

8. We find that
   The effects of commitment is affected by
   social influence
   Using checklists encourages task
   completion
   …
   8
   

   View Slide

9. So what did we do?
   9
   

   View Slide

10. Experimental Setup
    Conducted on Fall 2012 Stanford HCI
    Online class
    Waitlist control to minimize participant
    backlash
    10
    

    View Slide

11. Social Feedback / not
    Commitment / not
    Exp. 1 Social Feedback + Commitment
    2384
    participants
    2 x 2
    conditions
    •  How does social feedback and
    commitment affect student retention?
    •  How do these interact with each other?
    11
    

    View Slide

12. Exp. 1 Social Feedback + Commitment
    Both Social Feedback + Commitment Condition
    “I’ve started button” + List of friends
    (on Announcements Page & Assignment Page)
    12
    

    View Slide

13. Exp. 1 Social Feedback + Commitment
    Both Social Feedback + Commitment Condition
    Email before due date - friends
    who’ve started
    Email after due date – see your
    friends’ work
    13
    

    View Slide

14. Exp. 1 Social Feedback + Commitment
    Social Feedback Only Condition
    List of friends
    (on Announcements Page &
    Assignment Page)
    Generic email before due date
    See friends’ work email after
    due date
    14
    

    View Slide

15. Exp. 1 Social Feedback + Commitment
    Commitment Only Condition
    “I’ve started” button
    (on Announcements Page &
    Assignment Page)
    Generic email before due date
    Generic email after due date
    15
    

    View Slide

16. Exp. 1 Social Feedback + Commitment
    No Social, No Commitment Condition
    16
    

    View Slide

17. Exp. 2 Personalized & Generic Email
    Personalized Emails
    Generic Emails
    No Email
    1439
    participants
    3
    conditions
    •  Does sending emails affect retention?
    •  Are personalized emails more effective
    than generic emails?
    17
    

    View Slide

18. Exp. 2 Personalized & Generic Email
    Generic Email Condition
    Generic encouragement  
    Examples of good work  
    18
    

    View Slide

19. Exp. 2 Personalized & Generic Email
    Personalized Email Condition
    Number of videos
    watched,
    quiz score reporting
    Use of student’s name  
    Generic encouragement  
    Examples of good work  
    19
    

    View Slide

20. Exp. 2 Personalized & Generic Email
    No Email Condition
    20
    

    View Slide

21. Exp. 3 Action Plans
    Action Plans
    No Action Plans
    1439
    participants
    2
    conditions
    •  Do interactive checklists help people
    complete assignments?
    21
    

    View Slide

22. Exp. 3 Action Plans
    Interactive checklist at the top of an assignment
    Inline step completion 22
    

    View Slide

23. Facebook Friends
    have no noticeable impact on retention
    23
    

    View Slide

24. An “I’ve started” Button
    reduces assignment completion (and retention)
    24
    

    View Slide

25. But both together
    increase completion!
    (p < 0.05)
    25
    

    View Slide

26. Control Commitment
    0
    20
    40
    60
    80
    100
    Control Social Control Social
    social_feedback_condition
    count
    26
    

    View Slide

27. Emails that we sent
    have no noticeable impact on retention
    27
    

    View Slide

28. 0
    500
    1000
    None Generic Personalized
    intervention_condition
    count
    28
    

    View Slide

29. But! Personalized Emails
    discourage people from clicking on examples!
    (p < 0.05)
    Assignment Link  
    Example Link  
    Example Link  
    29
    

    View Slide

30. 3 4 5
    0
    50
    100
    150
    Generic Personalized Generic Personalized Generic Personalized
    factor(intervention_condition)
    count
    30
    

    View Slide

31. Action Plans
    increase assignment completion (and retention)
    (p < 0.05)
    31
    

    View Slide

32. 0
    100
    200
    300
    400
    500
    600
    Control Action plan
    Experimental condition
    Number of submissions
    32
    

    View Slide

33. To conclude,
    Commitment goes hand-in-hand with
    social influence
    Action plans or interactive assignments in
    general are promising
    33
    

    View Slide

34. What’s next?
    What are other ways to increase
    retention?
    34
    

    View Slide