Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Functional Programming for All! Scaling a MOOC for Students and Professionals Alike

Functional Programming for All! Scaling a MOOC for Students and Professionals Alike

Keynote at Trends in Functional Programming in Education 2017 (TFPIE'17), Canterbury, UK.

This talk covers how the Scala MOOCs work, and our intuitions for why they're successful.

Heather Miller

June 22, 2017
Tweet

More Decks by Heather Miller

Other Decks in Education

Transcript

  1. Functional
    Scaling a MOOC for Students
    and Professionals Alike
    TFPIE’17, Canterbury, UK
    June 21st, 2017
    Heather Miller
    Programming for All!

    View Slide

  2. First of all, this wasn’t all done by me alone.
    Lukas Rytz
    Vojin Jovanovic
    Manohar Jonnalagedda
    Aleksandar Prokopec
    Jorge Vicente Cantero
    Martin Odersky
    Viktor Kuncak
    Erik Meijer
    Tao Lee
    Tobias Schlatter
    Philipp Haller
    Julien Richard-Foy
    Fengyun Liu
    Others who helped make our MOOC
    story possible:

    View Slide

  3. Agenda
    the courses
    tools & infrastructure
    the data we collected
    our impressions
    (it’s open-source!)

    View Slide

  4. Agenda
    the courses
    tools & infrastructure
    the data we collected
    our impressions
    (it’s open-source!)
    My goal in this talk:
    To give you as complete of an impression as
    I can about the full experience of running a
    popular MOOC on functional programming.

    View Slide

  5. Our foray into MOOCs…
    Started in 2012:
    Functional Programming
    Principles in Scala
    ‐(in its infancy at the time)
    on:
    At a glance:
    To date, 6 MOOCs
    ~800,000 learners reached

    View Slide

  6. Daphne Koller visited us at EPFL
    in July 2012:

    View Slide

  7. • sdf
    By September 2012, our 1st
    MOOC was launched!

    View Slide

  8. • sdf
    Introduction of fundamentals +
    functional programming concepts
    GOAL:
    E.g., recursion, persistent/immutable
    data structures, higher-order
    functions, pattern matching, etc.

    View Slide

  9. Lecture videos.
    In-video quizzes.
    Auto-graded programming
    assignments.
    Preliminaries
    – each 6-8 minutes long
    – total 1.5-2 hours per week
    7 weeks.
    – workload: 5-7 hours per week
    – verbatim 50% of EPFL’s on-campus
    Functional Programming course
    (2nd year bachelor level)

    View Slide

  10. week 1: functions & evaluation, recursion
    Content:
    week 2: higher-order functions
    week 3: data and abstraction
    week 4: types and pattern matching
    week 5: functional lists
    week 6: list comprehensions + maps
    week 7: streams & lazy evaluation
    Taught by:
    Martin Odersky

    View Slide

  11. week 1: functions & evaluation, recursion
    Content:
    week 2: higher-order functions
    week 3: data and abstraction
    week 4: types and pattern matching
    week 5: functional lists
    week 6: list comprehensions + maps
    week 7: streams & lazy evaluation
    Ok. How’d it go?
    Taught by:
    Martin Odersky

    View Slide

  12. View Slide

  13. Number Enrolled
    http://www.katyjordan.com/MOOCproject.html

    View Slide

  14. http://www.katyjordan.com/MOOCproject.html
    Number Enrolled
    50,000 students
    19.2% completion rate
    6.5% completion rate
    AVERAGE:
    across all MOOCs

    View Slide

  15. http://www.katyjordan.com/MOOCproject.html
    Number Enrolled
    50,000 students
    19.2% completion rate
    Jordan, K. (2014)
    Initial trends in
    enrollment and
    completion of massive
    open online courses.
    The International Review
    of Research in Open and
    Distance Learning, 15(1),
    133-160.
    6.5% completion rate
    AVERAGE:
    across all MOOCs

    View Slide

  16. Why such a high
    completion rate?
    our completion rate was 3x the norm.

    View Slide

  17. We think it was the tooling &
    infrastructure.
    automated
    cloud-based
    graders
    interactive
    build tool
    decent choice
    of IDEs style checkers
    testing
    frameworks

    View Slide

  18. Interactive development/
    submission cycle.
    Scala’s interactive build tool, configured to
    submit student assignments to the automated
    cloud-based graders from the command line.
    Compile. Test. Submit.

    View Slide

  19. Automated grading/feedback.
    Custom cloud-based auto-grader.

    View Slide

  20. Automated grading/feedback.
    Custom cloud-based auto-grader.
    Provided two types of feedback:
    Massive suite of secret unit tests.
    Style-checker

    View Slide

  21. Automated grading/feedback.
    Custom cloud-based auto-grader.
    Provided two types of feedback:
    Massive suite of secret unit tests.
    Style-checker
    discourages:
    – mutable variables
    – return statements
    – the null value
    – while loops
    – magic numbers
    – overly long lines of code
    – non-standard capitalization
    – + more

    View Slide

  22. Automated grading/feedback.
    Custom cloud-based auto-grader.

    View Slide

  23. Automated grading/feedback.
    Custom cloud-based auto-grader.

    View Slide

  24. Automated grading/feedback.
    Custom cloud-based auto-grader.

    View Slide

  25. Automated grading/feedback.
    Custom cloud-based auto-grader.
    Provided two types of feedback:
    Massive suite of secret unit tests.
    Style-checker
    Importantly:
    Resubmissions welcome.
    Feedback arrives fast.
    seconds – 15 minutes

    View Slide

  26. IDEs
    Popular IDEs come with worksheets
    for easy experimenting:

    View Slide

  27. IDEs
    Popular IDEs come with worksheets
    for easy experimenting:
    Use sbt right from Eclipse/IntelliJ.
    code, compile, test, submit, all from the IDE.

    View Slide

  28. So, what
    does it mean?

    View Slide

  29. So, what
    does it mean?
    Students had a very tight
    feedback loop.

    View Slide

  30. If you scored >0, it was most likely that you got
    100% (80/80) in the course.

    View Slide

  31. Most people got a perfect score within 4
    submission attempts.
    5 10 15 20
    # Submissions
    0.0%
    5.0%
    10.0%
    15.0%
    20.0%
    25.0%
    30.0%
    35.0%
    40.0%
    45.0%
    Percentage of Students
    Fall 2012
    Spring 2013
    (The number of submissions required to achieve a perfect
    score.)

    View Slide

  32. also, this
    wasn’t just students
    professionals too!
    but,

    View Slide

  33. A vast majority of participants already had
    graduated from university – 87%.
    Participants’ highest degrees

    View Slide

  34. A vast majority of participants come from
    computer science or computer/software
    engineering – 71%.
    Participants’ fields of study

    View Slide

  35. A large portion of participants plan on applying
    what they’ve learned in the course at work –
    40%.
    Where do you plan to apply what you’ve
    learned in the course?

    View Slide

  36. And yet ~70% of professional respondents felt
    the course was well-worth their time.
    Do you feel the course was worth it?
    All respondents vs those who use Scala at
    work
    1
    (Disagree)
    2 3 4 5
    (Agree)
    0
    20
    40
    60
    80
    100
    Percentage of Students
    1 2
    7
    22
    68
    1 1
    6
    21
    71
    Fall 2012
    1
    (Disagree)
    2 3 4 5
    (Agree)
    0
    20
    40
    60
    80
    100
    1 2
    8
    24
    66
    1 1
    6
    22
    70
    Spring 2013
    All Respondents Respondents Using Scala at Work
    For Fall 2012, 71% amounts to 2,148/3,203 professional
    respondents.

    View Slide

  37. A vast majority of which received perfect
    scores, and felt that the course was well worth
    their time.
    we can conclude that there were indeed a
    significant number of professionals
    participating in the course.
    SO,

    View Slide

  38. How’d it fare on campus?
    Alongside of 50,000 MOOC learners,
    150 EPFL students took MOOC for credit.

    View Slide

  39. How’d it fare on campus?
    Alongside of 50,000 MOOC learners,
    150 EPFL students took MOOC for credit.
    MOOC Traditional
    offline course
    EPFL Semester:
    week 0 week 14
    week 7
    written
    midterm
    exam
    written
    final
    exam

    View Slide

  40. How did it differ for EPFL
    students?
    MOOC
    participants
    EPFL
    students
    5-7 videos each
    week, 8-12min
    LECTURES
    weekly
    programming
    exercises
    ASSIGNMENTS work in groups,
    with TAs on HW
    EXERCISE SESSIONS
    midterm & final
    WRITTEN EXAMS
    offline traditional
    2nd half of course
    SAME AS MOOC
    +

    View Slide

  41. What’d the EPFL students
    think?
    In the future, I'd prefer a course like this be...
    69% 17.8% 7% 6%
    Online,
    14 weeks
    Online 7wks,
    On-campus 7wks
    On-
    campus
    14wks
    No
    opinion

    View Slide

  42. Hang on,
    where did this data
    come from?

    View Slide

  43. The data
    Two iterations of Functional Programming
    Principles in Scala.
    Fall 2012
    Spring 2013

    View Slide

  44. The data
    Two iterations of Functional Programming
    Principles in Scala.
    Fall 2012
    Spring 2013
    Three sources per iteration:
    – Scores & submission data from Coursera
    – Survey data
    – EPFL specialized course survey

    View Slide

  45. Survey data
    Post-course survey:
    For the Fall 2012 course, 7,492 respondents out of ~50,000
    For the Spring 2013 course, 4,595 respondents out of ~37,000
    Total: 12,087 respondents

    View Slide

  46. Survey data
    Post-course survey:
    For the Fall 2012 course, 7,492 respondents out of ~50,000
    For the Spring 2013 course, 4,595 respondents out of ~37,000
    Total: 12,087 respondents
    Example questions:
    If applicable, what field of study was your highest degree in?
    What's your highest degree?
    How many years have you been programming?
    How difficult did you find the course overall?
    Where do you plan to apply what you've learned in this course?
    What experience do you have with other programming languages or paradigms?

    View Slide

  47. by the way, the data is open
    source
    + tools to generate visualizations of the data

    View Slide

  48. EPFL student data
    Post-course survey:
    Given to EPFL students who took both the MOOC as
    well as the regular on-campus in-person course.

    View Slide

  49. 6 - Excellent/Strongly Agree
    5
    4
    3
    2
    1 - Poor/Strongly Disagree
    No Opinion
    ine part of the course is:
    38.37% 15.11% 4%
    would like to get more online courses:
    23.26% 23.26% 9.6% 3%2%5%
    for the course is…:
    28.57% 28.57% 8.3% 2%11.9%
    exercise sessions for the course is…:
    15.11% 5.8%1% 48.83%
    prefer a course like this be...
    17.8% 7% 6%
    On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    Legend
    6 - Exce
    5
    4
    3
    2
    1 - Poor
    No Opin
    Overall, the online part of the course is:
    41.86% 38.37% 15.11% 4%
    (a)
    In the future, I would like to get more online courses:
    33.72% 23.26% 23.26% 9.6% 3%2%5%
    (b)
    The online help for the course is…:
    20.24% 28.57% 28.57% 8.3% 2%11.9%
    (c)
    The help in the exercise sessions for the course is…:
    4.6%24.41% 15.11% 5.8%1% 48.83%
    (d)
    In the future, I'd prefer a course like this be...
    69% 17.8% 7% 6%
    Online 14 weeks On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    (e)
    EPFL student data

    View Slide

  50. 6 - Excellent/Strongly Agree
    5
    4
    3
    2
    1 - Poor/Strongly Disagree
    No Opinion
    ine part of the course is:
    38.37% 15.11% 4%
    would like to get more online courses:
    23.26% 23.26% 9.6% 3%2%5%
    for the course is…:
    28.57% 28.57% 8.3% 2%11.9%
    exercise sessions for the course is…:
    15.11% 5.8%1% 48.83%
    prefer a course like this be...
    17.8% 7% 6%
    On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    Legend
    6 - Exce
    5
    4
    3
    2
    1 - Poor
    No Opin
    Overall, the online part of the course is:
    41.86% 38.37% 15.11% 4%
    (a)
    In the future, I would like to get more online courses:
    33.72% 23.26% 23.26% 9.6% 3%2%5%
    (b)
    The online help for the course is…:
    20.24% 28.57% 28.57% 8.3% 2%11.9%
    (c)
    The help in the exercise sessions for the course is…:
    4.6%24.41% 15.11% 5.8%1% 48.83%
    (d)
    In the future, I'd prefer a course like this be...
    69% 17.8% 7% 6%
    Online 14 weeks On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    (e)
    ~80% of students think the course was very good
    or excellent
    EPFL student data

    View Slide

  51. EPFL student data 6 - Excellent/Strongly Agree
    5
    4
    3
    2
    1 - Poor/Strongly Disagree
    No Opinion
    ine part of the course is:
    38.37% 15.11% 4%
    would like to get more online courses:
    23.26% 23.26% 9.6% 3%2%5%
    for the course is…:
    28.57% 28.57% 8.3% 2%11.9%
    exercise sessions for the course is…:
    15.11% 5.8%1% 48.83%
    prefer a course like this be...
    17.8% 7% 6%
    On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    Legend
    6 - Exce
    5
    4
    3
    2
    1 - Poor
    No Opin
    Overall, the online part of the course is:
    41.86% 38.37% 15.11% 4%
    (a)
    In the future, I would like to get more online courses:
    33.72% 23.26% 23.26% 9.6% 3%2%5%
    (b)
    The online help for the course is…:
    20.24% 28.57% 28.57% 8.3% 2%11.9%
    (c)
    The help in the exercise sessions for the course is…:
    4.6%24.41% 15.11% 5.8%1% 48.83%
    (d)
    In the future, I'd prefer a course like this be...
    69% 17.8% 7% 6%
    Online 14 weeks On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    (e)
    ~58% of students would like more online courses
    in the future

    View Slide

  52. EPFL student data 6 - Excellent/Strongly Agree
    5
    4
    3
    2
    1 - Poor/Strongly Disagree
    No Opinion
    ine part of the course is:
    38.37% 15.11% 4%
    would like to get more online courses:
    23.26% 23.26% 9.6% 3%2%5%
    for the course is…:
    28.57% 28.57% 8.3% 2%11.9%
    exercise sessions for the course is…:
    15.11% 5.8%1% 48.83%
    prefer a course like this be...
    17.8% 7% 6%
    On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    Legend
    6 - Exce
    5
    4
    3
    2
    1 - Poor
    No Opin
    Overall, the online part of the course is:
    41.86% 38.37% 15.11% 4%
    (a)
    In the future, I would like to get more online courses:
    33.72% 23.26% 23.26% 9.6% 3%2%5%
    (b)
    The online help for the course is…:
    20.24% 28.57% 28.57% 8.3% 2%11.9%
    (c)
    The help in the exercise sessions for the course is…:
    4.6%24.41% 15.11% 5.8%1% 48.83%
    (d)
    In the future, I'd prefer a course like this be...
    69% 17.8% 7% 6%
    Online 14 weeks On-
    campus
    14 weeks
    No Opinon
    Online 7wks/
    On-campus 7wks
    (e)
    ~69% of students would like their entire course to
    be online, with no on-campus component

    View Slide

  53. students seemed to overwhelmingly prefer the
    MOOC version of the course. In fact, students
    even preferred the MOOC forums to the exercise
    sessions.
    SO,
    Test performance remained the same, course
    ratings remained high.

    View Slide

  54. The result?
    Happier EPFL Students
    Uptaken and depended-on
    by professionals in industry
    Good performance,
    high course ratings

    View Slide

  55. Conclusions
    Positive experience for all
    Highest rate of retention for
    a course our size
    Both professionals and students alike
    had positive learning experiences.
    between 2012-2013

    View Slide

  56. 2013:
    New MOOC,
    Our foray into MOOCs continued…
    Principles of Reactive Programming
    ~67,000 registrants in first run

    View Slide

  57. View Slide

  58. 2013:
    New MOOC,
    Our foray into MOOCs continued…
    2016/2017:
    2 new MOOCs + capstone project bundled
    into a Scala mini-degree on Coursera
    Principles of Reactive Programming
    ~67,000 registrants in first run
    Parallel Programming
    Big Data Analysis with Scala & Spark
    400,000 registrants in first year
    for courses in mini-degree

    View Slide

  59. View Slide

  60. View Slide

  61. Financials?

    View Slide

  62. Financials?
    Well, it works*
    *mini-degrees, that is.

    View Slide

  63. Financials?
    ~2 million USD brought in in first fiscal year.
    Granted, how the money is actually split is a whole
    different issue.
    Well, it works*
    *mini-degrees, that is.

    View Slide

  64. Take aways…

    View Slide

  65. Take aways…
    Autograding + a tight feedback loop
    is key to retention.
    Recent results [1] at LAK’17 arrive at the
    same conclusion.
    [1] Follow the Successful Crowd: Raising MOOC Completion
    Rates through Social Comparison at Scale, LAK’17

    View Slide

  66. Would I do it again in 2017?
    Take aways…
    Autograding + a tight feedback loop
    is key to retention.
    Recent results [1] at LAK’17 arrive at the
    same conclusion.
    [1] Follow the Successful Crowd: Raising MOOC Completion
    Rates through Social Comparison at Scale, LAK’17

    View Slide

  67. Would I do it again in 2017?
    Take aways…
    Autograding + a tight feedback loop
    is key to retention.
    Recent results [1] at LAK’17 arrive at the
    same conclusion.
    Not sure.
    Did you notice I didn’t have data to
    show after 2013?
    [1] Follow the Successful Crowd: Raising MOOC Completion
    Rates through Social Comparison at Scale, LAK’17

    View Slide

  68. The MOOC-provider landscape has
    drastically changed
    Would I do it again in 2017?
    Take aways…
    Autograding + a tight feedback loop
    is key to retention.
    Recent results [1] at LAK’17 arrive at the
    same conclusion.
    Not sure.
    Did you notice I didn’t have data to
    show after 2013?
    [1] Follow the Successful Crowd: Raising MOOC Completion
    Rates through Social Comparison at Scale, LAK’17

    View Slide

  69. Thank you!
    Questions?

    View Slide