Interactive learning - A Scalable and Adaptive Learning Approach for Large Courses

Transcript

1. Interactive learning - A Scalable and Adaptive 
 Learning Approach

   for Large Courses Stephan Krusche 18.05.2021 Chair: Prof. Dr. Tobias Nipkow, TUM Department of Informatics Examiner: Prof. Dr. Bernd Brügge, TUM Department of Informatics Prof. Dr. Maria Bannert, TUM School of Education Habilitation

2. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Current state of learning in universities 2 0 1000 2000 3000 2013 2014 2015 2016 2017 2018 2019 2020 2.508 2.312 2.208 2.005 1.840 1.580 1.362 1.110 First year students (Informatics TUM) • Large effort for instructors, especially in the correction of exercises and exams • Impossible to interact with each student on an individual level • However: individual feedback is important for the learning experience [Iro07] Year Students [LB64] % unable to
 express idea size of group never talked had ideas
 which they
 did not express 0 12 24 36 48 0 10 20 30 40

3. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Problems in larger courses 3 No or little involvement Too much focus on lower cognitive skills Learning goals Learning 
 activities Constructive
 alignment Assessment Misaligned assessments Heterogeneous student groups

4. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Objectives and research process 4 Teaching philosophy Learning Teaching platform Application in case studies Dissemination Empirical evaluation

5. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Interactive learning* Definition: Instructors teach and exercise small chunks of content in short cycles using technology. They provide immediate feedback so that learners can reflect on the content and increase their knowledge incrementally. 5 “Tell me and I will forget. Show me and I will remember. Involve me and I will understand. Step back and I will act.” — Chinese Proverb Practice Example Feedback Student Reflection Theory * integrates aspects of active learning [BE91], blended learning [GK04] and experiential learning [Kol84] [KvFA17, KSBB17, KS18]

6. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Interactive learning (embedded in the syllabus) 6 Topic Topic Topic Topic Topic Course syllabus Practice Example Feedback Student Reflection Theory Learning sprint Knowledge increment Learning gain Learning goal Learning goal Lecture Learning goal ➡ Homework and tutor based exercises further deepen the knowledge (adapted from Scrum [Sch95] and experiential learning [Kol84]) based on constructive alignment [KS19]

7. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Artemis - interactive learning with individual feedback 7 Programming Modeling Text Quiz Team exercises | Lectures | Presentations | Exam mode | Questions and answers | Learning analytics Scalability: handle > 200 submissions per second Instant feedback: provide feedback in real time Usability: beginners 
 are able to use it

8. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Automatic assessment of programming exercises 8 Student Version control server 1 submit Continuous integration server 2 notify 3 compile, test & analyze 4 notify student with feedback [KS18]

9. Habilitation | Stephan Krusche | Interactive learning - A Scalable

   and Adaptive Learning Approach for Large Courses Programming exercises workflow 9 Student Artemis Instructor Start exercise Copy & configure repository Copy & configure build plan Clone repository Solve exercise Commit & push solution Build, test and analyze code Review results ok? Prepare exercise yes Review feedback no Solve 
 exercise in online 
 editor with interactive instructions [KS18]

10. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Online editor with interactive instructions 10 [KS18] Open source https://github.com/ls1intum/Artemis and free to use on https://artemis.ase.in.tum.de

11. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Artemis system architecture (v1) 11 Artemis client Local Build Agent Local Build Agent Local build agent University data center Version control server Continuous integration server Artemis server Version control client Remote Build Agent Remote Build Agent Remote build agent User management Student computer LTI Interface

12. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Artemis - interactive learning with individual feedback 12 Team exercises | Lectures | Presentations | Exam mode | Questions and answers | Learning analytics Scalability: handle > 200 submissions per second Instant feedback: provide feedback in real time Usability: beginners 
 are able to use it Programming Modeling Text Quiz

13. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Apollon: online modeling editor 13 Open source https://github.com/ls1intum/Apollon and free to use on https://apollon.ase.in.tum.de (without account) [KvFRB20]

14. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Semi-automatic assessment of modeling and text exercises 14 Reviewer Artemis Student Submit solution Review assessment Analyze assessment Model submission Assess automatically Assessment proposal Knowledge «use» Review assessment Assess manually Adjust assessment Assessment yes ok? Refine solution no Athene (Text) / Compass (Modeling) [BKKB21, BKB21]

15. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Vision: automatic assessment of modeling and text exercises 15 Reviewer Artemis Student Submit solution Review assessment Analyze assessment Model submission Assess automatically Assessment proposal Knowledge «use» Review assessment Assess manually Adjust assessment Assessment yes ok? Refine solution no Athene (Text) / Compass (Modeling) [BKKB21, BKB21]

16. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Assessments of UML models 16 Proposed assessment Example solution Grading criteria [KvFRB20]

17. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Artemis system architecture (v2) 17 Athene Artemis client
 University data center Version control server Continuous integration server Artemis Server
 Artemis Server
 Artemis server
 Version control client User management Student computer LTI Interface Apollon Compass Broker Discovery Local Build Agent Local Build Agent Local build agent Remote Build Agent Remote Build Agent Remote build agent Load balancer

18. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Application in case studies 18 Course Short Active
 students Program Instances Introduction to Software Engineering EIST up to 2,100 Bachelor (2nd sem) SS19 - SS21 Patterns in Software Engineering PSE up to 600 Bachelor + Master WS16/17 - WS20/21 Project Organization and Management POM up to 400 Bachelor + Master SS15 - SS19 MOOC: Software Engineering Essentials SEECx up to 700 Anyone SS17 - WS20/21

19. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Empirical evaluation: hypotheses H1 Scalability - Interactive learning can be used in large courses with more than 1,500 students participating in exercises at the same time H2 Engagement - Interactive learning increases the participation and motivation of students H3 Learning outcome - Interactive learning improves the learning outcome for students H4 Grading effort and feedback quality - Supervised machine learning reduces the grading effort while improving the feedback quality H5 Adaptability - Interactive learning adapts the difficulty of a course to each individual student by using machine learning 19

20. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses H2: Engagement 20 0 100 200 300 400 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 L12 L13 L14 L15 63 71 62 44 87 109 123 222 104 199 103 149 192 125 199 Participating students per lecture in POM 2014 Registered students: 345 58% 36% 56% 43% 30% 58% 30% 64% 36% 32% 25% 13% 18% 21% 18% 0 100 200 300 L1 L2 L3 L4 L5 L6 L7 L8 L9 L10 L11 112 128 143 160 149 183 180 191 196 173 154 Participating students per lecture in POM 2015 Registered students: 294 52% 59% 67% 65% 61% 62% 51% 54% 49% 44% 38% Lectures Traditional course Course with 
 interactive learning ~17% Participation ~46% Participation Lectures Increase by 165% [KSBB17]

21. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses H3: Learning outcome 21 20 % 40 % 60 % 80 % 63 % 61 % 57 % 55 % 46 % 65 % 55 % 49 % 41 % 36 % 61 % 59 % 53 % 41 % 36 % POM EIST PSE Exercise performance 20 % 40 % 60 % 80 % 100 % 0 % Average exam score (without bonus) Correlation between exercise performance (x) and average exam score (y) [KSBB17] Course POM EIST PSE Participants 294 1,128 324 2 (0.99;16) 83 547 48 p 5.8e-11 2.2e-16 4.7e-05 Cramer V 0.265 0.348 0.192 adj. contin- gency coeff 0.523 0.639 0.402 χ

22. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses 78 % 25 % 54 % 55 % 87 % 81 % 16 % 52 % 29 % 69 % 2018 exam (n=1128) 2019 exam (n=1225) H3: Learning outcome 22 1) Functional model 2) Structural model 3) Dynamic model 4) Architecture model 5) Model refactoring Improvement: 26 % (p = 2.2e-16, = 0.01) Improvement: 87 % (p = 2.2e-16, = 0.01) Improvement: 55 % (p = 6.4e-15, = 0.01) [KvFRB20] Exam assignments with UML modeling in EIST Control group Experimental group Results of a 2 sample t-test (1 tailed) Average scores per assignment in the final exams Average scores per assignment in the final exams

23. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses H4: Grading effort and feedback quality 23 ➡ Less effort ➡ Less complaints ➡ Perceived higher quality [BKKB21, BKB21] Model 1 (Reverse engineer tables, n=887) Model 2 (Build & release workflow, n=877) Model 3 (Analysis object model, n=836) Text 1 (Requirements, exam, n=446) Text 2 (Use cases, exam, n=425) Text 3 (Unified process and Scrum, n=959) 65 % 50 % 30 % 25 % 15 % 17 % 30 % 42 % 56 % 65 % 80 % 76 % Automatic Adjusted Manual 7 % 5 % 10 % 14 % 8 % 5 % Double blind grading Structured grading criteria Integrated training process Train on example submissions Grading leaderboard Homework or exam exercises

24. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Contributions 24 Practice Example Feedback Student Reflection Theory 1) Interactive learning Team, Lectures, Presentations, Exam mode, Q&A, Analytics Scalability Instant feedback Usability Programming Modeling Text Quiz 2) Artemis 3) Application in case studies • EIST • POM • PSE • SEECx

25. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Contributions 25 69% 29% 52% 16% 81% 87% 55% 54% 25% 78% 0 20 40 60 80 100 Functional Structural Dynamic Refactoring 2018 2019 Architecture n2018 = 1128 n2019 = 1225 5) Empirical evaluations ➡ 10 universities ➡ 63 courses with 
 30,000 students ➡ 31 exams with 
 8,500 students 4) Dissemination ✓ H1: Scalability ✓ H2: Increased engagement ✓ H3: Improved learning outcome ✓ H4: Reduced grading effort 
 and improved feedback quality X H5: Adaptability

26. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Future work 26 Learning analytics Adaptive learning Exam mode Modeling Programming Micro service 1 Micro service 2 Shared database Micro service 3 Micro services and micro frontends

27. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Research and development team 27

28. Interactive learning - A Scalable and Adaptive 
 Learning Approach

    for Large Courses Stephan Krusche 18.05.2021 Chair: Prof. Dr. Tobias Nipkow, TUM Department of Informatics Examiner: Prof. Dr. Bernd Brügge, TUM Department of Informatics Prof. Dr. Maria Bannert, TUM School of Education Habilitation Thank you! Artemis Apollon Compass Metis Athene Orion Ares

29. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses Relevant publications [KSBB17] Krusche, Seitz, Börstler, Bruegge: Interactive Learning: Increasing Student Participation through Shorter Exercise Cycles. ACE 2017. [KvFA17] Krusche, von Frankenberg, Afifi. Experiences of a Software Engineering Course based on Interactive Learning. SEUH 2017. [KBC+17] Krusche, Bruegge, Camilleri, Krinkin, Seitz, Wöbker: Chaordic Learning: A Case Study. ICSE 2017. [KS18] Krusche, Seitz: ArTEMiS: An Automatic Assessment Management System for Interactive Learning. SIGCSE 2018. [KDXB18] Krusche, Dzvonyar, Xu and Bruegge. Software Theater — Teaching Demo Oriented Prototyping. TOCE 2018 [KS19] Krusche, Seitz: Increasing the Interactivity in Software Engineering MOOCs - A Case Study. HICSS 2019. [LKvFB19] Laß, Krusche, von Frankenberg, Bruegge: Stager: Simplifying the Manual Assessment of Programming Exercises. SEUH 2019. [KvFRB20] Krusche, von Frankenberg, Reimer and Bruegge: An Interactive Learning Method to Engage Students in Modeling, ICSE 2020. [BKKB21] Bernius, Kovaleva, Krusche, Bruegge. Towards the Automation of Grading Textual Student Submissions to Open-ended Questions. ECSEE 2020. [BKB21] Bernius, Krusche, Bruegge. A Machine Learning Approach for Suggesting Feedback in Textual Exercises in Large Courses. L@S 2021. 29

30. Habilitation | Stephan Krusche | Interactive learning - A Scalable

    and Adaptive Learning Approach for Large Courses References [BEF+56] B. Bloom, M. Engelhart, E. Furst, W. Hill, and D. Krathwohl, “Taxonomy of educational objectives: The classification of educational goals,” 1956. [Big03] John Biggs. Aligning teaching and assessing to course objectives. Teaching and learning in higher education: New trends and innovations, 2:13–17, 2003. [PNI+18] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365, 2018. [LB64] Harold J Leavitt and Bernard M Bass. Organizational psychology. Annual Review of Psychology, 15(1):371– 398, 1964. [BE91] Charles Bonwell and James Eison. Active Learning: Creating Excitement in the Classroom. ASHE-ERIC Higher Education Reports, 1991. [Kol84] David Kolb. Experiential learning: Experience as the source of learning and development, volume 1. Prentice Hall, 1984. [Sch95] Ken Schwaber. Scrum development process. In Proceedings of the OOPSLA Workshop on Business Object Design and Information, 1995. [GK04] R. Garrison and H. Kanuka. Blended learning: Uncovering its transformative potential in higher education. The internet and higher education, 2004 [Iro07] Alastair Irons. Enhancing learning through formative assessment and feedback. Routledge, 2007. 30