Unit Testing Challenges with Automated Marking via EdStem

Abstract: Teaching software testing presents difficulties due to its abstract and conceptual nature. The lack of tangible outcomes and limited emphasis on hands-on experience further compound the challenge, often leading to difficulties in comprehension for students. This can result in waning engagement and diminishing motivation over time. In this paper, we introduce online unit testing challenges with automated marking as a learning tool via the EdStem platform to enhance students’ software testing skills and understanding of software testing concepts. Then, we conducted a survey to investigate the impact of the unit testing challenges with automated marking on student learning. The results from 92 participants showed that our unit testing challenges have kept students more engaged and motivated, fostering deeper understanding and learning, while the automated marking mechanism enhanced students’ learning progress, helping them to understand their mistakes and misconceptions quicker than traditional-style human-written manual feedback. Consequently, these results inform educators that the online unit testing challenges with automated marking improve overall student learning experience, and are an effective pedagogical practice in software testing.

Reference: https://arxiv.org/pdf/2310.06308

Published at the 30th Asia-Pacific Software Engineering Conference, APSEC 2023, Seoul, Republic of Korea, December 4-7, 2023

URL: https://ieeexplore.ieee.org/document/10479401/

Transcript

1. Dr. Kla Tantithamthavorn Senior Lecturer in Software Engineering Monash University, Australia http://chakkrit.com @klainfo Unit Testing Challenges with Automated Marking
2. Software Bugs Are Costly! How to Prevent Them? 2 However, one software defect can cost billions of dollars lost. For example, a failure in the Woolworth’s delivery system could leave Australians people hungry at home. Nowadays, software systems are very large and complex. How can we ensure that software is of high quality and meets user’s requirements?
3. Software Testing in a Nutshell Software testing (a discipline in software engineering) is a process to ensure the quality of software systems. To test the functionality of software systems (unit testing): we need 1. Test Input: The variables and their values in the test case. For an email login, it would be the username and password for the account. 2. Expected Output: The output expected after executing the test case 3. Actual Output: The actual output 4. Pass/Fail: Same outputs = Pass, Different outputs = Fail
4. An Example of Test Case A Test Scenario: 1+2+3 = 6 1. Test Input: [1, 2, 3] 2. Expected Output: [6] 3. Actual Output: 6 4. Pass/Fail: Passed Let’s say if we want to test a calculator app.
5. Teaching Software Testing is Quite Hard! 5 “Software testing is often not well accepted by students and the typical teaching practices often increase the cognitive load when learning software testing.” — Vahid Garousi Garousi, Vahid, et al. "Software-testing education: A systematic literature mapping." Journal of Systems and Software 165 (2020): 110570. Aniche, Maurício, Felienne Hermans, and Arie Van Deursen. "Pragmatic software testing education." Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 2019. Test Coverage is one of the most difficult topics in software testing. • Do not provide all the expected tests for a given piece of code • Write tests that are not totally correct How can we (as a teacher) motivate students’ interest in software testing and make teaching software testing more practical?
6. Teaching Software Testing is Harder than Teaching Programming 6 Teaching programming can be supported by online judges However, these online judges, coding competitions tools only evaluate functional correctness, without evaluating their testing ability Online judges aim to automatically mark if code is implemented correctly
7. Subject Structure FIT2107 Software Quality and Testing 321 Enrolled Year-2 Undergraduate Students (A core subject for B.SE.) Week Topics (Lecture + Studio) Assessment Schedule 1 Introduction to Software Quality Assurance 2 Introduction to Software Testing 3 Unit Testing / CI/CD Assessment 1: Quality Attributes, Test Plans & Unit Testing (25%) 4 Blackbox Testing 1 5 Blackbox Testing 2 6 Whitebox Testing 1 Assessment 2: Blackbox Testing (25%) 7 Whitebox Testing 2 8 Code Quality, Technical Debt, and Static Analysis Assessment 3: Whitebox Testing (25%) 9* Code Review 10 Software Metrics Assessment 4: Code Review and Code Quality (25%) 11 Software Security 12 Wrap Up / Guest Speaker Taught in both Australia and Malaysia campuses 1 Hour Lecture 2 Hours Workshop 12 weeks
8. Subject Design Principles • A responsive and iterative curriculum design approach • To enhance the learning experience and ensure the content remains relevant and effective • 12 Workshops to support active learning • To emphasize student engagement and participation • Actively practice software testing, instead of passively receiving information • Use EdStem (coding challenges) to support the constructivism learning theory • To help student construct their understanding • Use a student-centered approach • For each week, we run a survey to gather feedback from students as a continuous feedback loop • Allow educators to design, develop, and improve the subject
9. • We used Ed’s Coding Challenges (http://edstem.org) to create interactive learning modules of unit testings — to encourage students to actively participate in software testing classes. Introducing Unit Testing Challenges with Automated Marking 9 Click “Run” to get the coverage report. Following TDD, the student reads the problem requirement. Then, the student writes the test cases, and the code to pass the test cases.
10. Introducing Unit Testing Challenges with Automated Marking 10 Test cases allow the teachers to evaluate functional correctness / completeness Test coverage allows the teachers to evaluate the quality of students’ test suite. Rubrics: • Functionality completeness • Functionality correctness • Code Coverage
11. Developed a series of unit testing challenges 11
12. Developed a series of unit testing challenges 12
13. Developed a series of unit testing challenges 13
14. Marking Formula 14 Objective: To ensure students write the code correctly and design test cases to test their own code correctly. Marking Formula = % of passing test cases * % of code coverage Scenario 1: Implement code correct, but don’t write test cases = 100% passing test cases * 0% code coverage = 0% mark. Scenario 2: Implement code incorrect, but write all test cases = 0% passing test cases * 100% code coverage = 0% mark. Scenario 3: Implement code partially correct, and write test cases to partially execute their own code = 50% passing test cases * 50% code coverage = 25% mark. Scenario 4: Implement code correct, but write test cases to execute all lines and branches (Ideal!) = 100% passing test cases * 100% code coverage = 100% mark.
15. Research Question: How do the Online Unit Testing Challenges Impact Student Learning? 15 Anonymous survey questionnaire, approved by MUHRC Advertised within FIT2107 Software Quality and Testing, S2 2023 315 enrolled students, 234 from Australia, 81 from Malaysia
16. Benefits: More Engaged, Motivated, Foster Deeper Understanding 16 92 Valid Responses (29% Response Rate) • Develop self-regulated learning skills: “It helps me understand how to code and test it myself without just giving me the answers”. • Develop problem-solving skills: “Challenges the students to pay more attention to small details”. • Understand mistakes quicker than manual feedback: “Automatic feedback is good as it allows you to quickly check your answers then look for bugs if they are not passing”.
17. 17 Informal Feedback
18. Challenges: Suboptimal feedback, increasing cognitive workload 18 • “It may be helpful to give more in-depth instructions for the failed test cases.” • “It creates an overall annoying and long debug process for relatively simple problems.” Lessons learned: • Clear and comprehensive feedback is crucial. • For Workshops, the evaluation test cases are available to students so they can learn from mistakes. • For Assessment purposes, test cases should be hidden and educators should communicate clearly to students. A limited number of coding submissions could be considered. Future Work: • Assessment rubrics should go beyond code coverage (e.g., code quality, documentation, testing strategies, creativity)
19. Contact: Dr. Kla Tantithamthavorn http://chakkrit.com @klainfo Email: chakkrit@monash.edu Unit Testing Challenges with Automated Marking