Dr. Kla Tantithamthavorn Senior Lecturer in Software Engineering Monash University, Australia http://chakkrit.com @klainfo Unit Testing Challenges with Automated Marking 

Software Bugs Are Costly! How to Prevent Them? 2 However, one software defect can cost billions of dollars lost. For example, a failure in the Woolworth’s delivery system could leave Australians people hungry at home. Nowadays, software systems are very large and complex. How can we ensure that software is of high quality and meets user’s requirements? 

Software Testing in a Nutshell Software testing (a discipline in software engineering) is a process to ensure the quality of software systems. To test the functionality of software systems (unit testing): we need 1. Test Input: The variables and their values in the test case. For an email login, it would be the username and password for the account. 2. Expected Output: The output expected after executing the test case 3. Actual Output: The actual output 4. Pass/Fail: Same outputs = Pass, Different outputs = Fail 

An Example of Test Case A Test Scenario: 1+2+3 = 6 1. Test Input: [1, 2, 3] 2. Expected Output: [6] 3. Actual Output: 6 4. Pass/Fail: Passed Let’s say if we want to test a calculator app. 

Teaching Software Testing is Quite Hard! 5 “Software testing is often not well accepted by students and the typical teaching practices often increase the cognitive load when learning software testing.” — Vahid Garousi Garousi, Vahid, et al. "Software-testing education: A systematic literature mapping." Journal of Systems and Software 165 (2020): 110570. Aniche, Maurício, Felienne Hermans, and Arie Van Deursen. "Pragmatic software testing education." Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 2019. Test Coverage is one of the most difficult topics in software testing. ● Do not provide all the expected tests for a given piece of code ● Write tests that are not totally correct How can we (as a teacher) motivate students’ interest in software testing and make teaching software testing more practical? 

Teaching Software Testing is Harder than Teaching Programming 6 Teaching programming can be supported by online judges However, these online judges, coding competitions tools only evaluate functional correctness, without evaluating their testing ability Online judges aim to automatically mark if code is implemented correctly 

Subject Structure FIT2107 Software Quality and Testing 321 Enrolled Year-2 Undergraduate Students (A core subject for B.SE.) Week Topics (Lecture + Studio) Assessment Schedule 1 Introduction to Software Quality Assurance 2 Introduction to Software Testing 3 Unit Testing / CI/CD Assessment 1: Quality Attributes, Test Plans & Unit Testing (25%) 4 Blackbox Testing 1 5 Blackbox Testing 2 6 Whitebox Testing 1 Assessment 2: Blackbox Testing (25%) 7 Whitebox Testing 2 8 Code Quality, Technical Debt, and Static Analysis Assessment 3: Whitebox Testing (25%) 9* Code Review 10 Software Metrics Assessment 4: Code Review and Code Quality (25%) 11 Software Security 12 Wrap Up / Guest Speaker Taught in both Australia and Malaysia campuses 1 Hour Lecture 2 Hours Workshop 12 weeks 

Subject Design Principles ● A responsive and iterative curriculum design approach ● To enhance the learning experience and ensure the content remains relevant and effective ● 12 Workshops to support active learning ● To emphasize student engagement and participation ● Actively practice software testing, instead of passively receiving information ● Use EdStem (coding challenges) to support the constructivism learning theory ● To help student construct their understanding ● Use a student-centered approach ● For each week, we run a survey to gather feedback from students as a continuous feedback loop ● Allow educators to design, develop, and improve the subject 

● We used Ed’s Coding Challenges (http://edstem.org) to create interactive learning modules of unit testings — to encourage students to actively participate in software testing classes. Introducing Unit Testing Challenges with Automated Marking 9 Click “Run” to get the coverage report. Following TDD, the student reads the problem requirement. Then, the student writes the test cases, and the code to pass the test cases. 

Introducing Unit Testing Challenges with Automated Marking 10 Test cases allow the teachers to evaluate functional correctness / completeness Test coverage allows the teachers to evaluate the quality of students’ test suite. Rubrics: ● Functionality completeness ● Functionality correctness ● Code Coverage 

Developed a series of unit testing challenges 11 

Developed a series of unit testing challenges 12 

Developed a series of unit testing challenges 13 

Marking Formula 14 Objective: To ensure students write the code correctly and design test cases to test their own code correctly. Marking Formula = % of passing test cases * % of code coverage Scenario 1: Implement code correct, but don’t write test cases = 100% passing test cases * 0% code coverage = 0% mark. Scenario 2: Implement code incorrect, but write all test cases = 0% passing test cases * 100% code coverage = 0% mark. Scenario 3: Implement code partially correct, and write test cases to partially execute their own code = 50% passing test cases * 50% code coverage = 25% mark. Scenario 4: Implement code correct, but write test cases to execute all lines and branches (Ideal!) = 100% passing test cases * 100% code coverage = 100% mark. 

Research Question: How do the Online Unit Testing Challenges Impact Student Learning? 15 Anonymous survey questionnaire, approved by MUHRC Advertised within FIT2107 Software Quality and Testing, S2 2023 315 enrolled students, 234 from Australia, 81 from Malaysia 

Benefits: More Engaged, Motivated, Foster Deeper Understanding 16 92 Valid Responses (29% Response Rate) ● Develop self-regulated learning skills: “It helps me understand how to code and test it myself without just giving me the answers”. ● Develop problem-solving skills: “Challenges the students to pay more attention to small details”. ● Understand mistakes quicker than manual feedback: “Automatic feedback is good as it allows you to quickly check your answers then look for bugs if they are not passing”. 

17 Informal Feedback 

Challenges: Suboptimal feedback, increasing cognitive workload 18 ● “It may be helpful to give more in-depth instructions for the failed test cases.” ● “It creates an overall annoying and long debug process for relatively simple problems.” Lessons learned: ● Clear and comprehensive feedback is crucial. ● For Workshops, the evaluation test cases are available to students so they can learn from mistakes. ● For Assessment purposes, test cases should be hidden and educators should communicate clearly to students. A limited number of coding submissions could be considered. Future Work: ● Assessment rubrics should go beyond code coverage (e.g., code quality, documentation, testing strategies, creativity) 

Contact: Dr. Kla Tantithamthavorn http://chakkrit.com @klainfo Email: [email protected] Unit Testing Challenges with Automated Marking