Enhancement of Online Judge Systems by Gamification and Providing Similar Problems in Programming Assignments
Hironori Washizaki, “Enhancement of Online Judge Systems by Gamification and Providing Similar Problems in Programming Assignments,” 5th International Conference on Education Technology Management (ICETM 2022), Keynote, Online, December 17, 2022
Enhancement of Online Judge Systems by Gamification and Providing Similar Problems in Programming Assignments Hironori Washizaki Professor at Waseda University, Tokyo, Japan IEEE Computer Society Vice President Based on projects by Remin Kasahara, Ryoya Yoshimura, and Kazunori Sakamoto
Background ● In programming courses, online judge systems automate programming assignment grading ● It effectively fosters students' ability to write a program that meets a specification Students can instantly receive the judging (grading) result of their code 3 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Example 4 FizzBuzz is a game in which integers greater than or equal to 1 are said in sequence according to the following rules. • Fizz when divisible by 3 • Buzz when divisible by 5 • FizzBuzz when the number is divisible by both 3 and 5 • Otherwise, that number. Examples of game progress are shown below. 1, 2, Fizz, 4, Buzz, Fizz, 7, 8, Fizz, Buzz, 11, Fizz, 13, 14, FizzBuzz, 16, ...
Agenda • Online judge system for introductory programming • Gamifying good code writing • Providing similar program assignments • Conclusion 5 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
6 Problem & Approach ● Most students pay less attention to code quality because most programming courses do not emphasize code quality ○ One of the weak points of a previous online judge ○ How to motivate students to practice writing high-quality code...? ● Grades of class is a strong incentive, but it is undesirable ○ Grading assignments manually from multiple viewpoints (correctness and quality) is difficult and time-consuming ○ No all-around measurement metrics (for automation) ● Gamification + code metrics Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Gamification ● Gamification ○ The use of game design elements in non-game contexts (Deterding, 2011) ○ The goal of gamification is to motivate users' activities ● Gamification in software engineering ● Gamification in programming education ○ e.g. Visualizing learning progress, giving scores for learning activities Score, Level, Badge, Leaderboard, and so on 7 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Related Work ● Code quality in Programming education ○ Compare the code quality of 1st/2nd year students (Breuker, 2011) ○ New online judge framework (Zhou, 2018) ■ personalized feedback, code quality check, code similarity check, teaching adjustment ● Gamification in software engineering ○ Motivate developers to remove warnings of static bug finders (Arai, 2014) ○ Enforce coding conventions (Prause, 2015) ● Gamification in programming education ○ Visualize the progress of learning with progress bars and badges (Olsson, 2015) ○ Mutation testing game (Rojas, 2016) 8 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
10 Approach Gamification + Code metrics Simplify the approach for systematic evaluation ● Code metrics: cyclomatic complexity (CC) ○ Easy for students to understand and use, not too simple ○ Widely used in software analysis ○ You can change metrics in future work ● Gamification element: Leaderboard (Competition) ○ Code metrics do not have absolute threshold -> Comparing score with others instead of the predetermined goal ○ A familiar element in programming contests Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Method & Implementation 2/5 Problems Existing Online Judge Our Extensional Web App DB Leaderboard Code Metrics Tool Read a problem Submit solution code User "Correct" or "Incorrect" 12 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Method & Implementation 3/5 Proble Existing Online Judge Our Extensional Web App DB ms Leaderboard Code Metrics Tool User Measure the CC of the submitted code Using "Lizard" open source code analyser https://github.com/terryyin/lizard 13 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Method & Implementation 4/5 Existing Online Judge Our Extensional Web App DB Leaderboard Code Metrics Tool Sho Pw rob th le em sm s allest CC score of each user User Re-submit (optional) 14 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Method & Implementation 5/5 Existing Online Judge Our Extensional Web App Waseda Online Judge (WOJ) 15 WOJ Standings (WOJS) has used in Waseda University Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Use Case 1/4 A student reads a programming problem "Problem" Write a program which prints the result of FizzBuzz game 16 "Input" "Output" Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Use Case 2/4 The student coding a solution and submits it Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Use Case 3/4 The online judge system shows a judge result "Correct" Passed Test Case 18 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Use Case 4/4 Our extensional web app shows CC score of the submitted code Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
20 Research Questions RQ1 Does an online judge with a leaderboard using the CC change the number of problems solved by each student? RQ2 Does an online judge with a leaderboard using the CC change the CC of code written by students? RQ3 Does an online judge with a leaderboard using the CC change the code quality? Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Evaluation Intervention Our tool is available Intervention Our tool is available Our tool is NOT available Our tool is NOT available 1-2 2-2 3-2 4-2 5-2 6-2 1-1 2-1 4-1 5-1 Problem 1 (Easy) Problem 2 (Medium) C Programming Problems Week 1 Week 4 Group A 17 Group B 18 Participants 35 Overview of the experiment (no rewards for participation) and brief explanation of CC 21 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
19 Fig: Number of problems solved by group in the first half (1-1 to 3-2) and the second half (4-1 to 6-2) Result 1 - Number of Solved Problems Wilcoxon-Mann-Whitney test p=0.5985 p=0.7400 If p<0.05 significant difference Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Discussion 1 RQ1 Does an online judge with a leaderboard using the CC change the number of problems solved by each student? ● We could not find a statistically significant difference both in the first half and in the second half ● The difference in the actual values seems very small ● Our approach does not interfere with students' grades (i.e., the number of problems solved) 20 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Result 2 - CC Score Fig: Summary of the CC score by group for each problem Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Result 2 - CC Score (First Half) CC of Group A (white) tended to be smaller Fig: Summary of the CC score by group for each problem Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Result 2 - CC Score (Second Half) CC of Group B (grey) tended to be smaller Fig: Summary of the CC score by group for each problem Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
24 95% confidence interval [-1.35, -0.07] Fig: standardized mean difference (SMD) of the CC score combined by meta-analysis Result 2 - CC Score (Meta-analysis) Combine [-0.81, -0.01] if effect size < 0 intervention lowered CC Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Discussion 2 RQ2 Does an online judge with a leaderboard using the CC change the CC of code written by students? ● The leaderboard using the CC can reduce the CC of code written during the intervention ● The score and the leaderboard modify behavior in programming education without any additional rewards 25 Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
void A(int x) { if (x == -1) exit(0); } int main(void) { int x; char a[8][3] = {"N", "NW", "W", "SW", "S", "SE", "E", "NE"}; while (1) { scanf("%d", &x); A(x); x = x / 45; printf("%s¥n", a[x]); } return 0; } int main(void) { int n, i, A, E; scanf("%d", &n); for (i = 0; i < n; i++) { scanf("%d %d", &A, &E); if (A + E == 0) { printf("G¥n"); continue; } switch ((A + E) / 10) { case 10: printf("A+¥n"); break; case 9: printf("A+¥n"); break; case 8: printf("A¥n"); break; case 7: printf("B¥n"); break; case 6: printf("C¥n"); break; 26 Problem 2-1 with our tool Problem 5-1 Result 3 - Submitted Code (by Student A6) Small CC Using an array instead of multiple "if", "switch-case" Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
int main(void) { while (1) { int d; scanf("%d", &d); if (d == 0) printf("N¥n"); if (d == 45) printf("NW¥n"); if (d == 90) printf("W¥n"); if (d == 135) printf("SW¥n"); if (d == 180) printf("S¥n"); if (d == 225) printf("SE¥n"); if (d == 270) printf("E¥n"); if (d == 315) printf("NE¥n"); if (d == -1) break; } return 0; } void grade(int point); int main(void) { int N, i; int A, E; scanf("%d¥n", &N); for (i = 0; i < N; i++) { scanf("%d %d¥n", &A, &E); grade(A + E); } return 0; } void grade(int point) { char grade[12][4] = {"G¥n", "F¥n", "F¥n", "F¥n", "F¥n", "F¥n", "F¥n", "C¥n", "B¥n", "A¥n", "A+¥n", "A+¥n"}; point = (point != 0) * (point + 10); point /= 10; printf("%s", grade[point]); } 27 Problem 2-1 Problem 5-1 with our tool Result 3 - Submitted Code (by Student B4) Creating a function Lots of "if" Small CC Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
31 Discussion 3 RQ3 Does an online judge with a leaderboard using the CC change the code quality? ● Several techniques used by the students to reduce the CC ● Undesirable effect of competitive environment ○ One student in group B lowered the CC score to 1 with an array of function pointers void (*f[2])(); f[0] = function_for_false; f[1] = function_for_true; f[!!x](); Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
32 Conclusion ● We propose employing a leaderboard with code metric measurements to motivate students to improve code quality ● A leaderboard using the CC seems not to interfere with students' grades (RQ1) ● Our approach motivates students to improve code metrics without additional rewards (RQ2) ● The competitive environment may cause students to lower their CC greedily (RQ3) Future work ● Other metrics, tools, game elements Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
Agenda • Online judge system for introductory programming • Gamifying good code writing • Providing similar program assignments • Conclusion 33 Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Providing similar program assignments Background – Programming education course in university – Algorithm-related course using online judge Problem – If problem is too difficult, students will give up answering – It takes a lot of time to create hints manually Solution – Improve the learning effect by showing hint based on similar problems Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022. 34
Example (Student Perspective) 35 for(int i=0; ifor(int j=0; jdp[i][j+w[i]] = max( dp[i][j+w[i]], dp[i][j]+v[i]); … } } for(int i=1; ifor(int j=1;jdp[i][j]=min( dp[i][j], dp[i-1][j-1]+s[i]==t[j]?0:1); … } } Assignment: Solve the longest common subsequence problem Similar problem: Solve the knapsack problem If student cannot solve similar reference code Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Propose method（Teacher perspective） 36 Teacher System (1) Send: Correct answer to assignment (2) Recommend: Similar problem of assignment (3) Select: Similar problem to use (4) Recommend: Reference code of similar problem (5) Select: Reference code to use Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Key factors in recommendation • Recommend similar problem – TFIDF • A method used primarily for converting documents to feature vectors • Also used in fields such as code clone • Only the superficial features of the code can be converted, but the implementation and time complexity are light – Cosine similarity • A method for calculating the similarity of feature vectors • Recommend reference code – Cyclomatic Complexity (CC) • There is a correlation between CC and software quality – Number of tokens • Number of ints, +, etc. 37 Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Research question and Experimental Environment RQ – Does the system improve the number of correct answers for students' assignments? Experimental Environment – Population: Course at Waseda University – Course content: Algorithm practice center – Intervention period: 1-5 weeks of lecture (5 weeks) – Correct answer rate: 1-2 weeks of lecture (2 weeks-to make the same conditions as the comparison target) – Questionnaire: 5th week 38 Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Evaluation Comparison of number of correct answers (average for entire class) – Average number of correct answers • p-value(*1): <0.05 – It was suggested that the difficulty level of the assignment could be reduced, and the educational effect could be improved. – Both groups are almost perfect, so the higher the difficulty of the assignment, the wider the difference between the two groups will be. 39 Non-intervention year Intervention year Number of Students 89 80 Number of Students who answered all problems correctly 71 73 Average number of correct answers (9 problems in total) 8.37 8.77 *1. Mann-Whitney U test Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Conclusion and Future work 40 Conclusion • Propose a system that recommend similar problems and reference code • The percentage of correct answers for assignments in the entire course has improved compared to last year • It was suggested that the understanding of the task could be improved, and the educational effect of the assignment could be improved Future work • Expansion of problem collections to search for similar problems Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022.
Conclusion and future prospective • Online judge for introductory programming • Gamifying good code writing – We proposed employing a leaderboard with code metric measurements to motivate students to improve code quality – Our approach motivates students to improve code metrics without additional rewards – Future work: Other metrics, tools, game elements • Providing similar program assignments – We proposed a system that recommend similar problems and reference code – The percentage of correct answers for assignments in the entire course has improved significantly compared to last year – Future work: Expansion of problem collections to search for similar problems 42 Ryoya Yoshimura, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Recommendation System Providing Similar Problems Instead of Model Answers to Programming Assignments,” Applied System Innovation, Vol. 5, pp. 1-18, 2022. Remin Kasahara, Kazunori Sakamoto, Hironori Washizaki, Yoshiaki Fukazawa, “Applying Gamification to Motivate Students to Write High-Quality Code in Programming Assignments,” 24th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2019)
washizaki
harunakano
pylapp
mnori0211
natsuyo
tkimura12
espacio
wagnerst
rmcelreath
eumesy
signer
asial_edu
oscarfeeney
bermonpainter
gr2m
holman
mraible
morganepeng
bluesmoon
garrettdimon
mclloyd
schacon
rocio
philnash
bkeepers