Ikuma Uchida, Atom Scott, Hidehiko Shishido, and Yoshinari Kameda University of Tsukuba [email protected] Automated Offside Detection by Spatio-Temporal Analysis of Football Videos 4th International ACM Workshop on Multimedia Content Analysis in Sports at ACM Multimedia 2021 (ACM MMSPORTS 2021@Chengdu, China) October.20 2021 Computer Vision and Image Media Lab

1 SUMMARY INTRODUCTION METHOD EXPERIMENT /26 OUR PROPOSAL Computer Vision and Image Media Lab DISCUSSION Camera(L) Frames(L) Frames(R) Camera(R) This approach includes the following subtasks ❏ Player and ball tracking ❏ Detection of passers and receivers ❏ Offside detection algorithm We propose a new approach to detect offside automatically from football videos taken by two wide-area cameras input images system overview

2 SUMMARY INTRODUCTION METHOD EXPERIMENT DEMO by OUR SYSTEM Computer Vision and Image Media Lab DISCUSSION /26

❏ If a passer makes a pass and a referee judged that a receiver in an offside position had received the pass, it is a foul. 　 Where is the offside area? ➡The area behind the second player from the back of the defending team Defensive Player Attacking Players FIFA Competition Rules https://www.jfa.jp/rule/ 3 SUMMARY INTRODUCTION METHOD EXPERIMENT OFFSIDE RULE Computer Vision and Image Media Lab DISCUSSION ① Pass ② Receive Offside Area SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

Difficulties in judging offside by visual inspection ❏ Need to judge foul play in an instant ❏ Human error due to player occlusion ❏ Limits of human vision ❏ Accuracy of the judgments Currently available system ❏ Video Assistant Referee(VAR) A complex offside scene https://www.premierleague.com/news/1488423 4 SUMMARY INTRODUCTION METHOD EXPERIMENT RESEARCH BACKGROUND Computer Vision and Image Media Lab DISCUSSION Why do we need an automatic offside detection system? VAR decision https://www.fifa.com/technical/football-technolo gy/standards/video-assistant-referee /26

Detecting offside in an image(Panse et al, MMSPORTS’20) ❏ Pose estimation and team classification of the players in the image ❏ Obtaining the relative coordinates of the pitch and detecting offside Neeraj Panse, A Dataset & Methodology for Computer Vision based Offside Detection in Soccer, MM’Sports 2020 5 SUMMARY INTRODUCTION METHOD EXPERIMENT RELATED WORKS Computer Vision and Image Media Lab DISCUSSION Conventional offside detecting system SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION Limitation: offside is detected on an “image” ⇒Our solution: offside judgement should be made by understanding the dynamic position change of players and balls /26

Automatically detects offside in match videos with 2 cameras (to let the system understand the dynamic position change of players and a ball) 1. Player and ball tracking 2. Detection of passers and receivers 3. Offside detection algorithm 6 SUMMARY INTRODUCTION METHOD EXPERIMENT PURPOSE of THIS RESEARCH Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

7 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

8 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26 ① Player and ball tracking ① ① ② Detection of passers & receivers ② ③ Offside detection algorithm

9 SUMMARY INTRODUCTION METHOD EXPERIMENT Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION PIPELINE of PROPOSED SYSTEM /26

10 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

11 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

12 SUMMARY INTRODUCTION METHOD EXPERIMENT Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION PIPELINE of PROPOSED SYSTEM /26

Camera Calibration with Zhang's Method ①Take a picture so the checkerboard can be seen 　　(around 30 ~ 40 pieces in extent) Corner detection ②Calculate each parameter of the 　　 camera using the feature points ③Distortion correction by calculated parameters 13 SUMMARY INTRODUCTION METHOD EXPERIMENT CALIBRATION of INPUT FRAMES Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

Object Detection with YOLO v5 ❏ Left and right detection for the calibration image ❏ Projection of an object onto the pitch plane using the keypoints of the pitch Visualisation of projection results Right 14 SUMMARY INTRODUCTION METHOD EXPERIMENT OBJECT DETECTION Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

15 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

Detection of pass frames based on ball apparent speed ❏ Corrects the flight trajectory of the ball in the air ❏ Ball speed threshold setting ❏ Linear interpolation for positions above the threshold 16 SUMMARY INTRODUCTION METHOD EXPERIMENT PASS SITUATION DETECTION Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION Flying trajectory Fixed flying trajectory /26

17 SUMMARY INTRODUCTION METHOD EXPERIMENT PASS SITUATION DETECTION Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION velocity noise Detection of pass frames based on ball apparent speed ❏ Noise reduction by moving average ❏ Set appropriate window size to calculate average /26

18 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

❏ Get the colour palette in the BBox ❏ Estimation of the dominant colour from the calculation of the colour difference from the template 19 SUMMARY INTRODUCTION METHOD EXPERIMENT TEAM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION Team clasification /26

❏ Solving the GAP optimally under the constraint of the number of people ➡Constraint: Eleven players per team + 　three referees 20 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION Team clasification /26

Tracking by Kalman Filter and Linear Assignment Problem ❏ Prediction of position by KF assuming constant velocity motion ❏ Optimal assignment of LAPs using the Hungarian method Comparison of KF trajectories with observed values 21 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION Player tracking SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

Player identification based on distance between the ball and players ❏ Based on the results of team classification and tracking ❏ Player identification based on distance between a ball and a player 22 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION Passer & receiver detection SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

Offside decision based on output from previous subtasks 1. Identify the DF as the reference for the offside line 2. Identify potential offside players at the moment the pass is made 3. If the candidate receives a pass 　　➡"Offside" decision 23 SUMMARY INTRODUCTION METHOD EXPERIMENT PIPELINE of PROPOSED SYSTEM Computer Vision and Image Media Lab DISCUSSION Offside algorithm ① ② ③ SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

We recorded match videos and collect offside scenes ● 3 x 90 min matches taken by 2 wide-view cameras (30 fps) ● 10 offside scenes, 10 without offside, total 20 scenes (30 sec for each) (Corner kicks and throw-ins are excluded) 24 SUMMARY INTRODUCTION METHOD EXPERIMENT DATASETS Computer Vision and Image Media Lab DISCUSSION Camera(L) Frames(L) Frames(R) Camera(R) Datasets SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

Metrics for evaluation ❏ Evaluation of the accuracy of offside decisions ➡ Use Precision, Recall and F1 Score Results ❏ Precision: 60%, ❏ Recall: 67% ❏ F1 Score: 63.1% 25 SUMMARY INTRODUCTION METHOD EXPERIMENT RESULTS Computer Vision and Image Media Lab DISCUSSION Metrics for evaluation & Results SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26

26 SUMMARY INTRODUCTION METHOD EXPERIMENT CONCLUSION Computer Vision and Image Media Lab DISCUSSION We propose a new approach to detect offside automatically from football videos taken by two wide-area cameras 1. Player and ball tracking 2. Detection of passer & receiver 3. Offside detection algorithm SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /26 ① Player and ball tracking ① ① ② Detection of passer & receiver ② ③ Offside detection algorithm

How improve the result? ❏ Improve tracking accuracy ❏ ID switch problem ➡ Consideration of features other than positional coordinates 　　as KF observables ❏ Increase the number of cameras ❏ Large errors in the positions of the players at a distance from the camera ➡ Set up a camera on the other side of the pitch Reinforcing the baseline ❏ More datasets to use 24 SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION /15 Computer Vision and Image Media Lab DISCUSSION Discussion SUMMARY INTRODUCTION METHOD EXPERIMENT DISCUSSION