Presented By: Gregory Ditzler(1,2) (1)Department of Electrical & Computer Engineering (2)Machine Artificial Intelligence and Virtual Reality Center (MAVRC) Rowan University Glassboro, NJ DyViR: Dynamic Virtual Reality Dataset for Aerial Threat Object Detection Garrett Williams, George D. Lecakes Jr., Amanda Almon, Nikolas Koutsoubis, Kyle Naddeo, Thomas Kiel, Gregory Ditzler, and Nidhal C. Bouaynaya 

Outline ● UAV Threat/Object Detection using ML ● Motivating Data Augmentation using Synthetic Data ● Real-Time Rendering Technology ● What is DyViR and what does it produce? ○ YOLOv7-tiny Training and Validation ○ Results from YOLOv7-tiny ML models ● Conclusions and Future Directions 

The Bigger Picture ● Design & develop Virtual and Augmented Reality environments of armored and tactical vehicles, and threat scenarios. ● Develop AI/ML-based systems to enhance turret operations and to interrogate potential threats. ● Build situational awareness systems to provide secure, untethered, high-speed wireless communication between the vehicle and the soldier. Support by a grant from the Army Research Office W15QKN-21-C-0077. 

The Bigger Picture ● Design & develop Virtual and Augmented Reality environments of armored and tactical vehicles, and threat scenarios. ● Develop AI/ML-based systems to enhance turret operations and to interrogate potential threats. ● Build situational awareness systems to provide secure, untethered, high-speed wireless communication between the vehicle and the soldier Support by a grant from the Army Research Office W15QKN-21-C-0077. Sensors Human Factors VR/AR/MR AI/ML Sys. Int. 

UAV Object Detection using ML ● Unmanned Aerial Vehicles (UAVs) pose a major threat on the modern combat landscape ○ These objects tend to be small (e.g., contained on a small number of pixels), and ○ There are limited datasets available to learn an object detector. ■ Number of samples ■ Types of UAVs ■ Biomes where data are collected ■ Etc. ● Machine Learning (ML) is a potential solution in identifying and classifying friendly and hostile UAVs in large swarms; however, limited data will limit the performance of any ML model. 

Motivating Synthetic Data Augmentation ● The lack of datasets with UAVs in combat landscapes due to safety concerns and the emergence of new UAV technologies ● Synthetic training datasets are able to augment real datasets, allowing for training on large amounts of balanced data 

Real-Time Rendering Technology ● Advancements in computer graphics allow for the realistic renderings of 3D digital environments produced in real-time (30 frames per second +) ● Imagery of custom 3D assets in a simulated 3D environment is created by software such as the Unity Real-Time Development Platform (Unity) ○ The developer of the datasets are no longer limited by the biome, types of drones, etc. ○ This allows for users to develop more diverse datasets ● Real-time rendering allows for the rapid production of synthetic datasets 

What is DyViR and what does it produce? ● DyViR creates synthetic training datasets for UAV object detection built on the Unity Engine ● The 2D bounds of the synthetic UAVs is calculated within the engine ● Imagery and bounding box data is linked to produce training datasets in a compatible format for ML models Outputs images from Flaticon.com Compatible with YOLO formats 

DyViR Produced Datasets 

YOLOv7-tiny Training and Validation ● The You Only Look Once (YOLO) model is designed for object detection ● The YOLOv7-tiny model is a lightweight variant suitable for deployment on edge devices ○ YOLOv7-tiny classifies objects and places a bounding box identifying the extents of the object ● The Mean Average Precision (mAP) score is a metric for determine the performance of object detection 

● A real-world and DyViR model were trained, the former exclusively on real-world data and the latter trained first on synthetic DyViR data and fine-tuned on real-world Results from YOLOv7-tiny ML models Real-World Model images from Flaticon.com DyViR Model Train Fine-Tune Train 

Results from YOLOv7-tiny ML models 

● Over 100 Epochs, the DyViR model saw a 60.4% increase in the mAP 0.5:0.95 score over its real-world counterpart (0.653 to 0.407) Results from YOLOv7-tiny ML models 

● Access to real-world data to build a robust and trustworthy machine learning model can be challenging when real-world data are scarce ○ Real-world data can be challenging or near impossible to collect ○ Collecting data that covers all p(x) can be challenging regarding ■ Different biomes, UAVs, weather, etc. ● DyViR allows for rapid development of new datasets for aerial object detection. ● Future Work: DyViR intends to produce datasets that have more realistic flight patterns, allowing for object detection-based on temporal changes ○ Develop software to superimpose VR drones over real-scenes ○ Investigate curriculum learning methods to boost performance Conclusion 

Thank you! Questions? This work was supported by grants from the Army Research Office W15QKN-21-C-0077, and National Science Foundation CAREER #1943552. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the sponsors' views. Data https://github.com/RowanMAVRC/DyViR-For-Unity-SPIE-2023