Leveraging AI for Scene Detection in Video By learning a distance measure between shots By: Rik Heijdens 

Video Structure 

Scene Detection The task of finding Logical Story Units in Video Why? ● Automatic content indexing of (large) video libraries ● Automatic advertisement insertion 

Architecture Overview 

Feature extraction 

Extracting Visual Features ● Encode frames using Google's Inception CNN 

Extracting Audible Features ● Audio is often used to underline the development of a story ● Short-time Fourier Transforms (STFTs) ● Mel-scaled power spectrograms S. Dieleman et al. "End-to-end learning for music audio" (ICASSP 2014) 

Extracting Textual Features Extracted from transcripts Word2Vec embeddings 

Feed the extracted features into a Neural Network 1. Concatenate all the features into a single dense feature vector. 2. Feed this feature vector into a multimodal neural network that learns how to weight the components and maps high dimensional feature vectors into lower dimensional shot embeddings. 

Clustering Similarity Matrix 

Plot of similarity scores for temporally adjacent shots Clustering 

Clustering Scene Breaks and their confidence score plotted on top of the Similarity Matrix 

Questions? Approach me after the talk or send an email to [email protected]