A Single-Step Approach to Musical Tempo Estimation Using a Convolutional Neural Network

We present musical tempo estimation system based solely on a convolutional neural network (CNN). Contrary to existing systems our system estimates the tempo directly from a conventional mel-spectrogram in a single step. This is achieved by framing tempo estimation as a multi-class classification problem using a network architecture that is inspired by conventional approaches. The system’s CNN has been trained with the union of three datasets covering a large variety of genres and tempi using problem-specific data augmentation techniques. As input the system requires only 11.9s of audio and is therefore suitable for local as well as global tempo estimation. When used as a global estimator, it performs as well as or better than other state-of-the-art algorithms. Especially the exact estimation of tempo without tempo octave confusion is significantly improved. As local estimator it can be used to identify and visualize tempo drift in musical performances.
https://www.youtube.com/watch?v=w-fsuRbAVuo&t=1h21m55s

Transcript

1. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   A Single-Step Approach to Musical Tempo Estimation
   Using a Convolutional Neural Network
   Hendrik Schreiber

   tagtraum industries incorporated
   hs@tagtraum.com / @h_schreiber
   Meinard Müller

   International AudioLabs Erlangen
   meinard.mueller@audiolabs-erlangen.de
   

   View Slide

2. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   Tempo Estimation System
   Spectrogram
   Onset/Beat

   Detection
   Tempo Estimation
   Traditional System
   

   View Slide

3. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   Tempo Estimation System
   Mel-Spectrogram
   CNN-based Tempo
   Classification
   Proposed System
   “eliminating the middle-man”
   

   View Slide

4. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   CNN Architecture
   mel-spectrogram
   short pattern
   matching
   avg-pooling 5x1
   avg-pooling 2x1
   avg-pooling 2x1
   avg-pooling 2x1
   mf_mod
   mf_mod
   mf_mod
   mf_mod
   long pattern
   matching classification
   stepwise pooling along
   the frequency axis,
   long filters along time axis
   Input, 1x40x256
   Output, 256
   Batch Normalization Convolutional Layer
   Fully Connected Layer DO Dropout
   short filters
   along time
   axis
   dense layers,
   softmax
   mel-spectrogram
   short pattern
   matching
   avg-pooling 5x1
   avg-pooling 2x1
   avg-pooling 2x1
   avg-pooling 2x1
   mf_mod
   mf_mod
   mf_mod
   mf_mod
   long pattern
   matching classification
   stepwise pooling along
   the frequency axis,
   long filters along time axis
   Input, 1x40x256
   Output, 256
   Batch Normalization Convolutional Layer
   Fully Connected Layer DO Dropout
   short filters
   along time
   axis
   dense layers,
   softmax
   

   View Slide

5. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   CNN Architecture
   mel-spectrogram
   short pattern
   matching
   avg-pooling 5x1
   avg-pooling 2x1
   avg-pooling 2x1
   avg-pooling 2x1
   mf_mod
   mf_mod
   mf_mod
   mf_mod
   long pattern
   matching classification
   stepwise pooling along
   the frequency axis,
   long filters along time axis
   Input, 1x40x256
   Output, 256
   Batch Normalization Convolutional Layer
   Fully Connected Layer DO Dropout
   short filters
   along time
   axis
   dense layers,
   softmax
   mel-spectrogram
   short pattern
   matching
   avg-pooling 5x1
   avg-pooling 2x1
   avg-pooling 2x1
   avg-pooling 2x1
   mf_mod
   mf_mod
   mf_mod
   mf_mod
   long pattern
   matching classification
   stepwise pooling along
   the frequency axis,
   long filters along time axis
   Input, 1x40x256
   Output, 256
   Batch Normalization Convolutional Layer
   Fully Connected Layer DO Dropout
   short filters
   along time
   axis
   dense layers,
   softmax
   avg-pooling
   BN
   CONV
   24x1x32
   CONV
   24x1x64
   CONV
   24x1x96
   CONV
   24x1x128
   CONV
   24x1x192
   CONV
   24x1x256
   concatenation
   mf_mod
   CONV
   36x1x1
   next layer
   previous layer
   Pons, Jordi, and Xavier Serra. "Designing efficient architectures for modeling temporal features with convolutional neural networks.”
   IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, USA, March 2017.
   Inspired by
   Pons and Serra
   

   View Slide

6. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   CNN Architecture
   mel-spectrogram
   short pattern
   matching
   avg-pooling 5x1
   avg-pooling 2x1
   avg-pooling 2x1
   avg-pooling 2x1
   mf_mod
   mf_mod
   mf_mod
   mf_mod
   long pattern
   matching classification
   stepwise pooling along
   the frequency axis,
   long filters along time axis
   Input, 1x40x256
   Output, 256
   Batch Normalization Convolutional Layer
   Fully Connected Layer DO Dropout
   short filters
   along time
   axis
   dense layers,
   softmax
   mel-spectrogram
   short pattern
   matching
   avg-pooling 5x1
   avg-pooling 2x1
   avg-pooling 2x1
   avg-pooling 2x1
   mf_mod
   mf_mod
   mf_mod
   mf_mod
   long pattern
   matching classification
   stepwise pooling along
   the frequency axis,
   long filters along time axis
   Input, 1x40x256
   Output, 256
   Batch Normalization Convolutional Layer
   Fully Connected Layer DO Dropout
   short filters
   along time
   axis
   dense layers,
   softmax
   avg-pooling
   BN
   CONV
   24x1x32
   CONV
   24x1x64
   CONV
   24x1x96
   CONV
   24x1x128
   CONV
   24x1x192
   CONV
   24x1x256
   concatenation
   mf_mod
   CONV
   36x1x1
   next layer
   previous layer
   Pons, Jordi, and Xavier Serra. "Designing efficient architectures for modeling temporal features with convolutional neural networks.”
   IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, USA, March 2017.
   Inspired by
   Pons and Serra
   Pooling along
   frequency axis
   Convolution along
   time axis
   

   View Slide

7. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   Training Datasets
   • LMD Tempo—derived from Lakh MIDI Dataset [1]
   • MTG Tempo—tempo annotations for MTG Key [2]
   • Eball—Extended Ballroom [3] sans Ballroom
   [1] Colin Raffel. "Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching". PhD Thesis, 2016.

   [2] Ángel Faraldo, Sergi Jordà, and Perfecto Herrera. A multi-profile method for key estimation in EDM. In Proceedings of the AES International Conference on Semantic Audio, Erlangen, Germany, June 2017.

   [3] Ugo Marchand and Geoffroy Peeters. The extended ballroom dataset. In Late Breaking Demo of the International Conference on Music Information Retrieval (ISMIR), New York, NY, USA, 2016.
   NEW!
   NEW!
   Combined: 8,596 samples
   

   View Slide

8. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   Scale & Crop Data Augmentation
   Requires label
   adjustment!
   

   View Slide

9. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
   A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
   Local Tempo Estimation
   “Honky Tonk Women” by The Rolling Stones
   

   View Slide

10. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Local Tempo Estimation
    “Honky Tonk Women” by The Rolling Stones
    

    View Slide

11. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Local Tempo Estimation
    “Typhoon” by Foreign Beggars/Chasing Shadows
    

    View Slide

12. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Local Tempo Estimation
    “Typhoon” by Foreign Beggars/Chasing Shadows
    

    View Slide

13. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Global Tempo Estimation
    On ACM Mirum, ISMIR04, Ballroom, Hainsworth, GTzan, SMC, GiantSteps
    Combined: 74.2% (‐from 69.5%)
    Dataset Avg: 69.3% (‐from 66.7%)
    Accuracy1
    

    View Slide

14. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Global Tempo Estimation
    On ACM Mirum, ISMIR04, Ballroom, Hainsworth, GTzan, SMC, GiantSteps
    Combined: 92.1% (‑from 93.6%)
    Dataset Avg: 86.4% (‑from 89.9%)
    Accuracy2
    

    View Slide

15. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Summary
    

    View Slide

16. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Summary
    • Consolidates multi-component approach
    • Completely data-driven, no heuristics
    • Fewer octave-errors
    • Suitable for global and local tempo estimation
    

    View Slide

17. INTERNATIONAL AUDIO LABORATORIES ERLANGEN
    A joint institution of Fraunhofer IIS and Universität Erlangen-Nürnberg
    Thank you.
    New datasets are available at:
    http://www.tagtraum.com/tempo_estimation.html
    Tempo estimation code is available at:
    https://github.com/hendriks73/tempo-cnn
    

    View Slide