Environmental sound recognition with machine learning

Tatsuya Komatsu
LINE Speech Team Senior Researcher
https://linedevday.linecorp.com/2020/jp/sessions/5825
https://linedevday.linecorp.com/2020/en/sessions/5825

Transcript

2. Recognition of environmental sounds
3. Recognition of environmental sounds Speech
4. Recognition of environmental sounds Speech Coughing
5. Recognition of environmental sounds Speech Coughing Door knocking
6. Recognition of environmental sounds Scream Speech Coughing Door knocking
7. Example: Speech recognition Recognize “I've Never Been Out of the Village Before” by only-bee/CC BY 3.0 I‘ve Never Been…
8. Environmental sounds can tell us richer information Traffic noise Music People crowd I‘ve Never Been … Recognize “I've Never Been Out of the Village Before” by only-bee/CC BY 3.0
9. Large attention for sound-applications Automatic tagging of multi-media data › Diverse categories of environmental sounds City Surveillance › Scream › Shouting › Glass breaking Home Monitoring › Speech › Dog barking › home appliances
10. Large attention in the research field Annual international competition and workshop; DCASE 2020 2019 2018 2017 2016 0 100 200 300 400 500 Number of participant
11. Large attention in the research field Annual international competition and workshop; DCASE DCASE Community › Annual workshop › Workshop for environmental sound analysis › Increasing participants year by year
12. Large attention in the research field Annual international competition and workshop; DCASE Organizing committee member
13. DCASE Challenge: Annual international competition › An annual public evaluation event to accelerate the development of the research field › Open dataset › Open baseline method › Public benchmark
14. DCASE Challenge: Annual international competition › An annual public evaluation event to accelerate the development of the research field › LINE joined the challenge as the joint team with Nagoya Univ. and Johns Hopkins Univ. LINE Internship Student 2019
15. › Domestic sound recognition for sounds from YouTube and Vimeo › Noisy, diverse characteristics and low-quality sound labels DCASE2020 Challenge task4 Domestic-Environment-Sound Recognition Golden Retriever Dog Luke sings with piano music https://www.youtube.com/watch?v=-hDuDDv0lbQ crying baby https://www.youtube.com/watch?v=-3UBJIEYKVg
16. Result: 1st place!!! › 1st place among 21 teams, 72 system submission › 14.6 % higher than Baseline system › 3.3 % higher than 2nd place team submission Our team http://dcase.community/challenge2020/task-sound-event-detection-and-separation-in-domestic-environments-results
17. Environmental sound recognition and Our method that won 1st place
18. Environmental sound recognition = Understand “What situation, what sound” Wave signal Recognition Output Speech Coughing Door knocking Screaming
19. How can we handle sound data? Wave signal
20. How can we handle sound data? Wave signal › The wave signal itself is not so informative (for us)
21. Frequency analysis Wave signal Spectrogram Time Frequency
22. Frequency analysis Wave signal Spectrogram Can be handled like images Speech Coughing Door knocking Screaming Time Frequency
23. General procedure: Environmental sound recognition Recognition Spectral Feature Post-processing Spectral Model Sound Event
24. Characteristic of sound data 振幅 周波数 › Both spectral (local) and temporal (global) information are important › Sounds occur simultaneously, i.e. Overlapping → One of the key solution = Source separation Global information Local information Next Session
25. Convolutional-Recurrent Neural Networks › CNNs: Spectral (local) feature › RNNs: Temporal (global) feature › Almost all teams in DCASE2020 employs this CRNN-based method GRU Layer GRU Layer Sound input Time Frequency CNN Block CNN Block CNN Block Sound Classifier Recognition results Spectral information Temporal Information
26. Environmental sound recognition and Our method that won 1st place
27. › Domestic sound recognition for sounds from YouTube and Vimeo › Noisy, diverse characteristics and low-quality sound labels DCASE2020 Challenge task4 Domestic-Environment-Sound Recognition Golden Retriever Dog Luke sings with piano music https://www.youtube.com/watch?v=-hDuDDv0lbQ crying baby https://www.youtube.com/watch?v=-3UBJIEYKVg
28. › Sounds are not visible: Hard to annotate Challenges: Weakly labeled training Dog!! ? Where is dog?? Hard to make (strong) labels…
29. › Sounds are not visible: Hard to annotate Challenges: Weakly labeled training Dog!! Weak label ‘Dog is somewhere in this sound’
30. Challenges: Unlabeled and robustness › Recognition task for sounds in the ``wild’’ › Unlabeled data training for effective use of huge amounts of data on the web › Robust model to handle sounds with diverse characteristics Baby crying Music People speech
31. Our approach: Self-attention based weak supervised method › Self-attention (Transformer); outstanding performance in various fields (NLP, ASR,,,) › First application to this field [Miyazaki*+,2020] *LINE summer internship 2019 › Can capture global information effectively Multi-head Self-attention Sound input Time Frequency Sound Classifier Weak label estimation Neural Feature Extraction Stacked transformer encoder Feed Forward Sound Classifier Recognition results Special token for weak label × n times Concat CNN-based Feature extraction
32. Our approach+: Convolution-augmented transformer › Capture local and global information with CNNs and self-attention [Gulati+,2020] Feed Forward Conformer encoder Multi-head Self-attention Convolution Module Feed Forward Multi-head Self-attention Transformer encoder Feed Forward Additional Convolution module
33. For improvement of the performance: Unlabeled data training › Mean teacher [Tarvainen+, NIPS2017] Tarvainen et al., "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." in NIPS 2017. Teacher model Student model Recognition results Recognition results Sound input Parameter θ Moving average θ* < add noise/augment < add noise/augment Consistency loss
34. For improvement of the performance: Robust models by data augmentation › Time-shifting › Noise adding › Mix-up [Zhang+,2020] › Frequency masking
35. › Customized (Convolution-augmented) Transformer for weakly supervised training › +Mean-teacher for unlabeled training › +Data augmentation with time-shifting, frequency masking, noise adding and mix-up. Our method Sound input Sound Classifier Weak label estimation Neural Feature Extraction Stacked Conformer encoder Sound Classifier Recognition results Special token for weak label × n times Concat CNN-based Feature extraction Feed Forward Multi-head Self-attention Convolution Module Feed Forward
36. Result: 1st place among 72 systems from 21 teams › The implementation to reproduce the competition results will be public › ESPnet: end-to-end speech processing toolkit [https://espnet.github.io/espnet/] › 14.6 % higher than Baseline system › 3.3 % higher than 2nd place team submission Our team
37. Recent research activities in LINE
38. Barriers to Practical Applications › Data mismatch by recorded devices: diverse characteristics › ‘Meaning of sounds’ depends on scene › Richer information; want to know where the sounds come from.
39. Data mismatch depends on recording devices › Everybody can record sounds → Sounds are recorded by diverse devices › Spectral characteristic of sound data: Dependent on codecs, devices, environments,,, › Difficult to handle it as training data e.g. Differences by codec Airport - London 0 1 5 2 3 4 Frequency [kHz] 10 5 0 MJOFBS 1$. Airport - London 0 1 5 2 3 4 Frequency [kHz] 10 5 0 0 1 5 2 3 4 Time [sec] Frequency [kHz] 10 5 0 MJOFBS 1$. PHH 7PSCJT
40. Device- and codec-invariant classification using domain adaptation and knowledge distillation › Domain adaptation + knowledge distillation technique [Takeyama+, 2020] Takeyama et al., “Robust Acoustic Scene Classification to Multiple Devices Using Maximum Classifier Discrepancy and Knowledge Distillation” in EUSIPC02020 Device A model Device C model Device B model Student Model Knowledge distillation Robust for all devices Build by domain adaptation
41. Meaning of sounds depends on scenes Normal Anomaly!! › Same sound has different meaning in different scenes [Komatsu+, 2019, 2020]
42. Scene-aware sound recognition › Multi-task method for acoustic event detection and scene classification [Komatsu+,2020] › Condition event classifier using the estimated scene Multi-task network 58% lower error rate Komatsu et al., “SCENE-DEPENDENT ACOUSTIC EVENT DETECTION WITH SCENE CONDITIONING AND FAKE-SCENE-CONDITIONED LOSS” in ICASSP 2020 Shared feature Scene estimation Recognition Scene estimation Recognition results Condition!!
43. Localization of environmental sounds › Recognize and localize at the same time. › Use Multiple microphones and ‘phase’ information Sounds from Multiple microphones GRU Layer GRU Layer Amplitude & Phase Spectrogram Time Frequency Channel*2 CNN Block CNN Block CNN Block Fully Connected time Event class Event class time Recognition Fully Connected Localization Azimuth&Elevation Event Class Probability SED Layers DoA Estimation Layers Spectral Feature Extraction Temporal Information
44. Sound localization and detection with gated linear units (GLUs) › Focus on the differences in information required for classification and localization › New mechanism to automatically control input information by GLUs [Komatsu+, 2020] Komatsu et al., “Sound Event Localization and Detection using a Recurrent Convolutional Neural Network and Gated Linear Unit” in EUSIPC02020 Improved performance for both Batch norm Max pooling Drop out Sigmoid Batch norm Linear input GLU block Control information to be used Proposed feature extraction
45. Summary › Environmental sound recognition: One of the hottest research field of sounds › 1st place in the DCASE2020 Challenge with the joint team of Nagoya University, Johns Hopkins University and LINE › More advanced research activities in LINE › Codec-invariant environmental sound analysis › Scene-aware environmental sound recognition › Recognition and localization of environmental sounds
46. Thank you