Automated Annotation of Animal Vocalizations

talk given at Bird Song and Animal Communication 2019 conference

Transcript

1. automated annotation of animal vocalizations David Nicholson, Emory University, Biology Dept. Yarden Cohen, Boston University, Biology Dept.
2. Acknowledgements Gardner lab - Yarden Cohen - Alexa Sanchioni - Emily Mallaber - Vika Skidanova Sober lab - Jonah Queen hybrid-vocal-classifier + vak contributors: Varun Saravanan (Sober lab), Roman Ursu (Leblois), Bradley Colquitt + David Mets (Brainard), Ammon Perkes + Marc Badger (Schmidt) Vika Alexa Emily Yarden Jonah
3. Outline 1. Introduction a. Why automate annotation of vocalizations? 2. Methods 3. Results! 4. Discussion a. Details of software, plans for software development (now that you care) b. Why we need open source
4. Introduction Why automate annotation of vocalizations?
5. Introduction Why automate annotation of vocalizations? 1. save time
6. Introduction Why automate annotation of vocalizations? 1. save time 2. answer research questions a. answer usual questions, but increase statistical power
7. Introduction Why automate annotation of vocalizations? 1. save time 2. answer research questions a. answer usual questions, but increase statistical power b. answer new questions we couldn't answer before
8. Introduction What would a good auto-labeler do for us? Criterion Software we developed to meet this criterion • segment audio into vocalizations • predict labels for segments TweetyNet (neural network)
9. Introduction What would a good auto-labeler do for us? Criterion Software we developed to meet this criterion • segment audio into vocalizations • predict labels for segments TweetyNet (neural network) • make it easy for anyone to use vak (library)
10. Introduction What would a good auto-labeler do for us? Criterion Software we developed to meet this criterion • segment audio into vocalizations • predict labels for segments TweetyNet (neural network) • make it easy for anyone to use vak (library) • work with many different data formats vak, crowsetta (libraries)
11. Introduction What would a good auto-labeler do for us? A case study: Label many songs sung by several individuals of two species of songbirds, Bengalese finches and canaries
12. (Brief) Methods 1. benchmark our software on publicly available repositories of Bengalese Finch song ◦ compare error with previously published neural network 2. apply our software to canary song (lengthy bouts, large vocabulary of song syllables) ◦ measure error
13. (Brief) Methods 1. Measure error on a separate test set 2. Plot error as a function of training set size a. what's the lowest error we can get with the least amount of data
14. Methods Error metrics: 1. frame error rate ◦ "For every time bin, does the predicted label equal the true label?" ◦ Frame error is between 0 and 1 2. syllable error rate ◦ "For every predicted sequence, how many labels do I have to change to get back to the original sequence?" ◦ Edit distance, normalized by sequence length to compare across animals ◦ Syllable error rate can be greater than one. It is a distance.
15. Results TweetyNet achieves low frame error across individuals dataset: https://figshare.com/articles/BirdsongRecognition/3470165
16. Results TweetyNet achieves low frame error across individuals dataset: https://figshare.com/articles/BirdsongRecognition/3470165 <- frame error! (between 0 and 1)
17. Results TweetyNet achieves lower syllable error rate with less training data dataset: https://figshare.com/articles/BirdsongRecognition/3470165
18. Results TweetyNet achieves lower syllable error rate with less training data dataset: https://figshare.com/articles/BirdsongRecognition/3470165 <- syllable error! can be greater than 1
19. Results TweetyNet is accurate across large datasets of canary song with many syllables and lengthy bouts
20. Discussion TweetyNet: a hybrid convolutional-recurrent neural network that segments and labels birdsong and other vocalizations https://github.com/yardencsGitHub/tweetynet canary song segmented into phrases
21. TweetyNet: a hybrid convolutional-recurrent neural network that segments and labels birdsong and other vocalizations https://github.com/yardencsGitHub/tweetynet discussion convolutional layers
22. discussion TweetyNet: a hybrid convolutional-recurrent neural network that segments and labels birdsong and other vocalizations https://github.com/yardencsGitHub/tweetynet convolutional layers recurrent layers
23. discussion TweetyNet: a hybrid convolutional-recurrent neural network that segments and labels birdsong and other vocalizations https://github.com/yardencsGitHub/tweetynet convolutional layers recurrent layers output layers Labels
24. discussion Question: how do I use TweetyNet? Doing science is already hard enough, I don't want to have to learn how to program neural networks on top of that
25. vak discussion vak: automated annotation of vocalizations for everybody spectrograms in array files audio files train annotation files Vocalization Dataset predict learning_curve
26. discussion crowsetta A tool to work with any format for annotating vocalizations crowsetta Audactiy .adu Praat textgrid Annotation (data type) your lab's format Transcriber ("scribe") .csv file format file Annotation formats ...
27. Conclusion Open-source community-developed tools provide opportunity for high throughput automated annotation of vocalizations
28. Future work / links Development on Github • https://github.com/yardencsGitHub/tweetynet • https://github.com/NickleDave/vak • https://crowsetta.readthedocs.io/en/latest/ Next version out in time for Neuroscience 2019 in Chicago!
29. Questions / comments