VocalPy: a core Python package for acoustic communication research

Talk given at PyData Global 2023
https://global2023.pydata.org/cfp/talk/PFYEQQ/

Transcript

1. A core Python package for researchers studying acoustic communication David Nicholson PyData Global 2023 @nicholdav NickleDave @nicholdav@fosstodon.org nicholdav.info
2. Acknowledgements Dr. Yarden Cohen Dept of Brain Sciences Weizmann Institute of Science, Israel https://github.com/NeuralSyntaxLab https://www.weizmann.ac.il/brain-sciences/labs/cohen/ @YardenJCohen
3. A core package for animal acoustic communication research in Python VocalPy
4. A core package for animal acoustic communication research in Python VocalPy
5. A core package for animal acoustic communication research in Python VocalPy Adapted from: Bass Chagnaud 2012, Branstetter et al. 2016, Chen et al. 2020
6. Acoustic behavior and animal communication What makes us human? Language and speech How is speech like birdsong and bat calls? How did speech evolve? How do animals learn their vocalizations? Boraud Leblois Rougier 2014 https://inria.hal.science/hal-01874690
7. A core package for animal acoustic communication research in Python VocalPy • interdisciplinary (Wirthlin et al. 2019) • big team science (Hauser et al. 2002) • big data, automated analyses • → cutting edge computational methods ◦ Deep learning, AKA neural network models (Sainburg et al 2021, Stowell 2022) Hauser et al. 2002
8. • pykanto: https://github.com/nilomr/pykanto • koe: https://github.com/fzyukio/koe • Hundreds of different libraries and different formats for audio, array data, annotations: https://github.com/rhine3/bioacoustics-software Do we really need a core package?
9. A core package for animal acoustic communication research in Python Goals: • Develop a library that is general and robust • Enable better collaboration VocalPy
10. A core package for animal acoustic communication research in Python Features: • Data types • Works with a wide variety of formats • Classes for common steps in workflows • Designed for reproducibility • Allows scientist-coders to interactively build datasets then share as a database VocalPy
11. VocalPy • Built on the scientific Python stack (thank you NumFocus!) ◦ numpy ◦ scipy ◦ matplotlib ◦ pandas ◦ dask ◦ xarray
12. Case study: TweetyNet • A neural network model o Automates annotations o Avoids issues with segmenting audio Cohen et al., https://elifesciences.org/articles/63853
13. Annotating animal sounds manually, with a GUI
14. Annotating animals sounds Annotation requires segmenting audio into a sequence of units (Kershenbaum et al. 2016) Off-the-shelf signal processing algorithms for segmenting often don't work
15. Annotating animals sounds Off-the-shelf signal processing algorithms for segmenting often don't work
16. Annotating animals sounds Off-the-shelf signal processing algorithms for segmenting often don't work
17. Annotating animals sounds Off-the-shelf signal processing algorithms for segmenting often don't work → Neural network model
18. Statistical models of acoustic behavior Syntax McGregor et al. 2022 Cohen et al. 2022 Koparkar et al. 2023 Yang et al. 2022 Dialect Motor learning Fit to data annotated with TweetyNet
19. Case study: TweetyNet • TweetyNet, the code ◦ Started as a single Jupyter notebook, written in Tensorflow.
20. Case study: TweetyNet To be good computational scientists, we developed a neural network framework: vak • SciPy 2023 talk: https://www.youtube.com/watch?v=tpL0 m5UwpZM • proceedings paper: https://conference.scipy.org/proceeding s/scipy2023/david_nicholson.html Version 1.0 in alpha now • Lightning backend • Better abstractions for models, tasks, and datasets https://github.com/vocalpy/vak
21. To work with annotation formats, we developed crowsetta: https://github.com/vocalpy/crowsetta • now a pyOpenSci package ◦ https://www.pyopensci.org ◦ published in Journal of Open Source Software • PyCon 2023 lightning talk: https://youtu.be/54q_cPCNNS8?list=PL2 Uw4_HvXqvY2zhJ9AMUa_Z6dtMGF3gt b&t=1082 Case study: TweetyNet
22. Case study: TweetyNet • TweetyNet (the code) ◦ Separate repository to replicate results in paper: https://github.com/yardencsGitHub/tweetynet ▪ people often asked us for functionality ▪ that was hidden in this code -- hard to read ▪ and it was hard for them to adapt to their data
23. What's the problem? Hundreds of different libraries and different formats for audio, array data, annotations: https://github.com/rhine3/bioacoustics-software
24. What's the problem? Challenges: • GUIs ◦ Don't capture parameters / steps in analysis ◦ Other issues: proprietary language / no longer developed / single lab or PI or developer / closed source or source not easily accessible • Analysis scripts and libraries ◦ Deal with many low level details ◦ Because of data formats ◦ → dataset preparation + analysis tightly coupled to formats ◦ scripts are hard to read and hard for other groups to re-use ◦ libraries are hard for user to configure
25. What's the problem?
26. What's the problem? Key parameters hidden in defaults
27. What's the problem? Key parameters hidden in defaults Lack of data types leads to proliferation of variables
28. VocalPy
29. Data types for acoustic communication vocalpy.Audio • works with a wide array of formats Helper function to get paths to all audio files
30. Data types for acoustic communication vocalpy.Audio • works with a wide array of formats Create list of Audio instances with read method
31. Data types for acoustic communication vocalpy.Audio • works with a wide array of formats Audio encapsulates signal data along with samplerate and channels
32. Data types for acoustic communication vocalpy.Audio • works with a wide array of formats Audio captures metadata like path
33. vocalpy.Spectrogram • save expensive-to-compute spectrograms in array files Data types for acoustic communication Spectrogram encapsulates data with frequencies and times
34. Data types for acoustic communication vocalpy.Spectrogram • plot and inspect
35. Data types for acoustic communication vocalpy.Annotation • Load many different annotation formats using Crowsetta
36. Data types for acoustic communication • vocalpy.Annotation ◦ Plot with spectrogram
37. Classes for common steps in workflows vocalpy.Segmenter for segmentation of audio into sequences of units (Kershenbaum et al. 2016)
38. Classes for common steps in workflows vocalpy.Segmenter for segmentation of audio into sequences of units (Kershenbaum et al. 2016) Encourages explicit declaration of parameters
39. Classes for common steps in workflows vocalpy.Segmenter for segmentation of audio into sequences of units (Kershenbaum et al. 2016) Callbacks allow re-use of code
40. Classes for common steps in workflows vocalpy.SpectrogramMaker to compute spectrograms
41. Classes for common steps in workflows vocalpy.SpectrogramMaker to compute spectrograms Parallelize with
42. Build datasets interactively, share as databases vocalpy.datasets.SequenceDataset
43. Build datasets interactively, share as databases vocalpy.datasets.SequenceDataset
44. Build datasets interactively, share as databases vocalpy.datasets.SequenceDataset • Single-file database • built into Python • archival format
45. Results
46. Benchmarking neural network models Poster presented at Society for Neuroscience meeting 2023 https://github.com/vocalpy/Nicholson-Cohen-SfN-2023-poster
47. Benchmarking neural network models Cohen et al., https://elifesciences.org/articles/63853 Steinfath et al., https://elifesciences.org/articles/68837
48. Experimental set-up
49. Experimental set-up
50. Experimental set-up
51. Window size matters
52. Evaluating segmentation with VocalPy Evaluate with vocalpy.metrics.segmentation
53. Window size matters
54. TCNs tend to oversegment
55. Additional loss terms mitigate oversegmentation
56. Work in Progress
57. How do we measure song similarity? "A procedure for an automated measurement of song similarity", Tchernichovski et al. 1999
58. How do we measure song similarity? (predefined acoustic) features (!) • soundanalysispro.com ◦ http://soundanalysispro.com/matlab-sat • https://github.com/PaulEcoffet/birdsonganalysis • https://github.com/theresekoch/avn
59. Predefined Acoustic Features vocalpy.feature.sat
60. Predefined Acoustic Features vocalpy.feature.sat See also: https://github.com/theunissenlab/soundsig https://github.com/NeuroBatLab/SoundAnalysisBats
61. Predefined acoustic features: a use case for xarray
62. Roadmap
63. Roadmap Working towards a community-developed software for acoustic communication research Core package Annotation formats Neural networks
64. Roadmap: VocalPy community • We already have many contributors (thank you to all of you!)
65. Roadmap: VocalPy community • We already have a growing forum: https://forum.vocalpy.org/
66. Ways you can contribute • Star our repositories on GitHub • Join the forum ◦ Ask and answer questions ◦ Share examples • Contribute code • Attend development meetings
67. Links • VocalPy organization on GitHub: https://github.com/vocalpy ◦ https://github.com/vocalpy/vocalpy ◦ https://github.com/vocalpy/vak ◦ https://github.com/vocalpy/crowsetta • VocalPy forum: https://forum.vocalpy.org/ @nicholdav NickleDave @nicholdav@fosstodon.org nicholdav.info
68. References (that are not linked elsewhere) • Bass Chagnaud 2012: https://www.pnas.org/doi/abs/10.1073/pnas.1201886109 • Chen Wiens 2020: https://www.nature.com/articles/s41467-020-14356-3 (but see also Jorgewich-Cohen et al 2022 https://www.nature.com/articles/s41467-022-33741-8) • Wirthlin et al 2019: https://www.sciencedirect.com/science/article/pii/S0896627319308396 • Hauser Fitch Chomsky 2002: https://www.science.org/doi/full/10.1126/science.298.5598.1569?casa_token=iB9i4_rZEvUAAAAA%3AfhEc BMWcpTjRpmIrAFaxTe361utCWj-kB6eGvQ1UikCcPz6pDrtiWKywJJdJWl95pa2FwXJusxZ5xoc • Sainburg Gentner 2020: https://www.frontiersin.org/articles/10.3389/fnbeh.2021.811737/full • Stowell 2020: https://peerj.com/articles/13152/ • Kershenbaum et al. 2016: https://onlinelibrary.wiley.com/doi/abs/10.1111/brv.12160?casa_token=OzzQcd7O9AQAAAAA:jkcnLBuWBlia 0tHFmffTMziy9cXtAVwVraJm43mw7GQgDqGsmpZ-9omjv4X6FABVnd7KMcZST1Gl8mA