Intro to Acoustic Processing in Android by Serge Masse

Description of processing in Android Java for sound input to analysis and sound synthesis to output.

https://gdgmontreal.com/2020/01/22/february-meetup-2/

Transcript

1. Intro to Acoustic Processing in Android Serge Masse 2020
2. This talk is about • Acoustic processing: • Sound input to analysis • Sound synthesis to output
 • Not playing music/sound clips
3. 1. Input flow: 1. a2d analog to digital 2. capture: subset of voltage samples 3. analysis: FFT, wavelet, noise, ANN, recognition/ classification 4. Data, storage, sharing 2. Output flow: data, synthesis, d2a 3. Android issues
9. Output: synthesis Emission Max. Freq. = ~ 1/2 Acquisition Max. Freq.
12. Android Acoustic Issues Real-time = deterministic execution time <> Android near real-time
13. Android Acoustic Issues Internet Connectivity Intermittent for many acoustic applications Saving data remotely
14. Android Acoustic Issues Peripheral Connectivity Bluetooth ~< 44,100 sampling rate (CD) Heavy Compression Needed aptX aptX Low Latency
15. Android Acoustic Issues Java vs. Kotlin No significant gain Critical Components Inside OS C (aka. native): OpenSL ES, AAudio Games TBD
16. Android Acoustic Issues Threading 2-3 bg threads Special thread ‘priority’ to reduce latency https://developer.android.com/ndk/guides/ audio/audio-latency
17. Android Acoustic Issues Screen shots: app restarted, lose screen :-(
18. Android Acoustic Issues Device Capabilities Variability in models Determined by app at startup Open source app & lib on GitLab: sm.app.dac sm.lib.acoustic
19. Audio Sources: • Microphone (code 1) • Default Audio Source (code 0) • Unprocessed (raw) (code 9) • Camcorder (code 5) • Voice Comm (code 7) • Voice Recog (code 6) • …
20. Channels for Input: • Input Mono (or in Front) (code 16) • Input Default (code 1) • Input in Back (code 32) • Input in Back Processed (code 512) • Input in Front Processed (code 256) • Input in Left (code 4) • Input in Left Processed (code 64) • Input in Pressure (code 1024) • Input in Right (code 8) • Input in Right Processed (code 128) • Input Stereo (code 12) • Input in X Axis (code 2048) • Input in Y Axis (code 4096) • Input in Z Axis (code 8192) • …
21. Encodings for Input: • Encoding PCM 16-Bit (code 2), • Encoding PCM Float (code 4)
22. Channels for Output: • Output Mono (code 4) • Output Stereo (code 12) • Output Surround (code 1052) • …
 Output Mode: Stream, … Session: ID Generate, … Usage: Game, … Content Type: Sonification, … Encodings for Output: • Encoding PCM 16-Bit (code 2) • Encoding PCM Float (code 4)
23. Sample Rates Available for Configuration: • 1 of 9 = 8000 (the old emulator rate) • 2 of 9 = 11025 • 3 of 9 = 12000 • 4 of 9 = 16000 • 5 of 9 = 22050 • 6 of 9 = 24000 • 7 of 9 = 32000 • 8 of 9 = 44100 ( = native?) • 9 of 9 = 48000 ( = native?)
24. Bleeding Edge Wavelets Daubechies, Ingrid Julia, the language - lib.
25. –Anonymous “Ultimately, if our species survives, • it will be by being radically kind, • it will take a very long time, and • we may be a different species by then”