Intro to Acoustic Processing in Android by Serge Masse

Intro to Acoustic Processing in Android by Serge Masse

Description of processing in Android Java for sound input to analysis and sound synthesis to output.

https://gdgmontreal.com/2020/01/22/february-meetup-2/

1b77dd441f657f5aefb3e21283b252e6?s=128

GDG Montreal

February 04, 2020
Tweet

Transcript

  1. Intro to Acoustic Processing in Android Serge Masse 2020

  2. This talk is about • Acoustic processing: • Sound input

    to analysis • Sound synthesis to output
 • Not playing music/sound clips
  3. 1. Input flow: 1. a2d analog to digital 2. capture:

    subset of voltage samples 3. analysis: FFT, wavelet, noise, ANN, recognition/ classification 4. Data, storage, sharing 2. Output flow: data, synthesis, d2a 3. Android issues
  4. None
  5. None
  6. None
  7. None
  8. None
  9. Output: synthesis Emission Max. Freq. = ~ 1/2 Acquisition Max.

    Freq.
  10. None
  11. None
  12. Android Acoustic Issues Real-time = deterministic execution time <> Android

    near real-time
  13. Android Acoustic Issues Internet Connectivity Intermittent for many acoustic applications

    Saving data remotely
  14. Android Acoustic Issues Peripheral Connectivity Bluetooth ~< 44,100 sampling rate

    (CD) Heavy Compression Needed aptX aptX Low Latency
  15. Android Acoustic Issues Java vs. Kotlin No significant gain Critical

    Components Inside OS C (aka. native): OpenSL ES, AAudio Games TBD
  16. Android Acoustic Issues Threading 2-3 bg threads Special thread ‘priority’

    to reduce latency https://developer.android.com/ndk/guides/ audio/audio-latency
  17. Android Acoustic Issues Screen shots: app restarted, lose screen :-(

  18. Android Acoustic Issues Device Capabilities Variability in models Determined by

    app at startup Open source app & lib on GitLab: sm.app.dac sm.lib.acoustic
  19. Audio Sources: • Microphone (code 1) • Default Audio Source

    (code 0) • Unprocessed (raw) (code 9) • Camcorder (code 5) • Voice Comm (code 7) • Voice Recog (code 6) • …
  20. Channels for Input: • Input Mono (or in Front) (code

    16) • Input Default (code 1) • Input in Back (code 32) • Input in Back Processed (code 512) • Input in Front Processed (code 256) • Input in Left (code 4) • Input in Left Processed (code 64) • Input in Pressure (code 1024) • Input in Right (code 8) • Input in Right Processed (code 128) • Input Stereo (code 12) • Input in X Axis (code 2048) • Input in Y Axis (code 4096) • Input in Z Axis (code 8192) • …
  21. Encodings for Input: • Encoding PCM 16-Bit (code 2), •

    Encoding PCM Float (code 4)
  22. Channels for Output: • Output Mono (code 4) • Output

    Stereo (code 12) • Output Surround (code 1052) • …
 Output Mode: Stream, … Session: ID Generate, … Usage: Game, … Content Type: Sonification, … Encodings for Output: • Encoding PCM 16-Bit (code 2) • Encoding PCM Float (code 4)
  23. Sample Rates Available for Configuration: • 1 of 9 =

    8000 (the old emulator rate) • 2 of 9 = 11025 • 3 of 9 = 12000 • 4 of 9 = 16000 • 5 of 9 = 22050 • 6 of 9 = 24000 • 7 of 9 = 32000 • 8 of 9 = 44100 ( = native?) • 9 of 9 = 48000 ( = native?)
  24. Bleeding Edge Wavelets Daubechies, Ingrid Julia, the language - lib.

  25. –Anonymous “Ultimately, if our species survives, • it will be

    by being radically kind, • it will take a very long time, and • we may be a different species by then”