Intro to Acoustic Processing in Android by Serge Masse

Intro to Acoustic Processing in Android by Serge Masse

Description of processing in Android Java for sound input to analysis and sound synthesis to output.

https://gdgmontreal.com/2020/01/22/february-meetup-2/

1b77dd441f657f5aefb3e21283b252e6?s=128

GDG Montreal

February 04, 2020
Tweet

Transcript

  1. 2.

    This talk is about • Acoustic processing: • Sound input

    to analysis • Sound synthesis to output
 • Not playing music/sound clips
  2. 3.

    1. Input flow: 1. a2d analog to digital 2. capture:

    subset of voltage samples 3. analysis: FFT, wavelet, noise, ANN, recognition/ classification 4. Data, storage, sharing 2. Output flow: data, synthesis, d2a 3. Android issues
  3. 4.
  4. 5.
  5. 6.
  6. 7.
  7. 8.
  8. 10.
  9. 11.
  10. 14.

    Android Acoustic Issues Peripheral Connectivity Bluetooth ~< 44,100 sampling rate

    (CD) Heavy Compression Needed aptX aptX Low Latency
  11. 15.

    Android Acoustic Issues Java vs. Kotlin No significant gain Critical

    Components Inside OS C (aka. native): OpenSL ES, AAudio Games TBD
  12. 16.

    Android Acoustic Issues Threading 2-3 bg threads Special thread ‘priority’

    to reduce latency https://developer.android.com/ndk/guides/ audio/audio-latency
  13. 18.

    Android Acoustic Issues Device Capabilities Variability in models Determined by

    app at startup Open source app & lib on GitLab: sm.app.dac sm.lib.acoustic
  14. 19.

    Audio Sources: • Microphone (code 1) • Default Audio Source

    (code 0) • Unprocessed (raw) (code 9) • Camcorder (code 5) • Voice Comm (code 7) • Voice Recog (code 6) • …
  15. 20.

    Channels for Input: • Input Mono (or in Front) (code

    16) • Input Default (code 1) • Input in Back (code 32) • Input in Back Processed (code 512) • Input in Front Processed (code 256) • Input in Left (code 4) • Input in Left Processed (code 64) • Input in Pressure (code 1024) • Input in Right (code 8) • Input in Right Processed (code 128) • Input Stereo (code 12) • Input in X Axis (code 2048) • Input in Y Axis (code 4096) • Input in Z Axis (code 8192) • …
  16. 22.

    Channels for Output: • Output Mono (code 4) • Output

    Stereo (code 12) • Output Surround (code 1052) • …
 Output Mode: Stream, … Session: ID Generate, … Usage: Game, … Content Type: Sonification, … Encodings for Output: • Encoding PCM 16-Bit (code 2) • Encoding PCM Float (code 4)
  17. 23.

    Sample Rates Available for Configuration: • 1 of 9 =

    8000 (the old emulator rate) • 2 of 9 = 11025 • 3 of 9 = 12000 • 4 of 9 = 16000 • 5 of 9 = 22050 • 6 of 9 = 24000 • 7 of 9 = 32000 • 8 of 9 = 44100 ( = native?) • 9 of 9 = 48000 ( = native?)
  18. 25.

    –Anonymous “Ultimately, if our species survives, • it will be

    by being radically kind, • it will take a very long time, and • we may be a different species by then”