Tweet
Tweet

Slide 1

Slide 1 text

Intro to Acoustic Processing in Android Serge Masse 2020

Slide 2

Slide 2 text

This talk is about • Acoustic processing: • Sound input to analysis • Sound synthesis to output
 • Not playing music/sound clips

Slide 3

Slide 3 text

1. Input flow: 1. a2d analog to digital 2. capture: subset of voltage samples 3. analysis: FFT, wavelet, noise, ANN, recognition/ classification 4. Data, storage, sharing 2. Output flow: data, synthesis, d2a 3. Android issues

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Output: synthesis Emission Max. Freq. = ~ 1/2 Acquisition Max. Freq.

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Android Acoustic Issues Real-time = deterministic execution time <> Android near real-time

Slide 13

Slide 13 text

Android Acoustic Issues Internet Connectivity Intermittent for many acoustic applications Saving data remotely

Slide 14

Slide 14 text

Android Acoustic Issues Peripheral Connectivity Bluetooth ~< 44,100 sampling rate (CD) Heavy Compression Needed aptX aptX Low Latency

Slide 15

Slide 15 text

Android Acoustic Issues Java vs. Kotlin No significant gain Critical Components Inside OS C (aka. native): OpenSL ES, AAudio Games TBD

Slide 16

Slide 16 text

Android Acoustic Issues Threading 2-3 bg threads Special thread ‘priority’ to reduce latency https://developer.android.com/ndk/guides/ audio/audio-latency

Slide 17

Slide 17 text

Android Acoustic Issues Screen shots: app restarted, lose screen :-(

Slide 18

Slide 18 text

Android Acoustic Issues Device Capabilities Variability in models Determined by app at startup Open source app & lib on GitLab: sm.app.dac sm.lib.acoustic

Slide 19

Slide 19 text

Audio Sources: • Microphone (code 1) • Default Audio Source (code 0) • Unprocessed (raw) (code 9) • Camcorder (code 5) • Voice Comm (code 7) • Voice Recog (code 6) • …

Slide 20

Slide 20 text

Channels for Input: • Input Mono (or in Front) (code 16) • Input Default (code 1) • Input in Back (code 32) • Input in Back Processed (code 512) • Input in Front Processed (code 256) • Input in Left (code 4) • Input in Left Processed (code 64) • Input in Pressure (code 1024) • Input in Right (code 8) • Input in Right Processed (code 128) • Input Stereo (code 12) • Input in X Axis (code 2048) • Input in Y Axis (code 4096) • Input in Z Axis (code 8192) • …

Slide 21

Slide 21 text

Encodings for Input: • Encoding PCM 16-Bit (code 2), • Encoding PCM Float (code 4)

Slide 22

Slide 22 text

Channels for Output: • Output Mono (code 4) • Output Stereo (code 12) • Output Surround (code 1052) • …
 Output Mode: Stream, … Session: ID Generate, … Usage: Game, … Content Type: Sonification, … Encodings for Output: • Encoding PCM 16-Bit (code 2) • Encoding PCM Float (code 4)

Slide 23

Slide 23 text

Sample Rates Available for Configuration: • 1 of 9 = 8000 (the old emulator rate) • 2 of 9 = 11025 • 3 of 9 = 12000 • 4 of 9 = 16000 • 5 of 9 = 22050 • 6 of 9 = 24000 • 7 of 9 = 32000 • 8 of 9 = 44100 ( = native?) • 9 of 9 = 48000 ( = native?)

Slide 24

Slide 24 text

Bleeding Edge Wavelets Daubechies, Ingrid Julia, the language - lib.

Slide 25

Slide 25 text

–Anonymous “Ultimately, if our species survives, • it will be by being radically kind, • it will take a very long time, and • we may be a different species by then”