Designing High Impact Acoustics Apps for Cross-Species Communication Research by Serge Masse

Transcript

1. Designing High Impact Acoustics Apps for Cross-Species Communication Research Serge

   Masse Montreal 2018 !1

2. Data Example - Using an Android App !2

3. Valtamer/Samsung Android Tablet !3

4. Are you… Looking to improve the world with your apps?

   Looking for a field to disrupt with your work? Looking for very difficult apps to develop? Looking for great scientists to help? Looking for good reasons to go to sea and travel around the world? Don't know what you're looking for? !4

5. About Me • Informaticien – s/w engineer – since ~1973

   • claiming a few world firsts (ex.: in trains, telco micro-economics, 747 autopilot flight testing, stock market buy&sell, dolphins) • one app deemed impossible (ex.: an app that writes other apps in production for complex telco switches) • some life-critical (ex.: trains, airplanes, pesticides) • invented and used my own tools (ex.: event-based modelling, knowledge maps) • designed multiple expert systems in production (gofai) • said “no, sorry” to G twice • developing open-source library for acoustic communication apps, on GitLab • 2 apps on G Play, a third one coming soon !5

6. Do you know the species? !6 Species?

7. Gamers for Orcas – Trendy? https://tiltify.com/@gamersfororcas/official-livestream/about April 2018 – the

   streaming has now ended !7

8. Designing X-Species Comm Apps • Part 1: Markets, main players,

   potential users, partners, competitors, antagonists, and a bit of history • Part 2: Android – app design and technical issues – AI/aNN • Part 3: Features – Required & Recommended • Part 4: Usage issues !8

9. Your potential user/competitor: Dr Denise Herzing et al. • Wild

   Dolphin Project – WDP • http://www.wilddolphinproject.org/membership/the-chat-society/ • Dr. Thad Starner – Google Glass Tech Lead • Two-way device, with some AI • The Bahamas: > 30 years, same families of dolphins !9

10. Your newish competitors/partners/users: • Fabrice Schnöller: www.darewin.org – UN, VR,

    videos • www.gavagai.se – AI, Davos 2018, with Gov of Sweden • http://monacoexplorations.org/ – Prince of Monaco • https://www.dolphinembassy.org/ – Tenerife, mostly Europeans, esoteric, films ($), sail boats • www.kintocetaceans.org • cetalingua.com – AI?, $ for lessons? maybe a kind of scam? !10

11. Your potential user/partner: Scott Veirs et al. • https://www.orcasound.net/ “Orcasound

    connects your headphones to live hydrophones” • https://www.google.com/maps/d/u/0/viewer? mid=1F2tUPCjqmYNQhMS-767YRipppIE&ll=48.26009155791273%2C-123.00161468982 697&z=8 • Dr. Paul Spong (orcas) – whale sanctuary project – OrcaLab with Helena Symonds – ~50 years • http://www.orcasound.net/2018/03/29/shortcuts-for-listening-to-the-canadian-pacific/ • Working on an app, crowd-sourced • passive (no emission) only listening – one-way – limited !11

12. Fixed Hydrophone !12

13. Your IT helper: me • 2 Free Android apps, in

    the Play store: • Dolphin Communicator: https://play.google.com/store/apps/details? id=com.simplecode.leafy.seadragon.forandroid.free&hl=en • Spectrogram: https://play.google.com/store/apps/details? id=sm.app.spectro • Open-sourced on GitLab – access controlled • Considering the Alltab®, underwater tablet by Valtamer: https://youtu.be/ gxOlF_6XDFM (not partners) !13

14. Your potential user: Dr. David Rothenberg • Analog, music (clarinet)

    • Book: https://www.amazon.com/Thousand-Mile-Song-Whale-Music/dp/ 0465071287/ref=sr_1_1? ie=UTF8&qid=1521134677&sr=8-1&keywords=thousand+mile+song • www.thousandmilesong.com – diagram !14

15. Your potential IT partner: aguasonic.com • Mark Fischer – San

    Francisco • Advanced acoustic analysis using proprietary wavelet software • Algo, Matlab, C++, D, Java, AI/NN, • Beaked whales identification for mitigation • https://www.sciencephoto.com/search? subtype=contributors&searchstring=AAC&media_type=images&per_page=96 – black holes merging • https://get.google.com/albumarchive/113099471692993588785?source=pwa • https://news.cnrs.fr/articles/joseph-fourier-is-still-transforming-science !15

16. Your potential IT partners: Wolfram & son, Alpha programming language

    • http://wolfram.com/ • Arrival movie: • http://blog.wolfram.com/2017/01/31/analyzing-and-translating-an-alien- language-arrival-logograms-and-the-wolfram-language/ • seti.org • https://www.sciencefriday.com/segments/how-to-talk-with-aliens/ • Hawkins was opposed !16

17. Your clients/competitors/antagonists: Mil.-Indus. Robots – UAV • Example: • https://www.liquid-robotics.com/

    – Boeing, mil. • James Gosling, father of Java, now at AMZ !17

18. Your potential legacy users/competitors: DCP • Dolphin Communication Project –

    DCP • www.dolphincommunicationproject.org • Terramar – terramarresearch.org • www.whalestewardship.org !18

19. DCP • Dr Kathleen Dudzinski • “You can buy trips

    and participate in research” • Dr Toni Frohoff - Terramar - Whale Stewardship – whale sanctuary project • https://wearesonar.org/2017/07/21/conversing-with-dolphins-holy-grail- interspecies-communication/ • Catherine Kinsman • USA, Nova Scotia, The Bahamas, Central America, other places… !19

20. Your legacy potential users/antagonists: • Dr. Lori Marino – whale

    sanctuary project • Dr. Diana Reiss • Nora Bateson (father was Dr J C Lilly’s best friend) – Okeanos Foundation • Dr. Vergara, Vancouver Aquarium, St. Lawrence belugas • Richard Sears, et al., Îles Mingan, QC, blue • Dr. Lusseau, U. Aberdeen, “culture” • Dr. Peter Tyack, U. St. Andrews • Dr. Whitehead (Nova Scotia) – whale sanctuary project https://whalesanctuaryproject.org/our-team/ • Dr. Au • Dr. John Ford: orca dialects, since ~1977, BC, CA gov. employee? !20

21. Other important legacy players • Dr. Roger Payne: humpback songs,

    a first, with 2 others, risked career), now ecology, RV The Odyssey, Gloucester, US • Dr. Xanadu Halkias, acoustics analysis • The pirate, Captain Paul Watson ☠ • Sea Shepherd Conservation Society https://seashepherd.org • My advice: Avoid violence !21

22. Other potential clients/antagonists • $$$ oil and gas – https://www.edgewiseenvironmental.com/

    – Canada • http://www.soundandmarinelife.org/ $$$ • PAM passive acoustic monitoring s/w – Mellinger et al., Univ. St. Andrews • www.pamguard.org • paid jobs • open-source; Java (desktop, laptop); api; Matlab; only listening • resources links; community; “big players” • https://www.youtube.com/watch?v=exGJpYmws-M – 20 min. delay - Monterey, Calif. USA !22

23. Other markets • Non-radio comm – ultrasound, no animals –

    sensors • Ultrasonic security: https://ubeacsec.org/ • Bird Id: commercial app, db, $$ • Parrots, Dr. Irene Pepperberg; Alex; current human language recog s/w (NL) • Elephants (some infra-sound), cats $, dogs $$$ • GUI: Chimps, orangutans, gorillas; not much audio at first • 80+ species of cetaceans; primary Tt, Sf, Beluga, Minke… • http://www.ukogorter.com/merchandise/whales-of-world-poster.html !23

24. Brief History – Slide 1 of 2 • Dr Lou

    Herman, et al., U. Hawaii, 1980’s, visual, first artificial sentences • e.g., “get the blue frisbee and put it at the left corner of pool” • Dr. Roger Payne et al., 1970’s, first analysis of whale songs, Nat Geo • R. Cheney, USCG, 1960’s, audio, analog, two-way • sonars http://www.navy.mil/viewGallery.asp?id=103&page=7&r=4 !24

25. Brief History – Slide 2 of 2 • Dr J.

    C. Lilly, 1950s-1980s, Janus Project, audio, digital, two-way; completed?; US Gov; Gregory Bateson, George Lucas • https://en.wikipedia.org/wiki/John_C._Lilly • Dr Dwight (Wayne) Batteau, ~1964, first automated system, analog, emitter (two-way?), US Navy; daily ocean swimmer but drowned on day of making the device public !25

26. App Design Issues • Generic • Android/technical • aNN =

    artificial Neural Net = s/w dev tech = limited AI • Features/Requirements • Usage !26

27. Input Data Example !27 on Android sm Spectrogram app

28. DC - Dolphin Comm !28 Android app showing recognition chat

    in yellow

29. Android Technical – DC Threads !29

30. Android Audio • PCM – Pulse Code Modulation • 16-bit

    integer or Float? A: Float! recent, not all devices • Q: Java or C? • Q: C with OpenSL ES or AAudio? • A: Java because Java Audio is not the limiting factor, the graphics is (canvas) • https://developer.android.com/ndk/guides/audio/audio-latency !30

31. Android OK? !31 • Pros: • fast math (DSP, FFT)

    • inexpensive, widely available • underwater +$ • AI TensorFlow • two-way • acceptable sampling rate: • 48,000 sps input => ~11kHz output • Cons: • Slow graphics • 40 slices (frames) per sec., due to graphics • lack of legacy peripheral devices • connectors – Bluetooth is very limiting for audio • versions incompatibilities

32. Feasibility with Android • Proof-of-concept device feasible when starting with

    artificial whistles • Example: DC – Dolphin Communicator app • Sufficient device memory for acoustic data if using appropriate data structures !32

33. !33 DC aNN - An Example

34. aNN • Pre-model data collection & analysis • Existing recordings

    • Ex.: https://swfsc.noaa.gov/ textblock.aspxDivision=PRD&ParentMenuId=148&id=5776 • https://soundcloud.com/user-361819329/180211-sperm-whales-clicks !34

35. aNN • Post-session data analysis, audio, recognition, model training (ML)

    • NNAPI: https://developer.android.com/ndk/guides/neuralnetworks/ • Android 8.1 = API Level 27 • Longer term: Discover higher-level patterns, some possibly not in human languages !35

36. aNN • Consider using models from other contexts – “TransferLearning”

    – Inception V3 – TBD • Cheat sheet to pick ML: • https://blogs.sas.com/content/subconsciousmusings/2017/04/12/ machine-learning-algorithm-use/ !36

37. aNN • Training aNN model with two humans communicating with

    artificial whistles using one device each (in air) • Sources for artificial whistles: • Your own whistles • Your researcher’s whistles • Inspired by existing human whistle languages • https://en.wikipedia.org/wiki/Whistled_language !37

38. aNN • Is only one of many components of such

    a system • Is not absolutely required for proof-of-concept • May be advantageous after data accumulates from sessions !38

39. Feature – Two-Way • Two-way communication – just listening is

    not enough • Recognition & emission by same device • requires specific acoustic data structures !39

40. Feature – Off-the-Shelf • Lower costs, easy access, more users,

    more progress !40

41. Feature – Non-IT Users • Users are scientists, not IT:

    • can be very limited • human UI issues • the app must automate all possible tasks !41

42. Feature – Audio Commands • Audio UI for non-human participants:

    • some whistles are commands to system • DC includes some !42

43. Feature – Replication • Experiment Replication – required by science

    • Data Exchange between teams of scientists/citizens • Data in and out automatically, ex.: relational database, xml/json files !43

44. Feature – No Internet • Assume no internet during session

    • underwater • radio waves are absorbed • cables are cumbersome – but a possibility in some cases • And no significant internet even on boat when at sea • satellite links are slow and expensive !44

45. Feature – Artificial Whistles • First step: artificial whistles, dolphins

    learn from humans, at sea !45

46. Feature – Fast Edit • Second step: humans learn from

    dolphins, change vocab, repeat, improve system…additional species… • Difficulty: humans learning and changing vocab on-the-fly for keeping dolphins interested !46

47. Usage Issues – Medical • Potential future usage: • medical

    • species protection • survival (ex.: hurricane) • science • unknown usage today • Human-to-human underwater communication: diver to diver, diver to vessel !47

48. Usage Issues – WWW • Possible future world-wide network of

    devices: • underwater and on surface, at sea, far or near shore !48

49. Usage Issues – Legality • Attention: Underwater emission can be

    illegal without a “Take” license in some jurisdictions (e.g., USA, maybe Canada, other locations) • It seems that all US citizens are not allowed to emit underwater sound near cetaceans without a Take license, anywhere in the world !49

50. Usage Issues – My Advice • No Captivity • S/W

    not to be used in captivity, and not for hunting, not for harassment • Maybe open source but controlled distribution • Maybe special S/W license !50

51. Usage Issues – My Advice • No violence, harassment, etc.,

    towards cetaceans ... or humans • Do not support violent activists or activities, directly or indirectly, and take care on social media • Some pro-animal, some anti-whaling, some vegan, etc., can be violent !51

52. Related • The legal “person” definition: can cetaceans be persons,

    legally? • https://en.wikipedia.org/wiki/Personhood#Non-human_animals 
 !52

53. Last slide • Mind-meld !53