Unlocking Data Trapped in Audio and Video Files (PyTexas 2014)

As more and more apps record audio and video files we need to start thinking about what to do with those files. Playing them back isn't enough. Media files are full of data that developers can start exploiting thanks to an emergent category of signal and natural language processing APIs.

Let's look at some of the APIs that allow us to extract this data including words, emotion, and identity.

Transcript

1. Unlocking Data Trapped in Audio & Video Files Paul  Murphy   paul@clarify.io   @prmurphy  
2. College Station, we have a problem…
4. 4,500   4,000   3,500   3,000   2,500   2,000   1,500   1,000   500   0     2013   2010   2005   2000   40,000   BC   1440   1969   Cave  pain@ng   Inven@on  of  the   prin@ng  press   Inven%on  of   the  Internet   Exabytes  
5. Were building the problem,
6. and users aren’t helping.
7. It used to be different • ASCII • EBCDIC • Unicode
8. Remember what we could do? •  upper(), lower() •  split(), join() •  index() •  etc., etc.
10. But back to our problem
11. Now we feel like losers
12. What can you do with a video recording?
13. What can you do with a call recording?
14. Here’s what banks do
17. Let’s look at word options • Human transcription • Automated transcription
18. Lots of human transcription APIs • Rev.com • CastingWords.com • VerbalizeIt.com
19. POST /api/v1/orders Rev.com API • Ruby helper library only. • Ask for things like audio length. Huh?
20. Lots of automatic transcription APIs • Google • AT&T • Nuance • VoiceBase • Speechmatics
21. Any good? • Some are pretty standard REST. • Some are clearly an afterthought. • Some include timestamps. • Some include confidence scores. • Some include more information, comprehensible only to speech experts.
22. Now we’re back in the game! •  upper(), lower() •  split(), join() •  index() •  etc., etc.
23. And we feel like this
24. But hang on!
25. Audio is Быстрая   коричневая  лиса   прыгает  через   ленивую  собаку.   عيرسلا ينبلا بلعثلا زف  .لوسكلا بلكلا זירזה בלכ 速い茶色のキツネ は、のろまなイヌに 飛びかかった。   ???   The  quick  brown  fox   jumps  over  the  lazy   dog.  
26. Timeline   Audio is Text   ???  
27. One step at a time • A timeline is just a sequence • …(timestamp, word) 2-tuples • …(timestamp, word, probability) 3-tuples
28. But... • Emotion • Identity • Events
29. Emotions • Beyond Verbal • Moodzle
30. Beyond Verbal API • Have to ask for a Key. • No public docs.
31. Moodzle API • Pivoted to…
32. Moodzle API
33. Moodzle API • Pivoted to…
34. • Agnicio • VoiceVault • OneVault • Nuance Identity • No API
35. • Agnicio • VoiceVault • OneVault • Nuance Identity • No self-signup • PDF docs
36. • Agnicio • VoiceVault • OneVault • Nuance Identity • No API
37. • White Papers • Case studies • Benefits • Video • No API (I think) Identity • Agnicio • VoiceVault • OneVault • Nuance
38. Quick recap • Words – Lots of options! • Emotions – Bit tougher • Identity – Lots of nothing
39. Processing scenario Transcrip%on   API   Words  
40. But… • Speech   • Music   • Noise   • Silence  
41. Also, remember this? Быстрая   коричневая  лиса   прыгает  через   ленивую  собаку.   عيرسلا ينبلا بلعثلا زفقيو  .لوسكلا بلكلا قوف זירזה םוחה לעוש בלכה לעמ ץפוק  .ןלצעה The  quick  brown  fox   jumps  over  the  lazy   dog.  
42. Classification • Recording source • Content type • Language (speech) • Domain
43. Let’s (quickly) talk about video
44. Let’s (quickly) talk about video Full of data! •  Audio •  Images •  Motion
45. Video…the bad news
46. Under development •  Word recognition •  Object matching •  Object recognition •  Emotions •  ID •  Events
47. A few things to notice •  A lot of this data is complimentary •  Words •  Emotions •  ID •  We can use different signals to find the same data •  There is no standard way of accessing or representing it
48. Is there a solution? •  Lots of partial solutions, and a lot more coming. •  Classification is critical. •  Some sort of coherent way of describing this data will emerge. •  My company is working to speed things up.
49. Thank  you!   Paul  Murphy   paul@clarify.io   @prmurphy  
50. Any  Ques@ons?   PS. We’re hiring! Paul  Murphy   paul@clarify.io   @prmurphy