Slide 1
Slide 1 text
Can Neural Networks Make
Me a Better Parent?
A tale in three acts
Andrew Hao
@andrewhao
[email protected]
Slide 2
Slide 2 text
Act I
Night & Chaos
Slide 3
Slide 3 text
put the little one down
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
helplessness
frustration
anger
despair
Slide 7
Slide 7 text
I need help
I need data
Slide 8
Slide 8 text
sanity needed
Photo by Joshua Sortino on Unsplash
Slide 9
Slide 9 text
Act II
Enter the Machine
Slide 10
Slide 10 text
Parts List
• Raspberry Pi
• USB microphone
• DHT22 temp/humidity
sensor
Slide 11
Slide 11 text
$ arecord --device=hw:1,0 --format S16_LE --rate 22050 -c1 -
d 10 "${RECORDING_FILE}"
$ sox -V3 ${RECORDING_FILE} -n stats 2>&1 | grep dB
Pk lev dB -57.05
RMS lev dB -67.10
RMS Pk dB -64.94
RMS Tr dB -68.26
Slide 12
Slide 12 text
No content
Slide 13
Slide 13 text
Deep learning &
neural nets
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
convolutional neural network
https://www.fastcompany.com/3037882/how-flickrs-deep-
learning-algorithms-see-whats-in-your-photos
Slide 16
Slide 16 text
CNNs
Karpathy, Andrej "Convolutional Neural Networks (CNNs / ConvNets)" (http://cs231n.github.io/convolutional-networks/)
Slide 17
Slide 17 text
audio spectrogram = image
“yes” “Sheila” “wah, wah, wah”
frequency
time
frequency
time
frequency
time
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
(fingerprint_input)
v
[Conv2D]<-(weights)
v
[BiasAdd]<-(bias)
v
[Relu]
v
[MaxPool]
v
[Conv2D]<-(weights)
v
[BiasAdd]<-(bias)
v
[Relu]
v
[MaxPool]
v
[MatMul]<-(weights)
v
[BiasAdd]<-(bias)
v
‘cnn-trad-fpool3’
architecture
Slide 20
Slide 20 text
The Secret Life of ML Models
Clean
data
Test
model
✅
Get
data
Label
data
✍
Train
model
Deploy
model
Slide 21
Slide 21 text
• Record data
continuously in a
cronjob
• Upload to the cloud
Get
data
☁
Slide 22
Slide 22 text
Clean
data
• Align to a frame
• Normalize
volume
# Concatenate
sox crying/**/*.wav tmp/batch-crying.wav
Slide 23
Slide 23 text
Clean
data
• Align to a frame
• Normalize
volume
# Split
ffmpeg -i tmp/batch-crying.wav -
f segment -segment_time 10.1 -c
Slide 24
Slide 24 text
Clean
data
• Align to a frame
• Normalize
volume
# Trim to exactly 10000ms, boost 45
dB, resample @ 22050 hz
sox $in data/crying/$out vol 45 dB
trim 0 10 rate 22050
Slide 25
Slide 25 text
Using
EchoML
to
classify
data
Label
data
✍
Slide 26
Slide 26 text
• Bucket each
audio sample in a
folder by label
crying dog_barking city_noise
whining
Label
data
✍
Slide 27
Slide 27 text
python app/train.py
--data_url=
--data_dir=./data
--wanted_words=room_empty,whining,crying
--sample_rate=22050
--clip_duration_ms=10000
--how_many_training_steps=400,50
--train_dir=./training
Train
model
Slide 28
Slide 28 text
Train
model
Slide 29
Slide 29 text
Train
model
Slide 30
Slide 30 text
Test
model
✅
Slide 31
Slide 31 text
Freeze the model
$ python app/freeze.py
--start_checkpoint=./training/conv.ckpt-450
--output_file=./graph.pb
--clip_duration_ms=10000
--sample_rate=22050
--wanted_words=white_noise,room_empty,crying
--data_dir=./data
$ cp training/conv_labels.txt .
Deploy
model
Slide 32
Slide 32 text
Execute the model in prod
Deploy
model
> crying (score = 0.95350)
room_empty (score = 0.01888)
_silence_ (score = 0.01746)
$ arecord --format S16_LE --rate 22050 -c1
-d 10 $wav
$ python app/label_wav.py
--graph=./graph.pb
--labels=./conv_labels.txt
--wav=$wav
Slide 33
Slide 33 text
Will it work?
Slide 34
Slide 34 text
Yes!
Slide 35
Slide 35 text
..yes?
Slide 36
Slide 36 text
hmm.
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
Act III
The Illusion of Insight
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
We just looked at each other
Slide 41
Slide 41 text
No content
Slide 42
Slide 42 text
.
Systems,
Models,
Metrics,
Funnels
0
1
2
3
Slide 43
Slide 43 text
When we lose insights about the humans we serve,
It’s ultimately bad for business
Slide 44
Slide 44 text
In a Big Data world,
Let’s re-emphasize the human
experience
Slide 45
Slide 45 text
.
Systems,
Models,
Metrics,
Funnels
0
1
2
3
Slide 46
Slide 46 text
Are our assumptions right?
How do we find out?
Slide 47
Slide 47 text
Human-centered
UX research
Anthropology
Lean Startup: Customer interviews
Slide 48
Slide 48 text
Human-centered ML?
Slide 49
Slide 49 text
Personas
Photo by Toa Heftiba on Unsplash
Jenna
- Customer
- Recent college graduate
- Aspiring indie electronica
producer
- Wants to pay off college
loans
- Challenges: …
Photo by Philip Martin on Unsplash
Reeve
- Internal stakeholder
- Director, Digital Marketing
- Concerned about ad budget
spend
Photo by Jhon David on Unsplash Lydia
- Customer
- Civil engineer
- Mother of one & caretaker of her
aging parents
- Wants to save for her daughter’s
college fund
- Challenges: …
Slide 50
Slide 50 text
We need human-centered
design at all stages of the ML
lifecycle
Slide 51
Slide 51 text
Let’s start by connecting with
people first
Slide 52
Slide 52 text
Thank you
github.com/andrewhao/babblefish
Andrew Hao
@andrewhao
[email protected]