Word embeddings under the hood - Strata Data Conference

Word Embeddings Under the Hood
How Neural Networks Learn from Language

View Slide

View Slide

word2vec

View Slide

“Wait, what?”
Source: https://tinyurl.com/y9lb6e7j

View Slide

Good news! Text data is everywhere.

View Slide

Bad news… there is way too much.
We need computers to help!

View Slide

We started with the scallop dish as an appetizer, followed by
the spaghetti with tomato sauce and duck and foie gras ravioli.
How do we represent data like this?

View Slide

1 2 3 … V
we 1 0 0 … 0
started 0 1 0 … 0
with 0 0 1 … 0
… … … … … …
ravioli 0 0 0 … 1
One-Hot Encoding

View Slide

…but one-hot encoding
leaves a lot to be desired.
Are better word representations possible?

View Slide

y
-2
-1
0
1
2
x
-2 -1 0 1 2
beer
wine
cocktail
spoon
fork
knife
spaghetti
pasta
lasagna

View Slide

y
-2
-1
0
1
2
x
-2 -1 0 1 2
beer
wine
cocktail
spoon
fork
knife
spaghetti
pasta
lasagna
x y
spaghetti 1.0 1.5
pasta 1.2 1.3
… … …
fork 0.0 -0.7
spoon -0.5 -1.5

View Slide

“You shall know a word by the company it keeps.”
— J.R. Firth, 1957
Postulate #1

View Slide

“Neural networks learn useful, new data representations.”
— Rumelhart, Hinton & Williams, 1986
(paraphrased)
Postulate #2

View Slide

context clues
neural networks
?
=
+

View Slide

Context clues as training data?

View Slide

View Slide

spaghetti followed 1
spaghetti by 1
… … …
spaghetti sauce 1

View Slide

spaghetti followed 1
spaghetti by 1
… … …
spaghetti sauce 1
spaghetti we 0
spaghetti parking 0
… … …
spaghetti sushi 0

View Slide

with by 1
with the 1
… … …
with and 1
with appetizer 0
with loud 0
… … …
with up 0

View Slide

Minimum Viable Introduction to
Neural Networks

View Slide

View Slide

“Sigmoid” Activation Function
Weighted Input
Activation Value
(z) =
1
1 + e z
AAACBnicdVDLSsNAFJ3UV62vqEtBBotQEUMionYhFNy4rGBsoYllMp3UoTNJmJkIbcjOjb/ixoWKW7/BnX/j9CH4PHDhcM693HtPkDAqlW2/G4Wp6ZnZueJ8aWFxaXnFXF27lHEqMHFxzGLRDJAkjEbEVVQx0kwEQTxgpBH0Tod+44YISePoQvUT4nPUjWhIMVJaapubnqRdjiqDHXgCvVAgnDl55uySq2xvkOdts2xbVdupHjrwN3Ese4QymKDeNt+8ToxTTiKFGZKy5diJ8jMkFMWM5CUvlSRBuIe6pKVphDiRfjb6I4fbWunAMBa6IgVH6teJDHEp+zzQnRypa/nTG4p/ea1Uhcd+RqMkVSTC40VhyqCK4TAU2KGCYMX6miAsqL4V4mukw1A6upIO4fNT+D9x962qZZ8flGv1SRpFsAG2QAU44AjUwBmoAxdgcAvuwSN4Mu6MB+PZeBm3FozJzDr4BuP1A0VMmJI=

View Slide

“Sigmoid” Activation Function
Weighted Input
Activation Value
0.88
(z) =
1
1 + e z

View Slide

View Slide

A neural network for
learning context clues?

View Slide

(spaghetti, tomato, 1)

View Slide

View Slide

(weight matrix for the hidden layer)

View Slide

(weight matrix for the output layer)

View Slide

View Slide

Training our network
on the ﬁrst context clue

View Slide

Training our network
on the ﬁrst context clue
(there will be lots of these)

View Slide

1. Make a prediction

View Slide

2. Measure how wrong we are

View Slide

3. Tune the model to become
slightly less wrong

View Slide

slightly less wrong
4. Repeat with the next context clue

View Slide

View Slide

Weighted Input
Activation Value

View Slide

Weighted Input
Activation Value
0.51

View Slide

View Slide

“forward pass”

View Slide

slightly less wrong

View Slide

View Slide

“Loss” Function
Model Prediction
Penalty
L(ˆ
y) = ln(ˆ
y)
AAACAHicdVDLSgMxFM3UV62vUTeCm2AR6sIyI6J2IRTcuHBRwbFCO5RMmmlDM5khuSOUoW78FTcuVNz6Ge78G9OH4vPAhZNz7iX3niARXIPjvFm5qemZ2bn8fGFhcWl5xV5du9RxqijzaCxidRUQzQSXzAMOgl0lipEoEKwe9E6Gfv2aKc1jeQH9hPkR6UgeckrASC1746zU7BLI+oOd492mkJ+vll10yhXHrRy4+Ddxy84IRTRBrWW/NtsxTSMmgQqidcN1EvAzooBTwQaFZqpZQmiPdFjDUEkipv1sdMEAbxuljcNYmZKAR+rXiYxEWvejwHRGBLr6pzcU//IaKYRHfsZlkgKTdPxRmAoMMR7GgdtcMQqibwihiptdMe0SRSiY0AomhI9L8f/E2ytXys75frFam6SRR5toC5WQiw5RFZ2iGvIQRTfoDj2gR+vWureerOdxa86azKyjb7Be3gFNVZZs
right answer: 1

View Slide

“Loss” Function
Model Prediction
Penalty
right answer: 0
L(ˆ
y) = ln (1 ˆ
y)
AAACBnicdVDLSsNAFJ34rPEVdSnIYBHqoiERUbsQim5cuKhgbaENZTKdtEMnkzAzEULIzo2/4saFilu/wZ1/4/QFPg9cOJxzL/fe48eMSuU4H8bM7Nz8wmJhyVxeWV1btzY2b2SUCEzqOGKRaPpIEkY5qSuqGGnGgqDQZ6ThD86HfuOWCEkjfq3SmHgh6nEaUIyUljrWzmWp3UcqS/N989QstxnPSm55KuUdq+jYFcetHLnwN3FtZ4QimKDWsd7b3QgnIeEKMyRly3Vi5WVIKIoZyc12IkmM8AD1SEtTjkIivWz0Rw73tNKFQSR0cQVH6teJDIVSpqGvO0Ok+vKnNxT/8lqJCk68jPI4UYTj8aIgYVBFcBgK7FJBsGKpJggLqm+FuI8EwkpHZ+oQpp/C/0n9wK7YztVhsXo2SaMAtsEuKAEXHIMquAA1UAcY3IEH8ASejXvj0XgxXsetM8ZkZgt8g/H2CXbFmAQ=

View Slide

“Loss” Function
Model Prediction
Penalty
L(ˆ
y) = ln(ˆ
y)
right answer: 1

View Slide

“Loss” Function
Model Prediction
Penalty
0.51
0.67
L(ˆ
y) = ln(ˆ
y)
right answer: 1

View Slide

slightly less wrong

View Slide

“Loss” Function
Model Prediction
Penalty
0.51
0.67
L(ˆ
y) = ln(ˆ
y)
right answer: 1

View Slide

“Loss” Function
Model Prediction
Penalty
0.51
0.67
L(ˆ
y)
ˆ
y
=
1
ˆ
y
AAACKnicdZDLSsNAFIYnXmu8RV26GSyCLgxJVdouBNGNCxcVrApNKZPJxA6dXJg5EUrI+7jxVVzoQsWtD+K0RlHRAwM/338OZ87vp4IrcJwXY2JyanpmtjJnzi8sLi1bK6sXKskkZW2aiERe+UQxwWPWBg6CXaWSkcgX7NIfHI/8yxsmFU/icximrBuR65iHnBLQqGcdeaEk1My9gAkg+HTL6xPIh8V28cVKUpgH5o5Ztrsju+Q9q+rY9Wat0XSwYzvj0sKt7Tfqu9gtSRWV1epZD16Q0CxiMVBBlOq4TgrdnEjgVLDC9DLFUkIH5Jp1tIxJxFQ3H99a4E1NAhwmUr8Y8Jh+n8hJpNQw8nVnRKCvfnsj+JfXySBsdHMepxmwmH4sCjOBIcGj4HDAJaMghloQKrn+K6Z9otMAHa+pQ/i8FP8v2jW7aTtne9XDVplGBa2jDbSFXFRHh+gEtVAbUXSL7tETejbujEfjxXj9aJ0wypk19KOMt3dcg6bV
L(ˆ
y) = ln(ˆ
y)
right answer: 1

View Slide

Move in the opposite direction
from the gradient

View Slide

View Slide

It works! Now…

View Slide

It works! Now…
1. How do we know which direction
to nudge a weight?

View Slide

It works! Now…
1. How do we know which direction
to nudge a weight?
2. How can we calculate this
automatically for all the weights?

View Slide

View Slide

“back propagation”

View Slide

View Slide

Did it work?

View Slide

View Slide

“stochastic gradient descent”

View Slide

Context clues trained: 1

View Slide

slightly less wrong

View Slide

View Slide

Learning to predict Learning to represent

View Slide

View Slide

we can measure distances!

View Slide

Context clues trained: 1
topics
discerning
masked
sweets
carmelized
shelly
cue
prepare
“amazing” as focus word: 0
cheerful
succulent
adjusting
pop
antenna
suggesting
vinegary
brothers
“server” as focus word: 0
ignorant
sop
refrigerators
bags
recliner
introduce
covered
petco
“spaghetti” as focus word: 1

View Slide

Fast forward…

View Slide

Context clues trained: 2,000,000
awesome
delicious
super
here
)
$
customer
excellent
“amazing” as focus word: 1,854
thru
along
crab
tacos
/
windows
chef
1
“server” as focus word: 780
dollar
rings
loves
=
opened
wrapped
form
provided
“spaghetti” as focus word: 84

View Slide

Fast forward…

View Slide

Context clues trained: 100,000,000
incredible
awesome
outstanding
excellent
phenomenal
fabulous
superb
fantastic
“amazing” as focus word: 87,864
waiter
waitress
bartender
hostess
guide
technician
cashier
barista
“server” as focus word: 48,492
risotto
veal
katsu
goat
turkey
enchilada
raspberry
meatloaf
“spaghetti” as focus word: 3,600

View Slide

View Slide

bun american + mexican ⇡ tortilla
AAACLXicdZBLSwMxEMez9V1fVY9egkUQxLIrovZWEMGTKFgV2lKy6bQNzSZLMistpZ/Ii19FD4IPvPo1TNsVfA4E/vnNTCbzD2MpLPr+k5eZmJyanpmdy84vLC4t51ZWL61ODIcy11Kb65BZkEJBGQVKuI4NsCiUcBV2job5qxswVmh1gb0YahFrKdEUnKFD9dxxFaGL/TBRg+xOdnxhERhX4Mh2SiLojkGVxbHR3RSjNiikZIN6Lu8Xin5Q3A/obxEU/FHkSRpn9dxDtaF5EoFCLpm1lcCPsdZn7j0uwc1JLMSMd1gLKk4q9yVb64/WHdBNRxq0qY07CumIfu1wC1jbi0JXGTFs25+5IfwrV0mweVjrCxUnCIqPBzUTSVHToXe0IQxwlD0nGDfC/ZXyNjOMo3M460z43JT+L8q7hWIhON/Ll05TN2bJOtkgWyQgB6RETsgZKRNObsk9eSYv3p336L16b+PSjJf2rJFv4b1/AGJDqk4=

View Slide

“word2vec skip-gram negative sampling”

View Slide

No magical black box AI…
Just context clues and some arithmetic!
Bonus: now you know the fundamentals
of all neural network learning

View Slide

careers.spglobal.com

View Slide

Word embeddings under the hood - Strata Data Conference

Word embeddings under the hood - Strata Data Conference

Patrick Harrison

More Decks by Patrick Harrison

Other Decks in Technology

Featured

Transcript