Slide 6
Slide 6 text
y = 𝒂(ax+b), where a,b,x,y ∈R
What is Neural network?
x y
z
z = ax+b
Linear function
y = 𝝈(z)
Non-linear activation function
y = 𝝈(ax+b)
1
2
n
1
…
3
1
2
n
2
Layer 1 Layer 2
Neuron
Layer m
…
• Activation function a: 𝜎, 𝑅𝑒𝑙𝑢, tanh
• Quiz: Why need a non-linear activation function?
X
2
= a(WX
1
+b)
X
1
X
2
n
1
Neurons n
2
Neurons n
i
Neurons
W: weight matrix,
of dimension n
2
xn
1
X
i+1
= 𝒂i
(W
i
X
i
+b
i
),
where X
i
∈Rni , W
i
∈Rni+1xni, b
i
∈Rni+1
• Computation/ inference:
• Matrix multiplication, activation ->
• Matrix multiplication, activation
• …
• Quiz: How many parameters in this m layer neural network?
• Weights: σ
𝑖=1
𝑚−1 𝑛𝑖+1
∗ 𝑛𝑖
• Bias: σ
𝑖=1
𝑚−1 𝑛𝑖+1
x
1
x
2
x
3
1
2
n
ni
3
n
m
Neurons
y
X
i