Gabriel Peyré
May 05, 2023
3.4k

# The Mathematics of Neural Networks

Tutorial talk at the conference F2S "Science et Progrès" 2023

May 05, 2023

## Transcript

1. The Mathematics of

Neural Networks
Gabriel Peyré
É C O L E N O R M A L E
S U P É R I E U R E
www.numerical-tours.com

2. Overview
• Empirical Risk Minimization

• Perceptrons

• Optimization

• Convolutional Networks

• Residual Networks

• Transformers

3. Empirical Risk Minimization
Dataset: (xi, yi)n
i=1
xi
yi
2 R
y ⇡ f✓(x)
Empirical risk minimization:
min

1
n
n
X
i=1
`(f✓(xi), yi)
Least square: `(y, y0) = (y y0)2
f✓
f✓0
x
y

4. Empirical Risk Minimization
Dataset: (xi, yi)n
i=1
xi
yi
2 R
y ⇡ f✓(x)
Empirical risk minimization:
min

1
n
n
X
i=1
`(f✓(xi), yi)
Least square: `(y, y0) = (y y0)2
Logistic: `(y, y0) = log(1 + e yy0
)
yi
2 { 1, 1}
Classiﬁcation:
y ⇡ sign(f✓(x))
f✓
f✓0
x
y
yi = +1
yi = 1

5. Empirical Risk Minimization
Dataset: (xi, yi)n
i=1
xi
yi
2 R
y ⇡ f✓(x)
Empirical risk minimization:
min

1
n
n
X
i=1
`(f✓(xi), yi)
Least square: `(y, y0) = (y y0)2
Logistic: `(y, y0) = log(1 + e yy0
)
yi
2 { 1, 1}
Classiﬁcation:
y ⇡ sign(f✓(x))
f✓
f✓0
x
y
Overﬁtting, regularization, . . .
yi = +1
yi = 1

6. Linear model (1 layer)
f✓(x) = hx, ✓i =
P
k
xk✓k
x f(x) = 0
x
y y = f(x)
y = hx, ✓i
x1
xd
✓d
✓1
. . .
Regression
Classiﬁcation:

7. Linear model (1 layer)
f✓(x) = hx, ✓i =
P
k
xk✓k
x f(x) = 0
x
y y = f(x)
y = hx, ✓i
x1
xd
✓d
✓1
. . .
Regression
Classiﬁcation:
min

, 1
n
n
X
i=1
`(hxi, ✓i
i, yi)
Convex optimization:

8. Linear model (1 layer)
f✓(x) = hx, ✓i =
P
k
xk✓k
Deep learning methods: learn '(x)!
Kernel methods: replace x by '(x) 2 RD
(D d, even D = 1!)
x f(x) = 0
x
y y = f(x)
y = hx, ✓i
x1
xd
✓d
✓1
. . .
Regression
Classiﬁcation:
min

, 1
n
n
X
i=1
`(hxi, ✓i
i, yi)
Convex optimization:

9. Overview
• Empirical Risk Minimization

• Perceptrons

• Optimization

• Convolutional Networks

• Residual Networks

• Transformers

10. Multi-layer Perceptron
x
x1
x2
xD 1
y = xD
. . .
x0 x
Wk
2 Rdk+1
⇥dk
bk
2 Rdk+1
f✓(x0) = xD
xk+1
, (Wkxk + bk)
✓ = {(Wk, bk)}D 1
k=0
Frank Rosenblatt

11. Multi-layer Perceptron
x
x1
x2
xD 1
y = xD
. . .
x0 x
Wk
2 Rdk+1
⇥dk
bk
2 Rdk+1
f✓(x0) = xD
xk+1
, (Wkxk + bk)
✓ = {(Wk, bk)}D 1
k=0
s
(s)
s
(s)
ReLu
Non-linearity: must be non-polynomial to increase expressivity.
Frank Rosenblatt

12. Multi-layer Perceptron
x
x1
x2
xD 1
y = xD
. . .
x0 x
Wk
2 Rdk+1
⇥dk
bk
2 Rdk+1
f✓(x0) = xD
xk+1
, (Wkxk + bk)
✓ = {(Wk, bk)}D 1
k=0
Weight matrix: needs extra constraints (e.g. convolution & sub-sampling)
s
(s)
s
(s)
ReLu
Non-linearity: must be non-polynomial to increase expressivity.
Frank Rosenblatt

13. Two Layers Perceptron
Wx
x
w1
a1
p = 6 neurons p = 30 neurons p = 100 neurons
Input y = f(x)
! sum of “ridge” functions (hx, wi + b)
f✓(x) ,
p
X
s=1
as (hx, ws
i + bs)
wp
AABE+nictVxLcxxJES4vr8W8vHAhgksvWhNeQgjZ2EDEBhFra2RZa9mWPSPZux5bMT3TGo/dmh7Py4+x+DEEF4KAE3d+Bz+ACDjxF8hHVVf1THVntTDukFRdXV9mVnZVVmZWteNROphMNzf/ce6Dr339G9/81offPv+d737v+z+48NEPDyfZbNxNDrpZmo0fxZ1Jkg6GycF0ME2TR6Nx0jmJ0+Rh/GILnz+cJ+PJIBu2pm9GyZOTTn84OB50O1OoOrrw4zbReDzux08WG9fWN9c3rp1GnaPR0YW1zY1N+hetFi7rwprS//azjz7+p2qrnspUV83UiUrUUE2hnKqOmsD1WF1Wm2oEdU/UAurGUBrQ80SdqvOAnUGrBFp0oPYF/O7D3WNdO4R7pDkhdBe4pPAzBmSkLgImg3ZjKCO3iJ7PiDLWltFeEE2U7Q38jTWtE6idqmdQK+FMy1Ac9mWqjtVvqQ8D6NOIarB3XU1lRlpBySOnV1OgMII6LPfg+RjKXUIaPUeEmVDfUbcdev4vaom1eN/VbWfq3yTlRbgi1dS9z3IKHTUn+hG9zRk8Y3lS4NwHConuI5Zeka5PqPdDaL+A+rtwnVLJ6CSGa0G1p5XILbh8yC0RuQOXD7kjIvfg8iH3ROQ+XD7kvkYidkw69+ObcPnwTZHzfbh8yPsi8gFcPuQDEXkIlw95KCK/gsuH/EpE3oTLh7wpIm/D5UPeFpEtuHzIlog8gMuHPBCR23D5kNsaWT5Tx3BlRGcgzMrrUC7yQEuRQs11Ub4bZB192BsBc7pbgpVndQP++rGNAJ0mJdjtgHF3XIKVR94O2Eg/VrZFt2g18WFvidhdGAF+7K6I/UI9L8F+ETDTXpRg5bm2B+38WNn63oE7P/aOiL0LJT9WXqPuQY0fey9gxRiVYPdF7H31sgQbYvXHJVjZ7jfBrvix8jrVgvZ+bIg1nZVgZXt6CB6MHyuvVg+h1o99KGIfqdcl2Eci9kuw7n7slwEr7NsSrFljz9MK0id/JIEZW0Wtk89KLI2AWkfgn+ZrS0q+cQz1EqafY/qEOREROzliJxCxlyP2guWa5HZ0Qv6uzKWZI5qBiDhfm7A0Fdv38vZYSgMQjRzRWEJUeaT4rk1f5uRdmBoJOc1XLiyF9CnL7TeWEj0eqi2vQdwrIHhsP6ORv07REkZQqKkqas/yNZ6REd1XIV5R9GZ6aXjIuGluFVzUaxEVe1CxiHrjQb0RUTMPaiai5h7UXETZme/i2gEjwOof38WC7ngEsI9cfkXgFVyHVecWzNEIxs8+eIEPqOYe/G1S7C1dVZJhNI/rJGY5nhQs8RhKC7UG9TYqbFB8ndIMS0AybnlPx/h4h7mNhZ5zbIVP85U8yjMm4XQGJE8/p4PeYkTzqR6d21RzSt4dl+rhb+Xz3pTq4bdJ46fkxXOpHn6qpZ+eQfaWxrbOgG3CbBpp7dtyXRqcf2EapnyeVl20uPhWT/SYQXqva9Lf1W9m9wzvZYtKrB9brkdj4vRvUuhfHRpWzxNHz/WooPfEXq8pRbV7MtRxry3XlSGjVXSo5bB3dd8MtunpN2PK9Wjsg8e1RTH3winXHb2jvDe2XI/GoeK85yl58qZcj0af7lkftlyPBmZbOjrOt+W6lh01wLGzLde16kPKAmMOiMc811ivaEx+0kxTG5B/UJ2tcX3+1XUMczZP8xihmpL1bcvpxPlaVi2R8RcSsGrTmnKgfzFzfLAijYW6IsZXLMO0sL6v0rFrPGp+D7QYweznPQApZ56ChCYngdY7BYqXxair2DODuyLicJQcL6HaunYqeouWL2eNinVHVCvFZba3Vo9tstcTGnsj8gn3SLOSHvZK33AZRUlDewUNyfTq6O6tnq9F7W+KuNESYpSPtC7tCPFOWnWc6tN609HxRb3LM4WL93zs+MVs87G2NhjzZGSLUJYqnm47k0dy63BdXVc2x83PInqjaK/mZDUGtCM1EaNQky1mb3xB95b2Ae3JIQ+m0YX3GGkqI8W7ZphFx3x6RBbVtbcSb9SXydBxeUJW19jjanTfQfc96PoxzhasGHeh1IKY4QDuWgFRzvlcVxlpfKx+ke+OZvQGqyP6tGAhDQ22N0nBQlZF2c8KVF4BGkcDR+nhNJbpGHx7hZIc9fvksbFr0fJfpJ1bs7/doTFePprLMzE94nqFuEY0a3hXl++WObAEC++TK+S/VvcS+dXhiDZU4vrU4cx6GdKOf0IR7Ig845RmmzQ7iq3d/NTyE8NpX5m9c9zNzshCRmT/IlifMhqTEf24ZwfMDjpbhJRsZIjdGeTejc/XGYhjzPpxA8WnGux4S8iWzYi/oevOrgmNRY4YeB04XRrbRid75AsmxHWsrbud29WrDyLtOQl3lDBFO1YuEf9P6bf5MeNkbWVEoIbxDUy0rfO9j4xiFtRRh1b5ahtk2rpSfpLL8FRLbdc/K9MnBckaFHGhPLha94Bzl+6ZF46SMck9WWnD62hVNhcpj5b0iL09piie7X5fr8Ao9zqtkms059o0SvowCqZ5FGHaSlnkZb7VvIrUw2hP/i/Ura6LWkOKkbIZXNaQlN9PKFpzpUxhVPP4fUGzya/18VKraj5DGosnzlx+B7Ufw28jt7kPoxMXrMINGgNMwd5ZjXBNtNIijNeNAi8zMg0te2/52TFpWrk1Z4mv2brZGHtem8o+jZrXOmthymeh8dyh8TxQhy3aa7RaNPXGEh2JsUVL71aG8qvDrVWD8kykLHtkBjUIkNKNpcKo9kSqcoxvUG9FWpsirQ7MVnc3wJ3zIUj/XF+e3e/y1T1SN8m36ZIHxvFLj2bpgHwuU1sdqTEF5HxV21d39repBrnHZEGRMp/jxBnDu05duk5zSX+mV7aM7Ly1CObc0ivdxtjYNpV/tYI8oTkxoXlpEFepRaLld+WIlizShuNzRJT575BPxX5HdczstrbvJCr4Ezbe5FlleXGkMCT9S5m33ZXoddeJXyOKCWfau46BVv03jBQYYzIJfs9yQm8IVzneSWCPNib7uWqneBdv6Ei0QVIv1O8CbAxHvXasu2PL9Nj07efQErVu37qvhcwvDeYo8TvLjl6HVrUT7aMulu7PRqujV7nifZUeZkt8rT5m1MaNLGyUV8S01WfBXFiielwYE8KlXi/qyF9P8joy8+5UKGXT2lAuZhrYxjyjeEk6B4oIn3d3yevNfSr0I16hFxPWpcY1EiXMxmU6P+BaWsxKRUsRklsvrUmpsx6VrReWh7tqWDvOljIhK5gqKXfDrd0+tAvRipyNYQpdxSd7y+JEl+ZncOHvSPmiRMMxJIfYBD/3utpS2+/hVMRLXebMZkQ1aBN6SzF4R/ez2KJaRy8d6i79EA7hPAaga0n6Aa2odWVnyrLkLvVw+q/IGoxVIkpvW9bvg8tF7skqpzr9GZCFk3szUOabnLp9MRxCelLkEs6H9zekXhwr821TvT4Y6nIPihzq8DDnGcLeuW1dn5fLqVpfq1xCefA6YHZeDA53AMtjFtsuxEKNnTfy/jmgdTiuoG5Wi/+1H4aP5VSfVyi3CX1z9jzgrXO7RGdm0S+uP2cst5DRXM4xnGeW9856TX5+7P9Ftd5U5vTm/dNHv9SOAcNroTgfKkvHeHcUWXlDqeD+gE+GTP1H/f2c/FXCy5xGmRx1KJn9inJqpoVMzXx56eudeRYik6VTJlORmo0nmnQydkvtqpvws5V7gHVPifI3lfwXsf7vaHtQe0zWw2TTOYPQprqEsiB2N61H9/YcbZnEeKaXz/i2oAb3xPeoFs/73qX2eOa3Vehb+ZckPNfvqEz1CpHJ8i6fnVcx9KC4A8e5IPO9b0Rn6jmbxSfQTgL2GPkcFUdK5uvnBSF6FBcuS7oghBktVZRjL+WYziQlJbTjQt+6NMJHeqcf9x3wfH4nzy5F6pdU19GrA67UklT7HqkeU2YgJv1vQoR2Ta3D33Vd9ku6vyLphN5BUaLXzrPqk2Cn3nFhv2a8SHkwk6mb63YZRfV297A6E9so5cIn3qvx/Qp835GySW/rBcXdY1WdO5xV0Jxpmdz93KEyeU/WA0aznXx8VMfP8wpe84D+3y5F33Yk3QFZYsq2R7SfNyZ6qdbNNknP5yqr87a3KqQ1X20yTXuy0o4Dc0ayek8g1eOufPbzOUgpV5OU0HHnOp/IlE6LDLyU5Pk5CjgN0QnordzXkJ5KVGaiJLOAL5HnAbLMA+gcC9IcixT6oiTaPhxdWLu8/H99rBYOr2xc/vXG1ftX1z6/of8fkA/VT9RP1SVY+36jPofxv68OgNPv1R/VX9RfG+8af2j8qfFnbvrBOY35kSr8a/ztv8XMSIA=
ap

14. Two Layers Perceptron
Wx
x
w1
a1
p = 6 neurons p = 30 neurons p = 100 neurons
Input y = f(x)
! sum of “ridge” functions (hx, wi + b)
f✓(x) ,
p
X
s=1
as (hx, ws
i + bs)
wp
AABE+nictVxLcxxJES4vr8W8vHAhgksvWhNeQgjZ2EDEBhFra2RZa9mWPSPZux5bMT3TGo/dmh7Py4+x+DEEF4KAE3d+Bz+ACDjxF8hHVVf1THVntTDukFRdXV9mVnZVVmZWteNROphMNzf/ce6Dr339G9/81offPv+d737v+z+48NEPDyfZbNxNDrpZmo0fxZ1Jkg6GycF0ME2TR6Nx0jmJ0+Rh/GILnz+cJ+PJIBu2pm9GyZOTTn84OB50O1OoOrrw4zbReDzux08WG9fWN9c3rp1GnaPR0YW1zY1N+hetFi7rwprS//azjz7+p2qrnspUV83UiUrUUE2hnKqOmsD1WF1Wm2oEdU/UAurGUBrQ80SdqvOAnUGrBFp0oPYF/O7D3WNdO4R7pDkhdBe4pPAzBmSkLgImg3ZjKCO3iJ7PiDLWltFeEE2U7Q38jTWtE6idqmdQK+FMy1Ac9mWqjtVvqQ8D6NOIarB3XU1lRlpBySOnV1OgMII6LPfg+RjKXUIaPUeEmVDfUbcdev4vaom1eN/VbWfq3yTlRbgi1dS9z3IKHTUn+hG9zRk8Y3lS4NwHConuI5Zeka5PqPdDaL+A+rtwnVLJ6CSGa0G1p5XILbh8yC0RuQOXD7kjIvfg8iH3ROQ+XD7kvkYidkw69+ObcPnwTZHzfbh8yPsi8gFcPuQDEXkIlw95KCK/gsuH/EpE3oTLh7wpIm/D5UPeFpEtuHzIlog8gMuHPBCR23D5kNsaWT5Tx3BlRGcgzMrrUC7yQEuRQs11Ub4bZB192BsBc7pbgpVndQP++rGNAJ0mJdjtgHF3XIKVR94O2Eg/VrZFt2g18WFvidhdGAF+7K6I/UI9L8F+ETDTXpRg5bm2B+38WNn63oE7P/aOiL0LJT9WXqPuQY0fey9gxRiVYPdF7H31sgQbYvXHJVjZ7jfBrvix8jrVgvZ+bIg1nZVgZXt6CB6MHyuvVg+h1o99KGIfqdcl2Eci9kuw7n7slwEr7NsSrFljz9MK0id/JIEZW0Wtk89KLI2AWkfgn+ZrS0q+cQz1EqafY/qEOREROzliJxCxlyP2guWa5HZ0Qv6uzKWZI5qBiDhfm7A0Fdv38vZYSgMQjRzRWEJUeaT4rk1f5uRdmBoJOc1XLiyF9CnL7TeWEj0eqi2vQdwrIHhsP6ORv07REkZQqKkqas/yNZ6REd1XIV5R9GZ6aXjIuGluFVzUaxEVe1CxiHrjQb0RUTMPaiai5h7UXETZme/i2gEjwOof38WC7ngEsI9cfkXgFVyHVecWzNEIxs8+eIEPqOYe/G1S7C1dVZJhNI/rJGY5nhQs8RhKC7UG9TYqbFB8ndIMS0AybnlPx/h4h7mNhZ5zbIVP85U8yjMm4XQGJE8/p4PeYkTzqR6d21RzSt4dl+rhb+Xz3pTq4bdJ46fkxXOpHn6qpZ+eQfaWxrbOgG3CbBpp7dtyXRqcf2EapnyeVl20uPhWT/SYQXqva9Lf1W9m9wzvZYtKrB9brkdj4vRvUuhfHRpWzxNHz/WooPfEXq8pRbV7MtRxry3XlSGjVXSo5bB3dd8MtunpN2PK9Wjsg8e1RTH3winXHb2jvDe2XI/GoeK85yl58qZcj0af7lkftlyPBmZbOjrOt+W6lh01wLGzLde16kPKAmMOiMc811ivaEx+0kxTG5B/UJ2tcX3+1XUMczZP8xihmpL1bcvpxPlaVi2R8RcSsGrTmnKgfzFzfLAijYW6IsZXLMO0sL6v0rFrPGp+D7QYweznPQApZ56ChCYngdY7BYqXxair2DODuyLicJQcL6HaunYqeouWL2eNinVHVCvFZba3Vo9tstcTGnsj8gn3SLOSHvZK33AZRUlDewUNyfTq6O6tnq9F7W+KuNESYpSPtC7tCPFOWnWc6tN609HxRb3LM4WL93zs+MVs87G2NhjzZGSLUJYqnm47k0dy63BdXVc2x83PInqjaK/mZDUGtCM1EaNQky1mb3xB95b2Ae3JIQ+m0YX3GGkqI8W7ZphFx3x6RBbVtbcSb9SXydBxeUJW19jjanTfQfc96PoxzhasGHeh1IKY4QDuWgFRzvlcVxlpfKx+ke+OZvQGqyP6tGAhDQ22N0nBQlZF2c8KVF4BGkcDR+nhNJbpGHx7hZIc9fvksbFr0fJfpJ1bs7/doTFePprLMzE94nqFuEY0a3hXl++WObAEC++TK+S/VvcS+dXhiDZU4vrU4cx6GdKOf0IR7Ig845RmmzQ7iq3d/NTyE8NpX5m9c9zNzshCRmT/IlifMhqTEf24ZwfMDjpbhJRsZIjdGeTejc/XGYhjzPpxA8WnGux4S8iWzYi/oevOrgmNRY4YeB04XRrbRid75AsmxHWsrbud29WrDyLtOQl3lDBFO1YuEf9P6bf5MeNkbWVEoIbxDUy0rfO9j4xiFtRRh1b5ahtk2rpSfpLL8FRLbdc/K9MnBckaFHGhPLha94Bzl+6ZF46SMck9WWnD62hVNhcpj5b0iL09piie7X5fr8Ao9zqtkms059o0SvowCqZ5FGHaSlnkZb7VvIrUw2hP/i/Ura6LWkOKkbIZXNaQlN9PKFpzpUxhVPP4fUGzya/18VKraj5DGosnzlx+B7Ufw28jt7kPoxMXrMINGgNMwd5ZjXBNtNIijNeNAi8zMg0te2/52TFpWrk1Z4mv2brZGHtem8o+jZrXOmthymeh8dyh8TxQhy3aa7RaNPXGEh2JsUVL71aG8qvDrVWD8kykLHtkBjUIkNKNpcKo9kSqcoxvUG9FWpsirQ7MVnc3wJ3zIUj/XF+e3e/y1T1SN8m36ZIHxvFLj2bpgHwuU1sdqTEF5HxV21d39repBrnHZEGRMp/jxBnDu05duk5zSX+mV7aM7Ly1CObc0ivdxtjYNpV/tYI8oTkxoXlpEFepRaLld+WIlizShuNzRJT575BPxX5HdczstrbvJCr4Ezbe5FlleXGkMCT9S5m33ZXoddeJXyOKCWfau46BVv03jBQYYzIJfs9yQm8IVzneSWCPNib7uWqneBdv6Ei0QVIv1O8CbAxHvXasu2PL9Nj07efQErVu37qvhcwvDeYo8TvLjl6HVrUT7aMulu7PRqujV7nifZUeZkt8rT5m1MaNLGyUV8S01WfBXFiielwYE8KlXi/qyF9P8joy8+5UKGXT2lAuZhrYxjyjeEk6B4oIn3d3yevNfSr0I16hFxPWpcY1EiXMxmU6P+BaWsxKRUsRklsvrUmpsx6VrReWh7tqWDvOljIhK5gqKXfDrd0+tAvRipyNYQpdxSd7y+JEl+ZncOHvSPmiRMMxJIfYBD/3utpS2+/hVMRLXebMZkQ1aBN6SzF4R/ez2KJaRy8d6i79EA7hPAaga0n6Aa2odWVnyrLkLvVw+q/IGoxVIkpvW9bvg8tF7skqpzr9GZCFk3szUOabnLp9MRxCelLkEs6H9zekXhwr821TvT4Y6nIPihzq8DDnGcLeuW1dn5fLqVpfq1xCefA6YHZeDA53AMtjFtsuxEKNnTfy/jmgdTiuoG5Wi/+1H4aP5VSfVyi3CX1z9jzgrXO7RGdm0S+uP2cst5DRXM4xnGeW9856TX5+7P9Ftd5U5vTm/dNHv9SOAcNroTgfKkvHeHcUWXlDqeD+gE+GTP1H/f2c/FXCy5xGmRx1KJn9inJqpoVMzXx56eudeRYik6VTJlORmo0nmnQydkvtqpvws5V7gHVPifI3lfwXsf7vaHtQe0zWw2TTOYPQprqEsiB2N61H9/YcbZnEeKaXz/i2oAb3xPeoFs/73qX2eOa3Vehb+ZckPNfvqEz1CpHJ8i6fnVcx9KC4A8e5IPO9b0Rn6jmbxSfQTgL2GPkcFUdK5uvnBSF6FBcuS7oghBktVZRjL+WYziQlJbTjQt+6NMJHeqcf9x3wfH4nzy5F6pdU19GrA67UklT7HqkeU2YgJv1vQoR2Ta3D33Vd9ku6vyLphN5BUaLXzrPqk2Cn3nFhv2a8SHkwk6mb63YZRfV297A6E9so5cIn3qvx/Qp835GySW/rBcXdY1WdO5xV0Jxpmdz93KEyeU/WA0aznXx8VMfP8wpe84D+3y5F33Yk3QFZYsq2R7SfNyZ6qdbNNknP5yqr87a3KqQ1X20yTXuy0o4Dc0ayek8g1eOufPbzOUgpV5OU0HHnOp/IlE6LDLyU5Pk5CjgN0QnordzXkJ5KVGaiJLOAL5HnAbLMA+gcC9IcixT6oiTaPhxdWLu8/H99rBYOr2xc/vXG1ftX1z6/of8fkA/VT9RP1SVY+36jPofxv68OgNPv1R/VX9RfG+8af2j8qfFnbvrBOY35kSr8a/ztv8XMSIA=
ap

15. Universality of Perceptrons
George Cybenko
If f is continuous on a compact ⌦, for all " > 0
AABB9nictVzNchu5EYY3P7t2/rzJMZdJZKe8KUcrK64kVVupWlmSZa1pWzYp2btL2zUkR9TYQw49Q1KyuXqRHHJL5Zo3SK7JG+QNklNeId0NYIAhMdMYxTFKEgbE193oARrdDdC9SRLn042Nf1764Fvf/s53P/zo8pXvff8HP/zR1Y9/fJSns6wfHfbTJM2e9cI8SuJxdDiNp0n0bJJF4aiXRE97r7fx86fzKMvjdNyZvp1Ez0fhcBwfx/1wCk0vr96+1s3i4ck0zLL09FowTsfBm1k4BkrQYR4F3WSQTnNoD46zKAqS2bh/sv7y6trG+gb9C1Yrt1RlTah/B+nHwaHoioFIRV/MxEhEYiymUE9EKHIoX4tbYkNMoO25WEBbBrWYPo/EubgC2Bn0iqBHCK2v4fcQnr5WrWN4Rpo5ofvAJYGfDJCBuA6YFPplUEduAX0+I8rYWkV7QTRRtrfwt6dojaB1Kk6glcPpnr44HMtUHIvf0RhiGNOEWnB0fUVlRlpByQNrVFOgMIE2rA/g8wzqfUJqPQeEyWnsqNuQPv8X9cRWfO6rvjPxb5LyOpRAtNXo04JCKOZEP6C3OYPPpDwJcB4ChUiNEWunpOsRjX4M/RfQ/hDKOdW0TnpQFtR6XovchuJCbrPIPSgu5B6LbEFxIVss8gCKC3mgkIjNSOdufBuKC99mOT+G4kI+ZpFPoLiQT1jkERQX8ohFfgXFhfyKRd6F4kLeZZH3obiQ91lkB4oL2WGRh1BcyEMWuQvFhdxVyOqVmkFJiU7MrMotqJd5oKVIoGWLle8OWUcX9o7Hmu5XYPlVvQN/3dgdD51GFdhdj3l3XIHlZ94e2Eg3lrdF92g3cWHvsdh9mAFu7D6L/UK8qsB+4bHSXldg+bXWgn5uLG99H8CTG/uAxT6EmhvL71GPoMWNfeSxY0wqsAcs9rF4U4H1sfpZBZa3+22wK24sv091oL8b62NNZxVY3p4egQfjxvK71VNodWOfsthn4qwC+4zFfgnW3Y390mOHfVeB1XvsFdpBhuSPRLBi66iFxarE2gSohQz/pNhbEvKNe9DOYYYFZkiYEYvYKxB7nohWgWh5y5UXdjQnf5fn0i4QbU9Er9ibsDZl+w+K/lhLPBA7BWJnCVHnkeK71mOZk3ehWzjktNi5sOYzprSw31iL1Hyot7wa8aiEkHP7hGb+TYqWMIJCTdVROyn2eIkM6LkOcUrRmx6l5sHjpoVVsFFnLKrnQPVY1FsH6i2LmjlQMxY1d6DmLMqsfBvX9ZgBRv/4Lhb0JGeA9JGrSwBewRbsOvdgjQYwfw7AC3xCLY/gb5tib67USYbRPO6TmOV4XrLEGdQWYg3aTVS4Q/F1QissAslkz0cqxscnzG0s1JqTVvi82MmDImPiTycmeYYFHfQWA1pPzejcp5Zz8u5krRn+XrHuda0Zfpc0fk5evKw1w0+V9NMLyN5R2M4FsG1YTROlfVNvSkPmXyQNXb9Cuy5aXHyrIzVnkN5ZQ/r76s3sX+C9bFNN6sfUm9HIrfHlpfE1oWH0nFt6bkYFvSfp9epa0HgkYxX3mnpTGVLaRcdKDvPU9M1gn4F6M7rejMYBeFzbFHMvrHrT2TspRmPqzWgcCZn3PCdPXteb0RjSs9SHqTejgdmWUMX5pt7UsqMGZOxs6k2t+piywJgDknNethivKCM/aaaoxeQf1GdrbJ9/dR/DnM2LIkaop2R822o6vWIvq5dI+wsRWLVpQznQv5hZPliZxkJssvGVlGFa2t9X6Zg9HjXfAi0GsPrlGQCXM09AQp2TQOudAMVbbNRVHpnGbbI4nCXHS6iuap2y3qLhK7NG5baX1MrFZWa0Ro9dstc5zb0J+YQt0iynh1blG66iyGmoVdIQT6+J7t6p9VrW/gaLmywhJsVM69OJkDxJq49TXVpvWzq+rk55plDkmY+Zv5htPlbWBmOelGwRylLH0+6n80h2G+6rN4XJccvPAnqjaK/mZDViOpHK2ShUZ4ulN76gZ0P7kM7kkIek0Yf3GCgqEyFPzTCLjvn0gCyqbW853qgvnaGT9ZysrrbH9eihhR460M1jnG3YMR5CrQMxwyE8dTyinCuFrlLSeCZ+VZyOpvQG6yP6pGQhNQ1pb6KShayLsk9KVE4BjbNBRun+NJbpaHx3hRIf9bvkMbFr2fJfp5Nbfb4d0hyvns3VmZgBcd0krgGtGnmqK5+WOUgJFs5PNsl/rR8l8mvCEW0ox/WFxVnqZUwn/hFFsBPyjBNabdzqKPe281PLn2hOB0KfneNpdkoWMiD7F8D+lNKcDOjHvjugT9ClRUjIRvrYnbjwbly+TszOMePHxULeajDzLSJbNiP+mq69unKaizJikPvA+dLc1jppkS8YEddMWXeztut3H0SaexL2LJEUzVy5Qfw/od/6R8+TtZUZgRrGN5ArW+d6HynFLKijkHb5ehuk+9pSXitkeKGkNvufkelaSbIdirhQHtytB8C5T8+SF86SjOTOV/rIfbQum4uUJ0t6xNEeUxQv7f5Q7cAo903aJddozXVplgxhFkyLKEL35bLIy3zreZWp+9HO/y/Uja7LWkOKgTAZXKkhLr8fUbRmS5nArJbz9zWtJrfWs6Ve9XzGNBdH1lr+Blp/Br+13PrZj06vZBXu0ByQFMyT0YhsCVZ6+PG6U+KlZ6amZZ4NPzMndS+75SLxtbRuJsaeN6ZyQLPmTGUtdP0iNF5ZNF556rBDZ41Gi7pdW6KXbGzRUaeVvvyacOs0oDxjKfMemUbFHlLasZQf1QFLlY/xNeodS2uDpRXCarVPA+w174N0r/Xl1f1NsbsH4i75Nn3ywGT8MqBVGpPPpVvrIzVJATnfVvbVXv1dakHuPbKgSFne48QVI0+d+lTOC0l/oXa2lOy8sQj63tKp6qNtbJfqv15BjmhN5LQuNeI29YiU/LYcwZJFWrd8joAy/yH5VNLvqI+Z7d7mnQQlf8LEm3JVGV4yUhiT/rnM2/5K9Lpvxa8BxYQz5V33gFbzN4wUJEZnEtyeZU5vCHc5eZIgPdoe2c9VOyVP8caWROsk9UL83sPGyKjXzHV7bukR67H9Enqi1s1bd/Xg+SXeHDl+FznRC2lXGykfdbH0fDFaodrlys91epgt8TX6mFEfO7IwUV4Z0xWfeXOREjXjIjE+XJqNoon8zSRvIrM8nfKlrHtryuVMg7QxJxQvcfdAEeHy7m44vblPmHH0Vuj1CGtTky0cJczGpSo/YFtazEpdXtmHZOvl2t0osXaiqp1CU7d3C2O/pYWMyPolgsvZyN627N1SlMJnYSSFvpA3eqviQ5vmZ1DwdyBc0aHm6JM7bIN/uyW2xe57uA3xRtVlRjOgFrQFg6XYO1TjLPeo19Ebi7pN34eDP48YdM1JH9NO2lR2SZmX3KbuT/+UrEAmIlZ607P5GGwu/EhWOTUZT0yWjR9NLPR3cZqORXPwGUmZiz8fea7BjeJY6O80NRuDps6PoMyhCQ99j8HvnZvezXnZnOr1tcrFl4fcBfSJi8bhyV91rGL6+ViozHoj758DWofjGup6t/hfx6H5GE7Neflyy+m7Zq883rrsF6mMLPrDzdeM4eYzm6s5+vNMi9EZb8nNT/p9QaM3lVqjef/00R81c0DzWgiZB+Wlk3h7Fhl5fanguYBLhlT8R/z1Ev9thDcFjSo5mlDS5xTV1HQPnpr+xqVrdPozH5kMnSqZytRMHNGmG7HbYl/chZ/twgNsejtUfpdS/kWs+/uzA2g9Juuhs+gyc9CltoiyH+YUbUDP5v5slcR4l1fe7e1AC56Ft6gV7/k+pP5417dTGlv1N0jkWn8gUjEoRSTLp3tmXfVgBOWTN5kD0t/zDeguvcxiyZtnI4+zRX1/almiBX3C3yzoVeJ7lpR9mqsTdVaPJwd4wz4s8kOB+JTaQmXncc/lOB9Ucj5Y4pyTdsoczqzP6u9mVXHZtrgMitzZXPVLKc4253n1udGdSi7yDno9fliDH1pStkn7rykSzkR9Nm9WQ3OmZLJPWMdCZyKlHjDODIv3XR/Zzmt4zT3Gf78Sfd+SdA9k6VH+O6ATtozoJUo3uyS9vOlYn0m9VyOt+h7ly6trt5b/L4PVytHm+q3frN9+vLn2+R31/xx8JH4qfi5uwBr/rfgcqB2IQ+DwB/E38Xfxj62zrT9u/Wnrz7LrB5cU5iei9G/rL/8FKxG1ug==
! non quantitative . . . no free lunch.
for p large enough, there exists ✓ such that
8 x 2 ⌦, |f✓(x) f(x)| 6 "

16. Universality of Perceptrons
George Cybenko
Andrew Barron
If f is continuous on a compact ⌦, for all " > 0
AABB9nictVzNchu5EYY3P7t2/rzJMZdJZKe8KUcrK64kVVupWlmSZa1pWzYp2btL2zUkR9TYQw49Q1KyuXqRHHJL5Zo3SK7JG+QNklNeId0NYIAhMdMYxTFKEgbE193oARrdDdC9SRLn042Nf1764Fvf/s53P/zo8pXvff8HP/zR1Y9/fJSns6wfHfbTJM2e9cI8SuJxdDiNp0n0bJJF4aiXRE97r7fx86fzKMvjdNyZvp1Ez0fhcBwfx/1wCk0vr96+1s3i4ck0zLL09FowTsfBm1k4BkrQYR4F3WSQTnNoD46zKAqS2bh/sv7y6trG+gb9C1Yrt1RlTah/B+nHwaHoioFIRV/MxEhEYiymUE9EKHIoX4tbYkNMoO25WEBbBrWYPo/EubgC2Bn0iqBHCK2v4fcQnr5WrWN4Rpo5ofvAJYGfDJCBuA6YFPplUEduAX0+I8rYWkV7QTRRtrfwt6dojaB1Kk6glcPpnr44HMtUHIvf0RhiGNOEWnB0fUVlRlpByQNrVFOgMIE2rA/g8wzqfUJqPQeEyWnsqNuQPv8X9cRWfO6rvjPxb5LyOpRAtNXo04JCKOZEP6C3OYPPpDwJcB4ChUiNEWunpOsRjX4M/RfQ/hDKOdW0TnpQFtR6XovchuJCbrPIPSgu5B6LbEFxIVss8gCKC3mgkIjNSOdufBuKC99mOT+G4kI+ZpFPoLiQT1jkERQX8ohFfgXFhfyKRd6F4kLeZZH3obiQ91lkB4oL2WGRh1BcyEMWuQvFhdxVyOqVmkFJiU7MrMotqJd5oKVIoGWLle8OWUcX9o7Hmu5XYPlVvQN/3dgdD51GFdhdj3l3XIHlZ94e2Eg3lrdF92g3cWHvsdh9mAFu7D6L/UK8qsB+4bHSXldg+bXWgn5uLG99H8CTG/uAxT6EmhvL71GPoMWNfeSxY0wqsAcs9rF4U4H1sfpZBZa3+22wK24sv091oL8b62NNZxVY3p4egQfjxvK71VNodWOfsthn4qwC+4zFfgnW3Y390mOHfVeB1XvsFdpBhuSPRLBi66iFxarE2gSohQz/pNhbEvKNe9DOYYYFZkiYEYvYKxB7nohWgWh5y5UXdjQnf5fn0i4QbU9Er9ibsDZl+w+K/lhLPBA7BWJnCVHnkeK71mOZk3ehWzjktNi5sOYzprSw31iL1Hyot7wa8aiEkHP7hGb+TYqWMIJCTdVROyn2eIkM6LkOcUrRmx6l5sHjpoVVsFFnLKrnQPVY1FsH6i2LmjlQMxY1d6DmLMqsfBvX9ZgBRv/4Lhb0JGeA9JGrSwBewRbsOvdgjQYwfw7AC3xCLY/gb5tib67USYbRPO6TmOV4XrLEGdQWYg3aTVS4Q/F1QissAslkz0cqxscnzG0s1JqTVvi82MmDImPiTycmeYYFHfQWA1pPzejcp5Zz8u5krRn+XrHuda0Zfpc0fk5evKw1w0+V9NMLyN5R2M4FsG1YTROlfVNvSkPmXyQNXb9Cuy5aXHyrIzVnkN5ZQ/r76s3sX+C9bFNN6sfUm9HIrfHlpfE1oWH0nFt6bkYFvSfp9epa0HgkYxX3mnpTGVLaRcdKDvPU9M1gn4F6M7rejMYBeFzbFHMvrHrT2TspRmPqzWgcCZn3PCdPXteb0RjSs9SHqTejgdmWUMX5pt7UsqMGZOxs6k2t+piywJgDknNethivKCM/aaaoxeQf1GdrbJ9/dR/DnM2LIkaop2R822o6vWIvq5dI+wsRWLVpQznQv5hZPliZxkJssvGVlGFa2t9X6Zg9HjXfAi0GsPrlGQCXM09AQp2TQOudAMVbbNRVHpnGbbI4nCXHS6iuap2y3qLhK7NG5baX1MrFZWa0Ro9dstc5zb0J+YQt0iynh1blG66iyGmoVdIQT6+J7t6p9VrW/gaLmywhJsVM69OJkDxJq49TXVpvWzq+rk55plDkmY+Zv5htPlbWBmOelGwRylLH0+6n80h2G+6rN4XJccvPAnqjaK/mZDViOpHK2ShUZ4ulN76gZ0P7kM7kkIek0Yf3GCgqEyFPzTCLjvn0gCyqbW853qgvnaGT9ZysrrbH9eihhR460M1jnG3YMR5CrQMxwyE8dTyinCuFrlLSeCZ+VZyOpvQG6yP6pGQhNQ1pb6KShayLsk9KVE4BjbNBRun+NJbpaHx3hRIf9bvkMbFr2fJfp5Nbfb4d0hyvns3VmZgBcd0krgGtGnmqK5+WOUgJFs5PNsl/rR8l8mvCEW0ox/WFxVnqZUwn/hFFsBPyjBNabdzqKPe281PLn2hOB0KfneNpdkoWMiD7F8D+lNKcDOjHvjugT9ClRUjIRvrYnbjwbly+TszOMePHxULeajDzLSJbNiP+mq69unKaizJikPvA+dLc1jppkS8YEddMWXeztut3H0SaexL2LJEUzVy5Qfw/od/6R8+TtZUZgRrGN5ArW+d6HynFLKijkHb5ehuk+9pSXitkeKGkNvufkelaSbIdirhQHtytB8C5T8+SF86SjOTOV/rIfbQum4uUJ0t6xNEeUxQv7f5Q7cAo903aJddozXVplgxhFkyLKEL35bLIy3zreZWp+9HO/y/Uja7LWkOKgTAZXKkhLr8fUbRmS5nArJbz9zWtJrfWs6Ve9XzGNBdH1lr+Blp/Br+13PrZj06vZBXu0ByQFMyT0YhsCVZ6+PG6U+KlZ6amZZ4NPzMndS+75SLxtbRuJsaeN6ZyQLPmTGUtdP0iNF5ZNF556rBDZ41Gi7pdW6KXbGzRUaeVvvyacOs0oDxjKfMemUbFHlLasZQf1QFLlY/xNeodS2uDpRXCarVPA+w174N0r/Xl1f1NsbsH4i75Nn3ywGT8MqBVGpPPpVvrIzVJATnfVvbVXv1dakHuPbKgSFne48QVI0+d+lTOC0l/oXa2lOy8sQj63tKp6qNtbJfqv15BjmhN5LQuNeI29YiU/LYcwZJFWrd8joAy/yH5VNLvqI+Z7d7mnQQlf8LEm3JVGV4yUhiT/rnM2/5K9Lpvxa8BxYQz5V33gFbzN4wUJEZnEtyeZU5vCHc5eZIgPdoe2c9VOyVP8caWROsk9UL83sPGyKjXzHV7bukR67H9Enqi1s1bd/Xg+SXeHDl+FznRC2lXGykfdbH0fDFaodrlys91epgt8TX6mFEfO7IwUV4Z0xWfeXOREjXjIjE+XJqNoon8zSRvIrM8nfKlrHtryuVMg7QxJxQvcfdAEeHy7m44vblPmHH0Vuj1CGtTky0cJczGpSo/YFtazEpdXtmHZOvl2t0osXaiqp1CU7d3C2O/pYWMyPolgsvZyN627N1SlMJnYSSFvpA3eqviQ5vmZ1DwdyBc0aHm6JM7bIN/uyW2xe57uA3xRtVlRjOgFrQFg6XYO1TjLPeo19Ebi7pN34eDP48YdM1JH9NO2lR2SZmX3KbuT/+UrEAmIlZ607P5GGwu/EhWOTUZT0yWjR9NLPR3cZqORXPwGUmZiz8fea7BjeJY6O80NRuDps6PoMyhCQ99j8HvnZvezXnZnOr1tcrFl4fcBfSJi8bhyV91rGL6+ViozHoj758DWofjGup6t/hfx6H5GE7Neflyy+m7Zq883rrsF6mMLPrDzdeM4eYzm6s5+vNMi9EZb8nNT/p9QaM3lVqjef/00R81c0DzWgiZB+Wlk3h7Fhl5fanguYBLhlT8R/z1Ev9thDcFjSo5mlDS5xTV1HQPnpr+xqVrdPozH5kMnSqZytRMHNGmG7HbYl/chZ/twgNsejtUfpdS/kWs+/uzA2g9Juuhs+gyc9CltoiyH+YUbUDP5v5slcR4l1fe7e1AC56Ft6gV7/k+pP5417dTGlv1N0jkWn8gUjEoRSTLp3tmXfVgBOWTN5kD0t/zDeguvcxiyZtnI4+zRX1/almiBX3C3yzoVeJ7lpR9mqsTdVaPJwd4wz4s8kOB+JTaQmXncc/lOB9Ucj5Y4pyTdsoczqzP6u9mVXHZtrgMitzZXPVLKc4253n1udGdSi7yDno9fliDH1pStkn7rykSzkR9Nm9WQ3OmZLJPWMdCZyKlHjDODIv3XR/Zzmt4zT3Gf78Sfd+SdA9k6VH+O6ATtozoJUo3uyS9vOlYn0m9VyOt+h7ly6trt5b/L4PVytHm+q3frN9+vLn2+R31/xx8JH4qfi5uwBr/rfgcqB2IQ+DwB/E38Xfxj62zrT9u/Wnrz7LrB5cU5iei9G/rL/8FKxG1ug==
! non quantitative . . . no free lunch.
for p large enough, there exists ✓ such that
8 x 2 ⌦, |f✓(x) f(x)| 6 "
Barron’s functions:
||f||B
,
R
Rd
||!||| ˆ
f(!)|d! < +1
||f f✓
||L2(⌦)
6 2diam(⌦)||f||B
p
p
Theorem: for p large, there exists ✓ such that

17. Universality of Perceptrons
George Cybenko
Andrew Barron
If f is continuous on a compact ⌦, for all " > 0
AABB9nictVzNchu5EYY3P7t2/rzJMZdJZKe8KUcrK64kVVupWlmSZa1pWzYp2btL2zUkR9TYQw49Q1KyuXqRHHJL5Zo3SK7JG+QNklNeId0NYIAhMdMYxTFKEgbE193oARrdDdC9SRLn042Nf1764Fvf/s53P/zo8pXvff8HP/zR1Y9/fJSns6wfHfbTJM2e9cI8SuJxdDiNp0n0bJJF4aiXRE97r7fx86fzKMvjdNyZvp1Ez0fhcBwfx/1wCk0vr96+1s3i4ck0zLL09FowTsfBm1k4BkrQYR4F3WSQTnNoD46zKAqS2bh/sv7y6trG+gb9C1Yrt1RlTah/B+nHwaHoioFIRV/MxEhEYiymUE9EKHIoX4tbYkNMoO25WEBbBrWYPo/EubgC2Bn0iqBHCK2v4fcQnr5WrWN4Rpo5ofvAJYGfDJCBuA6YFPplUEduAX0+I8rYWkV7QTRRtrfwt6dojaB1Kk6glcPpnr44HMtUHIvf0RhiGNOEWnB0fUVlRlpByQNrVFOgMIE2rA/g8wzqfUJqPQeEyWnsqNuQPv8X9cRWfO6rvjPxb5LyOpRAtNXo04JCKOZEP6C3OYPPpDwJcB4ChUiNEWunpOsRjX4M/RfQ/hDKOdW0TnpQFtR6XovchuJCbrPIPSgu5B6LbEFxIVss8gCKC3mgkIjNSOdufBuKC99mOT+G4kI+ZpFPoLiQT1jkERQX8ohFfgXFhfyKRd6F4kLeZZH3obiQ91lkB4oL2WGRh1BcyEMWuQvFhdxVyOqVmkFJiU7MrMotqJd5oKVIoGWLle8OWUcX9o7Hmu5XYPlVvQN/3dgdD51GFdhdj3l3XIHlZ94e2Eg3lrdF92g3cWHvsdh9mAFu7D6L/UK8qsB+4bHSXldg+bXWgn5uLG99H8CTG/uAxT6EmhvL71GPoMWNfeSxY0wqsAcs9rF4U4H1sfpZBZa3+22wK24sv091oL8b62NNZxVY3p4egQfjxvK71VNodWOfsthn4qwC+4zFfgnW3Y390mOHfVeB1XvsFdpBhuSPRLBi66iFxarE2gSohQz/pNhbEvKNe9DOYYYFZkiYEYvYKxB7nohWgWh5y5UXdjQnf5fn0i4QbU9Er9ibsDZl+w+K/lhLPBA7BWJnCVHnkeK71mOZk3ehWzjktNi5sOYzprSw31iL1Hyot7wa8aiEkHP7hGb+TYqWMIJCTdVROyn2eIkM6LkOcUrRmx6l5sHjpoVVsFFnLKrnQPVY1FsH6i2LmjlQMxY1d6DmLMqsfBvX9ZgBRv/4Lhb0JGeA9JGrSwBewRbsOvdgjQYwfw7AC3xCLY/gb5tib67USYbRPO6TmOV4XrLEGdQWYg3aTVS4Q/F1QissAslkz0cqxscnzG0s1JqTVvi82MmDImPiTycmeYYFHfQWA1pPzejcp5Zz8u5krRn+XrHuda0Zfpc0fk5evKw1w0+V9NMLyN5R2M4FsG1YTROlfVNvSkPmXyQNXb9Cuy5aXHyrIzVnkN5ZQ/r76s3sX+C9bFNN6sfUm9HIrfHlpfE1oWH0nFt6bkYFvSfp9epa0HgkYxX3mnpTGVLaRcdKDvPU9M1gn4F6M7rejMYBeFzbFHMvrHrT2TspRmPqzWgcCZn3PCdPXteb0RjSs9SHqTejgdmWUMX5pt7UsqMGZOxs6k2t+piywJgDknNethivKCM/aaaoxeQf1GdrbJ9/dR/DnM2LIkaop2R822o6vWIvq5dI+wsRWLVpQznQv5hZPliZxkJssvGVlGFa2t9X6Zg9HjXfAi0GsPrlGQCXM09AQp2TQOudAMVbbNRVHpnGbbI4nCXHS6iuap2y3qLhK7NG5baX1MrFZWa0Ro9dstc5zb0J+YQt0iynh1blG66iyGmoVdIQT6+J7t6p9VrW/gaLmywhJsVM69OJkDxJq49TXVpvWzq+rk55plDkmY+Zv5htPlbWBmOelGwRylLH0+6n80h2G+6rN4XJccvPAnqjaK/mZDViOpHK2ShUZ4ulN76gZ0P7kM7kkIek0Yf3GCgqEyFPzTCLjvn0gCyqbW853qgvnaGT9ZysrrbH9eihhR460M1jnG3YMR5CrQMxwyE8dTyinCuFrlLSeCZ+VZyOpvQG6yP6pGQhNQ1pb6KShayLsk9KVE4BjbNBRun+NJbpaHx3hRIf9bvkMbFr2fJfp5Nbfb4d0hyvns3VmZgBcd0krgGtGnmqK5+WOUgJFs5PNsl/rR8l8mvCEW0ox/WFxVnqZUwn/hFFsBPyjBNabdzqKPe281PLn2hOB0KfneNpdkoWMiD7F8D+lNKcDOjHvjugT9ClRUjIRvrYnbjwbly+TszOMePHxULeajDzLSJbNiP+mq69unKaizJikPvA+dLc1jppkS8YEddMWXeztut3H0SaexL2LJEUzVy5Qfw/od/6R8+TtZUZgRrGN5ArW+d6HynFLKijkHb5ehuk+9pSXitkeKGkNvufkelaSbIdirhQHtytB8C5T8+SF86SjOTOV/rIfbQum4uUJ0t6xNEeUxQv7f5Q7cAo903aJddozXVplgxhFkyLKEL35bLIy3zreZWp+9HO/y/Uja7LWkOKgTAZXKkhLr8fUbRmS5nArJbz9zWtJrfWs6Ve9XzGNBdH1lr+Blp/Br+13PrZj06vZBXu0ByQFMyT0YhsCVZ6+PG6U+KlZ6amZZ4NPzMndS+75SLxtbRuJsaeN6ZyQLPmTGUtdP0iNF5ZNF556rBDZ41Gi7pdW6KXbGzRUaeVvvyacOs0oDxjKfMemUbFHlLasZQf1QFLlY/xNeodS2uDpRXCarVPA+w174N0r/Xl1f1NsbsH4i75Nn3ywGT8MqBVGpPPpVvrIzVJATnfVvbVXv1dakHuPbKgSFne48QVI0+d+lTOC0l/oXa2lOy8sQj63tKp6qNtbJfqv15BjmhN5LQuNeI29YiU/LYcwZJFWrd8joAy/yH5VNLvqI+Z7d7mnQQlf8LEm3JVGV4yUhiT/rnM2/5K9Lpvxa8BxYQz5V33gFbzN4wUJEZnEtyeZU5vCHc5eZIgPdoe2c9VOyVP8caWROsk9UL83sPGyKjXzHV7bukR67H9Enqi1s1bd/Xg+SXeHDl+FznRC2lXGykfdbH0fDFaodrlys91epgt8TX6mFEfO7IwUV4Z0xWfeXOREjXjIjE+XJqNoon8zSRvIrM8nfKlrHtryuVMg7QxJxQvcfdAEeHy7m44vblPmHH0Vuj1CGtTky0cJczGpSo/YFtazEpdXtmHZOvl2t0osXaiqp1CU7d3C2O/pYWMyPolgsvZyN627N1SlMJnYSSFvpA3eqviQ5vmZ1DwdyBc0aHm6JM7bIN/uyW2xe57uA3xRtVlRjOgFrQFg6XYO1TjLPeo19Ebi7pN34eDP48YdM1JH9NO2lR2SZmX3KbuT/+UrEAmIlZ607P5GGwu/EhWOTUZT0yWjR9NLPR3cZqORXPwGUmZiz8fea7BjeJY6O80NRuDps6PoMyhCQ99j8HvnZvezXnZnOr1tcrFl4fcBfSJi8bhyV91rGL6+ViozHoj758DWofjGup6t/hfx6H5GE7Neflyy+m7Zq883rrsF6mMLPrDzdeM4eYzm6s5+vNMi9EZb8nNT/p9QaM3lVqjef/00R81c0DzWgiZB+Wlk3h7Fhl5fanguYBLhlT8R/z1Ev9thDcFjSo5mlDS5xTV1HQPnpr+xqVrdPozH5kMnSqZytRMHNGmG7HbYl/chZ/twgNsejtUfpdS/kWs+/uzA2g9Juuhs+gyc9CltoiyH+YUbUDP5v5slcR4l1fe7e1AC56Ft6gV7/k+pP5417dTGlv1N0jkWn8gUjEoRSTLp3tmXfVgBOWTN5kD0t/zDeguvcxiyZtnI4+zRX1/almiBX3C3yzoVeJ7lpR9mqsTdVaPJwd4wz4s8kOB+JTaQmXncc/lOB9Ucj5Y4pyTdsoczqzP6u9mVXHZtrgMitzZXPVLKc4253n1udGdSi7yDno9fliDH1pStkn7rykSzkR9Nm9WQ3OmZLJPWMdCZyKlHjDODIv3XR/Zzmt4zT3Gf78Sfd+SdA9k6VH+O6ATtozoJUo3uyS9vOlYn0m9VyOt+h7ly6trt5b/L4PVytHm+q3frN9+vLn2+R31/xx8JH4qfi5uwBr/rfgcqB2IQ+DwB/E38Xfxj62zrT9u/Wnrz7LrB5cU5iei9G/rL/8FKxG1ug==
! non quantitative . . . no free lunch.
! non-constructive.
||f f✓
||
for p large enough, there exists ✓ such that
8 x 2 ⌦, |f✓(x) f(x)| 6 "
Barron’s functions:
||f||B
,
R
Rd
||!||| ˆ
f(!)|d! < +1
||f f✓
||L2(⌦)
6 2diam(⌦)||f||B
p
p
Theorem: for p large, there exists ✓ such that
! for p “large enough”
[Chizat-Bach 2018]
f✓
f

18. Overview
• Empirical Risk Minimization

• Perceptrons

• Optimization

• Convolutional Networks

• Residual Networks

• Transformers

Small ⌧`
Large ⌧` Optimal ⌧` = ⌧?
`
✓`+1 = ✓` ⌧`
rE(✓`)
min

E(✓) , 1
n
n
X
i=1
`(f✓(xi), yi)

Small ⌧`
Large ⌧` Optimal ⌧` = ⌧?
`
✓`+1 = ✓` ⌧`
rE(✓`)
min

E(✓) , 1
n
n
X
i=1
`(f✓(xi), yi)
Herbert Robbins
Sutton Monro
✓`+1 = ✓`

`
rE`(✓`)
E`(✓) , `(f✓(xi), yi)
i rand

21. The Complexity of Gradient Computation
Hypothesis: elementary operations (a ⇥ b, log(a),
p
a . . . )
and their derivatives cost O(1).
Setup: E : Rd ! R computable in K operations.
Question: What is the complexity of computing rE : Rd ! Rd?

22. The Complexity of Gradient Computation
Hypothesis: elementary operations (a ⇥ b, log(a),
p
a . . . )
and their derivatives cost O(1).
Setup: E : Rd ! R computable in K operations.
Question: What is the complexity of computing rE : Rd ! Rd?
Finite di↵erences:
AAA99HictVtfcxu3EYfStInVJnXSx850rlXcsVPHI6metjMZzcSWZFuxYssmJTsJbc2RPFFnn3g0j5RlM/omfev0tV+hn6MvfW6f+hW6uwAOOBJ3C6iuMJJwIH67iwWwf4Bjd5SlxWR19R9L7/3o/R//5IMPLy3/9Gcfffzzy598elDk03Ev2e/lWT5+2o2LJEuHyf4knWTJ09E4iU+6WfKk+3ITP39ymoyLNB+2J29GybOTeDBMj9JePIGmw8tnnWHczeKos51c7UyOr0WdeDQa52fLnaNx3Jutnc86yahIs3x4fnVZdfqdboo6/eRw7doXGnw9Wu5k/XxSuHv2Tc/la4eXV1ZvrNJPtFhZU5UVoX728k9+tS46oi9y0RNTcSISMRQTqGciFgWU78WaWBUjaHsmZtA2hlpKnyfiXCwDdgq9EugRQ+tL+DuAp+9V6xCekWZB6B5wyeB3DMhIXAFMDv3GUEduEX0+JcrYWkd7RjRRtjfwv6tonUDrRBxDK4fTPX1xOJaJOBJ/ojGkMKYRteDoeorKlLSCkkfWqCZAYQRtWO/D52Oo9wip9RwRpqCxo25j+vxf1BNb8bmn+k7Fv0nKK1Ai0VKjz0sKsTgl+hHN5hQ+k/JkwHkAFBI1Rqy9Jl2f0OiH0H8G7Q+gnFNN66QLZUat543ITSgu5CaLvAvFhbzLInehuJC7LHIPigu5p5CIHZPO3fgWFBe+xXJ+BMWFfMQiH0NxIR+zyAMoLuQBi/wOigv5HYu8A8WFvMMi70NxIe+zyDYUF7LNIvehuJD7LHIbigu5rZD1O3UMJSc6KbMrb0G9ygMtRQYtt1j5bpN1dGFve+zpXg2W39Vb8N+N3fLQaVKD3fZYd0c1WH7l3QUb6cbytugeeRMX9h6L3YEV4MbusNivxYsa7NceO+1lDZbfa7vQz43lre838OTGfsNiH0DNjeV91ENocWMfeniMUQ12j8U+Eq9qsD5Wf1yD5e1+C+yKG8v7qTb0d2N9rOm0Bsvb0wOIYNxY3ls9gVY39gmLfSrOarBPWey3YN3d2G89POzbGqz2scvkQQYUjySwY5uoxeWuxNoIqMUM/6z0LRnFxl1o5zCDEjMgzAmLuFsi7noidkvErrdcRWlHC4p3eS6tEtHyRHRL34S1Cdu/X/bHWuaB2CoRW3OIpogU51qP5ZSiC93CISel58Kaz5jy0n5jLVHrodnyasTDCkKu7WNa+dcpW8IMCjXVRO249PESGdFzE+I1ZW96lJoHj5uUVsFGnbGorgPVZVFvHKg3LGrqQE1Z1KkDdcqizM63cR2PFWD0j3Mxoye5AmSMXF8iiApugde5B3s0gvWzB1HgY2p5CP9blHtzpUkyzObRT+Ipx7OKJR5DbSZWoN1khVuUX2e0wxKQTPZ8qHJ8fMKzjZnac9IKn5eePCpPTPzppCTPoKSD0WJE+ymMzn1qOafoTtbC8PfKfa9rYfht0vg5RfGyFoafKOknF5C9rbDtC2BbsJtGSvumHkpDnr9IGrq+TF4XLS7O6olaM0jvLJD+jpqZnQvMyybVpH5MPYxGYY2vqIwvhIbRc2HpOYwKRk8y6tW1KHgkQ5X3mnqoDDl50aGSwzyFzgz26auZ0fUwGnsQcW1Szj2z6qGrd1SOxtTDaBwIee55TpG8rofRGNCz1Ieph9HA05ZY5fmmHmrZUQMydzb1UKs+pFNgPAOSa162mKhoTHHSVFFLKT5oPq2xY/5FP4ZnNs/LHKGZkolt6+l0S1/WLJGOFxKwapNAOTC+mFoxWJXGTKyz+ZWUYVLx74t0jI9Hze+CFiPY/fIOgDszz0BCfSaB1jsDimts1lUdmcatszhcJUdzqI5qnbDRouErT42qbYfUyuVlZrRGjx2y1wWtvRHFhLukWU4Pu7UzXEeR09BuRUM8vRDdvVX7tar9VRY3mkOMypXWoxsheZPWnKe6tN6ydHxF3fJMoMg7H7N+8bT5SFkbzHlyskUoSxNPu58+R7Lb0K9eF+aMW34W0YyivTolq5HSjVTBZqH6tFhG4zN6NrT36U4OeUgaPZjHSFEZCXlrhqfoeJ4ekUW17S3HG/WlT+hkvSCrq+1xM3pgoQcOdHiOswke4wHU2pAz7MNT2yPLWS51lZPGx+KL8nY0pxlszuizioXUNKS9SSoWsinLPq5QeQ1oXA0yS/enMU9H4zsLlPis3yWPyV2rlv8K3dzq++2Y1nj9aq4/iekT13XiGtGukbe68mmeg5Rg5vxkneLX5lEivxCOaEM5rs8tzlIvQ7rxTyiDHVFknNFu43ZHtbd9PjX/iea0J/TdOd5m52QhI7J/EfinnNZkRL/2uwP6Bl1ahIxspI/dScvoxhXrpOwaM3FcKuRbDWa9JWTLpsRf07V3V0FrUWYM0g+cz61trZNdigUT4jpW1t3s7Wbvg0jznoS9SiRFs1auEv9r9Ff/6nWysrAiUMM4A4Wyda75yClnQR3F5OWbbZDua0v5WSnDcyW18X9Gps8qkm1RxoXyoLfuA+cePUteuErGJHex0Ef60abTXKQ8mtMjjvaIsnhp9wfKA6Pc18lLrtCe69AqGcAqmJRZhO7LnSLP823mVaXuR7v4v1A3uq5qDSlGwpzgSg1x5/sJZWu2lBmsarl+X9Jucmt9PNermc+Q1uKJtZd/gNZfw18tt372o9OtWIXbtAYkBfNkNCJbooUefrxuV3jplalpmWfDz6xJ3ctuuUh+La2bybFPg6ns0ao5U6cWun4RGi8sGi88ddimu0ajRd2uLdEhm1u01W2lL78Qbu0AylOWMh+RaVTqIaWdS/lR7bNU+Rxfo96ytFZZWjHsVvs2wN7zPkj3Xp/f3T+U3j0Sdyi26VEEJvOXPu3SlGIu3dqcqUkKyPmmsq/27u9QC3LvkgVFyvI9Ttwx8tapR+W8lPS3yrPlZOeNRdDvLb1WfbSN7VD99wvIE9oTBe1LjbhJPRIlvy1HNGeRblgxR0Qn/zHFVDLuaM6Z7d5mTqJKPGHyTbmrDC+ZKQxJ/9zJ285C9rpj5a8R5YRTFV13gVb4DCMFidEnCe7IsqAZQi8nbxJkRNsl+7lop+Qt3tCS6AZJPRMbHjZGZr1mrdtrS49Yj+1z6IlaN7Pu6sHzy7w5cvwucqMXk1c7UTHqbO75YrRi5eWqz016mM7xNfqYUh87szBZXhXTEV96c5EShXGRGB8uYaMIkT9M8hCZ5e2UL2XdW1OunjRIG3NM+RL3HigiXNHdVWc0d40ZR3eBXpewNjXZwlHC07hcnQ/YlhZPpS4t+CHZeqnRG2WWJ6rzFJq67S2M/ZYWMiHrlwnuzEb2tmXvVLIU/hRGUugJ+UZvXX5o0/wSCv6NhCs71Bx9zg5bEN/eEpti+x28DfFK1eWJZkQtaAv6c7l3rMZZ7dGso1cWdZu+Dwd/HinompM+JU8aKrukzEtuU/en/5qswFgkrPSmZ/gYbC78SBY5hYwnJcvGjyYV+rs4oWPRHHxGUuXiz0fea3CjOBL6O01hY9DU+RFUOYTw0O8x+M256R3Oy+bUrK9FLr48pBfQNy4ahzd/9bmK6edjocbWjLx7Dmgdjhqoa2/xv45D8zGcwnn5civou2YvPGZd9kvUiSzGw+F7xnDzWc31HP155uXoTLTk5ifjvihopnJrNO+ePsajZg1oXjMhz0F56STeXkVGXl8qeC/gkiEX/xF/X+K/jfCqpFEnRwglfU9RT0334Knpb1y6Rqc/85HJ0KmTqUrN5BEteiN2U+yIO/C7WUaAoW+Hyu9Syv+IdX9/tg+tR2Q99Cm6PDnoUFtCpx/mFq1Pz+qM8fDyytr8t5AXKwfrN9b+cOPmo5srX91W31D+UPxS/AbykjXxR/GVuAfj3QeZ/rn0/tJHSx9vnG78eeMvG3+VXd9bUphfiMrPxt/+C2XT/cc=
rE(✓) ⇡
1
"
(E(✓ + " 1) E(✓), . . . E(✓ + " d) E(✓))
K(d + 1) operations, intractable for large d.

23. The Complexity of Gradient Computation
Hypothesis: elementary operations (a ⇥ b, log(a),
p
a . . . )
and their derivatives cost O(1).
Seppo Linnainmaa
This algorithm is reverse mode
automatic di↵erentiation
[Seppo Linnainmaa, 1970]
Theorem: there is an algorithm to compute rE in O(K) operations.
Setup: E : Rd ! R computable in K operations.
Question: What is the complexity of computing rE : Rd ! Rd?
Finite di↵erences:
AAA99HictVtfcxu3EYfStInVJnXSx850rlXcsVPHI6metjMZzcSWZFuxYssmJTsJbc2RPFFnn3g0j5RlM/omfev0tV+hn6MvfW6f+hW6uwAOOBJ3C6iuMJJwIH67iwWwf4Bjd5SlxWR19R9L7/3o/R//5IMPLy3/9Gcfffzzy598elDk03Ev2e/lWT5+2o2LJEuHyf4knWTJ09E4iU+6WfKk+3ITP39ymoyLNB+2J29GybOTeDBMj9JePIGmw8tnnWHczeKos51c7UyOr0WdeDQa52fLnaNx3Jutnc86yahIs3x4fnVZdfqdboo6/eRw7doXGnw9Wu5k/XxSuHv2Tc/la4eXV1ZvrNJPtFhZU5UVoX728k9+tS46oi9y0RNTcSISMRQTqGciFgWU78WaWBUjaHsmZtA2hlpKnyfiXCwDdgq9EugRQ+tL+DuAp+9V6xCekWZB6B5wyeB3DMhIXAFMDv3GUEduEX0+JcrYWkd7RjRRtjfwv6tonUDrRBxDK4fTPX1xOJaJOBJ/ojGkMKYRteDoeorKlLSCkkfWqCZAYQRtWO/D52Oo9wip9RwRpqCxo25j+vxf1BNb8bmn+k7Fv0nKK1Ai0VKjz0sKsTgl+hHN5hQ+k/JkwHkAFBI1Rqy9Jl2f0OiH0H8G7Q+gnFNN66QLZUat543ITSgu5CaLvAvFhbzLInehuJC7LHIPigu5p5CIHZPO3fgWFBe+xXJ+BMWFfMQiH0NxIR+zyAMoLuQBi/wOigv5HYu8A8WFvMMi70NxIe+zyDYUF7LNIvehuJD7LHIbigu5rZD1O3UMJSc6KbMrb0G9ygMtRQYtt1j5bpN1dGFve+zpXg2W39Vb8N+N3fLQaVKD3fZYd0c1WH7l3QUb6cbytugeeRMX9h6L3YEV4MbusNivxYsa7NceO+1lDZbfa7vQz43lre838OTGfsNiH0DNjeV91ENocWMfeniMUQ12j8U+Eq9qsD5Wf1yD5e1+C+yKG8v7qTb0d2N9rOm0Bsvb0wOIYNxY3ls9gVY39gmLfSrOarBPWey3YN3d2G89POzbGqz2scvkQQYUjySwY5uoxeWuxNoIqMUM/6z0LRnFxl1o5zCDEjMgzAmLuFsi7noidkvErrdcRWlHC4p3eS6tEtHyRHRL34S1Cdu/X/bHWuaB2CoRW3OIpogU51qP5ZSiC93CISel58Kaz5jy0n5jLVHrodnyasTDCkKu7WNa+dcpW8IMCjXVRO249PESGdFzE+I1ZW96lJoHj5uUVsFGnbGorgPVZVFvHKg3LGrqQE1Z1KkDdcqizM63cR2PFWD0j3Mxoye5AmSMXF8iiApugde5B3s0gvWzB1HgY2p5CP9blHtzpUkyzObRT+Ipx7OKJR5DbSZWoN1khVuUX2e0wxKQTPZ8qHJ8fMKzjZnac9IKn5eePCpPTPzppCTPoKSD0WJE+ymMzn1qOafoTtbC8PfKfa9rYfht0vg5RfGyFoafKOknF5C9rbDtC2BbsJtGSvumHkpDnr9IGrq+TF4XLS7O6olaM0jvLJD+jpqZnQvMyybVpH5MPYxGYY2vqIwvhIbRc2HpOYwKRk8y6tW1KHgkQ5X3mnqoDDl50aGSwzyFzgz26auZ0fUwGnsQcW1Szj2z6qGrd1SOxtTDaBwIee55TpG8rofRGNCz1Ieph9HA05ZY5fmmHmrZUQMydzb1UKs+pFNgPAOSa162mKhoTHHSVFFLKT5oPq2xY/5FP4ZnNs/LHKGZkolt6+l0S1/WLJGOFxKwapNAOTC+mFoxWJXGTKyz+ZWUYVLx74t0jI9Hze+CFiPY/fIOgDszz0BCfSaB1jsDimts1lUdmcatszhcJUdzqI5qnbDRouErT42qbYfUyuVlZrRGjx2y1wWtvRHFhLukWU4Pu7UzXEeR09BuRUM8vRDdvVX7tar9VRY3mkOMypXWoxsheZPWnKe6tN6ydHxF3fJMoMg7H7N+8bT5SFkbzHlyskUoSxNPu58+R7Lb0K9eF+aMW34W0YyivTolq5HSjVTBZqH6tFhG4zN6NrT36U4OeUgaPZjHSFEZCXlrhqfoeJ4ekUW17S3HG/WlT+hkvSCrq+1xM3pgoQcOdHiOswke4wHU2pAz7MNT2yPLWS51lZPGx+KL8nY0pxlszuizioXUNKS9SSoWsinLPq5QeQ1oXA0yS/enMU9H4zsLlPis3yWPyV2rlv8K3dzq++2Y1nj9aq4/iekT13XiGtGukbe68mmeg5Rg5vxkneLX5lEivxCOaEM5rs8tzlIvQ7rxTyiDHVFknNFu43ZHtbd9PjX/iea0J/TdOd5m52QhI7J/EfinnNZkRL/2uwP6Bl1ahIxspI/dScvoxhXrpOwaM3FcKuRbDWa9JWTLpsRf07V3V0FrUWYM0g+cz61trZNdigUT4jpW1t3s7Wbvg0jznoS9SiRFs1auEv9r9Ff/6nWysrAiUMM4A4Wyda75yClnQR3F5OWbbZDua0v5WSnDcyW18X9Gps8qkm1RxoXyoLfuA+cePUteuErGJHex0Ef60abTXKQ8mtMjjvaIsnhp9wfKA6Pc18lLrtCe69AqGcAqmJRZhO7LnSLP823mVaXuR7v4v1A3uq5qDSlGwpzgSg1x5/sJZWu2lBmsarl+X9Jucmt9PNermc+Q1uKJtZd/gNZfw18tt372o9OtWIXbtAYkBfNkNCJbooUefrxuV3jplalpmWfDz6xJ3ctuuUh+La2bybFPg6ns0ao5U6cWun4RGi8sGi88ddimu0ajRd2uLdEhm1u01W2lL78Qbu0AylOWMh+RaVTqIaWdS/lR7bNU+Rxfo96ytFZZWjHsVvs2wN7zPkj3Xp/f3T+U3j0Sdyi26VEEJvOXPu3SlGIu3dqcqUkKyPmmsq/27u9QC3LvkgVFyvI9Ttwx8tapR+W8lPS3yrPlZOeNRdDvLb1WfbSN7VD99wvIE9oTBe1LjbhJPRIlvy1HNGeRblgxR0Qn/zHFVDLuaM6Z7d5mTqJKPGHyTbmrDC+ZKQxJ/9zJ285C9rpj5a8R5YRTFV13gVb4DCMFidEnCe7IsqAZQi8nbxJkRNsl+7lop+Qt3tCS6AZJPRMbHjZGZr1mrdtrS49Yj+1z6IlaN7Pu6sHzy7w5cvwucqMXk1c7UTHqbO75YrRi5eWqz016mM7xNfqYUh87szBZXhXTEV96c5EShXGRGB8uYaMIkT9M8hCZ5e2UL2XdW1OunjRIG3NM+RL3HigiXNHdVWc0d40ZR3eBXpewNjXZwlHC07hcnQ/YlhZPpS4t+CHZeqnRG2WWJ6rzFJq67S2M/ZYWMiHrlwnuzEb2tmXvVLIU/hRGUugJ+UZvXX5o0/wSCv6NhCs71Bx9zg5bEN/eEpti+x28DfFK1eWJZkQtaAv6c7l3rMZZ7dGso1cWdZu+Dwd/HinompM+JU8aKrukzEtuU/en/5qswFgkrPSmZ/gYbC78SBY5hYwnJcvGjyYV+rs4oWPRHHxGUuXiz0fea3CjOBL6O01hY9DU+RFUOYTw0O8x+M256R3Oy+bUrK9FLr48pBfQNy4ahzd/9bmK6edjocbWjLx7Dmgdjhqoa2/xv45D8zGcwnn5civou2YvPGZd9kvUiSzGw+F7xnDzWc31HP155uXoTLTk5ifjvihopnJrNO+ePsajZg1oXjMhz0F56STeXkVGXl8qeC/gkiEX/xF/X+K/jfCqpFEnRwglfU9RT0334Knpb1y6Rqc/85HJ0KmTqUrN5BEteiN2U+yIO/C7WUaAoW+Hyu9Syv+IdX9/tg+tR2Q99Cm6PDnoUFtCpx/mFq1Pz+qM8fDyytr8t5AXKwfrN9b+cOPmo5srX91W31D+UPxS/AbykjXxR/GVuAfj3QeZ/rn0/tJHSx9vnG78eeMvG3+VXd9bUphfiMrPxt/+C2XT/cc=
rE(✓) ⇡
1
"
(E(✓ + " 1) E(✓), . . . E(✓ + " d) E(✓))
K(d + 1) operations, intractable for large d.

24. Backward Automatic Differentiation
return ✓R
✓r = gr(✓Parents(r)
)
for r = M + 1, . . . , R
function `(✓1, . . . , ✓M )
forward
for r = R 1, . . . , 1
rR` = 1
rr` =
X
s2Child(r)
@rgs(✓) rs`
backward
return (r1`, . . . , rM `)
function r`(✓1, . . . , ✓M )
computing `
computing r`
`(✓1, ✓2) def.
= ✓2e✓1
p
✓1 + ✓2e✓1
✓1
✓2
input
✓3
def.
= e✓1
✓4
def.
= ✓2✓3
✓5
def.
= ✓1 + ✓4
✓6
def.
=
p
✓5
output
✓7
def.
= ✓4✓6
g3
g4
g5
g7
g6
`

25. Overview
• Empirical Risk Minimization

• Perceptrons

• Optimization

• Convolutional Networks

• Residual Networks

• Transformers

26. Convolutional CNN
x
x1
y = xD
xk+1
, (Wkxk + bk)
! Leverage translation invariance of images.
! Sub-sampling: breaks invariance but increase receptive ﬁelds.
Pool
x
x1

27. Convolutional CNN
x
x1
y = xD
xk+1
, (Wkxk + bk)
! Leverage translation invariance of images.
! Sub-sampling: breaks invariance but increase receptive ﬁelds.
AlexNet, 2011
Pool
x
x1

28. Example of Activations

29. Overview
• Empirical Risk Minimization

• Perceptrons

• Optimization

• Convolutional Networks

• Residual Networks

• Transformers

30. ResNet-type Architectures [He et al’ 16]
ResNet-34