機械学習と自動微分 (2023)

⾒
͖͕ͨΘ ͍͕ͪ͘
2023 1 20 ( ) 15
4 14:55 - 16:25
https://itakigawa.github.io/

View Slide

( )

View Slide

( )
( )
/ (7 )
(7 )
JST (3.5 )
/

View Slide

( )
( )
/ (7 )
(7 )
JST (3.5 )
/
⾒ (4 )
(4 )
×
×
( )
AIP ATR ( )

View Slide

DAG
H
H
H
H
H
H
H
H
O
N
O
O
H
H
H
O
O
H
H
N
O
O
Cl
Cl
Cl
⾒⾒ etc

View Slide

. =
.
. ⾒
. Python
. Q & A
https://itakigawa.github.io/news.html

View Slide

=

View Slide

ίϯϐϡʔλ
ϓϩάϥϜ
ೖྗ ग़ྗ
=

View Slide

=
my_function
x1
x2
y
( )
x1 x2
y
y
x1
x2
AAACvnichVFNS9xQFD2mXzr9cNSN0M2jgzLD4HATFK0gDHbj0o+OClZDEt/YYL5IMsOMg3/AP+DChSh0UfwB/QGl0FW76sKfULq00E0X3mRCxQ6OL7y8c8+95+a8XDNw7CgmuhxQHjx89PjJ4FDu6bPnL4bzI6Prkd8ILVmzfMcPN00jko7tyVpsx47cDEJpuKYjN8z9N0l+oynDyPa9t3E7kNuusefZddsyYqb0/Ou2WBDFlq7uaGJKGLoqWunWRJkjLUE7WknkyqI4Zf7LmLpW0vMFqlC6RC9QM1BAtpb9/Ce8wy58WGjAhYSHmLEDAxE/W1BBCJjbRoe5kJGd5iUOkWNtg6skVxjM7vN7j6OtjPU4TnpGqdrirzi8Q1YKTNAP+khX9JUu6Cf9vbNXJ+2ReGnzaXa1MtCHj8bX/tyrcvmM8f5G1ddzjDrmUq82ew9SJrmF1dU3D46v1uZXJzqTdE6/2P8ZXdJnvoHX/G19WJGrJ338mOyF/xgPSP1/HL1gXauoMxVamS5UF7NRDeIlXqHI85hFFUtYRo37n+ILvuG7UlXqiqv43VJlINOM4dZSWtceQJ/s
y = (x2
1
a1x1x2 + a2x2
2
) + ( b1x2 + b2)

View Slide

( )
ίϯϐϡʔλϓϩάϥϜΛ࡞Δʹ͸ೖྗ͔Βग़ྗΛಘ
Δखॱ͕ΧϯϖΩʹΘ͔ͬͯͳ͚Ε͹͍͚ͳ͍ɻ
Ͱ΋ɺ࣮ࡍʹ͸؊৺ͷखॱ͕Α͘෼͔Βͳ͍͜ͱ͕ଟ͍
?
?
?
?

View Slide

A B
( )
? ? ?
? ? ?
..
?

View Slide

J’aime la
musique I love music
AI
( )

View Slide

診

View Slide

View Slide

OpenPose
Style Transfer
pix pix
face swapping
( DeepFake )
image
transformation
CycleGAN

View Slide

DALL·E ChatGPT, Codex

View Slide

OpenAI ChatGPT https://chat.openai.com/

View Slide

GitHub Copilot (OpenAI Codex)

View Slide

GPT (Generative Pre-trained Transformer)
ChatGPT
( )
Codex
GitHub Copilot
( )
Generative Pre-trained Transformer
(GPT- , GPT- . , )
GPT
×
TransformerDecoder (Pre-LN)
Masked (Multi-head) Self Attention
⾒
+
+
Layer Norm
Layer Norm

View Slide

=
( )

View Slide

= ( )
: 1 : 1

View Slide

=
x1
x2
y
( )
p1
...
AAACxHichVFNSxtRFD2OWq3VGnVTcBMMioKEOxITtQvFgnTpR6NCEsLM+BIH54uZl6Adxh/gH3DRVQsuSn9Af0A3Bbe68CdIlwpuuvDOZECsVO/w5p133j33nfeu7llmIImuupTunt5Xff2vB94MDr0dzoyMbgduyzdE2XAt19/VtUBYpiPK0pSW2PV8odm6JXb0gw/x/k5b+IHpOp/kkSdqttZ0zIZpaJKpema5MV3V7fAwep8Nq0m5it/UayHlSwVVLRVnKU/zc0U1BgUqFQuLUZxflftCalE0U8/kOCOJ7FOgpiCHNNbdzE9UsQcXBlqwIeBAMragIeCvAhUEj7kaQuZ8RmayLxBhgLUtzhKcoTF7wP8mryop6/A6rhkkaoNPsXj4rMxiki7pO93Qb/pB1/T3v7XCpEbs5YhnvaMVXn345N3W3Ysqm2eJ/QfVs54lGlhIvJrs3UuY+BZGR9/+fHqztbQ5GU7RN/rD/r/SFf3iGzjtW+NsQ2x+ecaPzl74xbhB6r/teAq25/LqfJ42CrmV1bRV/RjHBKa5HyWs4CPWUeb6ZzjHBS6VNcVSAqXVSVW6Us0YHoVyfA/wvKXW
f(x; ✓)
p2 p3 p5
p4
...
AAACi3ichVHLSsNAFL3Gd3206kZwEyyKq3BTY6PFRVEEl61aLWgpSZy2wbxIpgUt/oBLNy50o+BC/AA/wI0/4KKfIC4V3LjwJq2IC+sNk7lz5p47Z+bonmUGHLHVI/T29Q8MDg3HRkbHxuOJicndwK37BisYruX6RV0LmGU6rMBNbrGi5zPN1i22px+th/t7DeYHpuvs8GOPlWyt6pgV09A4QcUDXmNcK8vlRBIlVZFlNS2ihEuptBwmCqppZUWUJYwiCZ3IuYkHOIBDcMGAOtjAwAFOuQUaBPTtgwwIHmElaBLmU2ZG+wxOIUbcOlUxqtAIPaJ/lVb7HdShddgziNgGnWLR8Ikpwhw+4x2+4RPe4wt+/tmrGfUItRzTrLe5zCvHz6a3P/5l2TRzqP2wumrmUIHlSKtJ2r0ICW9htPmNk4u37czWXHMeb/CV9F9jCx/pBk7j3bjNs63LLnp00kIvRgZ9uyD+neymJHlJwrySzK51rBqCGZiFBfJDhSxsQg4KkQ/ncAlXwpiwKGSE1Xap0NPhTMGvEDa+AOhakw8=
✓1
AAACi3ichVHLSsNAFL3GV62vqhvBTbBUXIWbmjYqLkQRXNbWPkClJHHU0LxIpoVa/AGXblzoRsGF+AF+gBt/wIWfIC4ruHHhTVoRF+oNk7lz5p47Z+bonmUGHPG5R+jt6x8YjA3Fh0dGx8YTE5OlwK37BisaruX6FV0LmGU6rMhNbrGK5zPN1i1W1mvr4X65wfzAdJ1t3vTYnq0dOuaBaWicoMouP2Jcq6ariSRKqiLLalZECTPprBwmCqpZZUmUJYwiCd3IuYl72IV9cMGAOtjAwAFOuQUaBPTtgAwIHmF70CLMp8yM9hmcQJy4dapiVKERWqP/Ia12uqhD67BnELENOsWi4RNThBQ+4S228RHv8AU/fu3VinqEWpo06x0u86rjp9OF939ZNs0cjr5Zf2rmcACLkVaTtHsREt7C6PAbx+ftwnI+1ZrDa3wl/Vf4jA90A6fxZtxssfzFH3p00kIvRgZ9uSD+npTSkpyRcEtJrq51rYrBDMzCPPmhwipsQg6KkQ9ncAGXwqiwICwLK51SoafLmYIfIWx8Aup6kxA=
✓2
AAACi3ichVHLSsNAFL2N7/qquhHcBEvFVbipsdHiQhTBZWttLaiUJE5rMC+SaaEWf8ClGxe6UXAhfoAf4MYfcOEniMsKblx4k1bEhXrDZO6cuefOmTm6Z5kBR3yOCT29ff0Dg0Px4ZHRsfHExGQpcOu+wYqGa7l+WdcCZpkOK3KTW6zs+UyzdYvt6Efr4f5Og/mB6TrbvOmxfVurOWbVNDROUHmPHzKuVRYqiSRKqiLLakZECRfTGTlMFFQzyrIoSxhFErqRcxP3sAcH4IIBdbCBgQOccgs0COjbBRkQPML2oUWYT5kZ7TM4gThx61TFqEIj9Ij+NVrtdlGH1mHPIGIbdIpFwyemCCl8wlts4yPe4Qt+/NqrFfUItTRp1jtc5lXGT6cL7/+ybJo5HH6z/tTMoQpLkVaTtHsREt7C6PAbx+ftQnYr1ZrDa3wl/Vf4jA90A6fxZtzk2dbFH3p00kIvRgZ9uSD+npTSkrwoYV5Jrq51rRqEGZiFefJDhVXYhBwUIx/O4AIuhVFhQcgKK51SIdblTMGPEDY+AeyakxE=
✓3
AAACi3ichVHLSsNAFL3GV62vqhvBTbBUXIUbTVsVF6IILmtrH9CWksSphqZJSKaFWvwBl25c6EbBhfgBfoAbf8CFnyAuK7hx4U1aERfWGyZz58w9d87M0RzT8Djiy4AwODQ8MhoaC49PTE5NR2Zmc57dcHWW1W3Tdgua6jHTsFiWG9xkBcdlal0zWV6r7fj7+SZzPcO2DnjLYeW6emQZVUNXOUGFEj9mXK0olUgUpaQiy8mEiBLGVxKynyiYTCjroixhEFHoRcqOPEAJDsEGHRpQBwYWcMpNUMGjrwgyIDiElaFNmEuZEewzOIUwcRtUxahCJbRG/yNaFXuoRWu/pxewdTrFpOESU4QYPuMddvAJ7/EVP//s1Q56+FpaNGtdLnMq02fzmY9/WXWaORz/sPpq5lCFtUCrQdqdAPFvoXf5zZOLTmYjHWsv4Q2+kf5rfMFHuoHVfNdv91n6so8ejbTQi5FB3y6Ifye5FUmOS7ivRLe2e1aFYAEWYZn8SMIW7EEKsoEP53AJV8KksCpsCJvdUmGgx5mDXyHsfgHuupMS
✓4
AAACi3ichVHLTsJAFL3UFz5Q1I2JGyLBuGpukYcSF0Rj4lJBHgkY0tYBG0rbtAMJEn/ApRsXuNHEhfED/AA3/oALP8G4xMSNC28LxrhAbzOdO2fuuXNmjmLpmsMRX3zCyOjY+IR/cmp6JjA7F5xfyDtm01ZZTjV10y4qssN0zWA5rnGdFS2byQ1FZwWlvuPuF1rMdjTTOORtix015JqhVTVV5gQVy/yEcbliVYJhFJMxSUomQihiPJqQ3CSGyURsMySJ6EUYBrFvBh+gDMdgggpNaAADAzjlOsjg0FcCCRAswo6gQ5hNmebtMziDKeI2qYpRhUxonf41WpUGqEFrt6fjsVU6RadhEzMEEXzGO+zhE97jK34O7dXxerha2jQrfS6zKnPnS9mPf1kNmjmc/LD+1MyhChueVo20Wx7i3kLt81unl71sKhPprOINvpH+a3zBR7qB0XpXbw9YpvuHHoW00IuRQd8uhIYn+agoxUU8iIXT2wOr/LAMK7BGfiQhDXuwDznPhwvowpUQENaFlLDVLxV8A84i/Aph9wtuSZNO
✓p

View Slide

= ( )
( " " or )
where
⾒
...
f(x; ✓)
...
✓1
✓2
✓3
✓4
✓p
AAADN3icnVHPSxtBFH67WrWxNVEvgpdgsFhawtsQE+2hiD3ooeDPqJANYXcd45D9xe4kYJf9B/wHPPRUwUPx5qX03Iv/gAdP3sTSkyh46cG3mwVRqQXfMjvffPO+N9/M012T+wLxVJK7ul/09Pa9TPW/ej2QzgwOrflOyzNYxXBMx9vQNZ+Z3GYVwYXJNlyPaZZusnW9+SnaX28zz+eOvSp2XFaztIbNt7ihCaLqmSPV4nY9CNS4VNVr6LUA8+WiopRL7zGPk4WSEoEilkvF6VDVrUAV20xoYRhmP088R/c29S6rLlisoT1PXs/kKCOO7GOgJCAHSSw6mR+gwiY4YEALLGBggyBsggY+fVVQAMElrgYBcR4hHu8zCCFF2hZlMcrQiG3Sv0GrasLatI5q+rHaoFNMGh4pszCOJ/gdr/AYD/E3/v1nrSCuEXnZoVnvaJlbT++OrNz8V2XRLGD7TvWkZwFbMBV75eTdjZnoFkZH3/6yd7XyYXk8eIP7+If8f8NT/EU3sNvXxsESW/76hB+dvNCLUYOUh+14DNYKeWUyj0vF3Mxs0qo+GIUxmKB+lGEG5mERKmBII9JHaU6al3/KZ/K5fNFJlaVEMwz3Qr68BeQYyWw=
min
✓
L(✓) + ⌦(✓)
AAADI3icnVHNahNRFD4zVq3RmrTdCG4GQyWREs7UaeIPhaIbFy76Y9pCJg4z05vk0vnjzk1oHOYFfAEXgqDQhfgAPoAbwZ3oogsfQFxWcOOiZyYDosUKnsud+93vnu/c785xIo/HEvFQUc9MnT13fvpC6eKlmcvlyuzcVhwOhcvabuiFYsexY+bxgLUllx7biQSzfcdj287e/ex8e8REzMPgkRxHrOvb/YD3uGtLoqzKi4e1xMzLdETf6SbYaBm63mouYgOXl5p6BgxsNY3bqen4iSkHTNppWi+taGY89K2Er+jp40AzfVsOhJ8wIUKR1sYWX9R6tUyyn1r8rvZfl9TrVqVKOXloJ4FegCoUsRZW3oIJuxCCC0PwgUEAkrAHNsQ0OqADQkRcFxLiBCGenzNIoUTaIWUxyrCJ3aNvn3adgg1on9WMc7VLt3g0BSk1WMDP+BqP8D2+wa/486+1krxG5mVMqzPRssgqP72y+eOfKp9WCYNfqlM9S+jBrdwrJ+9RzmSvcCf60ZNnR5t3NhaS6/gKv5H/l3iI7+gFwei7e7DONp6f4schL/THqEH6n+04CbaWGvpyA9eN6uq9olXTcBWuQY360YJVeABr0AZXmVJuKDcVQz1QP6gf1U+TVFUpNPPwW6hfjgHcwcRG
L(✓) =
n
X
i=1
error(yi, f(xi; ✓)))

View Slide

= ( )
( " " or )
where
⾒
...
f(x; ✓)
...
✓1
✓2
✓3
✓4
✓p
min
✓
L(✓) + ⌦(✓)
AAADI3icnVHNahNRFD4zVq3RmrTdCG4GQyWREs7UaeIPhaIbFy76Y9pCJg4z05vk0vnjzk1oHOYFfAEXgqDQhfgAPoAbwZ3oogsfQFxWcOOiZyYDosUKnsud+93vnu/c785xIo/HEvFQUc9MnT13fvpC6eKlmcvlyuzcVhwOhcvabuiFYsexY+bxgLUllx7biQSzfcdj287e/ex8e8REzMPgkRxHrOvb/YD3uGtLoqzKi4e1xMzLdETf6SbYaBm63mouYgOXl5p6BgxsNY3bqen4iSkHTNppWi+taGY89K2Er+jp40AzfVsOhJ8wIUKR1sYWX9R6tUyyn1r8rvZfl9TrVqVKOXloJ4FegCoUsRZW3oIJuxCCC0PwgUEAkrAHNsQ0OqADQkRcFxLiBCGenzNIoUTaIWUxyrCJ3aNvn3adgg1on9WMc7VLt3g0BSk1WMDP+BqP8D2+wa/486+1krxG5mVMqzPRssgqP72y+eOfKp9WCYNfqlM9S+jBrdwrJ+9RzmSvcCf60ZNnR5t3NhaS6/gKv5H/l3iI7+gFwei7e7DONp6f4schL/THqEH6n+04CbaWGvpyA9eN6uq9olXTcBWuQY360YJVeABr0AZXmVJuKDcVQz1QP6gf1U+TVFUpNPPwW6hfjgHcwcRG
L(✓) =
n
X
i=1
error(yi, f(xi; ✓)))
AAACwXichVHLShxBFD22j6jxMTEbwY04KAoy3JZ5xJCFxI07XxkVpoehu1OOxfSL7poBbeYH8gNZZBUhi+AH+AFu3BrIwk8QlwrZZOHtnoaQSPQ21XXq1D23TtW1AkdGiui6T+sfGBx6MTwy+nJsfGIy92pqL/LboS2qtu/44YFlRsKRnqgqqRxxEITCdC1H7Fut9WR/vyPCSPreB3UciLprNj15KG1TMdXIvTM2XdE0F2MjrVULm1Y9pkKlqOuV8jIVqLRS1hNQpEq5uNo1LDc21JFQZre71MjlOSON2cdAz0AeWWz5uXMY+AgfNtpwIeBBMXZgIuKvBh2EgLk6YuZCRjLdF+hilLVtzhKcYTLb4n+TV7WM9Xid1IxStc2nODxCVs5inn7Sd7qjSzqjG/r931pxWiPxcsyz1dOKoDH5aXr317Mql2eFoz+qJz0rHOJN6lWy9yBlklvYPX3n5PPd7tud+XiBTumW/X+la7rgG3ide/vbttj58oQfi73wi3GD9H/b8RjsrRT0UoG2i/m191mrhjGDOSxyPypYwwa2UOX6p7jEFX5o65rUAi3spWp9meY1/gotfgDrHKSh
⌦(✓)
min
✓
L(✓) + ⌦(✓)

View Slide

my_function
x1
x2
y
a ,a ,b ,b
y

View Slide

my_function
x1
x2
y
y/ a
= a
y ?
( )
⾒

View Slide

= ( )
( ) ( " " or )
AAACp3ichVG7SgNBFD2u73fURrBZDIqChLtBUQRBtLEzPhIDRsLuOtHFfbE7CWiwF3/AwkrBQgRb7W38AQs/QSwVbCy82SyIinqX2Tlz5p47Z+Yavm2FkuixQWlsam5pbWvv6Ozq7ulN9PXnQq8cmCJrerYX5A09FLbliqy0pC3yfiB0x7DFhrG3WNvfqIggtDx3Xe77YsvRd1yrZJm6ZKqYGC4YTrUgd4XUD9U5dawOi9qEGqP0eDGRpBRFof4EWgySiCPjJW5QwDY8mCjDgYALydiGjpC/TWgg+MxtocpcwMiK9gUO0cHaMmcJztCZ3eP/Dq82Y9blda1mGKlNPsXmEbBSxQg90CW90D1d0RO9/1qrGtWoednn2ahrhV/sPR5ce/tX5fAssfup+tOzRAkzkVeLvfsRU7uFWddXDk5e1mZXR6qjdE7P7P+MHumOb+BWXs2LFbF6+ocfg73wi3GDtO/t+Aly6ZQ2laKVyeT8QtyqNgxhGGPcj2nMYwkZZLn+Ea5xg1tlXFlWckq+nqo0xJoBfAlF/wDm+Zyq
✓ = (✓1, ✓2)
AAACi3ichVHLTsJAFL3UF6II6sbEDZFgXJFbH9EQF0Rj4pKHPBIgpK0DNJS2aQcSJP6ASzcucKOJC+MH+AFu/AEXfIJxiYkbF15KE6NEvM10zpy5586ZubKpqTZH7HmEicmp6RnvrG9u3r8QCC4uZW2jaSksoxiaYeVlyWaaqrMMV7nG8qbFpIassZxcPxzs51rMslVDP+Ftk5UaUlVXK6oicaLyRV5jXCqL5WAYo+hEaBSILgiDGwkj+AhFOAUDFGhCAxjowAlrIIFNXwFEQDCJK0GHOIuQ6uwzOAcfaZuUxShDIrZO/yqtCi6r03pQ03bUCp2i0bBIGYIIvuA99vEZH/AVP/+s1XFqDLy0aZaHWmaWAxcr6Y9/VQ2aOdS+VWM9c6jAnuNVJe+mwwxuoQz1rbOrfjqWinTW8RbfyP8N9vCJbqC33pW7JEt1x/iRyQu9GDVI/N2OUZDdjIo7UUxuh+MHbqu8sAprsEH92IU4HEMCMk4fLqEL14Jf2BJiwv4wVfC4mmX4EcLRF0xPksM=
✓1
AAACi3ichVHNSgJRFD5Of2aZVpugjSRGKzlaUUgLKYKW/uQPqMjMdNPB+WPmKpj0Ai3btLBNQYvoAXqANr1ACx8hWhq0adFxHIiS7Ax37ne/e75zv3uPZKqKzRF7HmFicmp6xjvrm5v3LwSCi0t522haMsvJhmpYRUm0maroLMcVrrKiaTFRk1RWkBoHg/1Ci1m2YujHvG2yiibWdOVUkUVOVLHM64yL1Xg1GMYoOhEaBTEXhMGNlBF8hDKcgAEyNEEDBjpwwiqIYNNXghggmMRVoEOcRUhx9hmcg4+0TcpilCES26B/jVYll9VpPahpO2qZTlFpWKQMQQRf8B77+IwP+Iqff9bqODUGXto0S0MtM6uBi5Xsx78qjWYO9W/VWM8cTmHX8aqQd9NhBreQh/rW2VU/m8hEOut4i2/k/wZ7+EQ30Fvv8l2aZbpj/EjkhV6MGhT73Y5RkI9HY9tRTG+Fk/tuq7ywCmuwQf3YgSQcQQpyTh8uoQvXgl/YFBLC3jBV8LiaZfgRwuEXTm+SxA==
✓2
AAACunichVHLSiNBFD3pcZz4TBw3A24ag+IghBsZUURBxo0LwagTFYyE7rYSi/SL7kogBn9gfmAWrhRciB/gWtzIzNqFnyAuFWbjwptOwxBF5zbVderUPbdO1TV9W4aK6Dahfej62P0p2dPb1z8wmEoPfd4MvVpgiYLl2V6wbRqhsKUrCkoqW2z7gTAc0xZbZnWptb9VF0EoPfeHavhi1zEqrixLy1BMldLT5Ymi6TSLal8o4/CrvqCvdBKTenHVERWjgy2lM5SlKPTXIBeDDOLIe+kLFLEHDxZqcCDgQjG2YSDkbwc5EHzmdtFkLmAko32BQ/SytsZZgjMMZqv8r/BqJ2ZdXrdqhpHa4lNsHgErdYzRDZ3RA13TOd3R05u1mlGNlpcGz2ZbK/xS6ueXjb//VTk8K+z/U73rWaGM2cirZO9+xLRuYbX19YNfDxtz62PNcTqhe/Z/TLd0xTdw64/W6ZpYP3rHj8le+MW4QbmX7XgNNqeyueksrX3LLH6PW5XECEYxwf2YwSKWkUeB6x/hEr/xR5vXTE1q1Xaqlog1w+gITT0DYeWkRQ==
f(✓) = L(✓) + ⌦(✓)
AAACpnichVG7ThtBFD3e8DBvAw1SGoMFMo11jYiCqBBpqCKwMUbCyOwuYxixL+2OLTkr10j8QIpUQUoRRWnxB9DwAxR8AkpJJBoKrtcr8RJwV7tz7pl77p6Za3iWDBTRVUL70NPb158cGBwaHhkdS41PbAVu3TdFyXQt19829EBY0hElJZUltj1f6LZhibJx9KWzX24IP5Cus6manti19QNH1qSpK6aqqemKLZ1qWDHssKIOhdJbrXQt+yidr6YylKMo0i9BPgYZxLHuptqoYB8uTNRhQ8CBYmxBR8DPDvIgeMztImTOZySjfYEWBllb5yrBFTqzR/w94GwnZh3OOz2DSG3yXyx+fVamMUuX9Jtu6IL+0DXdvdorjHp0vDR5Nbpa4VXHTqaKt++qbF4VDh9Ub3pWqGEp8irZuxcxnVOYXX3j2/eb4nJhNpyjU/rH/n/SFZ3zCZzGf/PXhij8eMOPwV74xnhA+efjeAm2FnL5TznaWMysrMajSuIjZpDleXzGCtawjhL3P8ZfnKGtZbWvWkkrd0u1RKyZxJPQ9u4Ber2d5A==
min
✓
f(✓)

View Slide

https://arxiv.org/abs/1905.11946
2600ສύϥϝλ (ResNet-50)
8400ສύϥϝλ (ResNeXt-101)
M: Million = 100ສ

View Slide

1750ԯύϥϝλ (GPT-3)

View Slide

MLP ( )
import torch.nn as nn
nn.Sequential(nn.Linear(2, 3), nn.ReLU(), nn.Linear(3, 2), nn.ReLU(), nn.Linear(2, 2))
https://playground.tensorflow.org

View Slide

x =

x1
x2
AAACrHichVE9TxtBEH0+CB8mgIEGKQ2KRaCy5hAIRIWgoQRjA5LPsu6OtVlxX7pbWxiLP4BETUFFpBRRSgqaUNHwByj8E1BKIqWhYHw+ESUImNPuvn07b/btjRU4MlJE7ZTW0/uhr39gMD30cXhkNDM2vh359dAWRdt3/HDXMiPhSE8UlVSO2A1CYbqWI3asg7XO+U5DhJH0vYJqBqLsmjVPVqVtKqYqmRnDEjXptSzXVKE8PE43K7ph8DyXNoS398xXMlnKURxTL4GegCyS2PAzVzCwBx826nAh4EExdmAi4q8EHYSAuTJazIWMZHwucIw0a+ucJTjDZPaA5xrvSgnr8b5TM4rVNt/i8AhZOYVpuqPv9EC39IPu6fHVWq24RsdLk1erqxVBZfRkcuvPuyqXV4X9v6o3PStUsRR7lew9iJnOK+yuvnF09rC1nJ9ufaGv9Iv9X1CbbvgFXuO3/W1T5M/f8GOxF/5j3CD9/3a8BNtzOX0hR5vz2ZXVpFUD+ITPmOV+LGIF69hAkeuf4hI/ca3ltIJW0srdVC2VaCbwT2jVJ2Bsn+M=

y1
y2
AAACsHichVE9TxtBEH1cSAImBCdpkGgQFggpkjVngRKlQqFJaT4MljjrdHdezMJ96W5t2bH8B9JRpUgFEgXiB9AmouEPpOAnRJQg0VAwPp9AYOHMaWdn38ybfXtjh66MFdHFkPZi+OWr1yOjmbE3428nsu/eb8RBPXJEyQncICrbVixc6YuSksoV5TASlme7YtPeW+7mNxsiimXgr6tWKCqeVfPltnQsxZCZ/WjYoib9tu1ZKpLNTqZp6obBvsBezxjCr97nzGyO8pTYdH+gp0EOqRWD7CkMVBHAQR0eBHwojl1YiPnbgg5CyFgFbcYijmSSF+ggw9w6VwmusBjdY1/j01aK+nzu9owTtsO3uLwiZk5jlv7SMV3ROZ3QP7p9tlc76dHV0uLd7nFFaE78mFy7+S/L411h54E1ULPCNj4nWiVrDxOk+wqnx298/3m19mV1tj1Hh3TJ+g/ogs74BX7j2jlaEau/BuixWQv/MR6Q/nQc/cFGIa8v5mllIbf0NR3VCKYwg3mexycs4RuKKHH/fZziN/5oBa2smZrVK9WGUs4HPDJt9w4sH6D8
2
4
x1
x2
1
3
5
AAACyHichVG7ThtBFD1eIAGTBANNJBoLi4g01l0nEREVgiai4mVAYom1u4ztEfvS7tjBrNxQ5gdSUIFEgfiAfEAaOiok/AmIkkhpUuR6vYIkKOSuZufMmXvunJlrBY6MFFEno/X1Dzx5OjiUHX72/MVIbnRsPfIboS3Ktu/44aZlRsKRnigrqRyxGYTCdC1HbFi7C939jaYII+l7a6oViG3XrHmyKm1TMVXJLRiWqEkvtlxThXKvna1X9I/xtP66bRiMS7/hN3dYzxrC27nTVHIFKlIS+YdAT0EBaSz5ua8wsAMfNhpwIeBBMXZgIuJvCzoIAXPbiJkLGclkX6CNLGsbnCU4w2R2l/81Xm2lrMfrbs0oUdt8isMjZGUeU3RJp3RL53RG1/Tzn7XipEbXS4tnq6cVQWXk88vVH/9VuTwr1O9Vj3pWqOJ94lWy9yBhurewe/rm/pfb1dmVqfgVHdMN+z+iDn3jG3jN7/bJslg5fMSPxV74xbhB+t/teAjWS0X9XZGW3xbm5tNWDWICk5jmfsxgDh+whDLXP8UFrtDRFrVA+6S1eqlaJtWM44/QDn4BmRmpEA==
2
6
6
6
4
h(1)
1
h(1)
2
h(1)
3
1
3
7
7
7
5
bias = True
⾒
AAADT3ichVG7TtxAFL325gHOgwWaSDSrrECpVtc2KAgJCZGGkkd2QcLI2GZYRvgle3ZZ1vIP5AdSpCJSiogPyAekQXQpUvAJUUoi0USIa2MLsauQsTxz5p5z7tyZa4cujwXihSRXHj1+8nRkVHn2/MXLser4RCsOOpHDmk7gBtGWbcXM5T5rCi5cthVGzPJsl23ah+8yfrPLopgH/ntxHLIdz2r7fJ87lqCQWT03bNbmfmJ7loh4L1WOzERV09pMLQNaCfTUMDJKKymtpLSS0ktKLyldTxWD+Xt3yQcP65kqmXumRrM6oF0cUvdzdT9X9019QK+Y1To2MB+1YaAWoA7FWA2q38CAPQjAgQ54wMAHQdgFC2L6tkEFhJBiO5BQLCLEc55BCgp5O6RipLAoekhzm3bbRdSnfZYzzt0OneLSH5GzBtP4E7/iJZ7hKf7Cv//MleQ5slqOabVvvSw0xz682rj6r8ujVcDBnevBmgXsw3xeK6fawzyS3cK59Xf7Hy83Ftankxn8jL+p/hO8wO90A7/7x/myxtY/PVCPTbXQi1GD1MF2DIOW1lDnGrg2W19aLlo1AlPwGt5QP97CEqzAKjTBkVBqSaa0K/+Qr+TrSiGVpQJMwr1RGb0BM1nVwg==
2
4
w11 w12 w13
w21 w22 w23
w31 w32 w33
3
5
2
4
x1
x2
1
3
5 =
2
4
z1
z2
z3
3
5
AAADE3ichVE9T9xAEB0bCOCQcJAGQXPKQXRI6DQ+EoEiRULQpOTrAAkTy/Ytdyv8JXvvBFj3B1LQoRSpQKKI+AEUdEBBQ0nBT4hSgpQmBWOfgRwoZCx738zOe367Y/o2DwXilSS3tXe86OzqVl72vHrdm+nrXwq9WmCxkuXZXrBiGiGzuctKggubrfgBMxzTZsvmxky8v1xnQcg9d1Fs+WzNMSouX+eWIaikZ3Y0k1W4G5mOIQK+2VA0x9jMb+vqWBZHNe0uLbam43GqaMwtPxA/KY+lqrr6Jcqrow0iVvXiX3j8HrdqKHomhwVMIvsUqCnIQRqzXuYINCiDBxbUwAEGLgjCNhgQ0rMKKiD4VFuDiGoBIZ7sM2iAQtwadTHqMKi6Qd8KZatp1aU81gwTtkV/sekNiJmFEbzEH3iN53iIP/HPP7WiRCP2skWr2eQyX+/9OrDw+78sh1YB1QfWs54FrMNk4pWTdz+pxKewmvz69rfrhY/zI9E73Mdf5H8Pr/CUTuDWb6yDOTb//Rk/JnmhG6MBqY/H8RQsFQvqhwLOvc9NTaej6oIheAt5mscETMFnmIUS6d9Ig1JOGpZ35WP5RD5rtspSynkDLSFf3AJiaMED
2
4
max(z1, 0)
max(z2, 0)
max(z3, 0)
3
5 =
2
6
4
h(1)
1
h(1)
2
h(1)
3
3
7
5
nn.Linear(2, 3) nn.ReLU()
(ReLU)
MLP ( )

View Slide

x =

x1
x2

y1
y2
AAACsHichVE9TxtBEH1cSAImBCdpkGgQFggpkjVngRKlQqFJaT4MljjrdHdezMJ96W5t2bH8B9JRpUgFEgXiB9AmouEPpOAnRJQg0VAwPp9AYOHMaWdn38ybfXtjh66MFdHFkPZi+OWr1yOjmbE3428nsu/eb8RBPXJEyQncICrbVixc6YuSksoV5TASlme7YtPeW+7mNxsiimXgr6tWKCqeVfPltnQsxZCZ/WjYoib9tu1ZKpLNTqZp6obBvsBezxjCr97nzGyO8pTYdH+gp0EOqRWD7CkMVBHAQR0eBHwojl1YiPnbgg5CyFgFbcYijmSSF+ggw9w6VwmusBjdY1/j01aK+nzu9owTtsO3uLwiZk5jlv7SMV3ROZ3QP7p9tlc76dHV0uLd7nFFaE78mFy7+S/L411h54E1ULPCNj4nWiVrDxOk+wqnx298/3m19mV1tj1Hh3TJ+g/ogs74BX7j2jlaEau/BuixWQv/MR6Q/nQc/cFGIa8v5mllIbf0NR3VCKYwg3mexycs4RuKKHH/fZziN/5oBa2smZrVK9WGUs4HPDJt9w4sH6D8
2
4
x1
x2
1
3
5
AAACyHichVG7ThtBFD1eIAGTBANNJBoLi4g01l0nEREVgiai4mVAYom1u4ztEfvS7tjBrNxQ5gdSUIFEgfiAfEAaOiok/AmIkkhpUuR6vYIkKOSuZufMmXvunJlrBY6MFFEno/X1Dzx5OjiUHX72/MVIbnRsPfIboS3Ktu/44aZlRsKRnigrqRyxGYTCdC1HbFi7C939jaYII+l7a6oViG3XrHmyKm1TMVXJLRiWqEkvtlxThXKvna1X9I/xtP66bRiMS7/hN3dYzxrC27nTVHIFKlIS+YdAT0EBaSz5ua8wsAMfNhpwIeBBMXZgIuJvCzoIAXPbiJkLGclkX6CNLGsbnCU4w2R2l/81Xm2lrMfrbs0oUdt8isMjZGUeU3RJp3RL53RG1/Tzn7XipEbXS4tnq6cVQWXk88vVH/9VuTwr1O9Vj3pWqOJ94lWy9yBhurewe/rm/pfb1dmVqfgVHdMN+z+iDn3jG3jN7/bJslg5fMSPxV74xbhB+t/teAjWS0X9XZGW3xbm5tNWDWICk5jmfsxgDh+whDLXP8UFrtDRFrVA+6S1eqlaJtWM44/QDn4BmRmpEA==
2
6
6
6
4
h(1)
1
h(1)
2
h(1)
3
1
3
7
7
7
5
AAACvHichVG7ThtBFD1seMW8TGiQ0qBYIGisuxYIlCJCoaHkZUBiwdpdD/bI+9Lu2ApZ+QfyAxSIgkgUKB/AB9BQQEnBJ6CUREpDwfV6BQIE3NXsnDlzz50zc63AkZEiuu7QPnR2dff0fsz09Q8MDmWHP61Hfj20RdH2HT/ctMxIONITRSWVIzaDUJiu5YgNq7bQ2t9oiDCSvrem9gKx7ZoVT+5K21RMlbKzhiUq0ost11Sh/NHMVEv6TjxZmGoaBuPCA9YzhvDKD3mlbI7ylMTYS6CnIIc0lvzsKQyU4cNGHS4EPCjGDkxE/G1BByFgbhsxcyEjmewLNJFhbZ2zBGeYzNb4X+HVVsp6vG7VjBK1zac4PEJWjmGcruiEbumc/tAN3b1aK05qtLzs8Wy1tSIoDf0aXf3/rsrlWaH6qHrTs8Iu5hKvkr0HCdO6hd3WN37u365+XRmPJ+g3/WX/R3RNZ3wDr/HPPl4WKwdv+LHYC78YN0h/3o6XYL2Q12fytDydm/+etqoXn/EFk9yPWcxjEUsocv1DnOECl9o3razVNLedqnWkmhE8Ca1xD0x1pQY=
2
6
4
h(2)
1
h(2)
2
1
3
7
5

y1
y2
bias = True
⾒
nn.Linear(2, 3)
→ nn.ReLU()
nn.Linear(3, 2)
→ nn.ReLU()
nn.Linear(2, 2)
MLP ( )

View Slide

x =

x1
x2
x1
x2
1
1
w0
ji
w00
i
wkj +
+
+
+
+
1
torch.nn.Linear
in_features
out_features
bias (True/False) bias
...
...
1
bias
AAAChnichVG7SgNBFD2ur/hM1EawCQbFKtwVQ8QqaGOZh9FAlLC7jnHJvtjdBGLwBwRbU1gpWIgf4AfY+AMW+QSxjGBj4c1mQVTUu8zOmTP33DkzV3UM3fOJOgPS4NDwyGhkbHxicmo6GpuZ3fXsuquJomYbtltSFU8YuiWKvu4bouS4QjFVQ+ypta3e/l5DuJ5uWzt+0xEHplK19CNdU3ymCs2KXIklKElBxH8COQQJhJG1Y/fYxyFsaKjDhIAFn7EBBR5/ZcggOMwdoMWcy0gP9gVOMc7aOmcJzlCYrfG/yqtyyFq87tX0ArXGpxg8XFbGsURPdEtdeqQ7eqb3X2u1gho9L02e1b5WOJXo2Xzh7V+VybOP40/Vn559HGE98KqzdydgerfQ+vrGSbtb2MgvtZbpml7Y/xV16IFvYDVetZucyF/+4UdlL/xi3CD5ezt+gt3VpJxKUm4tkdkMWxXBAhaxwv1II4NtZFHk+lWc4wJtKSIlpZSU7qdKA6FmDl9CynwAVTiQmA==
y1
AAAChnichVG7SgNBFD2ur/iO2gg2YohYhRsxRKyCNpaJMSYQQ9hdJ3HJvtjdBGLwBwRbU1gpWIgf4AfY+AMWfoJYRrCx8GazIBqMd5mdM2fuuXNmrmLrmusRvQxJwyOjY+Ohicmp6ZnZufD8wqFr1R1V5FRLt5yCIrtC10yR8zRPFwXbEbKh6CKv1Ha7+/mGcFzNMg+8pi1Khlw1tYqmyh5T2WZ5oxyOUIz8WOkH8QBEEETaCj/gCMewoKIOAwImPMY6ZLj8FREHwWauhBZzDiPN3xc4wyRr65wlOENmtsb/Kq+KAWvyulvT9dUqn6LzcFi5gig90x116Inu6ZU+/6zV8mt0vTR5VnpaYZfnzpeyH/+qDJ49nHyrBnr2UMGW71Vj77bPdG+h9vSN03Ynu70fba3RDb2x/2t6oUe+gdl4V28zYv9qgB+FvfCLcYPiv9vRDw43YvFEjDKbkdRO0KoQlrGKde5HEinsIY0c16/iApdoSyEpJiWkZC9VGgo0i/gRUuoLV1iQmQ==
y2

y1
y2
MLP ( )

View Slide

X: (x , x )
y: ( or )
x x
x
x
y
x
x y =
y =
( ) ( )

View Slide

X: (x , x )
y: ( or )
y
x
x y =
y =
model = nn.Sequential(nn.Linear(2, 10), nn.ReLU(), nn.Linear(10, 2))
model = nn.Sequential(nn.Linear(2, 10), nn.Sigmoid(), nn.Linear(10, 2))

View Slide

https://itakigawa.page.link/autograd23-example

View Slide

(pytorch)

View Slide

(pytorch 7 !)
optimizer.zero_grad()
optimizer.step()
loss.backward()
loss = criterion(model(inputs), labels) (forward)
(backward)

View Slide

View Slide

= ( )

View Slide

Loss
n
✓1
✓2

View Slide

✓1
✓2

View Slide

(learning rate)
AAACh3ichVHLSsNQED3Gd3206kZwUyyKqzoRX7jysXHZh62CFknibRuaJiFJC1r8ARduFVwpuBA/wA9w4w+48BPEZQU3LpykAdFinZDMuefOmZx7R7UN3fWIXrqk7p7evv6BwcjQ8MhoNDY2nnetmqOJnGYZlrOnKq4wdFPkPN0zxJ7tCKWqGmJXrWz5+7t14bi6Ze54x7YoVJWSqRd1TfF8ipIkH8YSnIKItwM5BAmEkbJiDzjAESxoqKEKARMeYwMKXH72IYNgM1dAgzmHkR7sC5wiwtoaVwmuUJit8LfEq/2QNXnt93QDtcZ/Mfh1WBnHDD3THTXpie7plT7/7NUIevhejjmrLa2wD6Nnk9mPf1VVzh7K36qOnj0UsRp41dm7HTD+KbSWvn5y0cyuZWYas3RDb+z/ml7okU9g1t+127TIXHXwo7IXvjEekPx7HO0gv5CUl5KUXkysb4ajGsAUpjHH81jBOraRQo77l3GOC1xKg9K8tCyttkqlrlAzgR8hbXwB3TiQWA==
0.01 1 " " (step size)
✓1
✓2

View Slide

Q. A.
" "
AAADrHiclVE9bxNBEJ3L8RGOjzjQINFEWIlCY/aiIBBVBBSU+XISyWdZu+e1vcre7Wp3bRFO/gNI1BRUIFEgSgoaqGj4AxT5CYgySDQUzJ1PENsiKHO629k389693WFaCusIOfRm/DNnz52fvRBcvHT5ylxl/uqOVX0T83qspDJ7jFouRcrrTjjJ97ThNGGS77L9h3l9d8CNFSrddgeaNxPaTUVHxNQh1Jr3liLGuyLNWEKdEU+GQfSIS0cj1+OOtsIoGgdWcmDQVs5OVnQQ8bR9TEcbpZ0KpvQ1NU5QGSHQM0kmlbXD5Ygl2UhoeOv2n5ZjJk7NGnd6avrEcVqVKqmRIhamk7BMqlDGuqp8gAjaoCCGPiTAIQWHuQQKFp8GhEBAI9aEDDGDmSjqHIYQILePXRw7KKL7+O3irlGiKe5zTVuwY/yLxNcgcwEWyVfylhyRL+Qd+UZ+/VMrKzRyLwe4shGX69bcs+tbP//LSnB10PvLOtGzgw7cK7wK9K4LJD9FPOIPnr442rq/uZgtkdfkO/p/RQ7JZzxBOvgRv9ngmy9P8MPQC94YDiicHMd0srNSC+/UyMZqde1BOapZuAE3YRnncRfW4DGsQx1i77n33vvoffJr/rbf8Juj1hmv5FyDsfA7vwF/sgLy
2
6
6
6
4
✓1
✓2
.
.
.
✓p
3
7
7
7
5
/
2
6
6
6
4
@loss(✓)/@✓1
@loss(✓)/@✓2
.
.
.
@loss(✓)/@✓p
3
7
7
7
5

View Slide

=
=
AAAC+XiclVJNb9MwGHbCx0b5WAdHLtYqpE6UKum6Mg5I09hhxyHRbVITRY7rdNbsJLLfoFWWfwJXOHNDXPk1O/JP5qRFsJXLXsnSo+f9eF4/dloKriEIrjz/3v0HD9fWH7UeP3n6bKO9+fxEF5WibEwLUaizlGgmeM7GwEGws1IxIlPBTtOLD3X+9DNTmhf5J5iXLJZklvOMUwKOSjY9FGWKUBOVRAEnAmfdKJXm0m7bv9xlEtrWexwJLhNjokZ1omZpbIL+u71gZzfsBf1BuDMaDR0ImrDRIRNALI6gwIFtLWSyrpv1+o4jem6BQQ9H0wJ0jafb+A1uJq1m7F33s0m784fDqyBcgg5axnHS/u30aCVZDlQQrSdhUEJsaq+oYO6mlWYloRdkxiYO5kQyHZtmKYtfOWaKs0K5kwNu2H87DJFaz2XqKiWBc307V5P/y00qyPZiw/OyApbThVBWCezcr98dT7liFMTcAUIVd7tiek7cm4D7HTdUUll7Et52YBWcDPrhqD/8OOzsHyzdWUcv0RbqohC9RfvoCB2jMaLezPviffW++cb/7v/wfy5KfW/Z8wLdCP/XNSpG4vM=
@f(x)
@x1
= lim
!0
f(x1 + , x2, . . . , xd) f(x1, x2, . . . , xd)

View Slide

=
=
AAAC+XiclVJNb9MwGHbCx0b5WAdHLtYqpE6UKum6Mg5I09hhxyHRbVITRY7rdNbsJLLfoFWWfwJXOHNDXPk1O/JP5qRFsJXLXsnSo+f9eF4/dloKriEIrjz/3v0HD9fWH7UeP3n6bKO9+fxEF5WibEwLUaizlGgmeM7GwEGws1IxIlPBTtOLD3X+9DNTmhf5J5iXLJZklvOMUwKOSjY9FGWKUBOVRAEnAmfdKJXm0m7bv9xlEtrWexwJLhNjokZ1omZpbIL+u71gZzfsBf1BuDMaDR0ImrDRIRNALI6gwIFtLWSyrpv1+o4jem6BQQ9H0wJ0jafb+A1uJq1m7F33s0m784fDqyBcgg5axnHS/u30aCVZDlQQrSdhUEJsaq+oYO6mlWYloRdkxiYO5kQyHZtmKYtfOWaKs0K5kwNu2H87DJFaz2XqKiWBc307V5P/y00qyPZiw/OyApbThVBWCezcr98dT7liFMTcAUIVd7tiek7cm4D7HTdUUll7Et52YBWcDPrhqD/8OOzsHyzdWUcv0RbqohC9RfvoCB2jMaLezPviffW++cb/7v/wfy5KfW/Z8wLdCP/XNSpG4vM=
@f(x)
@x1
= lim
!0
f(x1 + , x2, . . . , xd) f(x1, x2, . . . , xd)
AAAClHichVHLSsNAFL2Nr1ofjQoiuAkWRUHKTVEUUSgWwZW01aqgJSRxWkPzIklLa/EH/AEXXSl2IX6AH+DGH3DRTxCXCm5ceJsGRMV6w2TOnLnnzpm5iq1rrofYCnE9vX39A+HByNDwyGiUHxvfd62yo7KcaumWc6jILtM1k+U8zdPZoe0w2VB0dqCUUu39gwpzXM0y97yazfKGXDS1gqbKHlESH60JG0JhviqJi0JVSixIfAzj6IfwG4gBiEEQaYu/h2M4AQtUKIMBDEzwCOsgg0vfEYiAYBOXhzpxDiHN32dwDhHSlimLUYZMbIn+RVodBaxJ63ZN11erdIpOwyGlALP4hLf4io94h8/48Wetul+j7aVGs9LRMluKXkztvv+rMmj24PRL1dWzBwVY9b1q5N32mfYt1I6+cnb5uruWna3P4TW+kP8rbOED3cCsvKnNDMs2uvhRyAu9GDVI/NmO32A/EReX45hZiiU3g1aFYRpmYJ76sQJJ2IY05PyeNeAGmtwkt86luK1OKhcKNBPwLbidTyDslEI=
y = f(x1, x2)
AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwEQ8QqaGOZh9FADGF3HeOSfbG7CWrwBwRbU1gpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV7F1zfWIngekwaHhkdHQ2PjE5NR0ODIzu+NaDUcVRdXSLaekyK7QNVMUPc3TRcl2hGwouthV6pud/d2mcFzNMre9Y1tUDLlmageaKntMFY6qyWokSnHyY7EXJAIQRRBZK3KPPezDgooGDAiY8BjrkOHyV0YCBJu5ClrMOYw0f1/gFOOsbXCW4AyZ2Tr/a7wqB6zJ605N11erfIrOw2HlImL0RLf0Ro90Ry/0+Wetll+j4+WYZ6WrFXY1fDZf+PhXZfDs4fBb1dezhwOs+V419m77TOcWalffPGm/FdbzsdYyXdMr+7+iZ3rgG5jNd/UmJ/KXffwo7IVfjBuU+N2OXrCTjCdSccqtRjMbQatCWMASVrgfaWSwhSyKXL+Gc1ygLYWkuJSS0t1UaSDQzOFHSJkvVTaQmA==
x2
AAAChnichVHLSsNAFD2Nr/quuhHciKXiqtyIUnFVdONSW/uAWkoSpzU0TUKSFmvxBwS3duFKwYX4AX6AG3/AhZ8gLhXcuPA2DYgW6w2TOXPmnjtn5qq2obse0XNIGhgcGh4Jj46NT0xOTUdmZrOuVXc0kdEsw3LyquIKQzdFxtM9Q+RtRyg11RA5tbrd2c81hOPqlrnvNW1RrCkVUy/rmuIxlT4uyaVIlOLkx2IvkAMQRRC7VuQeBziEBQ111CBgwmNsQIHLXwEyCDZzRbSYcxjp/r7AKcZYW+cswRkKs1X+V3hVCFiT152arq/W+BSDh8PKRcToiW7pjR7pjl7o889aLb9Gx0uTZ7WrFXZp+mw+/fGvqsazh6NvVV/PHsrY8L3q7N32mc4ttK6+cdJ+S2+mYq1luqZX9n9Fz/TANzAb79rNnkhd9vGjshd+MW6Q/LsdvSC7GpfX47S3Fk1uBa0KYwFLWOF+JJDEDnaR4foVnOMCbSksxaV1KdFNlUKBZg4/Qkp+AVMWkJc=
x1
AAACCHicdVDLSgMxFM34rPU16tJNsAh1M8yUqW13RTcuK9gHtKVk0kwbmmSGJFMsQ3/Ab3Cra3fi1r9w6Z+YPgQreuDC4Zx7ufeeIGZUadf9sNbWNza3tjM72d29/YND++i4oaJEYlLHEYtkK0CKMCpIXVPNSCuWBPGAkWYwup75zTGRikbiTk9i0uVoIGhIMdJG6tl2R6CAIRjmOwFP76cXPTvnOu4c0HUKlXKx7BtSKlZ8Q7yllQNL1Hr2Z6cf4YQToTFDSrU9N9bdFElNMSPTbCdRJEZ4hAakbahAnKhuOr98Cs+N0odhJE0JDefqz4kUcaUmPDCdHOmh+u3NxL+8dqLDcjelIk40EXixKEwY1BGcxQD7VBKs2cQQhCU1t0I8RBJhbcJa2RLwqcnk+3H4P2kUHO/S8W/9XPVqmU4GnIIzkAceKIEquAE1UAcYjMEjeALP1oP1Yr1ab4vWNWs5cwJWYL1/AS7mmf8=
rf(x)
AAACCXicdVDLTgIxFO3gC/EFunTTSExwIRkGBNwR3bjERB4JENIpHWhoO5O2o5IJX+A3uNW1O+PWr3Dpn9gBTMToSW5ycs69ufceN2BUadv+sBIrq2vrG8nN1Nb2zu5eOrPfVH4oMWlgn/my7SJFGBWkoalmpB1IgrjLSMsdX8Z+65ZIRX1xoycB6XE0FNSjGGkj9dOZ065ALkPQy3VdHt1PT/rprJ0/r5adMwfaeduuOMVyTJxKySnCglFiZMEC9X76szvwcciJ0JghpToFO9C9CElNMSPTVDdUJEB4jIakY6hAnKheNDt9Co+NMoCeL00JDWfqz4kIcaUm3DWdHOmR+u3F4l9eJ9RetRdREYSaCDxf5IUMah/GOcABlQRrNjEEYUnNrRCPkERYm7SWtrh8ajL5fhz+T5pOvlDOl65L2drFIp0kOARHIAcKoAJq4ArUQQNgcAcewRN4th6sF+vVepu3JqzFzAFYgvX+BZf+mjE=
rf(x)
(Gradient)
=
AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwEQ8QqaGOZh9FADGF3HeOSfbG7CWrwBwRbU1gpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV7F1zfWIngekwaHhkdHQ2PjE5NR0ODIzu+NaDUcVRdXSLaekyK7QNVMUPc3TRcl2hGwouthV6pud/d2mcFzNMre9Y1tUDLlmageaKntMFY6qyWokSnHyY7EXJAIQRRBZK3KPPezDgooGDAiY8BjrkOHyV0YCBJu5ClrMOYw0f1/gFOOsbXCW4AyZ2Tr/a7wqB6zJ605N11erfIrOw2HlImL0RLf0Ro90Ry/0+Wetll+j4+WYZ6WrFXY1fDZf+PhXZfDs4fBb1dezhwOs+V419m77TOcWalffPGm/FdbzsdYyXdMr+7+iZ3rgG5jNd/UmJ/KXffwo7IVfjBuU+N2OXrCTjCdSccqtRjMbQatCWMASVrgfaWSwhSyKXL+Gc1ygLYWkuJSS0t1UaSDQzOFHSJkvVTaQmA==
x2
AAAChnichVHLSsNAFD2Nr/quuhHciKXiqtyIUnFVdONSW/uAWkoSpzU0TUKSFmvxBwS3duFKwYX4AX6AG3/AhZ8gLhXcuPA2DYgW6w2TOXPmnjtn5qq2obse0XNIGhgcGh4Jj46NT0xOTUdmZrOuVXc0kdEsw3LyquIKQzdFxtM9Q+RtRyg11RA5tbrd2c81hOPqlrnvNW1RrCkVUy/rmuIxlT4uyaVIlOLkx2IvkAMQRRC7VuQeBziEBQ111CBgwmNsQIHLXwEyCDZzRbSYcxjp/r7AKcZYW+cswRkKs1X+V3hVCFiT152arq/W+BSDh8PKRcToiW7pjR7pjl7o889aLb9Gx0uTZ7WrFXZp+mw+/fGvqsazh6NvVV/PHsrY8L3q7N32mc4ttK6+cdJ+S2+mYq1luqZX9n9Fz/TANzAb79rNnkhd9vGjshd+MW6Q/LsdvSC7GpfX47S3Fk1uBa0KYwFLWOF+JJDEDnaR4foVnOMCbSksxaV1KdFNlUKBZg4/Qkp+AVMWkJc=
x1
AAACpXichVFNLyxBFD2a52M8DDYSm8mb+FhN7gghVsLGRuJrhsRIp7rVUNHT3emumWAyWwt/wMLqvcRC2OIH2PgDFn6CvCWJjYU7PZ0IgtuprlOn7rl1qq7lOyrURPdNRnPLr9a29o5E5++u7p5kb18+9MqBLXO253jBuiVC6ShX5rTSjlz3AylKliPXrN25+v5aRQah8txVve/LzZLYdlVR2UIzZSZThWIg7GrBF4FWwkkVa294zxyrJcxkmjIUReozyMYgjTgWveQ1CtiCBxtllCDhQjN2IBDyt4EsCD5zm6gyFzBS0b5EDQnWljlLcoZgdpf/27zaiFmX1/WaYaS2+RSHR8DKFIbojs7okW7pnB7o5cta1ahG3cs+z1ZDK32z52hg5flHVYlnjZ031beeNYqYirwq9u5HTP0WdkNfOTh+XJleHqoO0z/6z/7/0j3d8A3cypN9uiSXT77xY7EXfjFuUPZjOz6D/FgmO5GhpfH0zGzcqnYM4g9GuR+TmME8FpHj+oe4wCWujBFjwVg18o1UoynW9ONdGOYrpNKdFQ==
@f
@x2
AAACpXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsoxKQVJSZXxxQkFpVkJuYopNUi2BXxhrVc8QLKBnoGYKCAyTCEMpQZoCAgX2A7QwxDCkM+QzJDKUMuQypDHkMJkJ3DkMhQDITRDIYMBgwFQLFYhmqgWBGQlQmWT2WoZeAC6i0FqkoFqkgEimYDyXQgLxoqmgfkg8wsButOBtqSA8RFQJ0KDKoGVw1WGnw2OGGw2uClwR+cZlWDzQC5pRJIJ0H0phbE83dJBH8nqCsXSJcwZCB04XVzCUMagwXYrZlAtxeARUC+SIboL6ua/jnYKki1Ws1gkcFroPsXGtw0OAz0QV7Zl+SlgalBs/G4JwnoFmCIASPIED06MBlhRnqGpnoGgSbKDk7QqOJgkGZQYtAAxoc5gwODB0MAQyjQ/DaGNQxbGbYxqTP5MoUwhUGUMjFC9QgzoACmeACisJ0U
@f
@x1
AAAC2HicjVFNa9wwEJXdtPno1yY95iKyFFIoi704zeYQCC2UHFPIJqErs0iyvBGRZCPJyy7C0FvpNf8ut/afRHa2aT56yIDg8eaNZuYNKQU3NoqugvDZ0vMXyyuray9fvX7ztrO+cWKKSlM2pIUo9BnBhgmu2NByK9hZqRmWRLBTcvGlyZ9OmTa8UMd2XrJU4oniOafYemrc+e1Q+8lIT0jqol7Uxseo198b7AwSD3Z39pJBUiOFicAw30ZEuln9oYb7EBE24coRia3ms3oN5RpTh0qsLcfijvYfNxvHNUJPU/Zb5TQrrHlqSdaWMJXdDjXudP9uBR+DeAG6YBFH484flBW0kkxZKrAxozgqbeqaFlQwv2RlWInpBZ6wkYcKS2ZS15pYw/eeyWBeaP+UhS17t8JhacxcEq/0852bh7mG/F9uVNl8kDquysoyRW8a5ZWAtoDNWWHGNaNWzD3AVHM/K6Tn2Dtm/fHvdSGy8SR+6MBjcNLvxZ96ybeke/B54c4K2ARbYBvEYBccgENwBIaABl8DEVTBNPwe/gh/hr9upGGwqHkH7kV4eQ2Md+O1
rf(x) =
2
6
6
6
6
4
@f(x)
@x1
@f(x)
@x2
.
.
.
@f(x)
@xd
3
7
7
7
7
5

View Slide

⾒( )
:
( )
(Gradient descent)

View Slide

/ =
(LR)
LR
LR LR
LR

View Slide

(line search)
( )
( )
(line search)

View Slide

lr
( )
torch.optim.lr_scheduler

View Slide

LR
epoch
step
https://arxiv.org/abs/1608.03983v5
(or )

View Slide

(step size)
Loss
Loss
AAADNHicnVHLahRBFL3dvmJ8ZFQEwc3gEJnAMNwekjijBIIPcJmHkwSmx6a7rZkpUv2gqmbI2PQP+AMuXCm4EBfu9APc+AMuAu7EhQhuIrhx4e2eFtFgAlbTXadO3XP6VF0vFlxpxF3DPHL02PETUyenT50+c3amdO78hoqG0mdtPxKR3PJcxQQPWVtzLdhWLJkbeIJtetu3sv3NEZOKR+E9PY5ZN3D7Ie9x39VEOaVXtmY7WumxYHbg6oEMEhEplVYTOzfvyL7XTbDeai7ifKOGdcRmAxcJLKDVslqp7QWJrQdMu2k6t2SrYeAkfMlK74flX4ZMykim1V41q91JHV77L/Pa2OFzTqmSlWWjvB9YBahAMVai0huw4QFE4MMQAmAQgiYswAVFTwcsQIiJ60JCnCTE830GKUyTdkhVjCpcYrfp26dVp2BDWmeeKlf79BdBryRlGWbxPb7APXyHL/Ez/vinV5J7ZFnGNHsTLYudmUeX1r8fqgpo1jD4rTows4YeNPOsnLLHOZOdwp/oRw8f761fX5tNruIz/EL5n+IuvqUThKNv/vNVtvbkgDweZaEbowZZf7djP9ho1K2FOq7OV5ZvFq2agstwBarUj2uwDHdhBdrgGxeNG8Zt44752vxgfjQ/TUpNo9BcgD+G+fUn4TfN5g==
loss(✓) =
Pn
i=1
error(f(xi, ✓), yi)
AAACw3ichVFNSxtRFD2O2lpra9RNwY00KClIuBP8SFwUqQgu/WhMIAlhZvoSh8wXMy9BHeYP9A900VVLuyj+AH+Am9Jt24U/oXSp4MaFdyYDoqH2Dm/eeefdc9957+qeZQaS6HxIGR4ZffR47Mn404lnzyczU9P7gdv1DVE2XMv1q7oWCMt0RFma0hJVzxearVuionc24v1KT/iB6Tpv5ZEnGrbWdsyWaWiSqWbmdStX1+3wMFoM60m1mt/WGyHlS8UVWiosUp6oWKAVBsukltRSFKfX5YGQWhS9amaycUYcc4NATUEWaWy7mVPU8Q4uDHRhQ8CBZGxBQ8BfDSoIHnMNhMz5jMxkXyDCOGu7nCU4Q2O2w/82r2op6/A6rhkkaoNPsXj4rJzDPP2mb3RB3+mE/tD1P2uFSY3YyxHPel8rvObk+xd7V/9V2TxLHNyqHvQs0UIx8Wqydy9h4lsYfX3v+MPF3trufLhAn+kv+/9E53TGN3B6l8bXHbH78QE/OnvhF+MGqffbMQj2C3l1OU87S9n1N2mrxjCLl8hxP1axji1so8z1v+AHfuKXsql0FF+R/VRlKNXM4E4o0Q1MsaWX
f(x, ✓)
AAACuXichVHLSiNBFD22juPbqJsBNzJBcSHhdoia6EZmNi59RYUkhO62kvTYL7orAW3yA/6AoCsHXIgf4HZgNs7sZ+EniEsHZuPC252GQUW9TXWdOnXPrVN1dc8yA0l006V093zo/djXPzA4NDwymhob3w7cpm+IouFarr+ra4GwTEcUpSktsev5QrN1S+zo+1+j/Z2W8APTdbbkgScqtlZ3zJppaJKpaioXluMiJb+uV0LKFPILlMvOUYYon6UFBvOkFtRCu6zbYVk2hNTa7WoqHSVEMfUSqAlII4k1N3WFMvbgwkATNgQcSMYWNAT8laCC4DFXQcicz8iM9wXaGGBtk7MEZ2jM7vO/zqtSwjq8jmoGsdrgUywePiunME1/6ILu6Zou6ZYeXq0VxjUiLwc86x2t8KqjR582/72rsnmWaPxXvelZooZ87NVk717MRLcwOvrW4fH95tLGdDhD3+mO/Z/RDf3kGzitv8b5utg4fcOPzl74xbhB6vN2vATb2Yw6n6H1XHrlS9KqPkziM2a5H4tYwSrWUOT6J/iBX/itLCua0lC+dVKVrkQzgSehBI+WP6G1
✓

View Slide

(step size)
Loss
Loss
AAADNHicnVHLahRBFL3dvmJ8ZFQEwc3gEJnAMNwekjijBIIPcJmHkwSmx6a7rZkpUv2gqmbI2PQP+AMuXCm4EBfu9APc+AMuAu7EhQhuIrhx4e2eFtFgAlbTXadO3XP6VF0vFlxpxF3DPHL02PETUyenT50+c3amdO78hoqG0mdtPxKR3PJcxQQPWVtzLdhWLJkbeIJtetu3sv3NEZOKR+E9PY5ZN3D7Ie9x39VEOaVXtmY7WumxYHbg6oEMEhEplVYTOzfvyL7XTbDeai7ifKOGdcRmAxcJLKDVslqp7QWJrQdMu2k6t2SrYeAkfMlK74flX4ZMykim1V41q91JHV77L/Pa2OFzTqmSlWWjvB9YBahAMVai0huw4QFE4MMQAmAQgiYswAVFTwcsQIiJ60JCnCTE830GKUyTdkhVjCpcYrfp26dVp2BDWmeeKlf79BdBryRlGWbxPb7APXyHL/Ez/vinV5J7ZFnGNHsTLYudmUeX1r8fqgpo1jD4rTows4YeNPOsnLLHOZOdwp/oRw8f761fX5tNruIz/EL5n+IuvqUThKNv/vNVtvbkgDweZaEbowZZf7djP9ho1K2FOq7OV5ZvFq2agstwBarUj2uwDHdhBdrgGxeNG8Zt44752vxgfjQ/TUpNo9BcgD+G+fUn4TfN5g==
loss(✓) =
Pn
i=1
error(f(xi, ✓), yi)
AAACw3ichVFNSxtRFD2O2lpra9RNwY00KClIuBP8SFwUqQgu/WhMIAlhZvoSh8wXMy9BHeYP9A900VVLuyj+AH+Am9Jt24U/oXSp4MaFdyYDoqH2Dm/eeefdc9957+qeZQaS6HxIGR4ZffR47Mn404lnzyczU9P7gdv1DVE2XMv1q7oWCMt0RFma0hJVzxearVuionc24v1KT/iB6Tpv5ZEnGrbWdsyWaWiSqWbmdStX1+3wMFoM60m1mt/WGyHlS8UVWiosUp6oWKAVBsukltRSFKfX5YGQWhS9amaycUYcc4NATUEWaWy7mVPU8Q4uDHRhQ8CBZGxBQ8BfDSoIHnMNhMz5jMxkXyDCOGu7nCU4Q2O2w/82r2op6/A6rhkkaoNPsXj4rJzDPP2mb3RB3+mE/tD1P2uFSY3YyxHPel8rvObk+xd7V/9V2TxLHNyqHvQs0UIx8Wqydy9h4lsYfX3v+MPF3trufLhAn+kv+/9E53TGN3B6l8bXHbH78QE/OnvhF+MGqffbMQj2C3l1OU87S9n1N2mrxjCLl8hxP1axji1so8z1v+AHfuKXsql0FF+R/VRlKNXM4E4o0Q1MsaWX
f(x, ✓)
AAACuXichVHLSiNBFD22juPbqJsBNzJBcSHhdoia6EZmNi59RYUkhO62kvTYL7orAW3yA/6AoCsHXIgf4HZgNs7sZ+EniEsHZuPC252GQUW9TXWdOnXPrVN1dc8yA0l006V093zo/djXPzA4NDwymhob3w7cpm+IouFarr+ra4GwTEcUpSktsev5QrN1S+zo+1+j/Z2W8APTdbbkgScqtlZ3zJppaJKpaioXluMiJb+uV0LKFPILlMvOUYYon6UFBvOkFtRCu6zbYVk2hNTa7WoqHSVEMfUSqAlII4k1N3WFMvbgwkATNgQcSMYWNAT8laCC4DFXQcicz8iM9wXaGGBtk7MEZ2jM7vO/zqtSwjq8jmoGsdrgUywePiunME1/6ILu6Zou6ZYeXq0VxjUiLwc86x2t8KqjR582/72rsnmWaPxXvelZooZ87NVk717MRLcwOvrW4fH95tLGdDhD3+mO/Z/RDf3kGzitv8b5utg4fcOPzl74xbhB6vN2vATb2Yw6n6H1XHrlS9KqPkziM2a5H4tYwSrWUOT6J/iBX/itLCua0lC+dVKVrkQzgSehBI+WP6G1
✓
AAACwnichVHLTttAFD2Y0vJom1A2lbpBjaioWkXXEdAEqRIqXbArrwBSHEW2GYKLX9iTSGDyA/2BLlhB1QXiA/gANnRbxIJPQCxBYsOCa8cSAlR6rfGcOXPPnTNzDd+2Qkl02qF0Pul6+qy7p7fv+YuXmWz/q4XQawSmKJue7QVLhh4K23JFWVrSFkt+IHTHsMWisTYZ7y82RRBanjsvN3xRdfS6a61Ypi6ZqmU/f9C+OaKuD0daUqwS1I1qRPlScYxGCh8pT1Qs0BiDUVJLaqmlGU6kyVUh9VbrfS2bizPiGHwI1BTkkMa0lz2AhmV4MNGAAwEXkrENHSF/Fagg+MxVETEXMLKSfYEWelnb4CzBGTqza/yv86qSsi6v45phojb5FJtHwMpBDNEJ7dEFHdE+ndH1P2tFSY3YywbPRlsr/Frmx+u5q/+qHJ4lVm9Vj3qWWEEx8Wqxdz9h4luYbX1z8+fF3PjsUPSOdumc/e/QKR3yDdzmpfl7RsxuP+LHYC/8Ytwg9X47HoKFQl4dzdPMSG7iS9qqbrzBWwxzPz5hAlOYRpnr/8If/MWx8lX5rqwrYTtV6Ug1A7gTytYNacGk0A==
+⌦(✓)

View Slide

⾒
⾒

View Slide

View Slide

Loss = ⾒
x =

x1
x2
x1
x2
⾒

View Slide

Loss = ⾒
x =

x1
x2
x1
x2
⾒
AAAChnichVG7SgNBFD2ur/hM1EawEUPEKtwVQ8QqaGOZGPOAGMLuOsYl+2J3E9DgDwi2WlgpWIgf4AfY+AMWfoJYKthYeLNZEA3Gu8zOmTP33DkzV3UM3fOJngekwaHhkdHI2PjE5NR0NDYzW/TspquJgmYbtltWFU8YuiUKvu4bouy4QjFVQ5TUxlZnv9QSrqfb1q5/5IiqqdQt/UDXFJ+pfLMm12JxSlIQi71ADkEcYWTt2D32sA8bGpowIWDBZ2xAgcdfBTIIDnNVtJlzGenBvsAJxlnb5CzBGQqzDf7XeVUJWYvXnZpeoNb4FIOHy8pFJOiJbumNHumOXujzz1rtoEbHyxHPalcrnFr0dD7/8a/K5NnH4beqr2cfB1gPvOrs3QmYzi20rr51fPGW39hJtJfpml7Z/xU90wPfwGq9azc5sXPZx4/KXvjFuEHy73b0guJqUk4lKbcWz2yGrYpgAUtY4X6kkcE2sihw/TrOcI4LKSIlpZSU7qZKA6FmDj9CynwBTLCQlA==
u1
AAAChnichVHLSsNAFD3GV3226kZwIxbFVbgRS8VV0Y3Laq0tqJQkTmswTUIehVr8AcGtXbhScCF+gB/gxh9w4SeISwU3LrxJA6JivWEyZ87cc+fMXM0xDc8neuqRevv6BwYTQ8Mjo2PjydTE5I5nB64uirpt2m5ZUz1hGpYo+oZvirLjCrWumaKkHa2H+6WGcD3Dtrb9piP262rNMqqGrvpMFYLKUiWVJpmimP0NlBikEUfeTt1hDwewoSNAHQIWfMYmVHj87UIBwWFuHy3mXEZGtC9wgmHWBpwlOENl9oj/NV7txqzF67CmF6l1PsXk4bJyFvP0SDf0Sg90S8/08WetVlQj9NLkWetohVNJnk4X3v9V1Xn2cfil6urZRxUrkVeDvTsRE95C7+gbx+3XwurWfGuBruiF/V/SE93zDazGm369KbYuuvjR2Au/GDdI+dmO32BnSVYyMm0up3NrcasSmMEcFrkfWeSwgTyKXL+GM5yjLSUkWcpI2U6q1BNrpvAtpNwnTtCQlQ==
u2
AAAChnichVG7SgNBFD1ZXzG+ojaCTTBErMKNDyJWQRvLPIwJRAm76xiX7IvdTSAGf0CwNYWVgoX4AX6AjT9gkU8QSwUbC282C6Ki3mV2zpy5586ZuYqta65H1A1JA4NDwyPh0cjY+MTkVHR6Zte1Go4qiqqlW05ZkV2ha6Yoepqni7LtCNlQdFFS6lu9/VJTOK5mmTteyxb7hlwztUNNlT2mCo3qSjUapyT5EfsJUgGII4isFb3DHg5gQUUDBgRMeIx1yHD5qyAFgs3cPtrMOYw0f1/gBBHWNjhLcIbMbJ3/NV5VAtbkda+m66tVPkXn4bAyhgQ90g290APd0hO9/1qr7dfoeWnxrPS1wq5Onc4V3v5VGTx7OPpU/enZwyHWfa8ae7d9pncLta9vHndeChv5RHuRruiZ/V9Sl+75BmbzVb3OifzFH34U9sIvxg1KfW/HT7C7nEytJSm3Gs9sBq0KYx4LWOJ+pJHBNrIocv0aznCOjhSWktKalO6nSqFAM4svIWU+AFDwkJY=
u3
AAACjnichVHNTsJAEB7qH+IPqBcTL0Si8dRMCaiYGIleOCqIkgAhbV2gUtqmXUiQ8ALejQcTjSYejA/gA3jxBTzwCMajJl48OC0Y4wGdze7Ofjvf7LcziqVrDkfs+oSh4ZHRMf94YGJyajoYmpk9cMyGrbKsauqmnVNkh+mawbJc4zrLWTaT64rODpXajnt/2GS2o5nGPm9ZrFiXK4ZW1lSZE5Qv8CrjcqldO+6UQhEUMboej0XDKEbjmJAS5MRRSqzGwpKInkWgb7tm6AEKcAQmqNCAOjAwgJOvgwwOjTxIgGARVoQ2YTZ5mnfPoAMB4jYoilGETGiN1gqd8n3UoLOb0/HYKr2i07SJGYYlfMY7fMMnvMcX/ByYq+3lcLW0aFd6XGaVgqfzmY9/WXXaOVR/WH9q5lCGdU+rRtotD3F/ofb4zZPzt8xGeqm9jDf4SvqvsYuP9AOj+a7e7rH0xR96FNJCFaMGfXchPNg5iIpSXMS9WCS53W+VHxZgEVaoH2uQhBTsQtar6BlcwpUQElaFTWGrFyr4+pw5+GVC6gvHrpTF
✓kj
AAADCHiclVE9T9xAEB2bhA8n4Y7QEKWxciICEZ1mrbvAISEh0lDydYB0d7JsZzlW+Ev23gmwrqLjD1BQEYkiSpMO0SZN/kAKfgKiBIkmReZ8liJAIcpY6337dt7s2x07dEUsES8Ute/J0/6BwSHt2fMXw7n8yMv1OGhFDq86gRtEm7YVc1f4vCqFdPlmGHHLs12+Ye986O5vtHkUi8Bfk3shb3hW0xdbwrEkUWb+oGUyfU6vx6LpWRNaUk9L1qKm3UiwiMZMuWS8w6JRxgqrECgjq7wvdepym0vLTBjrdPRdk2lT/y01elJDmzTzBcpPQ38IWAYKkMVSkD+DOnyEABxogQccfJCEXbAgpq8GDBBC4hqQEBcREuk+hw5opG1RFqcMi9gd+jdpVctYn9bdmnGqdugUl0ZESh3G8Sd+xmv8gV/wEn/9tVaS1uh62aPZ7ml5aOYOx1Zv/6nyaJaw/Uf1qGcJWzCTehXkPUyZ7i2cnr69f3S9OrsynrzFT3hF/k/wAr/TDfz2jXO6zFeOH/Fjkxd6MWoQu9+Oh2DdKLJyEZdLhfmFrFWD8BrewAT1YxrmYRGWoEr1L5WcMqa8Ug/Vr+qZet5LVZVMMwp3Qv32GzQ1t+4=
u1 = (✓11x1 + ✓21x2)
AAADCHiclVG/T9tAFH42pYCBJsAC6mIRUVG1ip6tBBKkSqhdOvIrgJRElu0e4YR/yb5EpFambvwDHToVqQNi6Ya6wsI/wMCfgBhBYmHgxbFUtQgQzzrfd9+9791396zA4ZFAPJfkvhf9LwcGh5ThkdFXmezY+HrkN0ObVWzf8cNNy4yYwz1WEVw4bDMImelaDtuwdj519zdaLIy4762JdsDqrtnw+Ba3TUGUkf3WNHT1g1qLeMM1Z5W4lpSshg2rHmMe9VKxoL/HvF7EslYmUEStPFfo1MQ2E6YRa3qno+4amvLu2VK9J9WVt0Y2R/lJqPeBloIcpLHkZ4+gBl/ABxua4AIDDwRhB0yI6KuCBggBcXWIiQsJ8WSfQQcU0jYpi1GGSewO/Ru0qqasR+tuzShR23SKQyMkpQozeIYHeIWneIgXePtgrTip0fXSptnqaVlgZPYmV2+eVLk0C9j+q3rUs4AtKCVeOXkPEqZ7C7unb339frW6sDITv8F9vCT/P/EcT+gGXuva/rXMVn484sciL/Ri1CDt/3bcB+t6XivmcbmQW/yYtmoQXsM0zFI/5mERPsMSVKj+hZSRJqUpeU/+LR/Jf3qpspRqJuCfkI/vADtgt/E=
u2 = (✓12x1 + ✓22x2)
AAADCHiclVG/T9tAFH42UKhLmwALiMVqBKKiip5NUkglJARLR340gJRElu1ewgn/kn2JSq1M3fgHGJhaiQGxsKGusPQf6MCfgBhB6sLAi2MJASoVzzrfd9+9791396zA4ZFAPJfknt6+F/0DL5VXg6/fZLJDw+uR3wxtVrZ9xw83LTNiDvdYWXDhsM0gZKZrOWzD2l7q7G+0WBhx3/ssdgJWc82Gx+vcNgVRRvZ705hR59VqxBuuOaXE1aRkJWxYtRjzqM8VC/p7zOtFLGklAkXUSh8K7arYYsI0Ym2m3Va/Gpoy/Wyp3pXqyjsjm6P8JNTHQEtBDtJY9rMnUIUv4IMNTXCBgQeCsAMmRPRVQAOEgLgaxMSFhHiyz6ANCmmblMUowyR2m/4NWlVS1qN1p2aUqG06xaERklKFCfyDh3iFv/EIL/Dmn7XipEbHyw7NVlfLAiOzO7r2978ql2YBW3eqJz0LqMNc4pWT9yBhOrewu/rWt72rtY+rE/Ek/sRL8v8Dz/GMbuC1ru2DFba6/4Qfi7zQi1GDtIfteAzW9bxWzONKIbewmLZqAMbhLUxRP2ZhAT7BMpSp/oWUkUalMXlXPpZP5F/dVFlKNSNwL+TTW0KLt/Q=
u3 = (✓13x1 + ✓23x2)

View Slide

Loss = ⾒
x =

x1
x2
x1
x2
⾒
u1
u2
u3
✓kj
u1 = (✓11x1 + ✓21x2)
u2 = (✓12x1 + ✓22x2)
u3 = (✓13x1 + ✓23x2)
AAAChnichVG7SgNBFD1ZXzE+ErURbIIhYhXuiqJYBW0sNTGJoCHsrmNcsi92N4EY/AHB1hRWChbiB/gBNv6ART5BLCPYWHizWRAV9S6zc+bMPXfOzFUdQ/d8ok5EGhgcGh6JjsbGxicm44mp6aJn111NFDTbsN09VfGEoVui4Ou+IfYcVyimaoiSWtvs7ZcawvV029r1m44om0rV0o90TfGZyjcqciWRogwFkfwJ5BCkEMa2nbjHAQ5hQ0MdJgQs+IwNKPD424cMgsNcGS3mXEZ6sC9wihhr65wlOENhtsb/Kq/2Q9bida+mF6g1PsXg4bIyiTQ90S116ZHu6Jnef63VCmr0vDR5Vvta4VTiZ7P5t39VJs8+jj9Vf3r2cYS1wKvO3p2A6d1C6+sbJ+1ufj2Xbi3QNb2w/yvq0APfwGq8ajc7Inf5hx+VvfCLcYPk7+34CYpLGXklQzvLqexG2Koo5jCPRe7HKrLYwjYKXL+Kc1ygLUWljLQirfZTpUiomcGXkLIfTtKQlQ==
v1
AAAChnichVHLSsNAFD3GV62PVt0Iboql4irciEVxVXTjUlurQi0liWMNpklI0oAWf0BwqwtXCi7ED/AD3PgDLvoJ4lLBjQtv0oBoUW+YzJkz99w5M1dzTMPzido9Um9f/8BgYig5PDI6lkqPT2x5dtPVRVm3Tdvd0VRPmIYlyr7hm2LHcYXa0EyxrR2uhvvbgXA9w7Y2/SNHVBtq3TL2DV31mSoFtflaOksyRZHpBkoMsohj3U7fYxd7sKGjiQYELPiMTajw+KtAAcFhrooWcy4jI9oXOEGStU3OEpyhMnvI/zqvKjFr8Tqs6UVqnU8xebiszCBHT3RLr/RId/RMH7/WakU1Qi9HPGsdrXBqqdOp0vu/qgbPPg6+VH969rGPpcirwd6diAlvoXf0wfHFa2m5mGvN0jW9sP8ratMD38AK3vSbDVG8/MOPxl74xbhBys92dIOteVnJy7SxkC2sxK1KYBozmON+LKKANayjzPXrOMM5LqSEJEt5abGTKvXEmkl8C6nwCVDykJY=
v2
AAACjHichVHLLgRBFD3aa7wHG4nNxIRYdW5PZjBEIiRi6TUeQSbdrVD0K909k4yJH7C1sMCCxEJ8gA+w8QMWPkEsSWws3OkZEQvcSlXdOveeW6fqGp4lg5DoqU6pb2hsao61tLa1d3R2xbt7VgK34JsiZ7qW668ZeiAs6YhcKENLrHm+0G3DEqvGwUwlvloUfiBdZzkseWLL1ncduSNNPWRofdPbk/nyvjzKx5OkUmosk04lSE1lKKtl2cmQlh1JJzSVIkuiZvNu/A6b2IYLEwXYEHAQsm9BR8BjAxoIHmNbKDPmsyejuMARWplb4CzBGTqjB7zu8mmjhjp8rtQMIrbJt1g8fWYmMEiPdEOv9EC39Ewfv9YqRzUqWkq8G1Wu8PJdx31L7/+ybN5D7H2z/tQcYgdjkVbJ2r0IqbzCrPKLh6evS+OLg+UhuqIX1n9JT3TPL3CKb+b1glg8+0OPwVr4x7hBX11I/O6spFQto9JCOjk1XWtVDP0YwDD3YxRTmMM8clzfxgnOcaF0KmllQpmspip1NU4vfpgy+wmrrJPa
ji
AAADPXiclZHNStxQFMdPorYabR11IwgldLBYlOHcOKOOIEjropsWPzoqzAwhSa/jxXyRZAY0zAv4Ai66aqEL6QPozkU3fYEufAAXpcWNghsXnkkCpZUq3pDc/z33/E7+9x7Tt0UYIZ5Kcld3z6PHvX1K/8CTp4O5oeH10GsGFq9Ynu0Fm6YRclu4vBKJyOabfsANx7T5hrnzurO/0eJBKDz3fbTr87pjNFyxJSwjopCeO27pTF1Qa6FoOMaEEteSktWgYdZjLKA2VypqU1jQSlhmZRIlZOWZYrvmbws9ZqzdVps6UyYfCGopqD0YnE7BaeWlnstTdjLU24JlIg/ZWPZyR1CDD+CBBU1wgIMLEWkbDAjpqQIDBJ9idYgpFpASyT6HNijENimLU4ZB0R36NmhVzaIurTs1w4S26C82vQGRKozjDzzEC/yOX/EnXv+3VpzU6HjZpdlMWe7rg/uja1f3Ug7NEWz/oe70HMEWzCVeBXn3k0jnFFbKt/YOLtbmV8fjF/gZf5H/T3iK3+gEbuvS+rLCVz/e4cckL3Rj1CD2bztui3WtwEoFXCnmF19lreqFMXgOE9SPWViEN7AMFbCkZ9KS9FZ6J5/IZ/Jv+TxNlaWMGYG/hnx1Azx6x4g=
v1 = ( 11u1 + 21u2 + 31u3)
AAADPXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvsKIs3UrBViCnOTM9N1OCqjgEbGV2UnhRbbaBnYGRhamKkY6BnZGpgaWgJZJgaGFqamdTGFGRkxlcbGtXWKpTGG3Jpk6jRCKLRiGSNxhCNxlya8QLKQNVgoIDJMIQylBmgICBfYDtDDEMKQz5DMkMpQy5DKkMeQwmQncOQyFAMhNEMhgwGDAVAsViGaqBYEZCVCZZPZahl4ALqLQWqSgWqSASKZgPJdCAvGiqaB+SDzCwG604G2pIDxEVAnQoMqgZXDVYafDY4YbDa4KXBH5xmVYPNALmlEkgnQfSmFsTzd0kEfyeoKxdIlzBkIHThdXMJQxqDBditmUC3F4BFQL5Ihugvq5r+OdgqSLVazWCRwWug+xca3DQ4DPRBXtmX5KWBqUGz8bgnCegWYIgBI8gQPTowGWFGeoamegaBJsoOTtCo4mCQZlBi0ADGhzmDA4MHQwBDKEMyoxyjC6Mvox/TfqanTG+Y3kGUMjFC9QgzoACm7wBGcseM
v2 = ( 12u1 + 22u2 + 32u3)

View Slide

Loss = ⾒
x =

x1
x2
x1
x2
⾒
u1
u2
u3
✓kj
u1 = (✓11x1 + ✓21x2)
u2 = (✓12x1 + ✓22x2)
u3 = (✓13x1 + ✓23x2)
v1
v2
ji
v1 = ( 11u1 + 21u2 + 31u3)
v2 = ( 12u1 + 22u2 + 32u3)
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVuTiIxpiQTQmljzkkQAhu+sIG/aV3YEECT9gaWOBjSYWxg/wA2z8AQs+wVhiYmPhZdnEKBHvZnbOnLnnzpm5kqkqNkfseYSJyanpGe+sb27evxAILi7lbKNhySwrG6phFSTRZqqisyxXuMoKpsVETVJZXqofDvbzTWbZiqGf8JbJyppY1ZUzRRY5UYVSTeTtVidUCYYxik6ERkHMBWFwI2kEH6EEp2CADA3QgIEOnLAKItj0FSEGCCZxZWgTZxFSnH0GHfCRtkFZjDJEYuv0r9Kq6LI6rQc1bUct0ykqDYuUIYjgC95jH5/xAV/x889abafGwEuLZmmoZWYlcLGS+fhXpdHMofatGuuZwxnsOV4V8m46zOAW8lDfPL/qZ+LpSHsdb/GN/N9gD5/oBnrzXb5LsXR3jB+JvNCLUYNiv9sxCnKb0dhOFFPb4cSB2yovrMIabFA/diEBx5CErNOHS+jCteAXtoS4sD9MFTyuZhl+hHD0BaGDkus=
ˆ
y AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVuTiIxpiQTQmljzkkQAhu+sIG/aV3YEECT9gaWOBjSYWxg/wA2z8AQs+wVhiYmPhZdnEKBHvZnbOnLnnzpm5kqkqNkfseYSJyanpGe+sb27evxAILi7lbKNhySwrG6phFSTRZqqisyxXuMoKpsVETVJZXqofDvbzTWbZiqGf8JbJyppY1ZUzRRY5UYVSTeTtVidUCYYxik6ERkHMBWFwI2kEH6EEp2CADA3QgIEOnLAKItj0FSEGCCZxZWgTZxFSnH0GHfCRtkFZjDJEYuv0r9Kq6LI6rQc1bUct0ykqDYuUIYjgC95jH5/xAV/x889abafGwEuLZmmoZWYlcLGS+fhXpdHMofatGuuZwxnsOV4V8m46zOAW8lDfPL/qZ+LpSHsdb/GN/N9gD5/oBnrzXb5LsXR3jB+JvNCLUYNiv9sxCnKb0dhOFFPb4cSB2yovrMIabFA/diEBx5CErNOHS+jCteAXtoS4sD9MFTyuZhl+hHD0BaGDkus=
ˆ
y
AAACi3ichVG9TgJBEB7OP0QR1MbEhkgwVpc5BAViQTQmlqDykyghd+eqG467y91BgoQXsLSxwEYTC+MD+AA2voAFj2AsMbGxcDgwxgKcy97Ofjvf7Lf7KabGbQex4xHGxicmp7zTvplZ/1wgOL+Qt42apbKcamiGVVRkm2lcZzmHOxormhaTq4rGCkplp7dfqDPL5oZ+6DRMVqrKZzo/5arsEFQ8Nm1ebvJWORhGEaOJeCwaQjEax6SUpCSOUnIjFpJEdCMMg8gYwSc4hhMwQIUaVIGBDg7lGshg03cEEiCYhJWgSZhFGXf3GbTAR9waVTGqkAmt0P+MVkcDVKd1r6ftslU6RaNhETMEEXzFB+ziCz7iG34N7dV0e/S0NGhW+lxmlgOXSwef/7KqNDtw/ssaqdmBU0i4WjlpN12kdwu1z69fXHcPUvuR5ire4Tvpv8UOPtMN9PqHep9l++0RehTSQi9GBv24EBqe5KOiFBcxGwuntwdWeWEZVmCN/NiENOxBBnKuD1fQhhvBL6wLKWGrXyp4BpxF+BPC7je5G5Nx
i
AAAC/niclVHPS9xAFH5J/Rl/7FovQi+hi1KoLC9hV3cLwmIvPfqjq8LuEpI4roPZJCSzC9sQ8Ow/4MGTgoVSSo9699J/wIMnz8WjhV566NtsoKhU8YXJfPPN+958M8/yHR4KxCtJfjEwODQ8MqqMjU9MZrJTLzdCrx3YrGp7jhdsWWbIHO6yquDCYVt+wMyW5bBNa+99b3+zw4KQe+5H0fVZo2U2Xb7DbVMQZWS79V1TRN1YXVKVqJ7UqwVNqxFhHvVSsaDPY14vYlkrEyiiVl4oxHU/5EakxbHaMTTl7TN1eqLTFSObo9wk1IdAS0EO0ljxsmdQh23wwIY2tICBC4KwAyaE9NVAAwSfuAZExAWEeLLPIAaFtG3KYpRhErtH/yatainr0rpXM0zUNp3i0AhIqcIsXuIXvMUf+BV/4p//1oqSGj0vXZqtvpb5RuZgZv33k6oWzQJ2/6ke9SxgB0qJV07e/YTp3cLu6zufDm/X363NRnN4gjfk/xiv8IJu4HZ+2aerbO3oET8WeaEXowZp99vxEGzoea2Yx9VCrrKctmoEXsFreEP9WIQKfIAVqFL9a2lQmpQy8r78Wf4mf++nylKqmYY7IZ//BaU3tNc=
ˆ
y = 1v1 + 2v2

View Slide

Loss = ⾒
x =

x1
x2
x1
x2
⾒
u1
u2
u3
✓kj
u1 = (✓11x1 + ✓21x2)
u2 = (✓12x1 + ✓22x2)
u3 = (✓13x1 + ✓23x2)
v1
v2
ji
v1 = ( 11u1 + 21u2 + 31u3)
v2 = ( 12u1 + 22u2 + 32u3)
AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVuTiIxpiQTQmljzkkQAhu+sIG/aV3YEECT9gaWOBjSYWxg/wA2z8AQs+wVhiYmPhZdnEKBHvZnbOnLnnzpm5kqkqNkfseYSJyanpGe+sb27evxAILi7lbKNhySwrG6phFSTRZqqisyxXuMoKpsVETVJZXqofDvbzTWbZiqGf8JbJyppY1ZUzRRY5UYVSTeTtVidUCYYxik6ERkHMBWFwI2kEH6EEp2CADA3QgIEOnLAKItj0FSEGCCZxZWgTZxFSnH0GHfCRtkFZjDJEYuv0r9Kq6LI6rQc1bUct0ykqDYuUIYjgC95jH5/xAV/x889abafGwEuLZmmoZWYlcLGS+fhXpdHMofatGuuZwxnsOV4V8m46zOAW8lDfPL/qZ+LpSHsdb/GN/N9gD5/oBnrzXb5LsXR3jB+JvNCLUYNiv9sxCnKb0dhOFFPb4cSB2yovrMIabFA/diEBx5CErNOHS+jCteAXtoS4sD9MFTyuZhl+hHD0BaGDkus=
ˆ
y AAACi3ichVG7TgJBFL2sL0QR1MbEhkgwVuTiIxpiQTQmljzkkQAhu+sIG/aV3YEECT9gaWOBjSYWxg/wA2z8AQs+wVhiYmPhZdnEKBHvZnbOnLnnzpm5kqkqNkfseYSJyanpGe+sb27evxAILi7lbKNhySwrG6phFSTRZqqisyxXuMoKpsVETVJZXqofDvbzTWbZiqGf8JbJyppY1ZUzRRY5UYVSTeTtVidUCYYxik6ERkHMBWFwI2kEH6EEp2CADA3QgIEOnLAKItj0FSEGCCZxZWgTZxFSnH0GHfCRtkFZjDJEYuv0r9Kq6LI6rQc1bUct0ykqDYuUIYjgC95jH5/xAV/x889abafGwEuLZmmoZWYlcLGS+fhXpdHMofatGuuZwxnsOV4V8m46zOAW8lDfPL/qZ+LpSHsdb/GN/N9gD5/oBnrzXb5LsXR3jB+JvNCLUYNiv9sxCnKb0dhOFFPb4cSB2yovrMIabFA/diEBx5CErNOHS+jCteAXtoS4sD9MFTyuZhl+hHD0BaGDkus=
ˆ
y
AAACi3ichVG9TgJBEB7OP0QR1MbEhkgwVpc5BAViQTQmlqDykyghd+eqG467y91BgoQXsLSxwEYTC+MD+AA2voAFj2AsMbGxcDgwxgKcy97Ofjvf7Lf7KabGbQex4xHGxicmp7zTvplZ/1wgOL+Qt42apbKcamiGVVRkm2lcZzmHOxormhaTq4rGCkplp7dfqDPL5oZ+6DRMVqrKZzo/5arsEFQ8Nm1ebvJWORhGEaOJeCwaQjEax6SUpCSOUnIjFpJEdCMMg8gYwSc4hhMwQIUaVIGBDg7lGshg03cEEiCYhJWgSZhFGXf3GbTAR9waVTGqkAmt0P+MVkcDVKd1r6ftslU6RaNhETMEEXzFB+ziCz7iG34N7dV0e/S0NGhW+lxmlgOXSwef/7KqNDtw/ssaqdmBU0i4WjlpN12kdwu1z69fXHcPUvuR5ire4Tvpv8UOPtMN9PqHep9l++0RehTSQi9GBv24EBqe5KOiFBcxGwuntwdWeWEZVmCN/NiENOxBBnKuD1fQhhvBL6wLKWGrXyp4BpxF+BPC7je5G5Nx
i
ˆ
y = 1v1 + 2v2
AAACmXichVHNLgNRFP6Mv6q/+llIbCYawkJzKoRIJMVGxAJVJH6amXExMZ2ZzNw2aZu+gBewsEFigQfwADZewMIjiCWJjYXT6SSC4Ezu3O9+93znfvce3bVMXxI91in1DY1NzZGWaGtbe0dnrKt73XfyniEyhmM53qau+cIybZGRprTEpusJLadbYkM/mq/ubxSE55uOvSaLrtjJaQe2uW8ammQqG+tdUmfU4aI6qm4farJcrIzsjkWzsTglKAj1J0iGII4wlp3YLbaxBwcG8shBwIZkbEGDz98WkiC4zO2gzJzHyAz2BSqIsjbPWYIzNGaP+H/Aq62QtXldrekHaoNPsXh4rFQxSA90RS90Tzf0RO+/1ioHNapeijzrNa1ws53Hfem3f1U5niUOP1V/epbYx1Tg1WTvbsBUb2HU9IXSyUt6enWwPEQX9Mz+z+mR7vgGduHVuFwRq6d/+NHZC78YNyj5vR0/wfpYIjmRoJXxeGoubFUE/RjAMPdjEiksYBkZrl/CGa5wrfQrs8qCslhLVepCTQ++hJL+AFollg4=
L = (y ˆ
y)2
AAACoHichVE9TxtBEH1cEj5MwAYaJDcnW0QgRac5y9gklUUaOmzAgILR6e5Y4MR96W5t4VgUtPwBCqpEilCUJlT8gDT5Ayn4CRElSGlSZO7sKEoBmdXuzr6ZN/t2xwpdJ5ZEN0PKk6fPhkdGxzLjzycms7mp6c04aEe2aNqBG0TblhkL1/FFUzrSFdthJEzPcsWWdfQmiW91RBQ7gb8hu6HY9cwD39l3bFMyZOTy88eG/lI9NkoLasszw1gGauvQlL3uScbIFUmrlnW9WlFJo8VSRU+cMlUr5VeqrlFqRQysHuSu0cIeAthow4OAD8m+CxMxjx3oIISM7aLHWMSek8YFTpBhbpuzBGeYjB7xesCnnQHq8zmpGadsm29xeUbMVDFH3+kT3dE3+kw/6NeDtXppjURLl3erzxWhkT2bXf/5X5bHu8ThX9ajmiX2sZRqdVh7mCLJK+w+v/Pu/G799dpc7wV9oFvW/55u6Cu/wO/c2x8bYu3iET0Wa+Ef4wb96YL6sLNZ0vRFjRrlYm150KpR5FHAPPejihpWUEeT65/iEl9wpRSUFWVVafRTlaEBZwb/mPL2Nz6Kme4=
(x1, x2) 7! ˆ
y
⾒
AAACo3ichVE9TxtBEH0chA+HD0MapEjRCcsRkdBpzjI2UCFokEIBBgMStk53xwIn7kt3awvHoqPKH0hBRSSKhJJI+QFp8gco+AlRSiLRUDB3dhRRQGa1u7Nv5s2+3bFC14kl0U2P0tv3on9gcCjzcnhkdCw7PrEVB43IFlU7cINoxzJj4Tq+qEpHumInjITpWa7Yto6Wk/h2U0SxE/ibshWKumce+M6+Y5uSISP7ZvrY0GfUY6Mwo7beqTXPDGMZqLVDU7ZXTzJGNkdauajr5ZJKGs0WSnriFKlcKs6rukap5dC1tSD7HTXsIYCNBjwI+JDsuzAR89iFDkLIWB1txiL2nDQucIIMcxucJTjDZPSI1wM+7XZRn89JzThl23yLyzNipoo8XdMXuqWfdEm/6P7JWu20RqKlxbvV4YrQGPs4uXH3X5bHu8ThP9azmiX2MZdqdVh7mCLJK+wOv/nh0+3GQiXffkuf6TfrP6cb+sEv8Jt/7It1UTl7Ro/FWvjHuEF/u6A+7WwVNH1Wo/VibnGp26pBvMYUprkfZSxiBWuocv1TfMUVvil55b1SUTY7qUpPl/MKj0ypPwAIrpqk
(x1, x2, y) 7! ˆ
L

View Slide

⾒ (Chain Rule)
)
g f g
×
0(x) = (f(g(x)))0 = f0(g(x)) · g0(x)
(x) = f(g(x))
AAAC+niclVHNSutAFD7J9fpTr1qvG8FNuEVtQcpJqa0KgujGpX9Voa0liWMdTJOQTIsa+wK+gAtXCi7EreADuLkvcBdyn0BcKrhx4UkaEBUVT5jMN9+c78w3c3TH5J5AvJHkH20/2zs6u2Ldv3p6++L9v1c9u+4arGDYpu2u65rHTG6xguDCZOuOy7SabrI1fWcu2F9rMNfjtrUi9hxWrmlVi29xQxNEVeL1ZMnjVtIvhaWKblUv+5jOZ1U1nxvDNI5ncmoAspjPZSebyu5GpplKjSrTCim8b+pKxqYtkpndVCWeoJwwlPdAjUAColiw41dQgk2wwYA61ICBBYKwCRp49BVBBQSHuDL4xLmEeLjPoAkx0tYpi1GGRuwO/au0KkasReugpheqDTrFpOGSUoFh/IfneI9/8QJv8enDWn5YI/CyR7Pe0jKn0nc4uPz4papGs4DtF9WnngVswUTolZN3J2SCWxgtfWP/6H55amnYH8FTvCP/J3iD13QDq/FgnC2ypeNP/OjkhV6MGqS+bcd7sJpJq+NpXMwmZmajVnXCEPyBJPUjDzMwDwtQoPr/JUmKSd3ygXwqn8sXrVRZijQD8Crky2dHPbJe
(sin(x2))0 = cos(x2) · (2x)

View Slide

⾒ (Chain Rule)
)
g f g
×
0(x) = (f(g(x)))0 = f0(g(x)) · g0(x)
(x) = f(g(x))
AAAC+niclVHNSutAFD7J9fpTr1qvG8FNuEVtQcpJqa0KgujGpX9Voa0liWMdTJOQTIsa+wK+gAtXCi7EreADuLkvcBdyn0BcKrhx4UkaEBUVT5jMN9+c78w3c3TH5J5AvJHkH20/2zs6u2Ldv3p6++L9v1c9u+4arGDYpu2u65rHTG6xguDCZOuOy7SabrI1fWcu2F9rMNfjtrUi9hxWrmlVi29xQxNEVeL1ZMnjVtIvhaWKblUv+5jOZ1U1nxvDNI5ncmoAspjPZSebyu5GpplKjSrTCim8b+pKxqYtkpndVCWeoJwwlPdAjUAColiw41dQgk2wwYA61ICBBYKwCRp49BVBBQSHuDL4xLmEeLjPoAkx0tYpi1GGRuwO/au0KkasReugpheqDTrFpOGSUoFh/IfneI9/8QJv8enDWn5YI/CyR7Pe0jKn0nc4uPz4papGs4DtF9WnngVswUTolZN3J2SCWxgtfWP/6H55amnYH8FTvCP/J3iD13QDq/FgnC2ypeNP/OjkhV6MGqS+bcd7sJpJq+NpXMwmZmajVnXCEPyBJPUjDzMwDwtQoPr/JUmKSd3ygXwqn8sXrVRZijQD8Crky2dHPbJe
(sin(x2))0 = cos(x2) · (2x)
(Chain Rule) Leibniz
AAACbXicbZFbS8MwFMfTepudl6n4pEhwiD6Ndl5fhKEvPiq4C2xjpGm6haVpSVJxlH5Q8clP4Fcw7ars4oGEf37nnJzwjxsxKpVtfxjmyura+kZp0ypvbe/sVvb2WzKMBSZNHLJQdFwkCaOcNBVVjHQiQVDgMtJ2x49Zvv1GhKQhf1WTiPQDNOTUpxgpjQaVuOcLhBNvkibeewrv4cw5Hjgp7GEvVL80I9leX+L1nF/+cb3JxRqe1+gp1qBStWt2HnBZOIWogiKeB5UvfSOOA8IVZkjKrmNHqp8goShmJLV6sSQRwmM0JF0tOQqI7Ce5PSk808SDfij04grmdLYjQYGUk8DVlQFSI7mYy+B/uW6s/Lt+QnkUK8LxdJAfM6hCmHkNPSoIVmyiBcKC6rdCPELaDaV/ZG6KG6TaE2fRgWXRqtecm9r1y1W18VC4UwJH4BRcAAfcggZ4As+gCTD4NAzDMsrGt3loHpsn01LTKHoOwFyY5z9dVL0+
dy
dx
=
dy
du1
·
du1
du2
·
du2
du3
· · · · ·
dun
dux
AAACKXicbVDLSsQwFE3H1zi+qi7dBAfB1dCKr40w6MblCM4DpmVI03QmTPogScVS+h1+hN/gVtfu1J34I6adCvPwQODknHO5yXEiRoU0jE+tsrS8srpWXa9tbG5t7+i7ex0RxhyTNg5ZyHsOEoTRgLQllYz0Ik6Q7zDSdcY3ud99IFzQMLiXSURsHw0D6lGMpJIGuml5HOHUTbLUfczgFZy6xxm0sBvKPy0uMgO9bjSMAnCRmCWpgxKtgf5tuSGOfRJIzJAQfdOIpJ0iLilmJKtZsSARwmM0JH1FA+QTYafF1zJ4pBQXeiFXJ5CwUKcnUuQLkfiOSvpIjsS8l4v/ef1Yepd2SoMoliTAk0VezKAMYd4TdCknWLJEEYQ5VW+FeIRUD1K1ObPF8fNOzPkGFknnpGGeN87uTuvN67KdKjgAh+AYmOACNMEtaIE2wOAJvIBX8KY9a+/ah/Y1iVa0cmYfzED7+QX7+KjU
dy
dx
=
dy
du
·
du
dx
AAACE3icbZC7TsMwFIadcivlFmDsYlEhtUuVIG4LUgULY5HoRWqjynGd1KrjRLaDiKIOPATPwAozG2LlARh5E5w2A235JUuf/nOOzvHvRoxKZVnfRmFldW19o7hZ2tre2d0z9w/aMowFJi0cslB0XSQJo5y0FFWMdCNBUOAy0nHHN1m980CEpCG/V0lEnAD5nHoUI6WtgVlO4BX0qn71sVbLCPYxFRj6UBsDs2LVrangMtg5VECu5sD86Q9DHAeEK8yQlD3bipSTIqEoZmRS6seSRAiPkU96GjkKiHTS6Scm8Fg7Q+iFQj+u4NT9O5GiQMokcHVngNRILtYy879aL1bepZNSHsWKcDxb5MUMqhBmicAhFQQrlmhAWFB9K8QjJBBWOre5LW4w0ZnYiwksQ/ukbp/Xz+5OK43rPJ0iKIMjUAU2uAANcAuaoAUweAIv4BW8Gc/Gu/FhfM5aC0Y+cwjmZHz9Auvjm6Y=
y = f(g(x)) = f g(x)
AAACBHicbVC7SgNBFJ31GeMramkzGIQEJOyKryYQtLGMYB6QrGF2MpsMmZld5iEuIa3fYKu1ndj6H5b+iZNkC5N44MLhnHs5lxPEjCrtut/O0vLK6tp6ZiO7ubW9s5vb26+ryEhMajhikWwGSBFGBalpqhlpxpIgHjDSCAY3Y7/xSKSikbjXSUx8jnqChhQjbaWHpBwWTPEEmnKv8FTs5PJuyZ0ALhIvJXmQotrJ/bS7ETacCI0ZUqrlubH2h0hqihkZZdtGkRjhAeqRlqUCcaL84eTrETy2SheGkbQjNJyofy+GiCuV8MBucqT7at4bi/95LaPDK39IRWw0EXgaFBoGdQTHFcAulQRrlliCsKT2V4j7SCKsbVEzKQEf2U68+QYWSf205F2Uzu/O8pXrtJ0MOARHoAA8cAkq4BZUQQ1gIMELeAVvzrPz7nw4n9PVJSe9OQAzcL5+ATPCl7Y=
y = f(u), u = g(x) AAACBHicbVDLSgNBEOyNrxhfUY9eBoOQgIRd8XURgl48RjAPSNYwO5lNhszOLjOzYlhy9Ru86tmbePU/PPonTpI9mMSCboqqbropL+JMadv+tjJLyyura9n13Mbm1vZOfnevrsJYElojIQ9l08OKciZoTTPNaTOSFAcepw1vcDP2G49UKhaKez2MqBvgnmA+I1gb6WF45Rd7xafScd+0UidfsMv2BGiROCkpQIpqJ//T7oYkDqjQhGOlWo4daTfBUjPC6SjXjhWNMBngHm0ZKnBAlZtMvh6hI6N0kR9KU0Kjifp3I8GBUsPAM5MB1n01743F/7xWrP1LN2EiijUVZHrIjznSIRpHgLpMUqL50BBMJDO/ItLHEhNtgpq54gUjk4kzn8AiqZ+UnfPy2d1poXKdppOFAziEIjhwARW4hSrUgICEF3iFN+vZerc+rM/paMZKd/ZhBtbXLxG3l6A=
y = f(g(x), h(x))
in-out AAACbXicfVFbS8MwGE3rbc5bVXxSJDhEQRiteHsRhr74OMFdYB0jTdM1LL2QpGIp/aHik7/Av2DaFXSb+EHg5JzvcMKJEzMqpGm+a/rS8srqWm29vrG5tb1j7O51RZRwTDo4YhHvO0gQRkPSkVQy0o85QYHDSM+ZPBZ675VwQaPwRaYxGQZoHFKPYiQVNTIS2+MIZ26aZ+lbDu/h9G7HiEuKGPTyHzzOoY3dSFY77jjPXOW5+M/jz3n80jMyGmbTLAcuAqsCDVBNe2R82m6Ek4CEEjMkxMAyYznMihDMSF63E0FihCdoTAYKhiggYpiV9eTwVDEu9CKuTihhyf52ZCgQIg0ctRkg6Yt5rSD/0gaJ9O6GGQ3jRJIQT4O8hEEZwaJr6FJOsGSpAghzqt4KsY9UD1L9yEyKExSdWPMNLILuZdO6aV4/XzVaD1U7NXAITsA5sMAtaIEn0AYdgMGHpml1bUP70g/0I/14uqprlWcfzIx+9g0J7748
dy
yx
=
@f
@g
·
dg
dx
+
@f
@h
·
dh
dx
in ⾒

View Slide

u1 = (✓11x1 + ✓21x2)
u2 = (✓12x1 + ✓22x2)
u3 = (✓13x1 + ✓23x2)
v1 = ( 11u1 + 21u2 + 31u3)
v2 = ( 12u1 + 22u2 + 32u3)
ˆ
y = 1v1 + 2v2
AAACmXichVHNLgNRFP6Mv6q/+llIbCYawkJzKoRIJMVGxAJVJH6amXExMZ2ZzNw2aZu+gBewsEFigQfwADZewMIjiCWJjYXT6SSC4Ezu3O9+93znfvce3bVMXxI91in1DY1NzZGWaGtbe0dnrKt73XfyniEyhmM53qau+cIybZGRprTEpusJLadbYkM/mq/ubxSE55uOvSaLrtjJaQe2uW8ammQqG+tdUmfU4aI6qm4farJcrIzsjkWzsTglKAj1J0iGII4wlp3YLbaxBwcG8shBwIZkbEGDz98WkiC4zO2gzJzHyAz2BSqIsjbPWYIzNGaP+H/Aq62QtXldrekHaoNPsXh4rFQxSA90RS90Tzf0RO+/1ioHNapeijzrNa1ws53Hfem3f1U5niUOP1V/epbYx1Tg1WTvbsBUb2HU9IXSyUt6enWwPEQX9Mz+z+mR7vgGduHVuFwRq6d/+NHZC78YNyj5vR0/wfpYIjmRoJXxeGoubFUE/RjAMPdjEiksYBkZrl/CGa5wrfQrs8qCslhLVepCTQ++hJL+AFollg4=
L = (y ˆ
y)2
AAAC2HichVG/b9NAFP5qfpXwoyksSCwRUREDip5NUhqmChYGhrYhbUUcRWf3kp7q2Nb5EqlYltgQjCwMTCAxIP4AxMzCP8DQiZWKsUgsDLw4liqoKO90vu++e9+77/y8OFCJIdqbsU6cPHX6zOzZ0rnzFy7OlecvrSfRSPuy7UdBpDc9kchAhbJtlAnkZqylGHqB3PB27k3ON8ZSJyoKH5rdWHaHYhCqvvKFYapXbrl9LfzUjYU2SgSVB9khLqVufkFHD7xuSjVylhp15ybVnAY17SaDBtnNxXpWcc22NKKXOreyrJT1ylVOzqNyFNgFqKKIlaj8AS62EMHHCENIhDCMAwgkPDqwQYiZ6yJlTjNS+blEhhJrR5wlOUMwu8PfAe86BRvyflIzydU+3xLw1KysYIG+0Ds6oM/0nvbp1z9rpXmNiZddXr2pVsa9uedXWj//qxryarB9qDrWs0EfS7lXxd7jnJm8wp/qx49fHrTurC2k1+kNfWf/r2mPPvELwvEP/+2qXHt1jB+PvfAf4wbZf7fjKFh3anajRqv16vLdolWzuIpruMH9uI1l3McK2lz/I77iG/atR9YT66n1bJpqzRSay/gjrBe/AUIArcA=
@L
@✓23
AAACm3ichVHLSsNAFD3G97sqiCBCsSgVNNwURRGEYjciLtRaFVRCEkcNpklI0qJWf8AfcOFKwYXo3g9w4w+48BPEZQU3LrxNA6Ki3jCZM2fuuXNmru5aph8QPdVItXX1DY1NzS2tbe0dnbGu7lXfKXiGyBmO5XjruuYLy7RFLjADS6y7ntDyuiXW9P1MZX+tKDzfdOyV4NAVW3lt1zZ3TEMLmFJjvckDVRmNH6ipkZnkmCLTaDwl04gaS5BMYcR/AiUCCUSx6MTusIltODBQQB4CNgLGFjT4/G1AAcFlbgsl5jxGZrgvcIIW1hY4S3CGxuw+/3d5tRGxNq8rNf1QbfApFg+PlXEM0SNdU5ke6Iae6f3XWqWwRsXLIc96VStctfO0L/v2ryrPc4C9T9WfngPsYCr0arJ3N2QqtzCq+uLRWTk7vTxUGqZLemH/F/RE93wDu/hqXC2J5fM//OjshV+MG6R8b8dPsJqSlQmZlsYT6dmoVU3oxyCS3I9JpDGHReS4/jEucYNbaUDKSPPSQjVVqok0PfgSUu4D8m+VUg==
(x1, x2) = ( 1.0, 2.0)
AAACiHichVHLSsNAFD2Nr1pfVTeCG7EorsqttFQFodSNy7ZaLVSRJE51ME1CkhZq8QfcuFRxpeBC/AA/wI0/4MJPEJcKblx4mwZExXrDZM6cuefOmbmabUjXI3oKKV3dPb194f7IwODQ8Eh0dGzDtWqOLoq6ZVhOSVNdYUhTFD3pGaJkO0KtaobY1A5WWvubdeG40jLXvYYttqvqnikrUlc9poqN5WQ8tRONUZz8mPoNEgGIIYicFb3DFnZhQUcNVQiY8BgbUOHyV0YCBJu5bTSZcxhJf1/gCBHW1jhLcIbK7AH/93hVDliT162arq/W+RSDh8PKKczQI93QKz3QLT3Tx5+1mn6NlpcGz1pbK+ydkeOJtfd/VVWePex/qTp69lDBgu9VsnfbZ1q30Nv6+uHp69pSYaY5S1f0wv4v6Ynu+QZm/U2/zovCRQc/GnvhF+MGJX624zfYmI8nUnHKJ2OZbNCqMCYxjTnuRxoZrCKHIteXOMEZzpWIQkpaWWynKqFAM45voWQ/ATKxkPA=
y = 4.5
AAAD2nicpVE7b9RAEB7HPIJ55AINUpoTp0QUyWrW3CV3SEgRNJR5XRLpfDrZzuZuFb9k750I1jV0QItEkQokCsQPoKCk4Q9QpKSElEGioWDOtkAhIQixlr3ffLPffLMeJ/JkohD3tTH9zNlz58cvGBcvXb4yUZq8up6E/dgVTTf0wnjTsRPhyUA0lVSe2IxiYfuOJzacnXuj/MZAxIkMgzW1G4m2b3cDuS1dWxHVmdTWUiur0oq7TjtFhma9VjVnkZk1bPAGgRryxnx1aFiOn1qqJ5Q9HBp3LEd0ZZA6vq1i+WBoIOPlmfJcvnFmWhZRDcJUMk8sGJYItn4qZo1/c4568iRfnvvWWLVMliYzKZonYwrqrD5K3WL1/3ROTnQuXOY4Z3jUwOiUKlQ1W+XjgBegAsVaCktvwYItCMGFPvggIABF2AMbEnpawAEhIq4NKXExIZnlBQzBIG2fTgk6YRO7Q98uRa2CDSge1UwytUsuHr0xKcswjR/xNR7iB3yDn/H7H2ulWY1RL7u0O7lWRJ2JJ9dXv/1V5dOuoPdLdWrPCrahnvUqqfcoY0a3cHP94OHzw9XbK9PpDL7EA+r/Be7je7pBMPjqvloWK3un9ONQL/THaED893EcB+sm4zWGy9XK4t1iVOMwBTfgJs1jARbhPixBE1ztnfZJ+6Id6Jb+SH+sP82PjmmF5hocWfqzH51R9gc=
✓ =

0.1 0.1 1.2
0.9 0.0 0.7
, =
2
4
1.1 5.4
2.2 6.9
8.8 3.8
3
5 , =

6.9
11.0
⾒
AAACpXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsoxBRnpucmalRpKtgqxKQVJSZXG9ZWG2rHpFYUaOhWadbGCygb6BmAgQImwxDKUGaAgoB8ge0MMQwpDPkMyQylDLkMqQx5DCVAdg5DIkMxEEYzGDIYMBQAxWIZqoFiRUBWJlg+laGWgQuotxSoKhWoIhEomg0k04G8aKhoHpAPMrMYrDsZaEsOEBcBdSowqBpcNVhp8NnghMFqg5cGf3CaVQ02A+SWSiCdBNGbWhDP3yUR/J2grlwgXcKQgdCF180lDGkMFmC3ZgLdXgAWAfkiGaK/rGr652CrINVqNYNFBq+B7l9ocNPgMNAHeWVfkpcGpgbNxuOeJKBbgCEGjCBD9OjAZIQZ6Rma6hkEmig7OEGjioNBmkGJQQMYH+YMDgweDAEMoUDz2xjWMGxl2MakzuTLFMIUBlHKxAjVI8yAApjiAYn8m60=
(z) =
1
1 + exp( z)
Sigmoid

View Slide

AAAC2HichVG/b9NAFP5qfpXwoyksSCwRUREDip5NUhqmChYGhrYhbUUcRWf3kp7q2Nb5EqlYltgQjCwMTCAxIP4AxMzCP8DQiZWKsUgsDLw4liqoKO90vu++e9+77/y8OFCJIdqbsU6cPHX6zOzZ0rnzFy7OlecvrSfRSPuy7UdBpDc9kchAhbJtlAnkZqylGHqB3PB27k3ON8ZSJyoKH5rdWHaHYhCqvvKFYapXbrl9LfzUjYU2SgSVB9khLqVufkFHD7xuSjVylhp15ybVnAY17SaDBtnNxXpWcc22NKKXOreyrJT1ylVOzqNyFNgFqKKIlaj8AS62EMHHCENIhDCMAwgkPDqwQYiZ6yJlTjNS+blEhhJrR5wlOUMwu8PfAe86BRvyflIzydU+3xLw1KysYIG+0Ds6oM/0nvbp1z9rpXmNiZddXr2pVsa9uedXWj//qxryarB9qDrWs0EfS7lXxd7jnJm8wp/qx49fHrTurC2k1+kNfWf/r2mPPvELwvEP/+2qXHt1jB+PvfAf4wbZf7fjKFh3anajRqv16vLdolWzuIpruMH9uI1l3McK2lz/I77iG/atR9YT66n1bJpqzRSay/gjrBe/AUIArcA=
@L
@✓23
AAACi3ichVHLSsNAFL2Nr1qtrboR3BRLxY3lxkfVolAUwWUf9gG1SBKnNTRNQpIWavEHXLpxUTcKLsQP8APc+AMu+gnisoIbF96kAdFivWEyZ87cc+fMXFFXZNNC7Hi4oeGR0THvuG9i0j8VCE7P5EytbkgsK2mKZhREwWSKrLKsJVsKK+gGE2qiwvJidc/ezzeYYcqaemg1dVaqCRVVLsuSYBFV2FnmozF+K3YcDGMUnQj1A94FYXAjqQUf4QhOQAMJ6lADBipYhBUQwKSvCDwg6MSVoEWcQUh29hmcg4+0dcpilCEQW6V/hVZFl1Vpbdc0HbVEpyg0DFKGIIIveI9dfMYHfMXPP2u1nBq2lybNYk/L9OPAxVzm419VjWYLTr9VAz1bUIZNx6tM3nWHsW8h9fSNs6tuJp6OtBbxFt/I/w128IluoDbepbsUS7cH+BHJC70YNYj/3Y5+kFuJ8utRTK2FE7tuq7wwDwuwRP3YgAQcQBKyTh8uoQ3XnJ9b5eLcdi+V87iaWfgR3P4XVBCRYA==
= 1.6196

View Slide

⾒
( Toy Example)

View Slide

( Toy Example)
⾒
⾒

View Slide

forward:
5 0.2
0.3
0.4
1
-2

View Slide

forward:
5 0.2
0.3
0.4
1
-2
1.4

View Slide

forward:
5 0.2
0.3
0.4
1
-2
0.17
1.4

View Slide

forward:
5 0.2
0.3
0.4
1
-2
0.17
0.05
1.4

View Slide

forward:
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
1.4

View Slide

forward:
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
1.4

View Slide

forward:
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1
-0.99

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1
-0.99
0.3 0.17

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1
-0.99
0.3 0.17 5

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1
-0.99
0.3 0.17 5
-1 1

View Slide

forward
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1
-0.99
0.3 0.17 5
-1 1
5.9

View Slide

forward
⾒
torch.add, torch.sub, torch.mul, torch.div, ..
torch.sin, torch.cos, torch.tan, torch.asin,
torch.pow, torch.log, torch.square, torch.sqrt, ..
torch.matmul, torch.dot, torch.inverse, torch.det, ..
torch.sum, torch.mean, torch.max, torch.argmin, ..
⾒
https://pytorch.org/docs/stable/torch.html#math-operations

View Slide

forward
https://pytorch.org/tutorials/beginner/nlp/pytorch_tutorial.html
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4

View Slide

=
( ) "Back Propagation"
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4 ( )
AAACr3ichVHPSxtBFP5cW6uxNaleBC9iiMQSlrfRogiC6MWjUfMDok13t6MObnaX3UlAg/+AJ28ePLXQQ/FejxZ68R/w4J8gHhV66cGXzZai0vQNs/PNN+97880+y3dkqIiue7TeFy/7XvUPJAZfvxlKpt4Ol0KvEdiiaHuOF1QsMxSOdEVRSeWIih8Is245omztLbfPy00RhNJzN9S+L7bq5o4rt6VtKqZqqXfZTbUrlFkzcjHI/wHTUwtZ0vM50qd5zkzVUmnSKYrx58CIQRpxrHqpc2ziEzzYaKAOAReKsQMTIY8qDBB85rbQYi5gJKNzgUMkWNvgLMEZJrN7/N3hXTVmXd63a4aR2uZbHJ4BK8eRoSv6Rnd0SWd0Q7//WasV1Wh72efV6miFX0seja7/+q+qzqvC7l9VV88K25iLvEr27kdM+xV2R988OLlbn1/LtCbpC92y/890TT/5BW7z3v5aEGunXfxY7IX/GDfIeNqO56CU1433OhVm0otLcav6MYYJZLkfs1jEClZR5PrH+I4L/NAMrax90D52UrWeWDOCR6HJB6rFnZE=
(✓1, ✓2, ✓3) = (0.2, 0.3, 0.4)

View Slide

backward
⾒
5.9
5.9
5.9
1 5
-1 0.17
-1 0.3 -0.99 1
29.5
-1.00
1.75
1
-0.99
0.3 0.17 5
-1 1
5.9
=
=
=
× ×
× ×
× × × ×

View Slide

⾒
forward backward ( )
5 0.2
0.3
0.4
1
-2
0.17
0.05 1.0
0.95
8.7
1.4
1
-0.99
0.3 0.17 5
-1 1
5.9
loss

View Slide

" "
⾒ ( )
(step size)

View Slide

Python
PyTorch torch.autograd
micrograd
tinygrad
JAX

View Slide

https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html

View Slide

forward
( )
AAAClHichVHNSgJRFD5Of2Y/WkEEbYbEaJMctSiiQJKgVfiTP2AmM9PVBseZYWYUTHqBXqCFqyIX0QP0AG16gRY+QrQ0aNOi4zgQJdkZ7tzvfvd85373HlFXZNNC7Li4kdGx8Qn3pGdqembW65ubz5hazZBYWtIUzciJgskUWWVpS7YUltMNJlRFhWXFSqy3n60zw5Q19dhq6KxQFcqqXJIlwSKq6PMm+D0+wgunEX6dF0/DRZ8fg2gHPwhCDvCDE3HN9wgncAYaSFCDKjBQwSKsgAAmfXkIAYJOXAGaxBmEZHufwSV4SFujLEYZArEV+pdplXdYlda9mqatlugUhYZBSh4C+IL32MVnfMBX/PyzVtOu0fPSoFnsa5le9F4tpT7+VVVptuD8WzXUswUl2La9yuRdt5neLaS+vn5x3U3tJAPNVbzFN/J/gx18ohuo9XepnWDJ1hA/InmhF6MGhX63YxBkwsHQZhATG/7ovtMqNyzDCqxRP7YgCocQh7TdsxbcQZtb5Ha5GHfQT+VcjmYBfgR39AXXx5Oq
Q = 3a3 b2
AAACHXicbZDLTsJAFIaneEO8VV26mUhI3EDagujGhOjGJSRySaCQ6XQKE6aXzExNSMMT+BA+g1tduzNujUvfxAG6EPBPJvnyn3NyzvxOxKiQhvGtZTY2t7Z3sru5vf2DwyP9+KQlwphj0sQhC3nHQYIwGpCmpJKRTsQJ8h1G2s74blZvPxIuaBg8yElEbB8NA+pRjKSyBnqhAW9gGfawG0po9cuwCKt9S3lWRWG5qqhoWgM9b5SMueA6mCnkQar6QP/puSGOfRJIzJAQXdOIpJ0gLilmZJrrxYJECI/RkHQVBsgnwk7m35nCgnJc6IVcvUDCuft3IkG+EBPfUZ0+kiOxWpuZ/9W6sfSu7YQGUSxJgBeLvJhBGcJZNtClnGDJJgoQ5lTdCvEIcYSlSnBpi+NPVSbmagLr0LJKZrV02ajka7dpOllwBs7BBTDBFaiBe1AHTYDBE3gBr+BNe9betQ/tc9Ga0dKZU7Ak7esXwBOcwA==
Q = 3 · 23 62 = 24 36 = 12
AAACH3icbVDLTgIxFO3gC8cX6tJNIzG6IjNE0Y0J0Y1LTOSRMIR0ygUaOp1J2zEhEz7Bj/Ab3OranXHL0j+xA7MQ8CRNT865j/b4EWdKO87Uyq2tb2xu5bftnd29/YPC4VFDhbGkUKchD2XLJwo4E1DXTHNoRRJI4HNo+qP71G8+g1QsFE96HEEnIAPB+owSbaRu4dzzYcBEQs0MNbEJvsVlz7N9c1dsD0Qvc7qFolNyZsCrxM1IEWWodQs/Xi+kcQBCU06UartOpDsJkZpRDhPbixVEhI7IANqGChKA6iSzD03wmVF6uB9Kc4TGM/VvR0ICpcaBbyoDoodq2UvF/7x2rPs3nYSJKNYg6HxRP+ZYhzhNB/eYBKr52BBCJTNvxXRIJKHaZLiwxQ/STNzlBFZJo1xyK6Wrx8ti9S5LJ49O0Cm6QC66RlX0gGqojih6QW/oHX1Yr9an9WV9z0tzVtZzjBZgTX8BPC2hzg==
(
a = 2
b = 6

View Slide

backward
(.grad)
aka back propagation
forward
( )
Q = 3a3 b2
Q = 3 · 23 62 = 24 36 = 12
(
a = 2
b = 6

View Slide

AAACqnicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuoxqQVJSZXxxQkFpVkJuYoBNYi2Im1CrYKlgqJcUbxAsoGegZgoIDJMIQylB0s7l8Qv/VCMiBfYDtDDEMKQz5DMkMpQy5DKkMeQwmQncOQyFAMhNEMhgwGDAVAsViGaqBYEZCVCZZPZahl4ALqLQWqSgWqSASKZgPJdCAvGiqaB+SDzCwG604G2pIDxEVAnQoMqgZXDVYafDY4YbDa4KXBH5xmVYPNALmlEkgnQfSmFsTzd0kEfyeoKxdIlzBkIHThdXMJQxqDBditmUC3F4BFQL5Ihugvq5r+OdgqSLVazWCRwWug+xca3DQ4DPRBXtmX5KWBqUGz8bgnCegWYIgBI8gQPTowGWFGeoamegaBwJhyYoAADgZpBiUGDWB8mDM4MHgwBDCEAs3vZtjIsIthN5MOUxBTJFM0RCkTI1SPMAMKYEoBAL/solw=
@Q
@a
= 9a2
AAACqHichVG/SxxBFP5cjfFH1FMLIzaLoojg8U4IiiCINinvR84zeCIz65wO7u0uu3MHetwfkPwDKVIppAgpUiap06RMY2EaW8VSwcbCd3sHEkV9y+x888373nwzTwaujgzRSZvV3vGi82VXd0/vq77+gcTg0FrkV0JH5R3f9cN1KSLlak/ljTauWg9CJcrSVQW5t9rYL1RVGGnfe2f2A7VZFjueLmlHGKa2EhPFUiicWjEQodHCtTP1Oyzr9pI9Oyc5i5IUh/0QpFpgYnnh9O/Iv7PRtJ/4gSK24cNBBWUoeDCMXQhE/G0gBULA3CZqzIWMdLyvUEcPayucpThDMLvH/x1ebbRYj9eNmlGsdvgUl0fIShuTdExf6ZL+0Dc6p5tHa9XiGg0v+zzLplYFWwMfX+eun1WVeTbYvVM96dmghIXYq2bvQcw0buE09dWDT5e5xexkbYqO6IL9H9IJ/eYbeNUr50tGZT8/4UeyF34xblDqfjsegrW5ZOpNkjLcqRU0owtjGMc092Mey3iLNPJc/wO+4yd+WTNW2ipY75upVltLM4z/wpK3OXKhwA==
@Q
@b
= 2b
backward
(.grad)
aka back propagation
forward
( )
Q = 3a3 b2
Q = 3 · 23 62 = 24 36 = 12
(
a = 2
b = 6

View Slide

torch.no_grad
requires=True
torch.tensor numpy.array

View Slide

torch.no_grad
requires=True
requires=True
backward()

View Slide

torch.no_grad
backward()
⾒
torch.no_grad()
requires=True
requires=True
backward()

View Slide

backward vs torch.autograd
v
v.grad
out in
( torch.autograd
hessian jacobian )
backward()
torch.autograd.grad

View Slide

backward() .grad
2 backward()
a.grad: , -
b.grad: - , -

View Slide

backward() .grad
2 backward()
2
backward()
a.grad = = + (- )
b.grad = - = - + (- )
a.grad: , -
b.grad: - , -

View Slide

backward() .grad
2 backward()
2
backward()
a.grad = = + (- )
b.grad = - = - + (- )
(RNN ) ( )
a.grad: , -
b.grad: - , -

View Slide

torch.tensor
numpy (autograd)
https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html

View Slide

( Jacobian )
backward
AAACpnicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsoxqQVJSZXxxQkFpVkJuYoBNYisxVsFQzjBZQN9AzAQAGTYQhlKDvY379YoWvhHpAvsJ0hhiGFIZ8hmaGUIZchlSGPoQTIzmFIZCgGwmgGQwYDhgKgWCxDNVCsCMjKBMunMtQycAH1lgJVpQJVJAJFs4FkOpAXDRXNA/JBZhaDdScDbckB4iKgTgUGVYOrBisNPhucMFht8NLgD06zqsFmgNxSCaSTIHpTC+L5uySCvxPUlQukSxgyELrwurmEIY3BAuzWTKDbC8AiIF8kQ/SXVU3/HGwVpFqtZrDI4DXQ/QsNbhocBvogr+xL8tLA1KDZeNyTBHQLMMSAEWSIHh2YjDAjPUNTPYNAYEw5MUAAB4M0gxKDBjA+zBkcGDwYAhhCgea3M6xl2MawnUmDyY8plCkcopSJEapHmAEFMCUAAP2CoEU=
@Q
@Q
= 1

View Slide

PyTorch
optimizer.step()
loss.backward()
(backward)
model.parameters() grad 0
model.parameters() grad

View Slide

PyTorch
AAACkXichVHLSsNAFD3GV62vqhvBTbEorsqNKD5AKLoR3NjWqtDWksRRg2kSkmmhFn/AH1Bw1YIL8QP8ADf+gAs/QVwquHHhbQyIFvWGyZw5c8+dM3N11zJ9SfTYoXR2dff0Rvqi/QODQ8OxkdFt36l4hsgZjuV4u7rmC8u0RU6a0hK7rie0sm6JHf14rbW/UxWebzr2lqy5oljWDm3zwDQ0ydReQR4JqZXU+EqckrOlWIKSFES8HaghSCCMTSd2iwL24cBABWUI2JCMLWjw+ctDBcFlrog6cx4jM9gXOEWUtRXOEpyhMXvM/0Ne5UPW5nWrph+oDT7F4uGxMo4peqBreqF7uqEnev+1Vj2o0fJS41n/1Aq3NHw2nn37V1XmWeLoS/WnZ4kDLAZeTfbuBkzrFsanvnpy/pJdzkzVp6lJz+y/QY90xzewq6/GVVpkLv/wo7MXfjFukPqzHe1gezapzicpPZdIrYatimACk5jhfiwghXVsIsf1PVyggaYypiwpKSXMVTpCzRi+hbLxAX4clAw=
✓1 = 0.2
AAACkXichVHLSsNAFD3G97NVN4KbYqm4KrdV8QFC0Y3gRq21gtWSxGkbTJOQTAta/AF/QMGVggvxA/wAN/6Ai36CuKzgxoW3aUC0qDdM5syZe+6cmas5puFJonqH0tnV3dPb1z8wODQ8EgqPju16dsXVRUa3Tdvd01RPmIYlMtKQpthzXKGWNVNkteO15n62KlzPsK0deeKIg7JatIyCoauSqcOcLAmp5pORlQjFZ/PhKMXJj0g7SAQgiiA27fADcjiCDR0VlCFgQTI2ocLjbx8JEBzmDlBjzmVk+PsCZxhgbYWzBGeozB7zv8ir/YC1eN2s6flqnU8xebisjCBGz3RHDXqie3qhj19r1fwaTS8nPGstrXDyofOJ9Pu/qjLPEqUv1Z+eJQpY9L0a7N3xmeYt9Ja+enrRSC9vx2rTdEOv7P+a6vTIN7Cqb/rtlti++sOPxl74xbhBiZ/taAe7yXhiPk5bc9HUatCqPkxiCjPcjwWksI5NZLi+i0tc40YZV5aUlBLkKh2BZhzfQtn4BIJilA4=
✓2 = 0.3
AAACkXichVHLSsNAFD3G97NVN4KbYqm4Krc+8AFC0Y3gxldtoa0liVMNTZOQTAta/AF/QMGVggvxA/wAN/6ACz9BXFZw48LbNCBarDdM5syZe+6cmas5puFJopcOpbOru6e3r39gcGh4JBQeHdv37Iqri5Rum7ab0VRPmIYlUtKQpsg4rlDLminSWmm9sZ+uCtczbGtPnjgiX1aPLKNo6Kpk6iAnj4VUC3OR1QjF5wvhKMXJj0grSAQgiiC27PADcjiEDR0VlCFgQTI2ocLjL4sECA5zedSYcxkZ/r7AGQZYW+EswRkqsyX+H/EqG7AWrxs1PV+t8ykmD5eVEcTome6oTk90T6/0+Wetml+j4eWEZ62pFU4hdD6x+/GvqsyzxPG3qq1niSKWfK8Ge3d8pnELvamvnl7Ud1d2YrVpuqE39n9NL/TIN7Cq7/rttti5auNHYy/8YtygxO92tIL92XhiIU7b89HkWtCqPkxiCjPcj0UksYEtpLi+i0tc40YZV5aVpBLkKh2BZhw/Qtn8AoaolBA=
✓3 = 0.4
forward
( )

View Slide

AAADBnichVJNS9xQFL2J1dqx6lShFroJDorCTLiJymhhQOzGpR8dFYyEJD7Hh/kiyQxOwyy6K/0DXbhqwUUpdCduC930D7jwF4i4tNCNC28yGURFvSF59513zn3nvRvTt3kYIZ4KYtez7p7nvS9yfS/7Bwbzr4bWQq8eWKxqebYXbJhGyGzusmrEI5tt+AEzHNNm6+be+2R9vcGCkHvuh6jpsy3HqLl8h1tGRJCe/6SZrMbd2KIaYSunmU6839IVqSJNzMhYlBQZJ4vSeFNXKqVZWdW0DkVNKKVywil3OGqlNCfP3HCmEo4iq0VpukOZqpRQnstpzN3O9tTzBZQxDel+omRJAbJY8vLHoME2eGBBHRxg4EJEuQ0GhPRsggIIPmFbEBMWUMbTdQYtyJG2TixGDIPQPfrWaLaZoS7Nk5phqrZoF5vegJQSjOEJ/sBL/Is/8RyvHqwVpzUSL00azbaW+frglzer/59UOTRGsHujetRzBDswm3rl5N1PkeQUVlvf+Pj1cvXdylg8jt/xgvx/w1P8QydwG/+sw2W2cvCIH5O80I1Rg5S77bifrKmyQn/M8nRhfiFrVS+8hVGYoH6UYR4WYQmqVP9M6BdeCyPiZ/GXeCQet6mikGmG4VaIv68BjXO1NA==
8
>
<
>
:
x1 = (5.0, 1.0), y1 = 8.2
x2 = ( 7.0, 7.0), y2 = 9.5
x3 = (1.2, 4.0), y3 = 0.9
(3 )

View Slide

(forward )
(3 )
AAACrXichVFNLwNRFD0d39/FRmLTaIiF1J0iRCJp2Fi2KBKlmRlPTUxnJjOvTWj8AQtbCysSC7GVWLGx8Qcs/ASxJLGxcDudEAR3MvPuOe+eO+e9q7uW6Uuih4hSV9/Q2NTc0trW3tHZFe3uWfadkmeIrOFYjreqa76wTFtkpSktsep6QivqlljRd+aq+ytl4fmmYy/JXVesF7WCbW6ZhiaZykeHc3JbSC2vzlAiORILUZLR2AcaYzSej8YpQUHEfiZqmMQRRtqJXiGHTTgwUEIRAjYk5xY0+PysQQXBZW4dFeY8zsxgX2AfrawtcZXgCo3ZHf4WGK2FrM242tMP1Ab/xeLXY2UMg3RP5/RMd3RBj/T2a69K0KPqZZdXvaYVbr7roG/x9V9VkVeJ7U/Vn54ltjAVeDXZuxsw1VMYNX157+h5cXphsDJEp/TE/k/ogW75BHb5xTjLiIXjP/zo7IVvjAekfh/Hz2Q5mVAnEpQZj6dmw1E1ox8DGOZ5TCKFeaSR5f6HuMQ1bpRRJavklI1aqRIJNb34EkrhHa51nT0=
✓1 = 0.2, ✓2 = 0.3, ✓3 = 0.4
AAADBnichVJNS9xQFL2J1dqx6lShFroJDorCTLiJymhhQOzGpR8dFYyEJD7Hh/kiyQxOwyy6K/0DXbhqwUUpdCduC930D7jwF4i4tNCNC28yGURFvSF59513zn3nvRvTt3kYIZ4KYtez7p7nvS9yfS/7Bwbzr4bWQq8eWKxqebYXbJhGyGzusmrEI5tt+AEzHNNm6+be+2R9vcGCkHvuh6jpsy3HqLl8h1tGRJCe/6SZrMbd2KIaYSunmU6839IVqSJNzMhYlBQZJ4vSeFNXKqVZWdW0DkVNKKVywil3OGqlNCfP3HCmEo4iq0VpukOZqpRQnstpzN3O9tTzBZQxDel+omRJAbJY8vLHoME2eGBBHRxg4EJEuQ0GhPRsggIIPmFbEBMWUMbTdQYtyJG2TixGDIPQPfrWaLaZoS7Nk5phqrZoF5vegJQSjOEJ/sBL/Is/8RyvHqwVpzUSL00azbaW+frglzer/59UOTRGsHujetRzBDswm3rl5N1PkeQUVlvf+Pj1cvXdylg8jt/xgvx/w1P8QydwG/+sw2W2cvCIH5O80I1Rg5S77bifrKmyQn/M8nRhfiFrVS+8hVGYoH6UYR4WYQmqVP9M6BdeCyPiZ/GXeCQet6mikGmG4VaIv68BjXO1NA==
8
>
<
>
:
x1 = (5.0, 1.0), y1 = 8.2
x2 = ( 7.0, 7.0), y2 = 9.5
x3 = (1.2, 4.0), y3 = 0.9

View Slide

AAAC13ichVE9bxNBEH05voL5iAMNEs2KKMiWwJpzhIKQIkVQQJk4OAnKJae7Y22vcl+6XVuYw6JDiI6KggokCuAHQE/DH6BIQwuIMkg0FIzPFggiwpz29u3bebNvd/w0VNoQ7UxYBw4eOnxk8mjp2PETJ6fK06dWddLNAtkMkjDJ1n1Py1DFsmmUCeV6mkkv8kO55m9fG+6v9WSmVRLfNP1UbkZeO1YtFXiGKbfccCLPdLIoDxOtB6WKYzrSeK59QYxR/Reaq4rSgnB0N3JztWAPtuZEpe8qcVG0Ko4f5XcGrqpWt+pueYZqVITYC+wxmFm8Ll457r32UlJ+Awe3kSBAFxEkYhjGITxo/jZgg5Ayt4mcuYyRKvYlBiixtstZkjM8Zrf53+bVxpiNeT2sqQt1wKeEPDJWCszSB3pJu/SeXtMX+vHPWnlRY+ilz7M/0srUnXp0ZuX7f1URzwad36p9PRu0cLnwqth7WjDDWwQjfe/uk92VK43Z/Dw9p6/s/xnt0Du+Qdz7FrxYlo2n+/jx2Qu/GDfI/rsde8FqvWZfqtEyd+oqRjGJsziHCvdjHou4gSU0uf5bfMQnfLZuWfetB9bDUao1Mdacxh9hPf4Jws+xGw==
loss(✓1, ✓2, ✓3) =
3
X
i=1
(yi f(xi))2

View Slide

View Slide

optimizer
step()
p.grad.zero_()
p.add_(p.grad, alpha=-self.lr)
zero_grad()
grad

View Slide

(pytorch 7 !)
optimizer.step()
loss.backward()
(backward)

View Slide

AAACmHicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsIx+QlJuUkxuQmlmQU5Vbn5BcX18YLKBvoGYCBAibDEMpQtj/OZVuw+DpXQL7AdoYYhhSGfIZkhlKGXIZUhjyGEiA7hyGRoRgIoxkMGQwYCoBisQzVQLEiICsTLJ/KUMvABdRbClSVClSRCBTNBpLpQF40VDQPyAeZWQzWnQy0JQeIi4A6FRhUDa4arDT4bHDCYLXBS4M/OM2qBpsBckslkE6C6E0tiOfvkgj+TlBXLpAuYchA6MLr5hKGNAYLsFszgW4vAIuAfJEM0V9WNf1zsFWQarWawSKD10D3LzS4aXAY6IO8si/JSwNTg2bjcU8S0C3AEANGkCF6dGAywoz0DE31DAINlB2cGCCAg0GaQYlBAxgf5gwODB4MAQyhQPMrGeYzrGBYySTF5MDkzuQJUcrECNUjzIACmIIAlUqblQ==
rloss
AAAC0nichVHPaxNBFH7d+qNN1US9CF4WSyRCCW8LRRECpV68CG3StIVuXXbXSTJk9gczk9Bk2YN4E+89SA8VPIhHD/4BXvwHRHNtT8VjFS8efNkEpS2tb5mZb75535tv9nmx4EojDiaMyQsXL12ems7NXLl6LV+4fmNNRR3ps7ofiUhueK5igoesrrkWbCOWzA08wda99qPh+XqXScWjcFX3YrYVuM2QN7jvaqKcwhNbs22tdE8w0w5c3ZJBIiKlUrNi2qoTOAmvWOnTxELE9G8GkzKSaalRsr0g2U4dfm/O7NHsFGaxjFmYp4E1BrOLxZ/fvu7mmstR4SPY8Awi8KEDATAIQRMW4IKibxMsQIiJ24KEOEmIZ+cMUsiRtkNZjDJcYts0N2m3OWZD2g9rqkzt0y2ChiSlCUX8gu/wCD/jezzE32fWSrIaQy89Wr2RlsVO/uWt2q//qgJaNbT+qc71rKEBDzKvnLzHGTN8hT/Sd/s7R7WH1WJyF9/gd/K/hwP8RC8Iuz/8tyus+vocPx55oT9GDbJOtuM0WJsvWwtlXKFOLcEopuA23IES9eM+LMJjWIY61f8AA9iHA2PV6BvPjRejVGNirLkJx8J49Qfd17NZ
loss =
P1000
i=1
error(f(xi), yi)
AAAC03ichVHLahRBFD1pX3F8ZKIbwU1jGBlBhtuCGIRA0I0rycNJAunYdLc1kyLVD6pqhoxNb8SV4NqFuFBwIW4FP8CNPyAat7oRl1HcuPBOz4BoMN6mqk6duufWqb5RrqSxRDsTzoGDhw4fmTxaO3b8xMmp+vSpFZP1dCzacaYyvRaFRiiZiraVVom1XIswiZRYjbauD89X+0IbmaW37CAXG0nYTWVHxqFlKqjf9K3YtsYOlHD9JLSbOilUZkzpzrk13/SSoJBzXnm78IhKZsYpQutMl81O04+SYrsM5IWL7oDnoD5DLarC3Qu8MZiZb3z/8P5JrbuQ1V/Dxx1kiNFDAoEUlrFCCMPfOjwQcuY2UDCnGcnqXKBEjbU9zhKcETK7xXOXd+tjNuX9sKap1DHfonhoVrpo0Dt6Qbv0ll7SF/r5z1pFVWPoZcBrNNKKPJh6cGb5x39VCa8Wm79V+3q26GC28irZe14xw1fEI33/7qPd5atLjeI8PaOv7P8p7dAbfkHa/xY/XxRLj/fxE7EX/mPcIO/vduwFK5da3uUWLXKnrmEUkziLc2hyP65gHjewgDbXf4WP+ITPTtspnHvO/VGqMzHWnMYf4Tz8BdCMs0c=
loss =
P100
i=1
error(f(xi), yi)
AAAC1nichVFNSxxBEC1HTczmw41eAl6GyIYVZKkRgkYQJLnk6GpWFxwdZia9a2PPB929i+sw3kIgp9xyCDkoeJD8AH9ALvkDQT3kmiAeTfDiwdrZhaCiqaG7X7+uV/16yosFVxrxqMfo7eu/c3fgXu7+g4ePBvOPhxZV1JA+q/iRiGTVcxUTPGQVzbVg1VgyN/AEW/LWX7XPl5pMKh6Fb3QrZiuBWw95jfuuJsrJl23NNrTSLcFMO3D1mgwSESmVmjNmzlaNwEn4zAu00tXEQsSUyG4WkzKSabFWtL0g2UgdPjZutmh28qNYwizM68DqgtHZwt/Dgy+5+lyU3wcb3kIEPjQgAAYhaMICXFD0LYMFCDFxK5AQJwnx7JxBCjnSNiiLUYZL7DrNddotd9mQ9u2aKlP7dIugIUlpQgF/4B6e4nf8isd4fmOtJKvR9tKi1etoWewMfniycPZfVUCrhrV/qls9a6jBVOaVk/c4Y9qv8Dv65uan04Xp+ULyDHfwhPxv4xF+oxeEzT/+bpnNf77Fj0de6I9Rg6yr7bgOFidK1vMSlqlTL6ETAzACT6FI/ZiEWXgNc1Ch+vvwE37Bb6NqbBnvjPedVKOnqxmGS2F8vADMV7P+
loss =
P1000
i=901
error(f(xi), yi)
rloss
rloss
rloss
(batch_size= ⾒, 100 loss )
epoch
torch.utils.data.DataLoader
iterate
AAAC1XichVG/SxxBFH6u+WEuMZ6mCdgskZMT5HgrBEUQJGksT815gqvL7mbuHJz9wczc4blsJynSpLQQCwULsY9/gE3+gZAIaRMQSxNsUuTd3oEkEn3LzHzzzfvefLPPiwVXGvGsx+i9d//Bw75HucdP+p8O5AeHllTUkD6r+JGI5LLnKiZ4yCqaa8GWY8ncwBOs6m28bp9Xm0wqHoVvdCtmq4FbD3mN+64mysmXbc02tdItwUw7cPW6DBIRKZWaM2bOVo3ASfiMhVa6lkwgpsR1k5iUkUyLtaLtBclm6vCxcbNFs5MfwRJmYd4EVheMzBZ+ff2yl6uXo/wJ2PAWIvChAQEwCEETFuCCom8FLECIiVuFhDhJiGfnDFLIkbZBWYwyXGI3aK7TbqXLhrRv11SZ2qdbBA1JShMK+BmP8BI/4TGe4+//1kqyGm0vLVq9jpbFzsD754tXd6oCWjWsX6tu9ayhBlOZV07e44xpv8Lv6JtbO5eL0wuFZBQP8IL87+MZntILwuZP/3CeLeze4scjL/THqEHWv+24CZYmStbLEs5Tp15BJ/pgGF5AkfoxCbMwB2WoUP2P8A2+ww+jaqTGtvGuk2r0dDXP4K8wPvwBGS6zvQ==
loss =
P200
i=101
error(f(xi), yi)

View Slide

micrograd: Python ( )
https://github.com/karpathy/micrograd
engine.py
nn.py
micrograd
(97 lines)
(60 lines)
( Andrej Karpathy)
(
backward )

View Slide

tinygrad Python ( )
https://github.com/geohot/tinygrad

View Slide

geohot
micrograd pytorch
1000
(GPU)
ANE (Apple Neural Engine)
E cientNet Transformer (YOLO BERT ?)
YouTube
https://www.youtube.com/c/georgehotzarchive/search?query=tinygrad
github initial commit
tinygrad Python

View Slide

tinygrad

View Slide

geohot comma.ai
2007 8 (17 )
iPhone SIM lock
iOS jailbreaks
2009
Playstation
SCE
2014 6 Android root
Towelroot root
180 Galaxy S
2016
Comma.ai

View Slide

Deep Learning ト
3
DeZero
DeZero
60
PyTorch TensorFlow
Chainer

View Slide

JAX:
https://github.com/google/jax
JAX is NumPy on the CPU, GPU, and TPU, with great automatic
differentiation for high-performance machine learning research.
Google ColabͰ࢖͑ΔͷͰGPU/TPUΠϯελϯεͰࢼͯ͠ΈͯͶ

View Slide

NumPy

View Slide

JAX
https://slideslive.com/38923687/jax-accelerated-machinelearning-
research-via-composable-function-transformations-in-python
Talk and demo by Skye Wanderman-Milne @ NeurIPS2019

View Slide

https://github.com/google/jax
JAX

View Slide

https://sjmielke.com/jax-purify.htm
From PyTorch to JAX: towards neural net frameworks that purify stateful code
JAX

View Slide

: JAX vs PyTorch

View Slide

. =
.
. ⾒
. Python
. Q & A
https://itakigawa.github.io/news.html

View Slide

機械学習と自動微分 (2023)

機械学習と自動微分 (2023)

itakigawa

More Decks by itakigawa

Other Decks in Technology

Featured

Transcript