Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mark Peletier (Technical University of Eindhove...

Jia-Jie Zhu
March 15, 2024
120

Mark Peletier (Technical University of Eindhoven, Netherlands) Singular-limit Analysis of Training with Noise Injection

WORKSHOP ON OPTIMAL TRANSPORT
FROM THEORY TO APPLICATIONS
INTERFACING DYNAMICAL SYSTEMS, OPTIMIZATION, AND MACHINE LEARNING
Venue: Humboldt University of Berlin, Dorotheenstraße 24

Berlin, Germany. March 11th - 15th, 2024

Jia-Jie Zhu

March 15, 2024
Tweet

More Decks by Jia-Jie Zhu

Transcript

  1. Singular-limit analysis of training with noise injection Mark Peletier TU

    Eindhoven with Anna Shalova (TU/e) and André Schlichting (Münster)
  2. The challenge <latexit sha1_base64="PfCN1NPT5KCsLzeOrtZO5k0tHKk=">AAAFi3icjVRLbxMxEF4egRJeBY5cDEklDlAlrXioAomHEBwQgtJCJFxVXu9s1tRrW7Y3IZi98mu4wn/h3zDeTaqGgoSl1Y6+x3jGYzk1Ujg/GPw6cfLU6c6ZsyvnuucvXLx0efXK1fdOV5bDLtdS21HKHEihYNcLL2FkLLAylfAhPXgW+Q8TsE5oteNnBvZKNlYiF5x5hPZXCbWQhSei3CI1yYXKSH/aJ67iBfEF86S7v9obrA+aRY4Hw3nQS+brzf6V0+9opnlVgvJcMuc+DgfG7wVmveAS6i6tHBjGD9gYQtNATdYQykiuLX7KkwZd0pXMtNIlFFRVCg/lMspK52Zlehv/MZvD9CXzhftTFMHbaflXujEa4PVaU9ULISV5xxSKMshZJX0U5MB8ZcGFV2LcRo92YIT5iNekYBPAIwTCKq9xA8EdkQsd0TlBaZc68BZZFdOFHVEi9RqmZDtidUM73LVhD2tA3IKCKdclqrJAc1YKOZsXVgfq8kV8tKePSj99auReiN0a9kXHQpUmMSSpRFGqmc1IqmW25Kt8/mAvCGUqD4q3w8orGZuMN4pkwgL3ckYY5zjzinmcJS+YZdzjzVue2GfPrNXT2AQWSUdC+d4w0FgTL7Tg0CWEBDqKBM2EM5LNnJ9JoB69TRR6w7peO6o7pKjjVph/yY6Qi/g/5P9nojcoehZdtW31NnqbIcQhpvrz4BEt8Bf62G4kcE7R0K+jG/2TeHZgw0KFkn5d0wOw6s76XTrNBvU8e8ZcgdbQHN6durvWwhDohFkwTkitFlIxac9Wm/Zvy4BYvUjlKmOOCSJ4qNgOLZ6mZHuBlTqLM5OQ+6+9IbViXPivyEnAkbmlqbUGLGjcqPBSGK1VhZtZkCmz9JMWCkO6TNfdLjXM4nOEh0Lw+YjP0PDPR+d48H5jfXhv/d7bjd7j5/MHaSW5ntxMbiXD5H7yOHmZvEl2E558S74nP5KfnYudzc5W52ErPXli7rmWLK3O898u7ula</latexit> Aim: find w such that <latexit

    sha1_base64="i5Cg85UL8fJIQRMuOcXfiOuYgiw=">AAAFkXicjVTrbiM1FJ6Whu2Gy7b05/4xJEVF2lZJVxQQqlQVrUACodJrpHVVeTxnEm89trE9ueDOG/A0+xdehLfheCapmi1Iaymak+9yfI6P5dRI4Xyv98/K6gdrrQ+frD9tf/TxJ58+29j87NLp0nK44FpqO0iZAykUXHjhJQyMBVakEq7S2x8ifzUG64RW535m4LpgQyVywZlH6Gbjy26+M/1+8hWhzBirp4QMd6b47/eSZYTm2jIpybR7s9Hp7fXqRR4H/XnQSebr5GZz7YxmmpcFKM8lc+51v2f8dWDWCy6hatPSgWH8lg0h1G1UZBuhjOCe+FOe1OiSrmCmkS6hoMpCeCiWUVY4NyvSF/iN2RymL5gfuXdFEXyRFv9J10YDvNquq/pR4FmcMYWiDHJWSh8FOTBfWnDhFzFsosNzGGA+4jUZsTEQPwLCSq9xA8EdkQsd0TlBaZs68BZZFdOFc1Eg9StMyGnEqpp2uGvN3teAuAUFE64LVGWB5qwQcjYvrArU5Yv4YU+vlT4+NvI6xG4N+0PHQpUmMSSpRFGqmc1IqmW25Ct9/u11EMqUHhRvhpWXMjYZ7xXJhAXu5YwwznHmJfM4Sz5ilnGP9295YlPPrNWT2AQWSQdC+U4/0FgTH2nBoU0ICXQQCZoJZySbOT+TQD166yh0+lW1/VB3T1HHrTD/J3tALuL3kL+fiX5O0bPoqmmrs995GUIcYqqnvUM6wk/oYruRwDlFQ7eKbvSP49mBDQsVSrpVRW/Bqt29r+kk61Xz7BlzI7SG+vB2q/Z2A0OgY2bBOCG1WkjFuDlbbZqvLQJi1SKVK415JIjgveI0NHiaktMFVugszkxC7u86fWrFcOTvkJOAI3NLU2sMWNCwVuGlMFqrEjezIFNm6RstFIZ0ma7abWqYFSrDQyH4fLTxGeq/++g8Di739/oHewe/7XeOXs0fpPXkefJFspP0k2+So+Sn5CS5SHjyZ/I2+Sv5u7XV+q511DpupKsrc89WsrRaP/8LUULrkg==</latexit> f (x; w) ⇡ g(x) 8x <latexit sha1_base64="9vuSxefKiE7+JP19frMe4yGXPuE=">AAAFr3icjVRbbxw1FJ6WLpTllsIjL4ZNpERqop3QFgSKVAUQPFRRSZNmpTpZPJ4zuya+yfbsZnHnD/FreC2/huOZ3SjbgFRLozn6Lsfn+FgurBQ+DIdv7tx9717v/Q/uf9j/6ONPPv1s48HnL72pHYdTbqRxo4J5kELDaRBBwsg6YKqQcFZc/pj4sxk4L4w+CQsL54pNtKgEZwGh8cZPVAk9npNn2/Md8v0BoZVjPI/7Rw2hvlbjKA7y5uKI0EMx2a62r8biBxTukkkKdxK6c7E/3hgM94btIreDfBkMsuV6Pn5w7wUtDa8V6MAl8/5VPrThPDIXBJfQ9GntwTJ+ySYQ2yYbsoVQSSrj8NOBtOiaTjHbSddQ0LUSAdQ6ypT3C1U8xH/K5jG9YmHq3xYl8GGh/pNujRZ4s9VW9YuQkrxgGkUlVKyWIQkqYKF24OMzMemigxMYYT4SDJmyGZAwBcLqYHADwT2RKx0xFUFpn3oIDlmd0sUToZA6gjk5TljT0h53bdnrGhB3oGHOjUJVGWnFlJCLZWFNpL5axTd7eqXN4aGV5zF1a9mfJhWqDUkhKSSKCsNcSQojyzVfHarvzqPQtg6geTesqpapyXTrSCkc8CAXhHGOM69ZwFnyKcPLFvB2rk/sKjDnzDw1gUXSkdBhkEeaauJTIzj0CSGRjhJBS+GtZAsfFhJoQG8bxUHeNFs3ddcU9dwJ+3+yG+Qqfgf5u5noVxQ9q666tgb7g29iTEMszNXwgE7xFzex3UTgnJJhs0lu9M/S2YGLKxVKNpuGXoLTu3uP6bwcNsvsJfNTtMb28Hab/lYHQ6Qz5sB6IY1eScWsO1tju79TEbFmlcrX1t4SJPBacRw7vCjI8QpTpkwzk1CF14OcOjGZhtfIScCR+bWpdQYsaNKq8FJYY3SNmzmQBXP0DyM0hnSdbvp9apkTusRDIfh89PEZyt9+dG4HL/f38id7T357NHj68/JBup99mX2dbWd59m32NPs1e56dZjz7K/s7e5P908t7Z72L3u+d9O6dpeeLbG31xL9AVfXu</latexit> min w L(w) := 1 2N N Â i=1 ⇣ f (xi ; w) g(xi) ⌘2 <latexit sha1_base64="cQCxz2elW7aOqkGvBangrF8EV0E=">AAAFknicjVTbbhMxEF0ugRJuLfDGiyGpAKmtkiIuQkKCIgQPCEFpIRKuKq93NjH12pbtTRvc/QS+hlf4EP6G8W5SNRQkHCWenHNmPOOxJjVSON/r/Tp1+szZ1rnzCxfaFy9dvnJ1cenaR6dLy2Gba6ntIGUOpFCw7YWXMDAWWJFK+JTuvYj8pzFYJ7Ta8hMDOwUbKpELzjxCu4t3ujQVw7sHu2Kl/r0X/97rEoqfjHlGjBbKO7K72Omt9epFThr9qdFJpuvd7tLZDzTTvCxAeS6Zc5/7PeN3ArNecAlVm5YODON7bAihrqMiywhlJNcWv8qTGp3TFcw00jkUVFkID8U8ygrnJkW6gnuM5jB8wfzI/SmK4Epa/JWuHQ3warnO6pWQknxgCkUZ5KyUPgpyYL604MIbMWysp1swwHjEazJiYyB+BISVXuMBgjsiZzqic4LSNnXgLbIqhgtbokDqLeyTzYhVNe3w1Jo9ygFxCwr2uS5QlQWas0LIyTSxKlCXz+zjNX1WemPDyJ0QqzXsq46JKk2iSVKJolQzm5FUy2zOr/T5450glCk9KN40Ky9lLDI+LJIJC9zLCWGcY89L5rGXfMQs4x4f4HzHDjyzVu/HIjBJOsA31ukHGnPiIy04tAkhgQ4iQTPhjGQT5ycSqEff2gqdflUtH9cdUdRxK8y/ZMfImf0f8v9zorco+syqasrqrHfuhxCbmOqD3lM6wi10sdxIYJ+iQ7eK3ug/jncHNsxUKOlWFd0Dq1bXHtD9rFdNo2fMjdA11Je3WrWXGxgCHTMLxgmp1Uwqxs3datPstgiIVbNQrjTmhCCCR4rN0OBpSjZnWKGz2DMJuT/s9KkVw5E/RE4CtszNda1xwISGtQofhdFalXiYBZkyS7/glEGTztNVu00Ns0JleCkEx0cbx1D/z6Fz0vi4vtZ/uPbw/Xrn2cvpQFpIbia3k7tJP3mUPEteJ++S7YQn35LvyY/kZ+tG60nreetFIz19aupzPZlbrTe/Ab9z69o=</latexit> xi , g(xi) data points In practice: How good? It depends …
  3. The challenge <latexit sha1_base64="PfCN1NPT5KCsLzeOrtZO5k0tHKk=">AAAFi3icjVRLbxMxEF4egRJeBY5cDEklDlAlrXioAomHEBwQgtJCJFxVXu9s1tRrW7Y3IZi98mu4wn/h3zDeTaqGgoSl1Y6+x3jGYzk1Ujg/GPw6cfLU6c6ZsyvnuucvXLx0efXK1fdOV5bDLtdS21HKHEihYNcLL2FkLLAylfAhPXgW+Q8TsE5oteNnBvZKNlYiF5x5hPZXCbWQhSei3CI1yYXKSH/aJ67iBfEF86S7v9obrA+aRY4Hw3nQS+brzf6V0+9opnlVgvJcMuc+DgfG7wVmveAS6i6tHBjGD9gYQtNATdYQykiuLX7KkwZd0pXMtNIlFFRVCg/lMspK52Zlehv/MZvD9CXzhftTFMHbaflXujEa4PVaU9ULISV5xxSKMshZJX0U5MB8ZcGFV2LcRo92YIT5iNekYBPAIwTCKq9xA8EdkQsd0TlBaZc68BZZFdOFHVEi9RqmZDtidUM73LVhD2tA3IKCKdclqrJAc1YKOZsXVgfq8kV8tKePSj99auReiN0a9kXHQpUmMSSpRFGqmc1IqmW25Kt8/mAvCGUqD4q3w8orGZuMN4pkwgL3ckYY5zjzinmcJS+YZdzjzVue2GfPrNXT2AQWSUdC+d4w0FgTL7Tg0CWEBDqKBM2EM5LNnJ9JoB69TRR6w7peO6o7pKjjVph/yY6Qi/g/5P9nojcoehZdtW31NnqbIcQhpvrz4BEt8Bf62G4kcE7R0K+jG/2TeHZgw0KFkn5d0wOw6s76XTrNBvU8e8ZcgdbQHN6durvWwhDohFkwTkitFlIxac9Wm/Zvy4BYvUjlKmOOCSJ4qNgOLZ6mZHuBlTqLM5OQ+6+9IbViXPivyEnAkbmlqbUGLGjcqPBSGK1VhZtZkCmz9JMWCkO6TNfdLjXM4nOEh0Lw+YjP0PDPR+d48H5jfXhv/d7bjd7j5/MHaSW5ntxMbiXD5H7yOHmZvEl2E558S74nP5KfnYudzc5W52ErPXli7rmWLK3O898u7ula</latexit> Aim: find w such that <latexit

    sha1_base64="i5Cg85UL8fJIQRMuOcXfiOuYgiw=">AAAFkXicjVTrbiM1FJ6Whu2Gy7b05/4xJEVF2lZJVxQQqlQVrUACodJrpHVVeTxnEm89trE9ueDOG/A0+xdehLfheCapmi1Iaymak+9yfI6P5dRI4Xyv98/K6gdrrQ+frD9tf/TxJ58+29j87NLp0nK44FpqO0iZAykUXHjhJQyMBVakEq7S2x8ifzUG64RW535m4LpgQyVywZlH6Gbjy26+M/1+8hWhzBirp4QMd6b47/eSZYTm2jIpybR7s9Hp7fXqRR4H/XnQSebr5GZz7YxmmpcFKM8lc+51v2f8dWDWCy6hatPSgWH8lg0h1G1UZBuhjOCe+FOe1OiSrmCmkS6hoMpCeCiWUVY4NyvSF/iN2RymL5gfuXdFEXyRFv9J10YDvNquq/pR4FmcMYWiDHJWSh8FOTBfWnDhFzFsosNzGGA+4jUZsTEQPwLCSq9xA8EdkQsd0TlBaZs68BZZFdOFc1Eg9StMyGnEqpp2uGvN3teAuAUFE64LVGWB5qwQcjYvrArU5Yv4YU+vlT4+NvI6xG4N+0PHQpUmMSSpRFGqmc1IqmW25Ct9/u11EMqUHhRvhpWXMjYZ7xXJhAXu5YwwznHmJfM4Sz5ilnGP9295YlPPrNWT2AQWSQdC+U4/0FgTH2nBoU0ICXQQCZoJZySbOT+TQD166yh0+lW1/VB3T1HHrTD/J3tALuL3kL+fiX5O0bPoqmmrs995GUIcYqqnvUM6wk/oYruRwDlFQ7eKbvSP49mBDQsVSrpVRW/Bqt29r+kk61Xz7BlzI7SG+vB2q/Z2A0OgY2bBOCG1WkjFuDlbbZqvLQJi1SKVK415JIjgveI0NHiaktMFVugszkxC7u86fWrFcOTvkJOAI3NLU2sMWNCwVuGlMFqrEjezIFNm6RstFIZ0ma7abWqYFSrDQyH4fLTxGeq/++g8Di739/oHewe/7XeOXs0fpPXkefJFspP0k2+So+Sn5CS5SHjyZ/I2+Sv5u7XV+q511DpupKsrc89WsrRaP/8LUULrkg==</latexit> f (x; w) ⇡ g(x) 8x <latexit sha1_base64="9vuSxefKiE7+JP19frMe4yGXPuE=">AAAFr3icjVRbbxw1FJ6WLpTllsIjL4ZNpERqop3QFgSKVAUQPFRRSZNmpTpZPJ4zuya+yfbsZnHnD/FreC2/huOZ3SjbgFRLozn6Lsfn+FgurBQ+DIdv7tx9717v/Q/uf9j/6ONPPv1s48HnL72pHYdTbqRxo4J5kELDaRBBwsg6YKqQcFZc/pj4sxk4L4w+CQsL54pNtKgEZwGh8cZPVAk9npNn2/Md8v0BoZVjPI/7Rw2hvlbjKA7y5uKI0EMx2a62r8biBxTukkkKdxK6c7E/3hgM94btIreDfBkMsuV6Pn5w7wUtDa8V6MAl8/5VPrThPDIXBJfQ9GntwTJ+ySYQ2yYbsoVQSSrj8NOBtOiaTjHbSddQ0LUSAdQ6ypT3C1U8xH/K5jG9YmHq3xYl8GGh/pNujRZ4s9VW9YuQkrxgGkUlVKyWIQkqYKF24OMzMemigxMYYT4SDJmyGZAwBcLqYHADwT2RKx0xFUFpn3oIDlmd0sUToZA6gjk5TljT0h53bdnrGhB3oGHOjUJVGWnFlJCLZWFNpL5axTd7eqXN4aGV5zF1a9mfJhWqDUkhKSSKCsNcSQojyzVfHarvzqPQtg6geTesqpapyXTrSCkc8CAXhHGOM69ZwFnyKcPLFvB2rk/sKjDnzDw1gUXSkdBhkEeaauJTIzj0CSGRjhJBS+GtZAsfFhJoQG8bxUHeNFs3ddcU9dwJ+3+yG+Qqfgf5u5noVxQ9q666tgb7g29iTEMszNXwgE7xFzex3UTgnJJhs0lu9M/S2YGLKxVKNpuGXoLTu3uP6bwcNsvsJfNTtMb28Hab/lYHQ6Qz5sB6IY1eScWsO1tju79TEbFmlcrX1t4SJPBacRw7vCjI8QpTpkwzk1CF14OcOjGZhtfIScCR+bWpdQYsaNKq8FJYY3SNmzmQBXP0DyM0hnSdbvp9apkTusRDIfh89PEZyt9+dG4HL/f38id7T357NHj68/JBup99mX2dbWd59m32NPs1e56dZjz7K/s7e5P908t7Z72L3u+d9O6dpeeLbG31xL9AVfXu</latexit> min w L(w) := 1 2N N Â i=1 ⇣ f (xi ; w) g(xi) ⌘2 <latexit sha1_base64="cQCxz2elW7aOqkGvBangrF8EV0E=">AAAFknicjVTbbhMxEF0ugRJuLfDGiyGpAKmtkiIuQkKCIgQPCEFpIRKuKq93NjH12pbtTRvc/QS+hlf4EP6G8W5SNRQkHCWenHNmPOOxJjVSON/r/Tp1+szZ1rnzCxfaFy9dvnJ1cenaR6dLy2Gba6ntIGUOpFCw7YWXMDAWWJFK+JTuvYj8pzFYJ7Ta8hMDOwUbKpELzjxCu4t3ujQVw7sHu2Kl/r0X/97rEoqfjHlGjBbKO7K72Omt9epFThr9qdFJpuvd7tLZDzTTvCxAeS6Zc5/7PeN3ArNecAlVm5YODON7bAihrqMiywhlJNcWv8qTGp3TFcw00jkUVFkID8U8ygrnJkW6gnuM5jB8wfzI/SmK4Epa/JWuHQ3warnO6pWQknxgCkUZ5KyUPgpyYL604MIbMWysp1swwHjEazJiYyB+BISVXuMBgjsiZzqic4LSNnXgLbIqhgtbokDqLeyTzYhVNe3w1Jo9ygFxCwr2uS5QlQWas0LIyTSxKlCXz+zjNX1WemPDyJ0QqzXsq46JKk2iSVKJolQzm5FUy2zOr/T5450glCk9KN40Ky9lLDI+LJIJC9zLCWGcY89L5rGXfMQs4x4f4HzHDjyzVu/HIjBJOsA31ukHGnPiIy04tAkhgQ4iQTPhjGQT5ycSqEff2gqdflUtH9cdUdRxK8y/ZMfImf0f8v9zorco+syqasrqrHfuhxCbmOqD3lM6wi10sdxIYJ+iQ7eK3ug/jncHNsxUKOlWFd0Dq1bXHtD9rFdNo2fMjdA11Je3WrWXGxgCHTMLxgmp1Uwqxs3datPstgiIVbNQrjTmhCCCR4rN0OBpSjZnWKGz2DMJuT/s9KkVw5E/RE4CtszNda1xwISGtQofhdFalXiYBZkyS7/glEGTztNVu00Ns0JleCkEx0cbx1D/z6Fz0vi4vtZ/uPbw/Xrn2cvpQFpIbia3k7tJP3mUPEteJ++S7YQn35LvyY/kZ+tG60nreetFIz19aupzPZlbrTe/Ab9z69o=</latexit> xi , g(xi) data points In practice: <latexit sha1_base64="0r1Bywg9HUyTxTL8jRzK9polhH8=">AAAFmXicjVTdjhM3FB4ooTRQWNrLvXGbXQlUNsps6bY3K1GqCoRQRfcHIuFV5PGcyZh4bMv2JKRm3qJP09v2JXibHs8kqw0LEpZGc/T9HJ/jYzkzUjg/Gr2/cvWLa73rX974qn/z1te372zd/eal07XlcMq11HacMQdSKDj1wksYGwusyiS8yma/Rf7VHKwTWp34pYGzik2VKARnHqHJ1pBOLYAKi0mY/ZA25JAsJjOyRyiTpmSEKpZJNlmQ5/cQv99Mtgaj4ahd5HKQroJBslovJnevHdNc87oC5blkzr1OR8afBWa94BKaPq0dGMZnbAqhbachuwjlpNAWP+VJi27oKmY66QYKqq6Eh2oTZZVzyyp7gP+YzWH6ivnSfSiK4IOs+ijdGg3wZret6omQkhwzhaIcClZLHwUFMF9bcOG5mHbR4QmMMR/xmpRsDsSXQFjtNW4guCNyrSO6ICjtUwfeIqtiunAiKqT+gAU5iljT0g53bdnzGhC3oGDBdYWqPNCCVUIuV4U1gbpiHV/s6bXSjx8beRZit4b9pWOhSpMYEhw5n2Wa2ZxkWuYbvtoXv5wFoUztQfFuWEUtY5PxfpFcWOBeLgnjHGdeM4+z5CWzjHu8h5sTe+uZtXoRm8Ai6VgoP0gDjTXxUgsOfUJIoONI0Fw4I9nS+aUE6tHbRmGQNs3uRd05RR23wnxKdoFcx58h/zwT/Y6iZ91V19Zgf/BjCHGImX47OqQl/sIOthsJnFM07DTRjf55PDuwYa1CyU7T0BlYtTf8iS7yUbPKnjNXojW0h7fX9Hc7GAKdMwvGCanVWirm3dlq0/1tFRBr1qlcbcwlQQTPFUehw7OMHK2xSudxZhIK/26QUiumpX+HnAQcmduYWmfAgqatCi+F0VrVuJkFmTFL32ihMKSbdNPvU8OsUDkeCsHno4/PUPrho3M5eLk/TA+GB38+HDz6ffUg3Ui2k++Te0ma/Jw8Sp4mL5LThCd/J/8k/yb/9bZ7v/ae9p510qtXVp5vk43VO/4fUUfvTw==</latexit> wk+1 = wk arwL(wk) time-discretization of <latexit sha1_base64="JP8vwBMmCQKFvQX4fT6yvxye7XQ=">AAAFlnicjVTbbhMxEF0u4RJuBV6QeDGklYpoq4T7SxECIXhAqJSWRsJV5PXOJqZe27K9CcHsR/A1vMJn8DeMd5OqoSBhabWjcxnPeCynRgrnu91fJ06eOt06c/bc+faFi5cuX1m6eu2D06XlsMu11LafMgdSKNj1wkvoGwusSCXspQcvIr83BuuEVjt+amC/YEMlcsGZR2iwdJcOLYAK1DDrBZMDT/ZW/R2ySdapYqlkgwl5sxqhO9VgqdPd6NaLHA96s6CTzNbW4Orp9zTTvCxAeS6Zcx97XeP3Q9yKS6jatHRgGD9gQwh1LxVZQSgjubb4KU9qdEFXMNNIF1BQZSE8FIsoK5ybFuka/mM2h+kL5kfuT1EE19Lir3RtNMCrlbqqV0JK8p4pFGWQs1L6KMiB+dKCC2/EsIk2d6CP+YjXZMTGQPwICCu9xg0Ed0TOdUTnBKVt6sBbZFVMF3ZEgdRbmJDtiFU17XDXmj2sAXELCiZcF6jKAs1ZIeR0VlgVqMvn8dGePir9/LmR+yF2a9gXHQtVmsSQ4Mz5QaqZzUiqZbbgK33+ZD8IZUoPijfDyksZm4yXi2TCAvdyShjnOPOSeZwlHzHLuMdLuDixz55ZqyexCSyS9oXynV6gsSY+0oJDmxASaD8SNBPOSDZ1fiqBevTWUej0qmrlqO6Qoo5bYf4lO0LO4/+Q/5+J3qLomXfVtNW517kfQhxiqj93N+kIf2EZ240EzikalqvoRv84nh3YMFehZLmq6AFYtb7xkE6ybjXLnjE3QmuoD2+9aq80MAQ6ZhaME1KruVSMm7PVpvnbIiBWzVO50phjgggeKrZDg6cp2Z5jhc7izCTk/munR60YjvxX5CTgyNzC1BoDFjSsVXgpjNaqxM0syJRZ+kkLhSFdpKt2O75NQmV4KASfjzY+Q70/H53jwYd7G71HG4/ePeg8ezl7kM4lN5PbyWrSSx4nz5LXyVaym/DkW/I9+ZH8bN1oPW29bL1qpCdPzDzXk4XV2voNr6TtzA==</latexit> ∂tW(t) = rwL(W(t)) <latexit sha1_base64="sIUUoOvV5wIvwqLu9SK4LIcG7fY=">AAAFi3icjVRbaxQxFB4v62W9VX30JbpbUKhlt8ULYkEU0QcRra0umFIymTM7cTNJSDJb1ziv/hpf9b/4bzyZ2S1dq2BgmMN3OTknJyQ1Ujg/GPw6cfLU6c6Zs+fOdy9cvHT5ysrVa++driyHXa6ltqOUOZBCwa4XXsLIWGBlKuFDOnkW+Q9TsE5oteNnBvZKNlYiF5x5hPZXyO0+ZdIUjHo96K+Rvt+fkC3SYmTSv7O/0husD5pFjgfDedBL5uvN/tXT72imeVWC8lwy5z4OB8bvBWa94BLqLq0cGMYnbAyhaaAmqwhlJNcWP+VJgy7pSmZa6RIKqiqFh3IZZaVzszJdw3/M5jB9yXzh/hRFcC0t/0o3RgO8Xm2qeiGkJO+YQlEGOaukj4IcmK8suPBKjNtoawdGmI94TQo2BeILIKzyGjcQ3BG50BGdE5R2qQNvkVUxXdgRJVKv4YBsR6xuaIe7NuxhDYhbUHDAdYmqLNCclULO5oXVgbp8ER/t6aPST58auRdit4Z90bFQpUkMSSpRlGpmM5JqmS35Kp8/3AtCmcqD4u2w8krGJuONIpmwwL2cEcY5zrxiHmfJC2YZ93jzlif22TNr9UFsAoukI6F8bxhorIkXWnDoEkICHUWCZsIZyWbOzyRQj94mCr1hXa8e1R1S1HErzL9kR8hF/B/y/zPRmxQ9i67atnobvc0Q4hBT/XmwRQv8hT62GwmcUzT06+hG/zSeHdiwUKGkX9d0AlbdXb9HD7JBPc+eMVegNTSHd7furrYwBDplFowTUquFVEzbs9Wm/dsyIFYvUrnKmGOCCB4qtkOLpynZXmClzuLMJOT+a29IrRgX/ityEnBkbmlqrQELGjcqvBRGa1XhZhZkyiz9pIXCkC7TdbdLDbNCZXgoBJ+PLj5Dwz8fnePB+4314f31+283ek+ezx+kc8mN5FZyOxkmD5InycvkTbKb8ORb8j35kfzsXOpsdh51HrfSkyfmnuvJ0uo8/w0wcOiS</latexit> (a ! 0, tk = ak) Overparametrized case: <latexit sha1_base64="kjG70s4miyQSg/OEcnKyW68D9lA=">AAAFlHicjVRbaxQxFJ6q62W9VQVffInuFhRs2a14QSxIi+iDiNZWF0wpmcyZ3dhMEpLMrms6v8Ff46v+Dv+NJzO7pWsVDAxz+C4n5+SEpEYK53u9X0unTp9pnT13/kL74qXLV64uX7v+wenSctjlWmo7SJkDKRTseuElDIwFVqQSPqYHW5H/OAbrhFY7fmpgr2BDJXLBmUdof/lel75kRcHI0w1Cw+QpeX13cm+jRyvSJcIRaiELktkhVPvLnd5ar17kZNCfBZ1ktt7uXzvznmaalwUozyVz7lO/Z/xeYNYLLqFq09KBYfyADSHUnVRkBaGM5Nripzyp0QVdwUwjXUBBlYXwUCyirHBuWqT38R+zOUxfMD9yf4oieD8t/krXRgO8WqmreimkJO+ZQlEGOSulj4IcmC8tuPBaDJtoYwcGmI94TUZsDMSPgLDSa9xAcEfkXEd0TlDapg68RVbFdGFHFEi9gQnZjlhV0w53rdmjGhC3oGDCNQ5QZYHmrBByOiusCtTl8/h4T5+U3tw0ci/Ebg37qmOhSpMYklSiKNXMZiTVMlvwlT5/sheEMqUHxZth5aWMTcarRTJhgXs5JYxznHnJPM6Sj5hl3OMVXJzYF8+s1ZPYBBZJB0L5Tj/QWBMfacGhTQgJdBAJmglnJJs6P5VAPXrrKHT6VbVyXHdEUcetMP+SHSPn8X/I/89Eb1P0zLtq2uqsdx6EEIeY6i+9DTrCX+hiu5HAOUVDt4pu9I/j2YENcxVKulVFD8Cq1bWHdJL1qln2jLkRWkN9eKtVe6WBIdAxs2CckFrNpWLcnK02zd8WAbFqnsqVxpwQRPBIsR0aPE3J9hwrdBZnJiH3h50+tWI48ofIScCRuYWpNQYsaFir8FIYrVWJm1mQKbP0sxYKQ7pIV+02NcwKleGhEHw+2vgM9f98dE4GH9bX+o/WHr1b7zx/MXuQzie3kjvJ3aSfPE6eJ6+St8luwpNvyffkR/KzdbP1rLXVmmlPLc08N5KF1XrzGz+Z7Fw=</latexit> G := {w : L(w) = 0} is large
  4. Central idea Adding noise may ‘help’ It helps by moving

    to ‘better parameter points’ The lack of generalization ability is due to the fact that large-batch methods tend to converge to sharp minimizers of the training function. These minimizers are characterized by a signif- icant number of large positive eigenvalues in r2f(x), and tend to generalize less well. In contrast, small-batch methods converge to flat minimizers characterized by having numerous small eigenvalues of r2f(x). We have observed that the loss function landscape of deep neural networks is such that large-batch methods are attracted to regions with sharp minimizers and that, unlike small-batch methods, are unable to escape basins of attraction of these minimizers. The concept of sharp and flat minimizers have been discussed in the statistics and machine learning literature. (Hochreiter & Schmidhuber, 1997) (informally) define a flat minimizer ¯ x as one for which the function varies slowly in a relatively large neighborhood of ¯ x. In contrast, a sharp minimizer ˆ x is such that the function increases rapidly in a small neighborhood of ˆ x. A flat minimum can be de- scribed with low precision, whereas a sharp minimum requires high precision. The large sensitivity of the training function at a sharp minimizer negatively impacts the ability of the trained model to generalize on new data; see Figure 1 for a hypothetical illustration. This can be explained through the lens of the minimum description length (MDL) theory, which states that statistical models that require fewer bits to describe (i.e., are of low complexity) generalize better (Rissanen, 1983). Since flat minimizers can be specified with lower precision than to sharp minimizers, they tend to have bet- ter generalization performance. Alternative explanations are proffered through the Bayesian view of learning (MacKay, 1992), and through the lens of free Gibbs energy; see e.g. Chaudhari et al. (2016). Flat Minimum Sharp Minimum Training Function Testing Function f(x) Figure 1: A Conceptual Sketch of Flat and Sharp Minima. The Y-axis indicates value of the loss function and the X-axis the variables (parameters) 2.2 NUMERICAL EXPERIMENTS Maybe: ‘wider valleys are better’: Keskar et al. 2016 (Hochreiter-Schmidhuber 97, Neyshabur et al. 17, Tsuzuku et al. 20, Petzka et al. 21) How could we recognize ‘better’ points?
  5. Question How can we understand the effect of noise on

    gradient descent? (and is there a relation with ‘wide valleys’?)
  6. <latexit sha1_base64="/XAOa71NNkGAYrP8syQ1lGb6Qx4=">AAAFbXicjVRbaxQxFB6162W9Kz4pEt0WFWrZ9f5SEEX0QURrqwumlEzmzE5sJglJZtc1nb/gq/41f4V/wZOZ3dK1CgaGOXyXk3NyQlIjhfP9/s8jR48tdY6fOHmqe/rM2XPnL1y89MHpynLY4lpqO0yZAykUbHnhJQyNBVamEj6mu88j/3EM1gmtNv3UwHbJRkrkgjMfode3J3d2LvT6a/1mkcPBYBb0ktl6u3Nx6T3NNK9KUJ5L5tynQd/47cCsF1xC3aWVA8P4LhtBaEqsyQpCGcm1xU950qALupKZVrqAgqpK4aFcRFnp3LRMV/EfszlMXzJfuD9FEVxNy7/SjdEAr1eaql4KKcl7plCUQc4q6aMgB+YrCy68FqM2Wt+EIeYjXpOCjYH4AgirvMYNBHdEznVE5wSlXerAW2RVTBc2RYnUG5iQjYjVDe1w14bdrwFxCwomXJeoygLNWSnkdFZYHajL5/HBnj4p/eyZkdshdmvYVx0LVZrEkKQSRalmNiOpltmCr/L5k+0glKk8KN4OK69kbDLeGZIJC9zLKWGc48wr5nGWvGCWcY93a3FiXzyzVk9iE1gkHQrle4NAY0280IJDlxAS6DASNBPOSDZ1fiqBevQ2UegN6nrloG6foo5bYf4lO0DO4/+Q/5+J3qDomXfVttW717sfQhxiqr/012mBv7CM7UYC5xQNy3V0o38czw5smKtQslzXdBesurv2kE6yfj3LnjFXoDU0h3e37q60MAQ6ZhaME1KruVSM27PVpv3bMiBWz1O5yphDggjuKzZCi6cp2Zhjpc7izCTkfq83oFaMCr+HnAQcmVuYWmvAgkaNCi+F0VpVuJkFmTJLP2uhMKSLdN3tUsOsUBkeCsHno4vP0ODPR+dw8OHe2uDR2qN3D3pPX8wepJPJ1eRmcjsZJI+Tp8mr5G2ylfCkSL4l35MfS786VzrXOtdb6dEjM8/lZGF1bv0GWyHfuA==</latexit> L(w) <latexit sha1_base64="oAhOm3WiRgTKsQlbJUaLReQSqb0=">AAAFhXicjVRLbxMxEN5CwyO8CtzgYkgqgUSrpEDhgkBFCA4IQWkhEq4qr3c2MfXalu1NCO5K/Bqu8Hf4N4x3k6qhILHSyqPvMZ7xWE6NFM73er+WTp1ebp05e+58+8LFS5evrFy99sHp0nLY5VpqO0iZAykU7HrhJQyMBVakEj6mB88j/3EM1gmtdvzUwF7BhkrkgjOP0P7KjaFlmQDlSQaOx1XnpPv6zuRud3+l01vv1R85GfRnQSeZfW/3ry6/p5nmZYFpuGTOfer3jN8LzHrBJVRtWjowjB+wIYS69IqsIpSRXFv8cfMaXdAVzDTSBRRUWQgPxSLKCuemRXoP15jNYfqC+ZH7UxTBe2nxV7o2GuDVal3VSyElec8UijLIWSl9FOTAfGnBhddi2ERPdmCA+YjXZMTGQPwICCu9xg0Ed0TOdfF4UdqmDrxFVsV0YUcUSL2BCdmOWFXTDnet2aMaELegYMJ1gaos0JwVQk5nhVWBunweH+/pk9JbW0buhditYV91LFRpEkOSShSlmtmMpFpmC77S54/3glCm9KB4M6y8lLHJeJdIJixwL6eE8Xh1SuZxlnzELOMe79zixL54Zq2exCawSDoQynf6gcaa+EgLDm1CSKCDSNBMOCPZ1PmpBOrRW0eh06+q1eO6I4o6boX5l+wYOY//Q/5/JnqLomfeVdNWZ6NzP4Q4xFR/6T2hI1xCF9uNBM4pGrpVdKN/HM8ObJirUNKtKnoAVq2tP6STrFfNsmfMjdAa6sNbq9qrDQyBjpkF44TUai4V4+ZstWlWWwTEqnkqVxpzQhDBI8V2aPA0JdtzrNBZnJmE3B92+tSK4cgfIicBR+YWptYYsKBhrcJLYbRWJW5mQabM0s9aKAzpIl2129QwK1QW3yJ8Ptr4DPX/fHROBh821vub65vvNjrPXswepHPJzeR2cifpJ4+SZ8mr5G2ym/DkW/I9+ZH8bJ1trbUetDYb6amlmed6svC1nv4GspfndQ==</latexit> gradient descent of L(w) <latexit

    sha1_base64="Cr51tge5akkZa5UOpVFq6V9WkiM=">AAAFdXicjVRLb9QwEA7Q5bG84YiQDNsiDrBseF8qIRCCA0JQWlgJV5XjTDamjm3ZzpbF5G9whb/FL+HKONmtuhQkLEUZfY/xjMdyZqRwfjT6eeTosZXe8RMnT/VPnzl77vyFi5feO11bDltcS23HGXMghYItL7yEsbHAqkzCh2z3WeQ/TME6odWmnxnYrthEiUJw5hGilElTMrJORsN058JgNBy1ixwO0nkwSObrzc7FlXc017yuQHkumXMf05Hx24FZL7iEpk9rB4bxXTaB0JbakDWEclJoi5/ypEWXdBUznXQJBVVXwkO1jLLKuVmV3cJ/zOYwfcV86f4URfBWVv2Vbo0GeLPWVvVCSEneMYWiHApWSx8FBTBfW3DhlZh00fomjDEf8ZqUbArEl0BY7TVuILgjcqEjuiAo7VMH3iKrYrqwKSqkXsMe2YhY09IOd23Z/RoQt6Bgj+sKVXmgBauEnM0LawJ1xSI+2NNHpZ8+NXI7xG4N+6JjoUqTGJJMoijTzOYk0zJf8tW+eLwdhDK1B8W7YRW1jE3Gu0NyYYF7OSOMc5x5zTzOkpfMMu7xji1P7LNn1uq92AQWScdC+UEaaKyJl1pw6BNCAh1HgubCGclmzs8kUI/eNgqDtGnWDur2Keq4FeZfsgPkIv4P+f+Z6DWKnkVXXVuDu4N7IcQhZvrzaJ2W+Aur2G4kcE7RsNpEN/qn8ezAhoUKJatNQ3fBqtvDB3QvHzXz7DlzJVpDe3i3m/5aB0OgU2bBOCG1WkjFtDtbbbq/rQJizSKVq405JIjgvmIjdHiWkY0FVuk8zkxC4b8OUmrFpPRfkZOAI3NLU+sMWNCkVeGlMFqrGjezIDNm6SctFIZ0mW76fWqYFSrHQyH4fPTxGUr/fHQOB+/vDtOHw4dv7w+ePJ8/SCeTK8n15GaSJo+SJ8nL5E2ylfDEJN+S78mPlV+9q73V3o1OevTI3HM5WVq9O78BhJfiYg==</latexit> a = 0.1 noisy gradient descent <latexit sha1_base64="kC5mqpeUA2aPRHS6XU0UhaXS2Kc=">AAAFjHicjVTdbhw1FJ6GbFsWSFO45MZlN1JRk2g3tAWBIkVFQC+iKk3zs1IcRR7PmR0Tj23Znt0u7lzzNNzCs/A2HM/sRtmESrE08pnvxz7Hx3JqpHB+MPj33sonq537Dx5+2v3s8y/WHq0//vLE6cpyOOZaajtKmQMpFBx74SWMjAVWphJO08ufI386AeuEVkd+ZuC8ZGMlcsGZR+hi/UmfTkUGBfNk/+l0k4Jn35Ifd+PPs+anf7HeG2wPmkFuB8N50Evm4+Di8eo7mmlelaA8l8y5s+HA+PPArBdcQt2llQPD+CUbQ2gqqMkGQhnJtcVPedKgS7qSmVa6hIKqSuGhXEZZ6dysTDdxjqs5XL5kvnA3RRHcTMv/pRujAV5vNFn9JqQk75hCUQY5q6SPghyYryy4sC/GbbR7BCNcj3hNCjYB4gsgrPIaNxDcEbnQEZ0TlHapA2+RVXG5cCRKpN7AlBxGrG5oh7s27FUOiFtQMOW6RFUWaM5KIWfzxOpAXb6Ir9d0pvSrV0aeh1itYX/omKjSJIYklShKNbMZSbXMlnyVz384D0KZyoPibbPySsYi45UimbDAvZwRxjn2vGIee8kLZhn3ePWWO/beM2v1NBaBSdKRUL43DDTmxAstOHQJIYGOIkEz4YxkM+dnEqhHbxOF3rCuN67rrijquBXmY7Jr5CK+g/xuJvqEomdRVVtWb6f3XQixial+P9ilBU6hj+VGAvsUDf06utE/iWcHNixUKOnXNb0Eq7a2X9BpNqjnq2fMFWgNzeFt1d2NFoZAJ8yCcUJqtZCKSXu22rSzLQNi9WIpVxlzSxDBK8VhaPE0JYcLrNRZ7JmE3H/oDakV48J/QE4Ctswtda01YELjRoWXwmitKtzMgkyZpb9roTCky3Td7VLDrFAZHgrB56OLz9Dw5qNzOzjZ2R6+3H75dqe398v8QXqYfJ18kzxNhsn3yV7yOjlIjhOe/Jn8lfyd/NNZ6zzv/NTZbaUr9+aer5Kl0fn1P2p26QI=</latexit> b L(w, h) := L(w + h) <latexit sha1_base64="ZFwVazWe+R0SmN0wJ6jGK5yRumQ=">AAAFiHicjVRLbxQ5EG4eswsNuxvgiJAMk0h7CNF0eB8ioSAEB4TYkMBIOIrc7uppE7dt2e4Jg+kTv4Yr/Br+DeXumShDFomWRq75HuUql+XcSOH8aPT9zNlz5wd//HnhYnrp8l9//7Ny5eobpxvLYY9rqe04Zw6kULDnhZcwNhZYnUt4mx8+ifzbKVgntNr1MwP7NZsoUQrOPEIHKzcok6ZiZIuMNrJ1Qgl1YlL3/0ebBytDXLqPnA6yeTBM5t+rgyvnX9NC86YG5blkzr3LRsbvB2a94BLalDYODOOHbAKhq74lawgVpNQWf8qTDl3S1cz00iUUVFMLD/UyymrnZnW+jmvM5jB9zXzlfhZFcD2v/5fujAZ4u9ZV9UxISV4zhaICStZIHwUlMN9YcOGFmPTR1i6MMR/xmlRsCsRXQFjjNW4guCNyoSO6JChNqQNvkVUxXdgVNVIv4YjsRKztaIe7duxxDYhbUHDEdY2qItCS1ULO5oW1gbpyEZ/s6Z3S29tG7ofYrWEfdSxUaRJDkksU5ZrZguRaFku+xpcP94NQpvGgeD+sspGxyXidSCEscC9nhHGOM2+Yx1nyilnGPV675Yl98MxafRSbwCLpWCg/zAKNNfFKCw4pISTQcSRoIZyRbOb8TAL16O2iMMzadu2k7piijlthfiU7QS7i35D/nonepOhZdNW3Ndwc3gkhDjHXH0ZbtMIlrGK7kcA5RcNqG93on8azAxsWKpSsti09BKtub9yjR8WonWcvmKvQGrrDu92maz0MgU6ZBeOE1GohFdP+bLXpV1sHxNpFKtcYc0oQwWPFTujxPCc7C6zWRZyZhNJ/GmbUiknlPyEnAUfmlqbWG7CgSafCS2G0Vg1uZkHmzNL3WigM6TLdpik1zApV4KEQfD5SfIaynx+d08GbzY3s/sb9/+4OHz+dP0gXkuvJreTfJEseJI+T58mrZC/hyefkS/I1+TZIB6PBg8GjXnr2zNxzLVn6Bts/AKuO5rA=</latexit> a = 0.1, s = 0.02 First reported by Li-Wang-Arora 22, What Happens After SGD Reaches Zero Loss?
  7. Overview of the talk • De fi ne ‘noisy gradient

    descent’ • First case: ‘non-degenerate’ • Second case: ‘degenerate’ • Wrap-up
  8. Noisy gradient descent <latexit sha1_base64="0r1Bywg9HUyTxTL8jRzK9polhH8=">AAAFmXicjVTdjhM3FB4ooTRQWNrLvXGbXQlUNsps6bY3K1GqCoRQRfcHIuFV5PGcyZh4bMv2JKRm3qJP09v2JXibHs8kqw0LEpZGc/T9HJ/jYzkzUjg/Gr2/cvWLa73rX974qn/z1te372zd/eal07XlcMq11HacMQdSKDj1wksYGwusyiS8yma/Rf7VHKwTWp34pYGzik2VKARnHqHJ1pBOLYAKi0mY/ZA25JAsJjOyRyiTpmSEKpZJNlmQ5/cQv99Mtgaj4ahd5HKQroJBslovJnevHdNc87oC5blkzr1OR8afBWa94BKaPq0dGMZnbAqhbachuwjlpNAWP+VJi27oKmY66QYKqq6Eh2oTZZVzyyp7gP+YzWH6ivnSfSiK4IOs+ijdGg3wZret6omQkhwzhaIcClZLHwUFMF9bcOG5mHbR4QmMMR/xmpRsDsSXQFjtNW4guCNyrSO6ICjtUwfeIqtiunAiKqT+gAU5iljT0g53bdnzGhC3oGDBdYWqPNCCVUIuV4U1gbpiHV/s6bXSjx8beRZit4b9pWOhSpMYEhw5n2Wa2ZxkWuYbvtoXv5wFoUztQfFuWEUtY5PxfpFcWOBeLgnjHGdeM4+z5CWzjHu8h5sTe+uZtXoRm8Ai6VgoP0gDjTXxUgsOfUJIoONI0Fw4I9nS+aUE6tHbRmGQNs3uRd05RR23wnxKdoFcx58h/zwT/Y6iZ91V19Zgf/BjCHGImX47OqQl/sIOthsJnFM07DTRjf55PDuwYa1CyU7T0BlYtTf8iS7yUbPKnjNXojW0h7fX9Hc7GAKdMwvGCanVWirm3dlq0/1tFRBr1qlcbcwlQQTPFUehw7OMHK2xSudxZhIK/26QUiumpX+HnAQcmduYWmfAgqatCi+F0VrVuJkFmTFL32ihMKSbdNPvU8OsUDkeCsHno4/PUPrho3M5eLk/TA+GB38+HDz6ffUg3Ui2k++Te0ma/Jw8Sp4mL5LThCd/J/8k/yb/9bZ7v/ae9p510qtXVp5vk43VO/4fUUfvTw==</latexit> wk+1 = wk arwL(wk) Gradient

    descent: <latexit sha1_base64="nId5DiYWqvINCysP4pqggxoNKOY=">AAAFqHicjVRfbxw1EN+WHpTjT1N45MVwiVQgie4KFF4iVUUIHioU0qQJqqPTrHf21pzXtmzvXQ93Pw2fhld44tsw3r2Lcg1ItbTa0e/PeMZjObdK+jAe/3Pr9lt3Bm+/c/fd4Xvvf/DhvZ37Hz33pnECz4RRxl3k4FFJjWdBBoUX1iHUucLzfP594s8X6Lw0+jSsLF7WMNOylAICQdOdI56rBuNyGudfTlp2xJbTOTtgHJStgHENuYLpkvGlLLCCwJ4+IME+xwDT+eftdGc0Phx3i90MJutglK3X8fT+nWe8MKKpUQehwPsXk7ENlxFckEJhO+SNRwtiDjOMXXct2yOoYKVx9OnAOnRLV4PtpVso6qaWAettFGrvV3W+T/+UzVP6GkLlXxclcD+v/5PujBZFu9dV9aNUij0DTaICS2hUSIISITQOfXwqZ310dIoXlI8FwypYIAsVMmiCoQ2k8ExtdMyUjKRD7jE4YnVKF09lTdTPuGQnCWs72tOuHXtVA+EONS6FqUlVRF5CLdVqXVgbuS838fWeXmjz5IlVlzF1a+F3kwrVhqWQ0RUQ89yAK1huVLHla0L53WWU2jYBteiHVTYqNZmuGyukQxHUioEQNPMGAs1SVOBABLqW2xN7GcA5s0xNUJH8QuowmkSeahKVkQKHjLHILxLBC+mtgpUPK4U8kLeL4mjStnvXdVcU98JJ+3+ya+QmfgP5m5n4p5w8m676tkYPR1/FmIaYm5fjI17RL+5Su4mgOSXDbpvc5F+ks0MXNyqS7LYtn6PTB4ff8GUxbtfZC/AVWWN3eAftcK+HMfIFOLReKqM3Urnoz9bY/u/qSFi7SeUba28IEnilOIk9nufsZIPVpkgzU1iGV6MJd3JWhVfEKaSR+a2p9QYqaNap6FJYY3RDmzlUOTj+m5GaQr5Nt8Mht+CkLuhQGD0fQ3qGJq8/OjeD5w8PJ48OH/3y9ejxD+sH6W72SfZZ9iCbZN9mj7OfsuPsLBPZH9mf2V/Z34MvBseD88GvvfT2rbXn42xrDfJ/AYeQ9Wo=</latexit> wk+1 = wk arw b L(wk , hk) <latexit sha1_base64="yCNl+vkJSapNadwMYJjWANJ3two=">AAAF0XicjVRLjxM5EG5YwiOwMLBHLob0SHsYos6gZbmMhFih5YAQzItIeBi53dWJGbdt2e4MWdPSiit/jT/A3+DKHrbcSY8mDEhYirr0PcpVLse5kcL5LPt87vwvF3oXL12+0r967dfrN9Zu3tp3urYc9riW2o5z5kAKBXteeAljY4FVuYRX+dFfkX81A+uEVrt+buCgYhMlSsGZR+hwraS5rCGkFDw7PHrzlqTEzohQJKXb6QahRAzFsBjGCKGK+Wmekycn6i2Spac4W4V9ZhuEqROTir3ZJNTrLG0O1wbZMGsXORuMlsEgWa4Xhzcv7NBC87oC5blkzr0eZcYfBGa94BKaPq0dGMaP2ARCewwNWUeoIKW2+FOetOiKrmJmIV1BQdWV8FCtoqxybl7lG/iN2Rymjx26b0UR3Mir79Kt0QBv1tuq/hZSkh2mUFRAyWrpo6AE5msLLjwTk0W0tQtjzEe8JlM2A+KnQFjtNW4guCOy0xFdEpT2qQNvkVUxXdgVFVLP4ZhsR6xpaYe7tuxJDYhbUHDMdYWqItCSVULOl4U1gbqyi0/39Frpx4+NPAixW8P+0bFQpUkMSS5RlGtmC5JrWaz4al8+PAhCmdqD4othlbWMTcZ7SQphgXs5J4xznHnNPM6ST5ll3OP9XZ3YO8+s1cexCSySjoXyg1ForyCfasGhTwgJdBwJWghnJJs7P5dAPXrbKAxGTbN+WndCUcetMD+SnSK7+CfkP2eidyh6uq4WbQ02B/dDiEPM9btsi07xE1JsNxI4p2hIm+hG/yyeHdjQqVCSNg09AqvuDf+gx0XWLLMXzE3RGtrDu9f01xcwBDpjFowTUqtOKmaLs9UmdH9zxJoulauNOSOI4IliO4Tu7djusEoXcWYSSv9+MKJWTKb+PXIScGRuZWoLAxY0aVV4KYzWqsbNLMicWfpWC4UhXaWbfp8aZoUq8FAIPh99fIZG3z46Z4P9zeHowfDBy83BoyfLB+lycju5m/yejJI/k0fJ0+RFspfw5FPyJfma/Nfb6c17//Y+LKTnzy09vyUrq/fxf1wAAVM=</latexit> h j k rv in R, i.i.d., Eh j k = 0, Var = s2 ! 0 <latexit sha1_base64="eikrTu+8Dzwaj8c5QAGLAzgin2w=">AAAFkHicjVTrbhM7EF4u4RIOUOAnfwxJJJBKlRRxEVLFTUcghI44pYVIuKq83tmsqde2bG9CMPsEPA1/D0/C2zDeTaqGHiQsrXb0XcYzHsupkcL54fDHiZOnTnfOnD13vnvhr4uXLq9dufrO6cpy2OVaajtOmQMpFOx64SWMjQVWphLepwfPI/9+CtYJrXb83MBeySZK5IIzj9D+2oBr5XAbUHz+iPTpTGRQME9e35qtD2+TrRjc7u+v9YYbw2aR48FoEfSSxXqzf+X0W5ppXpWgPJfMuQ+jofF7gVkvuIS6SysHhvEDNoHQdFGTAUIZybXFT3nSoCu6kplWuoKCqkrhoVxFWencvEzX8R+zOUxfMl+4X0URXE/L/6UbowFeD5qqXggpyVumUJRBzirpoyAH5isLLrwWkzba2oEx5iNek4JNgfgCCKu8xg0Ed0QudUTnBKVd6sBbZFVMF3ZEidQ/MCPbEasb2uGuDXtYA+IWFMy4LlGVBZqzUsj5orA6UJcv46M9fVD62TMj90Ls1rDPOhaqNIkhSSWKUs1sRlItsxVf5fOHe0EoU8Wr0g4rr2RsMl4rkgkL3Ms5YZzjzCvmcZa8YJZxj9dvdWKfPLNWz2ITWCQdC+V7o0BjTbzQgkOXEBLoOBI0E85INnd+LoF69DZR6I3qenBUd0hRx60wv5MdIZfxH8j/zERvUPQsu2rb6m327oYQh5jqT8MtWuAv9LHdSOCcoqFfRzf6p/HswIalCiX9uqYHYNWdjXt0lg3rRfaMuQKtoTm8O3V30MIQ6JRZME5IrZZSMW3PVpv2b8uAWL1M5SpjjgkieKjYDi2epmR7iZU6izOTkPsvvRG1YlL4L8hJwJG5lam1Bixo0qjwUhitVYWbWZAps/SjFgpDukrX3S41zAqV4aEQfD66+AyNfn10jgfvNjdG9zfu/7vZe/L34kE6l1xPbia3klHyIHmSvEzeJLsJT74m35L/ku+dq52Hncedp6305ImF51qysjqvfgK14+rV</latexit> consistency: b L(w, 0) = L(w) <latexit sha1_base64="M8/dSLSL2UGCd97j9XloTd4K/ZU=">AAAF1HicjVTNbxw1FJ9+LLTDR9Ny5GLYBHFoop3SFi6RqiIEB4RKmqQrxUvk8bzZdeOxXduzydbxCXHlX+PMH8IVeJ7ZjbIJSLU0mqff7/ee34f1SiOF86PRnzdu3ro9eO/9O3fzDz786ON7G/cfHDrdWg4HXEttxyVzIIWCAy+8hLGxwJpSwqvy5NvEv5qDdUKrfb8wMGnYVIlacOYROt4QFDw7PvnlNdkltISpUIFjOBfz7YLQN29aVpEvCPVw5oOxuiTRUHpEX2s/yWlVW8aDiaHYNpHQK0IEcwqqWgY83hiOdkbdIdeNYmkMs+V5cXz/9ktaad42oDyXzLmjYmT8JDDrBZcQc9o6MIyfsCmErhORbCFUkVpb/JQnHbqma5jppWsoqLYRHpp1lDXOLZryIf5TNIfhG+Zn7qoogQ/L5j/pztEAj1tdVt8LKclLplBUQc1a6ZOgBuZbCy78KKa9tbsPY4xHvCYzNgfiZ0BY6zVeILgjcqUjuiYozakDb5FVKVzYFw1SP8Ep2UtY7GiHt3bsRQ6IW1BwynWDqirQmjVCLpaJxUBdvbIv13Sk9PPnRk5CqtawtzolqjRJJiklikrNbEVKLas1v9bX30yCUKb1oHg/rLqVqcj0NEklLHAvF4RxjjNvmcdZ8hnDR+bxCa9P7Mwza/VpKgKTpGOh/LAINOXEZ1pwyAkhgY4TQSvhjGQL5xcSuifaWWFYxLh1WXdBUcetMP8nu0Su7HeQv5sT/Yyiz6qqvqzho+FXIaQhlvpstEtn+AubWG4icE7JYTMmb/Sfp96BDSsVSjZjpCdg1fbOE3pajeIyesXcDF1D17ztmG/1MAQ6ZxaME1KrlVTM+95q0/9tExCLq1CuNeaaIIEXir3Q42VJ9lZYo6s0Mwm1Px8W1IrpzJ8jJwFH5tam1jtgQtNOhY/CaK1avMyCLJnFjSQUmnSdjnlODbNCVdgUgusjxzVUXF06143DRzvF052nPz8ePvtuuZDuZJ9mn2dfZkX2dfYs+yF7kR1kPPsj+yv7O/tncDg4H/w6+K2X3ryx9PkkWzuD3/8FAyoHkw==</latexit> h j k = 8 < : 1 prob p p 1 p prob 1 p 3. Bernoulli Dropout: <latexit sha1_base64="V6RrjQg5Vx2GocV7zeD+XdCbXtc=">AAAFm3icjVTrihQ5FC4v7aVddVx/ihDtGRhxHLrc9YIwIIooy7DoOKMNZhhSqVNd2UklIUl128Z6DZ/Gv/oO+zZ7UtU9TDsrGCjq8F1OzskJyYwUzg+H/546feZs79z5Cxf7l367fOXqyrXf3zldWw57XEttRxlzIIWCPS+8hJGxwKpMwvvs8Hnk30/AOqHVrp8Z2K/YWIlCcOYROlgZ0qnIoWSebK9PNyh4doc82SLbNBPj9SnVufZkPb3bEhG7c7AyGG4O20VOBuk8GCTz9frg2tm3NNe8rkB5LplzH9Kh8fuBWS+4hKZPaweG8UM2htA21JA1hHJSaIuf8qRFl3QVM510CQVVV8JDtYyyyrlZlW3gP2ZzmL5ivnQ/iiK4kVX/S7dGA7xZa6t6KaQkb5lCUQ4Fq6WPggKYry24sC3GXbS1CyPMR7wmJZsA8SUQVnuNGwjuiFzoiC4ISvvUgbfIqpgu7IoKqb9hSnYi1rS0w11b9qgGxC0omHJdoSoPtGCVkLN5YU2grljEx3v6oPSzZ0buh9itYZ90LFRpEkOSSRRlmtmcZFrmS77aF4/3g1Cm9qB4N6yilrHJeMNILixwL2eEcY4zr5nHWfKSWcY93sTliX30zFo9jU1gkXQklB+kgcaaeKkFhz4hJNBRJGgunJFs5vxMAvXobaMwSJtm7bjuiKKOW2F+JjtGLuJfkP+aid6i6Fl01bU1uD/4I4Q4xEx/HG7REn9hFduNBM4pGlab6Eb/JJ4d2LBQoWS1aeghWHVv8wGd5sNmnj1nrkRraA/vXtNf62AIdMIsGCekVgupmHRnq033t1VArFmkcrUxJwQRPFLshA7PMrKzwCqdx5lJKPznQUqtGJf+M3IScGRuaWqdAQsatyq8FEZrVeNmFmTGLP1HC4UhXaabfp8aZoXK8VAIPh99fIbSHx+dk8G7+5vpw82Hb/4cPH0xf5AuJDeS28l6kiaPkqfJq+R1spfw5EvyNfmWfO/d7D3v/dXb7qSnT80915Ol1dv7D2YB7yI=</latexit> b L(w, h) := L w (1 + h) 2. Label noise: 4. Gaussian Dropout: <latexit sha1_base64="2m81fz8ZcFZIL/XnRyD8Wc2+pTQ=">AAAFj3icjVTdbiM1FJ4uDSyBhRYuuTGklXalbpUU2HJTVBWhBQlVpdufSHW38njOJN76T7YnbXDnBXgabpc34W04nkmqpgWplkZz9P0cn+NjObdS+NDv/7P05IPlzocfPf24+8mnzz77fGX1ixNvKsfhmBtp3DBnHqTQcBxEkDC0DpjKJZzmlz8l/nQCzgujj8LUwrliIy1KwVlA6GJljUJgF5dv3xHqhSJUsTDmTJL95/0NREaKvd16cbHS62/2m0UeBoNZ0Mtm6+BidfkNLQyvFOjAJfP+bNC34TwyFwSXUHdp5cEyfslGEJsmarKOUEFK4/DTgTTogk4x20oXUNCVEgHUIsqU91OVb+A/ZfOYPnXm74sSuJGr/6QbowVerzdVvRZSkjdMo6iAklUyJEEJLFQOfPxNjNpo5wiGmI8EQ8ZsAiSMgbAqGNxAcE/kXEdMSVDapR6CQ1andPFIKKT24YocJqxuaI+7NuxtDYg70HDFjUJVEWnJlJDTWWF1pL6cx3d7OtNmb8/K85i6tewPkwrVhqSQ5BJFuWGuILmRxYKvCuUP51FoWwXQvB1WWcnUZLpVpBAOeJBTwjjHmVcs4Cz5mDnGA96+xYldB+acuUpNYJF0KHToDWJ79cZGcOgSQiIdJoIWwlvJpj5MJdCA3iaKvUFdr9/V3VLUcyfs/8nukPP4EfLHmejXFD3zrtq2elu9b2NMQ8zNdX+HjvEX17DdROCckmGtTm70T9LZgYtzFUrW6ppegtMvN7+nV0W/nmUvmB+jNTaH97LurrcwRDphDqwX0ui5VEzaszW2/TsVEavnqXxl7QNBAm8Vh7HF85wczjFlijQzCWW46Q2oE6NxuEFOAo7ML0ytNWBBo0aFl8IaoyvczIHMmaPvjNAY0kW67napZU7oAg+F4PPRxWdocP/ReRicbG0OXm2++v273u7PswfpafZV9k32PBtk29lu9kt2kB1nPPsz+yt7n/3dWe1sd37s7LbSJ0szz5fZwur8+i+30etx</latexit> h j k ⇠ N (0, s2) <latexit sha1_base64="6G8IUmuIBJDKdjXPhVQWUmklhJ0=">AAAFvnicjVRbb9s2FFa7euvcXdLtcS9snQAJkARWtnXDhgBBhqF9KII2l9ZAmRgUdWRxoUiCpOx4rH7Yfkqf9rr9ix5KdhA3G1ACgg6+y+E5PAQzI4Xzw+G7O3c/udf79LP7n/cffPHlV1+vPfzmldO15XDGtdR2lDEHUig488JLGBkLrMokvM4uf4v86ylYJ7Q69XMD5xWbKFEIzjxC47UTOhM5lMyT55uzbQqebZFf9gktLONp2DtqCHV1NQ5iP20ujgg9FJPNYvNqLH6dbe2QSYy2yA6JxgsR2a2LvfHaYLg7bBe5HaSLYJAs1ovxw3snNNe8rkB5Lplzb9Kh8eeBWS+4hKZPaweG8Us2gdD23JANhHJSaIuf8qRFV3QVM510BQVVV8JDtYqyyrl5lW3jP2ZzmL5ivnQfiiK4nVX/SbdGA7zZaKt6KqQkJ0yhKIeC1dJHQQHM1xZceC4mXbR/CiPMR7wmJZsC8SUQVnuNGwjuiFzqiC4ISvvUgbfIqpgunIoKqSOYkeOINS3tcNeWva4BcQsKZlxXqMoDLVgl5HxRWBOoK5bxzZ7eKH14aOR5iN0a9qeOhSpNYkgyiaJMM5uTTMt8xVf74ufzIJSpPSjeDauoZWwyXkKSCwvcyzlhnOPMa+ZxlrxkeOk8XtbViV15Zq2exSawSDoSyg/SQGNNvNSCQ58QEugoEjQXzkg2d34ugXr0tlEYpE2zcVN3TVHHrTD/J7tBLuOPkH+ciT6i6Fl21bU12Bt8H0IcYqavhvu0xF9Yx3YjgXOKhvUmutE/jWcHNixVKFlvGnoJVu3s/khn+bBZZM+ZK9Ea2sPbafobHQyBTpkF44TUaikV0+5sten+tgqINctUrjbmliCC14rj0OFZRo6XWKXzODMJhX87SKkVk9K/RU4CjsytTK0zYEGTVoWXwmitatzMgsyYpX9ooTCkq3TT71PDrFA5HgrB56OPz1D64aNzO3i1t5s+2X3y8ofBwe+LB+l+8l3yONlM0uSn5CB5lrxIzhKe/JX8nfyT/Ns76BW9qqc76d07C8+3ycrqXb0HYsz7vA==</latexit> b L(w, h) := 1 2N N Â i=1 ⇣ f (xi ; w) g(xi) hi ⌘2 1. Mini-batch noise Examples
  9. What can we expect? <latexit sha1_base64="mEvOs+SNOOoWqgBurlufV+6IPT0=">AAAFsnicjVRfbxw1EN+WHpTjX1oeeTFcIhWRRHelLbxEqopQeahQSZM2ok5Ps97ZW3Ne27K9dz3c/Uh8Gp6Q4Lsw3r2Lcg1IXWnl0e/PeMZjObdK+jAe/3Xt+ns3Bu9/cPPD4Ucff/LpZzu3bj/3pnECT4VRxp3l4FFJjadBBoVn1iHUucIX+fyHxL9YoPPS6JOwsnhew0zLUgoIBE13HvNcNRiX0zj/ZtK+kuyALadzWo8o4qBsBYxryBVM4/KVbBlfygIrCOzJHRLucwwwnX/dTndG48Nx97GrwWQdjLL193R668YzXhjR1KiDUOD9y8nYhvMILkihsB3yxqMFMYcZxq7Plu0RVLDSOPp1YB26pavB9tItFHVTy4D1Ngq196s636c1ZfOUvoZQ+bdFCdzP6/+kO6NF0e51VT2WSrFnoElUYAmNCklQIoTGoY9P5KyPjk7wjPKxYFgFC2ShQgZNMLSBFJ6pjY6ZkpF0yD0GR6xO6eKJrIn6GZfsOGFtR3vatWMvaiDcocalMDWpishLqKVarQtrI/flJr7c00ttHj2y6jymbi38blKh2rAUMroGYp4bcAXLjSq2fE0ovz+PUtsmoBb9sMpGpSbTxWOFdCiCWjEQgmbeQKBZigociEAXdHtirwM4Z5apCSqSn0kdRpPIU02iMlLgkDEW+VkieCG9VbDyYaWQB/J2URxN2nbvsu6C4l44af9PdoncxO8gfzcT/5KTZ9NV39bo7ujbGNMQc/N6fMQrWuIutZsImlMy7LbJTf5FOjt0caMiyW7b8jk6fXB4ny+LcbvOXoCvyBq7wztoh3s9jJEvwKH1Uhm9kcpFf7bG9qurI2HtJpVvrL0iSOCF4jj2eJ6z4w1WmyLNTGEZ3owm3MlZFd4Qp5BG5rem1huooFmnokthjdENbeZQ5eD4b0ZqCvk23Q6H3IKTuqBDYfR8DOkZmrz96FwNnt89nDw4fPDLvdHDH9cP0s3si+yr7E42yb7LHmY/ZU+z00xkf2R/Zn9n/wzuDX4dwED00uvX1p7Ps61voP4FH2P5aA==</latexit> wi k+1 wi k

    = arwi b L(wk , hk) <latexit sha1_base64="h8F/D9RYSTmhPgmmJ/9OMRM+MQM=">AAAF7XicjVRPjxs1FJ+0DZQplG05gMTFJbtSkXZXSWkLl0pVEYJDhcp2t420TqM3njcZE49tbE+ywZ0L34Eb4spn4hvwMWrPJKtNl0q1NJqn35/n9/wsZ1pw64bDf3tXrl7rf/Dh9Y/SGx9/cvPTnVu3X1hVG4YnTAllxhlYFFziieNO4FgbhCoT+DKbfx/5lws0lit57FYaJxXMJC84Axeg6c4fNBM1Eu8bClobdUYOKAhdAqESMgH0ztQvX/GG0CXPsQRHnt5dTuf7w69T+lsNOTkg79BrMI6DmFJ0cNl9GuHpfJI26XRnMDwctotcDkbrYJCs17PprWvPaa5YXaF0TIC1p6OhdhMfN2QCm5TWFjWwOczQt0fUkL0A5aRQJnzSkRbd0lWgO+kWirKuuMNqG4XK2lWV7Yd/zGZD+gpcad8WRXA/q/6Xbo0aWbPXVvUjF4I8BxlEORZQCxcFBYKrDVr/lM+66NExjkM+4hQpYYHElUigdipswJklYqMjqiBBmlKLzgRWxnT+mFeB+hmX5ChiTUvbsGvLntcQcIMSl0xVQZV7WkDFxWpdWOOpLTbxxZ5OpXryRIuJj91q+F3FQqUiMSThbrB5psDkJFMi3/LVrvhu4rnUtUPJumEVtYhNxjtLcm6QObEiwFiYeQ0uzJKVYIC5cLe3J3bmwBi1jE2EIumYSzcYeRprYqXiDFNCiKfjSNCcWy1gZd1KIHXB20Z+MGqavYu6c4paZrh+l+wCuYnfQ/5+JnqHBs+mq66twb3BN97HIWbqbPiIluHnd0O7kQhziobdJrqDfxHPDo3fqIJkt2noHI08OHxAl/mwWWfPwZbB6tvDO2jSvQ5GTxdgUFsulNxI+aI7W6W7v6l8wJpNKltrfUkQwXPFke/wLCNHG6xSeZyZwMK9Hoyo4bPSvQ6cwDAyuzW1zhAKmrWqcCm0UrIOmxkUGRj6q+IyhHSbbtI0vlBc5uFQSHg+4jM0evvRuRy8uHc4enj48Jf7g8c/rB+k68mXyVfJ3WSUfJs8Tn5KniUnCUv+693sfd77oq/6f/b/6v/dSa/01p7Pkq3V/+cNwNsNhw==</latexit> ⇡ arwi b L(wk , 0) arwi ∂h b L(wk , 0)[hk] <latexit sha1_base64="I9O5rqdPtNZTLxsrXRBp/iP9u4U=">AAAF53icjVRdjxs1FJ2UBpbwtVseeXHJrlSkdJUEKLysVIoqeKhQ2e5uI9W7kcdzJ2PisY3tSRqc+Q28IV75WbzyS7ieZFabLki1FM3Nuedc3w/rpkYK54fDvzt33rnbffe9vfd7H3z40cef7B/cu3C6shzOuZbaTlLmQAoF5154CRNjgZWphJfp/Pvof7kA64RWZ35l4LJkMyVywZlHaLq/piXzRZqSp+SEDAeE/lqxjDSgLcMFszXilElTsKsxoU7MysZYfzed0/V0fDUekN5GhEjkKpZKRu9Pw/JK1Nu/UwqeEboUGRTMk2cPltP5YPjFdL8/PB42h9w2Rlujn2zP8+nB3Rc007wqQXkumXOvRkPjLwOzXnAJdY9WDgzjczaD0DSnJkcIZSTXFn/Kkwbd4ZXMbKg7KKiqFB7KXZSVzq3KdIDfGM1h+Ngq9yYpgoO0/E93IzTA66Mmqx+ElOQFU0jKIGeV9JGQA/OVBReeidnGOjmDCcYjXpOCLYD4AgirvMYLBHdEtjyic4LUHnXgLXpVDBfORImun2BJTiNWN26Htzbe6xwQt6BgyXWJrCzQnJVCrraJ1YG6vLVv1vRK6SdPjLwMsVrDftMxUaVJNAnOn89TzWxGUi2zHV3l828vg1Cm8qD4Zlh5JWOR8bWSTFjgXq4I4xxnXjGPs+QFs4x7fNW7E3vtmbV6GYvAJOlEKN8fheYt80ILDj1CSKCT6KCZcEaylfMrCdSjtrFCf1TXRzd51y7quBXm/2g3nK39FvS3E9H7FDVtVZuy+uP+lyHEIab69fCEFvgJh1hudOCcouCwjmrUL2LvwIaWhZTDuqZzsOrh8dd0mQ3rbfSMuQKloWnew7p3tIEh0AWzYJyQWrVUsdj0VpvQ7gvE6jaUq4y5RYjgNeM0hHb7nLZYqbM4Mwm5X/dH1IpZ4dfok4AjcztT2wgwoVnDwkdhtFYVXmZBpszSX7RQaNJdd93rUcOsUBk2heD66OEaGr25dG4bF+Pj0aPjRz9/1X/8dLuQ9pLPks+TB8ko+SZ5nPyYPE/OE57809nrHHTudUX39+4f3T831DudrebTZOd0//oXsQEJEQ==</latexit> E = 0, Var = a2 s2kAkk2 2 , Ak = rwi rh b L(wk , 0) <latexit sha1_base64="2ikxvJu2am14lBK63p63rQeJDX0=">AAAF4XicjVRRbxtFED6HGopbIIVHXrY4kYqURHaAwkukqlDBQ4VKmqSRuqk1tzfnW7y3e93ds2O29wN4Q7zyy5D4McyefVHcgNSVrBt93zezMzvjSSslnR+N/u5tvXer//4Htz8c3Ln70cefbN/79MyZ2go8FUYZe56CQyU1nnrpFZ5XFqFMFb5IZ99H/sUcrZNGn/hlhRclTLXMpQBP0GR7zkvwRZqyJ+yI7fPcguCgqgLYIeNOTkt4RYaGVAHj9ydh8Uo2jP+AysOEoydwITMswLOnDxaT2d7oS7bH+OvXNWSsDW3LcAa2oejc46UPrgSlmsn2cHQwag+7aYzXxjBZn2eTe7ee88yIukTthQLnXo5Hlb8IYL0UCpsBrx1WIGYwxdA+S8N2CcpYbiz9tGctuqEroVpJN1DUdSk9lpsolM4ty3SPvjGao/CxPPe2KIJ7afmfdOtYoWh226x+lEqx56BJlGEOtfJRkCP42qILT+V0ZR2d4DnFY96wAubIfIEMam/oAikcU52OmZyRdMAdekusjuHCiSyJ+hkX7DhiTUs7urVlr3Ig3KLGhTAlqbLAcyilWq4TawJ3eWdfr+mlNo8fV+oixGor+M3ERLVh0WQ0NGKWGrAZS43KNvxqn393EaSuao9arJqV1yoWGeeUZdKi8GrJQAjqeQ2eeikKoAH1NM+bHbv0YK1ZxCIoSX4utR+OQzt/ojBS4IAxFvh5JHgmXaVg6fxSYTuTrRWG46bZva67orgTVlb/J7tGdvY7yN/Nid/n5NNVtSpreDj8KoTYxNRcjo54QZ+wQ+VGgvoUHXaa6E3+8/h2aEOnIslO0/AZWr1/8A1fZKNmHT0DV5BraB9vvxnsrmAMfA4WKyeV0Z1Uzldva6rQ/ccJa7pQrq6qG4IIXimOQ+j2znGHlSaLPVOY+zfDMbdyWvg3xCmklrmNrq0cKKFpq6KhqIzRNV1mUaVg+a9GajL5Jt0MBrwCK3VGj8JofQxoDY3fXjo3jbPDg/HDg4e/fD189GS9kG4nnydfJA+ScfJt8ij5KXmWnCYi+ae31bvTu9sX/d/7f/T/XEm3emufz5KN0//rXyeaCHo=</latexit> E = a 2 s2rwi Dh b L(wk , 0), Var = small <latexit sha1_base64="PsuLZ8VdIPvFZaswEUL05jgs28w=">AAAFu3icjVRfbxw1EN+WHpTlXwqPvBguqXhIT3uXNikPkaqiCh4QCmmSnlSHyOudvXXjtS3be+nh7tfiuyDxCp+D8e5tyAGVaum0o9+f8YzH59xI4XyW/X7r9nt3Ru9/cPfD9KOPP/n0s617n5853VgOp1xLbec5cyCFglMvvIS5scDqXMKL/PK7yL9YgnVCqxO/MnBes4USpeDMI3SxdXSfMmkqRgj1mmSUpvdpzXyV5+QZoeDZxSU5vIHbOpwx29Ldf0jqxKJmv8z6DBdb42zy7eP92aMZySZZdjDb24/B7ODhbI9MEYlrnKzX0cW9O89poXlTg/JcMudeTjPjzwOzXnAJbUobB4bxS7aA0DXckh2EClJqiz/lSYdu6GpmeukGCqqphYd6E2W1c6s638VvzOYwfezU/VsUwd28/l+6Mxrg7U5X1fdCSvKcKRQVULJG+igogfnGggs/ikUfHZ7AHPMRPLiKLYH4CghrvMYNBHdEDjqiS4LSlDrwFlkV04UTUSP1E1yR44i1He1w1469rgFxCwquuK5RVQRaslrI1bqwNlBXDvHNnl4q/fSpkechdmvYrzoWqjSJIcklinLNbEFyLYsNX+PLx+dBKNN4ULwfVtnI2GS8gaQQFriXK8I4x5k3zOMsecUs4x5v6ubEXntmrb6KTWCRdC6UH09DdxV5pQWHlBAS6DwStBDOSLZyfiWBevR2URhP23bnpu6aoo5bYd4mu0EO8TvI381Ev6LoGbrq2xrPxnshxCHm+nV2SCv8hG1sNxI4p2jYbqMb/ct4dmDDoELJdtvSS7DqweQRvSqydp29YK5Ca+gO70Gb7vQwBLpkFowTUqtBKpb92WoThr87Yu2QyjXG/EcQwWvFcQjD+3E8YLUu4swklP7NeEqtWFT+DXIScGRuY2q9AQtadCq8FEZr1eBmFmTOLH2lhcKQbtJtmlLDrFAFHgrB5yPFZ2h4a8jbg7PZZLo/2f/54fjJs/WDdDf5Mvk6+SaZJgfJk+SH5Cg5TXjyW/JH8mfy1+hwxEevRrKX3r619nyRbKxR8zdN1vrM</latexit> a ! 0 Ehk = 0 Var hk = s2 ! 0 leading order <latexit sha1_base64="4eCcIZWwQxyB/tiHUSqBEi26akw=">AAAFi3icjVRfjxM3EDeUFEhLOcojL4bcSVTiTgm0FFBPOlFV9AFV9LiDSPiIvN7ZrBuvbdnehNTsaz9NX9vvwrdhvJucLlwrYWm1o9+f8YzHcmaV9GE4/HDh4heXel9evnK1/9XX1765vnXj21fe1E7AsTDKuHHGPSip4TjIoGBsHfAqU/A6m/2c+NdzcF4afRSWFk4qPtWykIIHhCZbdJ/uMq5sySnTPFOc3Z7ExVvZ0Od3F5PZd3SyNRjuDdtFzwejVTA4eEza9WJy49JLlhtRV6CDUNz7N6OhDSeRuyCFgqbPag+WixmfQmwbaOgOQjktjMNPB9qiG7qK2066gYKuKxmg2kR55f2yyu7hP2XzmL7iofSfihJ4L6v+k26NFkSz01b1TCpFX3KNohwKXquQBAXwUDvw8bmcdtH+EYwxHw2GlnwONJRAeR0MbiCFp2qto6agKO0zD8Ehq1O6eCQrpH6DBT1MWNPSHndt2dMaEHegYSFMhao8soJXUi1XhTWR+WIdn+3pjTZPn1p1ElO3lv9pUqHa0BRSHLyYZYa7nGZG5Ru+OhSPTqLUtg6gRTesolapyXSjaC4diKCWlAuBM695wFmKkjsuAt68zYm9C9w5s0hNYJFsLHUYjCJLNYnSSAF9Smlk40SwXHqr+NKHpQIW0NtGcTBqmp2zulOKeeGk/T/ZGXIdf4b880zsNkPPuquurcH9wYMY0xAz8264z0r8xW1sNxE4p2TYbpIb/fN0duDiWoWS7aZhM3B6d+8HtsiHzSp7zn2J1tge3m7T3+lgiGzOHVgvldFrqZx3Z2ts93dVRKxZp/K1tecECTxVHMYOzzJ6uMYqk6eZKSjC+8GIOTktw3vkFODI/MbUOgMWNG1VeCmsMbrGzRyojDv2h5EaQ7ZJN/0+s9xJneOhUHw++vgMjT59dM4Hr+7vjR7uPfz9+8HBT917RK6QW+QOuUtG5EdyQH4lL8gxEeQv8jf5h/zbu9Z70HvSW2kvXlh5bpKN1fvlIzzo6fQ=</latexit> = arwi L(wk) <latexit sha1_base64="huaKS7ThmeGsOhKxGm+6GDa0o8c=">AAAFf3icjVRfbxQ3EHcoR+FoaSi88eJyF6UPcNoLKgWJSBE80AeEaEjgJBwFr3f2zsRrW7b3wmH2u/AK34hvw3g3F+VIK2FptaPfn/GMx3JulfQhy76uXfjpYu/Sz5ev9K/+8uu139av//7Km9oJ2BdGGTfJuQclNewHGRRMrANe5Qpe50dPEv96Ds5Lo/fCwsJBxadallLwgNDh+s23b4fb2ZAaTYfsKa8qPtzcPFwfZKOsXfR8MD4JBjsPSbteHF6/+JIVRtQV6CAU9/7NOLPhIHIXpFDQ9FntwXJxxKcQ26IbuoFQQUvj8NOBtuiKruK2k66goOtKBqhWUV55v6jyO/hP2Tymr3iY+e9FCbyTV/9Jt0YLotloq3oqlaIvuUZRASWvVUiCEnioHfj4TE67aHsPJpiPBkNnfA40zIDyOhjcQApP1VJHTUlR2mcegkNWp3RxT1ZIPYdjupuwpqU97tqypzUg7kDDsTA4IV1EVvJKqsVJYU1kvlzGZ3t6o83jx1YdxNSt5R9MKlQbmkKaKxTlhruC5kYVK746lA8OotS2DqBFN6yyVqnJdItoIR2IoBaUC4Ezr3nAWYoZd1wEvG2rE3sfuHPmODWBRbKJ1GEwjizVJGZGCuhTSiObJIIV0lvFFz4sFLCA3jaKg3HTbJzVnVLMCyft/8nOkMv4B+Q/ZmJ/MPQsu+raGmwN7sWYhpib99k2m+EvDrHdROCckmHYJDf65+nswMWlCiXDpmFH4PTd0V/suMiak+wF9zO0xvbw7jb9jQ6GyObcgfVSGb2Uynl3tsZ2f1dFxJplKl9be06QwFPFbuzwPKe7S6wyRZqZgjJ8HIyZk9NZ+IicAhyZX5laZ8CCpq0KL4U1Rte4mQOVc8feGakxZKt00+8zy53UBR4Kxeejj8/Q+PtH53zwams0vj+6/+/WYOdR9x6Ry+QWuU3+JGPyN9kh/5AXZJ8I8oF8Ip/Jl95ab7M36mWd9MLaiecGWVm9h98A1q/kSw==</latexit> “= 0 on G” <latexit sha1_base64="GDFdex3JZQXfuOpeVeMRYxKOUW0=">AAAF3nicjVRPbxQ3FJ9QloYppaE99mK6iQRqEu2mLe0FCVFV9IAqGhJYKV62Hs+bHXc9trE9u2zNXHuruPar8Wna55ndKEtAYqSRn35/nt/zs5wZKZwfDN5uXfnkau/ap9vX089ufH7zi51bXz5zurYcTrmW2o4y5kAKBadeeAkjY4FVmYTn2eznyD+fg3VCqxO/NDCu2FSJQnDmEZrsOJrJGkgIDTkgtLCMUyZNyY4IVSyTjN6ehMUL0RBqmPWCyRdHEwqeEboQOZTMk8d3FpPZ/uDuWYQx6pZxSl++rFlOVsu3hPJce0fSJp3s9AeHg/Yjl4PhKugnq+/J5NbVpzTXvK5AeS6Zc2fDgfHjECviEpqU1g4M4zM2hdCeSUP2EMpJoS3+ypMW3dBVzHTSDRRUXQkP1SbKKueWVbaPa8zmMH3FfOneFUVwP6veS7dGA7zZa6t6JKQkT5lCUQ4Fq6WPggKYry248FhMu+j+CYwwH/GalGwOxJdAWO01biC4I3KtI7ogKE2pA2+RVTFdOBEVUr/BghxHrGlph7u27HkNiFtQsOC6QlUeaMEqIZerwppAXbGOL/Z0pvTDh0aOQ+zWsL90LFRpEkOCl4fPMs1sTjIt8w1f7YufxkEoU3tQvBtWUcvYZLykJBcWuJdLwjjHmdfM4yx5yfB6erzMmxN75Zm1ehGbwCLpSCjfHwYaa+KlFhxSQkigo0jQXDgj2dL5pQTq0dtGoT9smr2LunOKOm6F+ZDsArmOP0L+cSZ6m6Jn3VXXVv+o/10IcYiZfjW4T0tcwi62GwmcUzTsNtGN/nk8O7BhrULJbtPQGVh1cPgDXeSDZpU9Z65Ea2gP76BJ9zoYAp0zC8YJqdVaKubd2WrTrbYKiDXrVK425pIggueK49DhWUaO11il8zgzCYV/3R9SK6alf42cBByZ25haZ8CCpq0KL4XRWtW4mQWZMUv/1EJhSDfpJk3jEyZUjodC8PmIz9Dw3UfncvDs6HB47/De79/3H/yyepC2k6+Tb5I7yTD5MXmQ/Jo8SU4TnrxN/tva3rre+6P3d++f3ptOemVr5fkq2fh6//4PuFEIVg==</latexit> a 2 rwi ∂2 h b L(wk , 0)[hk , hk] + · · ·
  10. <latexit sha1_base64="h8F/D9RYSTmhPgmmJ/9OMRM+MQM=">AAAF7XicjVRPjxs1FJ+0DZQplG05gMTFJbtSkXZXSWkLl0pVEYJDhcp2t420TqM3njcZE49tbE+ywZ0L34Eb4spn4hvwMWrPJKtNl0q1NJqn35/n9/wsZ1pw64bDf3tXrl7rf/Dh9Y/SGx9/cvPTnVu3X1hVG4YnTAllxhlYFFziieNO4FgbhCoT+DKbfx/5lws0lit57FYaJxXMJC84Axeg6c4fNBM1Eu8bClobdUYOKAhdAqESMgH0ztQvX/GG0CXPsQRHnt5dTuf7w69T+lsNOTkg79BrMI6DmFJ0cNl9GuHpfJI26XRnMDwctotcDkbrYJCs17PprWvPaa5YXaF0TIC1p6OhdhMfN2QCm5TWFjWwOczQt0fUkL0A5aRQJnzSkRbd0lWgO+kWirKuuMNqG4XK2lWV7Yd/zGZD+gpcad8WRXA/q/6Xbo0aWbPXVvUjF4I8BxlEORZQCxcFBYKrDVr/lM+66NExjkM+4hQpYYHElUigdipswJklYqMjqiBBmlKLzgRWxnT+mFeB+hmX5ChiTUvbsGvLntcQcIMSl0xVQZV7WkDFxWpdWOOpLTbxxZ5OpXryRIuJj91q+F3FQqUiMSThbrB5psDkJFMi3/LVrvhu4rnUtUPJumEVtYhNxjtLcm6QObEiwFiYeQ0uzJKVYIC5cLe3J3bmwBi1jE2EIumYSzcYeRprYqXiDFNCiKfjSNCcWy1gZd1KIHXB20Z+MGqavYu6c4paZrh+l+wCuYnfQ/5+JnqHBs+mq66twb3BN97HIWbqbPiIluHnd0O7kQhziobdJrqDfxHPDo3fqIJkt2noHI08OHxAl/mwWWfPwZbB6tvDO2jSvQ5GTxdgUFsulNxI+aI7W6W7v6l8wJpNKltrfUkQwXPFke/wLCNHG6xSeZyZwMK9Hoyo4bPSvQ6cwDAyuzW1zhAKmrWqcCm0UrIOmxkUGRj6q+IyhHSbbtI0vlBc5uFQSHg+4jM0evvRuRy8uHc4enj48Jf7g8c/rB+k68mXyVfJ3WSUfJs8Tn5KniUnCUv+693sfd77oq/6f/b/6v/dSa/01p7Pkq3V/+cNwNsNhw==</latexit> ⇡ arwi b L(wk , 0) arwi ∂h

    b L(wk , 0)[hk] <latexit sha1_base64="I9O5rqdPtNZTLxsrXRBp/iP9u4U=">AAAF53icjVRdjxs1FJ2UBpbwtVseeXHJrlSkdJUEKLysVIoqeKhQ2e5uI9W7kcdzJ2PisY3tSRqc+Q28IV75WbzyS7ieZFabLki1FM3Nuedc3w/rpkYK54fDvzt33rnbffe9vfd7H3z40cef7B/cu3C6shzOuZbaTlLmQAoF5154CRNjgZWphJfp/Pvof7kA64RWZ35l4LJkMyVywZlHaLq/piXzRZqSp+SEDAeE/lqxjDSgLcMFszXilElTsKsxoU7MysZYfzed0/V0fDUekN5GhEjkKpZKRu9Pw/JK1Nu/UwqeEboUGRTMk2cPltP5YPjFdL8/PB42h9w2Rlujn2zP8+nB3Rc007wqQXkumXOvRkPjLwOzXnAJdY9WDgzjczaD0DSnJkcIZSTXFn/Kkwbd4ZXMbKg7KKiqFB7KXZSVzq3KdIDfGM1h+Ngq9yYpgoO0/E93IzTA66Mmqx+ElOQFU0jKIGeV9JGQA/OVBReeidnGOjmDCcYjXpOCLYD4AgirvMYLBHdEtjyic4LUHnXgLXpVDBfORImun2BJTiNWN26Htzbe6xwQt6BgyXWJrCzQnJVCrraJ1YG6vLVv1vRK6SdPjLwMsVrDftMxUaVJNAnOn89TzWxGUi2zHV3l828vg1Cm8qD4Zlh5JWOR8bWSTFjgXq4I4xxnXjGPs+QFs4x7fNW7E3vtmbV6GYvAJOlEKN8fheYt80ILDj1CSKCT6KCZcEaylfMrCdSjtrFCf1TXRzd51y7quBXm/2g3nK39FvS3E9H7FDVtVZuy+uP+lyHEIab69fCEFvgJh1hudOCcouCwjmrUL2LvwIaWhZTDuqZzsOrh8dd0mQ3rbfSMuQKloWnew7p3tIEh0AWzYJyQWrVUsdj0VpvQ7gvE6jaUq4y5RYjgNeM0hHb7nLZYqbM4Mwm5X/dH1IpZ4dfok4AjcztT2wgwoVnDwkdhtFYVXmZBpszSX7RQaNJdd93rUcOsUBk2heD66OEaGr25dG4bF+Pj0aPjRz9/1X/8dLuQ9pLPks+TB8ko+SZ5nPyYPE/OE57809nrHHTudUX39+4f3T831DudrebTZOd0//oXsQEJEQ==</latexit> E = 0, Var = a2 s2kAkk2 2 , Ak = rwi rh b L(wk , 0) <latexit sha1_base64="2ikxvJu2am14lBK63p63rQeJDX0=">AAAF4XicjVRRbxtFED6HGopbIIVHXrY4kYqURHaAwkukqlDBQ4VKmqSRuqk1tzfnW7y3e93ds2O29wN4Q7zyy5D4McyefVHcgNSVrBt93zezMzvjSSslnR+N/u5tvXer//4Htz8c3Ln70cefbN/79MyZ2go8FUYZe56CQyU1nnrpFZ5XFqFMFb5IZ99H/sUcrZNGn/hlhRclTLXMpQBP0GR7zkvwRZqyJ+yI7fPcguCgqgLYIeNOTkt4RYaGVAHj9ydh8Uo2jP+AysOEoydwITMswLOnDxaT2d7oS7bH+OvXNWSsDW3LcAa2oejc46UPrgSlmsn2cHQwag+7aYzXxjBZn2eTe7ee88yIukTthQLnXo5Hlb8IYL0UCpsBrx1WIGYwxdA+S8N2CcpYbiz9tGctuqEroVpJN1DUdSk9lpsolM4ty3SPvjGao/CxPPe2KIJ7afmfdOtYoWh226x+lEqx56BJlGEOtfJRkCP42qILT+V0ZR2d4DnFY96wAubIfIEMam/oAikcU52OmZyRdMAdekusjuHCiSyJ+hkX7DhiTUs7urVlr3Ig3KLGhTAlqbLAcyilWq4TawJ3eWdfr+mlNo8fV+oixGor+M3ERLVh0WQ0NGKWGrAZS43KNvxqn393EaSuao9arJqV1yoWGeeUZdKi8GrJQAjqeQ2eeikKoAH1NM+bHbv0YK1ZxCIoSX4utR+OQzt/ojBS4IAxFvh5JHgmXaVg6fxSYTuTrRWG46bZva67orgTVlb/J7tGdvY7yN/Nid/n5NNVtSpreDj8KoTYxNRcjo54QZ+wQ+VGgvoUHXaa6E3+8/h2aEOnIslO0/AZWr1/8A1fZKNmHT0DV5BraB9vvxnsrmAMfA4WKyeV0Z1Uzldva6rQ/ccJa7pQrq6qG4IIXimOQ+j2znGHlSaLPVOY+zfDMbdyWvg3xCmklrmNrq0cKKFpq6KhqIzRNV1mUaVg+a9GajL5Jt0MBrwCK3VGj8JofQxoDY3fXjo3jbPDg/HDg4e/fD189GS9kG4nnydfJA+ScfJt8ij5KXmWnCYi+ae31bvTu9sX/d/7f/T/XEm3emufz5KN0//rXyeaCHo=</latexit> E = a 2 s2rwi Dh b L(wk , 0), Var = small Var hk = s ! 0 leading order <latexit sha1_base64="4eCcIZWwQxyB/tiHUSqBEi26akw=">AAAFi3icjVRfjxM3EDeUFEhLOcojL4bcSVTiTgm0FFBPOlFV9AFV9LiDSPiIvN7ZrBuvbdnehNTsaz9NX9vvwrdhvJucLlwrYWm1o9+f8YzHcmaV9GE4/HDh4heXel9evnK1/9XX1765vnXj21fe1E7AsTDKuHHGPSip4TjIoGBsHfAqU/A6m/2c+NdzcF4afRSWFk4qPtWykIIHhCZbdJ/uMq5sySnTPFOc3Z7ExVvZ0Od3F5PZd3SyNRjuDdtFzwejVTA4eEza9WJy49JLlhtRV6CDUNz7N6OhDSeRuyCFgqbPag+WixmfQmwbaOgOQjktjMNPB9qiG7qK2066gYKuKxmg2kR55f2yyu7hP2XzmL7iofSfihJ4L6v+k26NFkSz01b1TCpFX3KNohwKXquQBAXwUDvw8bmcdtH+EYwxHw2GlnwONJRAeR0MbiCFp2qto6agKO0zD8Ehq1O6eCQrpH6DBT1MWNPSHndt2dMaEHegYSFMhao8soJXUi1XhTWR+WIdn+3pjTZPn1p1ElO3lv9pUqHa0BRSHLyYZYa7nGZG5Ru+OhSPTqLUtg6gRTesolapyXSjaC4diKCWlAuBM695wFmKkjsuAt68zYm9C9w5s0hNYJFsLHUYjCJLNYnSSAF9Smlk40SwXHqr+NKHpQIW0NtGcTBqmp2zulOKeeGk/T/ZGXIdf4b880zsNkPPuquurcH9wYMY0xAz8264z0r8xW1sNxE4p2TYbpIb/fN0duDiWoWS7aZhM3B6d+8HtsiHzSp7zn2J1tge3m7T3+lgiGzOHVgvldFrqZx3Z2ts93dVRKxZp/K1tecECTxVHMYOzzJ6uMYqk6eZKSjC+8GIOTktw3vkFODI/MbUOgMWNG1VeCmsMbrGzRyojDv2h5EaQ7ZJN/0+s9xJneOhUHw++vgMjT59dM4Hr+7vjR7uPfz9+8HBT917RK6QW+QOuUtG5EdyQH4lL8gxEeQv8jf5h/zbu9Z70HvSW2kvXlh5bpKN1fvlIzzo6fQ=</latexit> = arwi L(wk) <latexit sha1_base64="huaKS7ThmeGsOhKxGm+6GDa0o8c=">AAAFf3icjVRfbxQ3EHcoR+FoaSi88eJyF6UPcNoLKgWJSBE80AeEaEjgJBwFr3f2zsRrW7b3wmH2u/AK34hvw3g3F+VIK2FptaPfn/GMx3JulfQhy76uXfjpYu/Sz5ev9K/+8uu139av//7Km9oJ2BdGGTfJuQclNewHGRRMrANe5Qpe50dPEv96Ds5Lo/fCwsJBxadallLwgNDh+s23b4fb2ZAaTYfsKa8qPtzcPFwfZKOsXfR8MD4JBjsPSbteHF6/+JIVRtQV6CAU9/7NOLPhIHIXpFDQ9FntwXJxxKcQ26IbuoFQQUvj8NOBtuiKruK2k66goOtKBqhWUV55v6jyO/hP2Tymr3iY+e9FCbyTV/9Jt0YLotloq3oqlaIvuUZRASWvVUiCEnioHfj4TE67aHsPJpiPBkNnfA40zIDyOhjcQApP1VJHTUlR2mcegkNWp3RxT1ZIPYdjupuwpqU97tqypzUg7kDDsTA4IV1EVvJKqsVJYU1kvlzGZ3t6o83jx1YdxNSt5R9MKlQbmkKaKxTlhruC5kYVK746lA8OotS2DqBFN6yyVqnJdItoIR2IoBaUC4Ezr3nAWYoZd1wEvG2rE3sfuHPmODWBRbKJ1GEwjizVJGZGCuhTSiObJIIV0lvFFz4sFLCA3jaKg3HTbJzVnVLMCyft/8nOkMv4B+Q/ZmJ/MPQsu+raGmwN7sWYhpib99k2m+EvDrHdROCckmHYJDf65+nswMWlCiXDpmFH4PTd0V/suMiak+wF9zO0xvbw7jb9jQ6GyObcgfVSGb2Uynl3tsZ2f1dFxJplKl9be06QwFPFbuzwPKe7S6wyRZqZgjJ8HIyZk9NZ+IicAhyZX5laZ8CCpq0KL4U1Rte4mQOVc8feGakxZKt00+8zy53UBR4Kxeejj8/Q+PtH53zwams0vj+6/+/WYOdR9x6Ry+QWuU3+JGPyN9kh/5AXZJ8I8oF8Ip/Jl95ab7M36mWd9MLaiecGWVm9h98A1q/kSw==</latexit> “= 0 on G” <latexit sha1_base64="GDFdex3JZQXfuOpeVeMRYxKOUW0=">AAAF3nicjVRPbxQ3FJ9QloYppaE99mK6iQRqEu2mLe0FCVFV9IAqGhJYKV62Hs+bHXc9trE9u2zNXHuruPar8Wna55ndKEtAYqSRn35/nt/zs5wZKZwfDN5uXfnkau/ap9vX089ufH7zi51bXz5zurYcTrmW2o4y5kAKBadeeAkjY4FVmYTn2eznyD+fg3VCqxO/NDCu2FSJQnDmEZrsOJrJGkgIDTkgtLCMUyZNyY4IVSyTjN6ehMUL0RBqmPWCyRdHEwqeEboQOZTMk8d3FpPZ/uDuWYQx6pZxSl++rFlOVsu3hPJce0fSJp3s9AeHg/Yjl4PhKugnq+/J5NbVpzTXvK5AeS6Zc2fDgfHjECviEpqU1g4M4zM2hdCeSUP2EMpJoS3+ypMW3dBVzHTSDRRUXQkP1SbKKueWVbaPa8zmMH3FfOneFUVwP6veS7dGA7zZa6t6JKQkT5lCUQ4Fq6WPggKYry248FhMu+j+CYwwH/GalGwOxJdAWO01biC4I3KtI7ogKE2pA2+RVTFdOBEVUr/BghxHrGlph7u27HkNiFtQsOC6QlUeaMEqIZerwppAXbGOL/Z0pvTDh0aOQ+zWsL90LFRpEkOCl4fPMs1sTjIt8w1f7YufxkEoU3tQvBtWUcvYZLykJBcWuJdLwjjHmdfM4yx5yfB6erzMmxN75Zm1ehGbwCLpSCjfHwYaa+KlFhxSQkigo0jQXDgj2dL5pQTq0dtGoT9smr2LunOKOm6F+ZDsArmOP0L+cSZ6m6Jn3VXXVv+o/10IcYiZfjW4T0tcwi62GwmcUzTsNtGN/nk8O7BhrULJbtPQGVh1cPgDXeSDZpU9Z65Ea2gP76BJ9zoYAp0zC8YJqdVaKubd2WrTrbYKiDXrVK425pIggueK49DhWUaO11il8zgzCYV/3R9SK6alf42cBByZ25haZ8CCpq0KL4XRWtW4mQWZMUv/1EJhSDfpJk3jEyZUjodC8PmIz9Dw3UfncvDs6HB47/De79/3H/yyepC2k6+Tb5I7yTD5MXmQ/Jo8SU4TnrxN/tva3rre+6P3d++f3ptOemVr5fkq2fh6//4PuFEIVg==</latexit> a 2 rwi ∂2 h b L(wk , 0)[hk , hk] + · · · <latexit sha1_base64="NCgqXr1kyXacp4Wek19zk/Owdpo=">AAAGwHicjVTdbhtFFF6XEsry18AlNwfsVImURHYQBSGCQqhULioU0qSJlAnW7Oyxd/DszGZm1o673SfjSbjkFl6CM7t2iJsidaTVHn3nO/8zJymUdL7f/7Nz753776699+D9+IMPP/r4k4frn75wprQCT4VRxp4n3KGSGk+99ArPC4s8TxSeJZOfgv5sitZJo0/8vMDLnI+1HEnBPUHD9c4Js5hWzOO1T0bVSZbXNbCrkqcQ/+jByxzBCa4Qen44ge/2gSWqRGBcFRlnTo5z/tseTHrbwKA364EwmuKN0YE30DsLOItnmRQZOKOm6GJ2EbMEx1JXXMmxxrSOH7GCWy+5Gno42/RbsA87wHLuM5vD0ZA95XnOgWmeKD6sZpThdptHtSBVxziu681gu9VEfPRG/WyrqWBkuRjsAXuCyvMhQ0++ZzLFjHt4tjnb7m9BzFCn/yXIrpqWLH4hDJMalnmRR2O5UuB/6MfsMo6HD7v93X5z4K4wWAjdgydRc46G6/efs9SIMkftheLOXQz6hb+sQleEQkqgdFhwMeFjrJqp17BBUAoUmT7toUFXeDkvWuoKirrMpcd8FeW5c/M82aZ/8ObIfeice50UwO0kf6O6MSxQ1BtNVk8ldeQ510RKccRL5QNhhNyXFl31TI5baf8Ez8lfuC8ZnyL4DIGX3lAAKRyoJQ/MCIgaM4feklYHd9UJ3VAHv+AMjgNWN2pHURvtTQ6EW9Q4E4YmRpNlI55LNV8kVlfMjZby7ZoutDk8LNRlFaot+EsTEtUGggh0GcUkMdymkBiVrtiVfvTtZSV1UXrUoh3WqFShyPAMIZUWhVdz4ELQzEvuaZYi43QzPT3X1Ylde26tmYUiKEl2LrXvDtqrLTIjBcYAULHzoGCpdIXic+fnCptn3UhVd1DXG7d5NyrmhJXF/9FuKZfyW9Dfzoh9Qa/IL6tqy+rudb+qqjDExFz391lGv6pH5QYFzSkY9OpgTfbT0Du01ZJFlF5dswlavbP7NZul/XrhPeUuI9Oqad5OHW+0MC2IKbdYOKmMXlLltO2tKW7WB2H10pUri+IOIYA3jOOqxZMEjpdYbtIwM4Uj/6o7YFaOM/+KdAppZG5laq0BJTRuWHQpCmN0ScEsqoRb9ruRmkS2qq7jOKxRqVNqCtD6CGto8PrSuSu82NsdPN59/Ote9+D7dh9FD6LPoy+jzWgQfRMdRD9HR9FpJDp/dP7q/N35Z+1wLVsza1ct9V5nYfNZtHLWXv4LZRNUig==</latexit> Thm At time scale tk := as2k, w converges to W, which solves ∂tW(t) = PGrw Reg(W(t)), Reg(w) := 1 2 Dh b L(w, 0) W(t) 2 G 8t > 0
  11. Example: Dropout (DropConnect) <latexit sha1_base64="2GP4lW/rSROU/YJDZLMxLEvwpPI=">AAAGL3icjVTdbhw1FJ5tu1CGv7RccmPYDSrQRDtBFIQUqQpCcIFQSJM2UEeRx3Nm18Rju7Znt4vrh+IdeAfEDeKWt+B4ZjfKNiDV0miOvvOdz+dPLo0Uzk8mfw5u3Lw1fO3122/kb7719jvvbt25+9jp1nI44Vpqe1oyB1IoOPHCSzg1FlhTSnhSXnyd/E/mYJ3Q6tgvDZw1bKpELTjzCJ1v/X4AVulWSvEVoc9aVpExoZUjxacUPCP7hJYwFSpwvMPFfEI+ItTDcx8WhkRiCDUzprxuQohhJxaRPutEVqR7hlnWgAdLLoSUUH0cKc1pVVvGSRGKHRM3BQsUQdFe5JqGBcdZp5JTUNU6qfH51miyO+kOuW4UK2OUrc7h+Z1bj2ileduA8lwy554WE+PPArNecAko3zowjF+wKYSuyZFsI1SRWlv8lCcdusFrmOmpGyiothEemk2UNc4tm/I+/pOaQ/mG+Zl7mZTA+2Xzn+4u0ACP211W32KDySOmkFRBzVrpE6EG5lvsW/heTHtr/xhOUY94TWZsDsTPgLAWR4gbwR2Rax7RNUFqTh14i16V5MKxaND1AyzIUcJi53Z4a+e9zAFxCwoWXDfIqgKtWSPkcpVYDNTVa/tqTU+VPjgw8iykag37VadElSbJJKVEUqmZrUipZbUR1/r6y7MglGk9KN4Pq25lKjJtPamEBe7lkjDOceYt8zhLPsPN4rhYm33FpWPW6kUqApOkp0L5URFoyonPtOCQE0ICPU0OWglnJFs6v5TQLWxnhVER4/ZV3qWLOm6F+T/aFefafgX6qwXRDyjGrKvqyxrtjT4LIQ2x1M8n+3SGvzDGcpMD55QCxjFFY/w89Q5sWLOQMo6RXuATsrP7OV1Uk7hSr5ibYWjomrcT8+0ehkDnzIJxQmq1pop531tt+r9tAmJxLeVaY64REnjJOAo9XpbkaI01ukozk1D7F6OCWjGd+Rfok4AjcxtT6wMwoWnHwqUwWqsWL7MgS2bpL1ooNOmmO+Y5xadJqAqbQvD5yPEZKl5+dK4bj/d2iwe7D37cGz38ZvUg3c7ezz7M7mVF9kX2MPsuO8xOMj74ZHA4+Gnw8/C34R/Dv4Z/99Qbg1XMe9nGGf7zLx33Jzw=</latexit> Bernoulli: 1 + h =

    8 < : 0 wp p (parameter killed) 1 1 p wp 1 p (parameter rescaled) <latexit sha1_base64="Rn707XvUUMEUdjvlyjXo/bN3tZw=">AAAFpHicjVRLjxw1EO48BsLw2pAjF8PMoiA2q+lFJIAUKQpC4RBFYR/JSPFmVe2unjHrth3bPZvB6d/Cr+EKZ/4N5e6Z1U4WpFhqdel7lKtclgurpA+TyT9Xrl67Pnjv/RsfDD/86ONPPt26+dkzbxon8EgYZdy0AI9KajwKMiicWodQFwqfF6c/Jf75Ap2XRh+GpcXjGmZaVlJAIOhk64dH0HgvQf/I+KsGSjZmvPQs/4ZjAMa9rBmvIcwFKPbkdr5DyKyGl3tfj0+2RpPdSbfY5SBfBaNstZ6e3Lx+wEsjmhp1EAq8f5FPbDiO4IIUCtshbzxaEKcww9h11rJtgkpWGUefDqxDN3Q12F66gaJuahmw3kSh9n5ZFzv0T9k8pU+t+bdFCdwp6v+kO6NF0W53VT2SSrED0CQqsYJGhSSoEELj0MfHctZH9w9xSvlYMGwOC2RhjgyaYGgDKTxTax0zFSPpkHsMjlid0sVDWRP1BM/YfsLajva0a8ee10C4Q41nwtSkKiOvoJZquSqsjdxX6/hiTy+0efjQquOYurXwu0mFasNSyApFosKAK1lhVLnha0L1/XGU2jYBteiHVTUqNZmuGiulQxHUkoEQNPMGAs1SzMGBCHQlNyf2OoBz5iw1QUXyqdRhlMf+7s2NFDhkjEU+TQQvpbcKlj4sFfJA3i6Ko7xtty/qzinuhZP2/2QXyHX8DvJ3M/EvOHnWXfVtjfZG38aYhliY15P7fE6/OKZ2E0FzSoZxm9zkX6SzQxfXKpKM25afotN3dr/jZ+WkXWUvwc/JGrvDu9MOt3sYI1+AQ+ulMnotlYv+bI3t/66OhLXrVL6x9pIggeeK/djjRcH211htyjQzhVV4M8q5k7N5eEOcQhqZ35hab6CCZp2KLoU1Rje0mUNVgOO/Gakp5Jt0OxxyC07qkg6F0fMxpGcof/vRuRw829vN7+7e/XVv9ODn1YN0I/s8+zK7neXZvexB9kv2NDvKRPZH9mf2V/b34KvB48HB4KiXXr2y8tzKNtbg5b+4w/KD</latexit> Gaussian: 1 + h ⇠ N (1, s2) <latexit sha1_base64="m3ClBQRBrlqP+dPkQgzpa+pK8Go=">AAAFhXicjVRLbxMxEF4eoRBeBW5wMSSVONAqW6BwqUAgBAeEoLQQCZfK651NTL22ZXtTglmJX8MV/g7/hvFuUjUUJCytdvQ9xjMey5mRwvnB4NeJk6dOd84snT3XPX/h4qXLy1euvnO6shx2uJbaDjPmQAoFO154CUNjgZWZhPfZ/tPIv5+AdUKrbT81sFuykRKF4MwjtLd8vU+dGJXs4zrZJDQvLOPEhHTV1P295d5gbdAscjxIZ0Evma3Xe1dOv6W55lUJynPJnPuQDozfDcx6wSXUXVo5MIzvsxGEpvSarCCUk0Jb/JQnDbqgK5lppQsoqKoUHspFlJXOTcvsDv5jNofpS+bH7k9RBO9k5V/pxmiA1ytNVc+FlOQtUyjKoWCV9FFQAPOVBRdeilEbbW7DEPMRr8mYTYD4MRBWeY0bCO6InOuILghKu9SBt8iqmC5sixKpV3BAtiJWN7TDXRv2sAbELSg44LpEVR5owUohp7PC6kBdMY+P9vRB6SdPjNwNsVvDvuhYqNIkhiSTKMo0sznJtMwXfJUvHu4GoUzlQfF2WEUlY5PxLpFcWOBeTgnjHGdeMY+z5GOGl8jjnVuc2GfPrNUHsQkskg6F8r000FgTH2vBoUsICXQYCZoLZySbOj+VQD16myj00rpeOao7pKjjVph/yY6Q8/g/5P9nojcpeuZdtW311nt3Q4hDzPTnwSYd4y/0sd1I4JyioV9HN/on8ezAhrkKJf26pvtg1erafXqQD+pZ9py5MVpDc3irdXelhSHQCbNgnJBazaVi0p6tNu3flgGxep7KVcYcE0TwULEVWjzLyNYcK3UeZyah8F97KbViNPZfkZOAI3MLU2sNWNCoUeGlMFqrCjezIDNm6SctFIZ0ka67XWqYFSrHQyH4fHTxGUr/fHSOB+/W19KNtY03673Hz2YP0tnkRnIruZ2kyYPkcfIieZ3sJDz5lnxPfiQ/O0ud1c69zkYrPXli5rmWLKzOo98Lvuck</latexit> s2 = p 1 p <latexit sha1_base64="l6cdkZXq3zqff1jK+wFxXO8Jr7w=">AAAFhnicjVRLbxMxEF4egRJeBY5cDEklDhBtigpckKoiBAeEoLQQCZfK651NTL22ZXvTBrMHfg1X+Dn8G8a7SdVQkLC02tH3GM94LGdGCufT9NeZs+fOdy5cXLnUvXzl6rXrqzduvne6shx2uZbajjLmQAoFu154CSNjgZWZhA/ZwbPIf5iCdUKrHT8zsFeysRKF4MwjtL96u0+dGJfs0zqhXpOUUFEUxGCc9vdXe+kgbRY5HQznQS+Zrzf7N86/o7nmVQnKc8mc+zhMjd8LzHrBJdRdWjkwjB+wMYSm9pqsIZSTQlv8lCcNuqQrmWmlSyioqhQeymWUlc7Nyuw+/mM2h+lL5ifuT1EE72flX+nGaIDXa01VL4SU5B1TKMqhYJX0UVAA85UFF16JcRs93YER5iN4iBM2BeInQFjlNW4guCNyoSO6ICjtUgfeIqtiurAjSqRewyHZjljd0A53bdjjGhC3oOCQ6xJVeaAFK4WczQurA3XFIj7Z00elt7aM3AuxW8O+6Fio0iSGJJMoyjSzOcm0zJd8lS+e7AWhTOVB8XZYRSVjk/EykVxY4F7OCOMcZ14xj7PkE2YZ93jplid25Jm1+jA2gUXSkVC+Nww01sQnWnDoEkICHUWC5sIZyWbOzyRQj94mCr1hXa+d1B1T1HErzL9kJ8hF/B/y/zPROxQ9i67atnrrvYchxCFm+ih9Sif4C31sNxI4p2jo19GN/mk8O7BhoUJJv67pAVj1YLBBD/O0nmfPmZugNTSH96DurrUwBDplFowTUquFVEzbs9Wm/dsyIFYvUrnKmFOCCB4rtkOLZxnZXmClzuPMJBT+a29IrRhP/FfkJODI3NLUWgMWNG5UeCmM1qrCzSzIjFn6WQuFIV2m626XGmaFyvFQCD4fXXyGhn8+OqeD9+uD4aPBo7frvc3n8wdpJbmd3E3uJcPkcbKZvEzeJLsJT74l35Mfyc/OSmfQ2eg8bqVnz8w9t5Kl1dn8DS47514=</latexit> s2 ! 0 () p ! 0 Hinton et al. 13, Srivastava et al. 14, Wan et al. 13 <latexit sha1_base64="pyqZIGh3MbCfVPR0bpdkivFZZhE=">AAAF3XicjVTdb9s2EFe6es28r3R93As7J4CDJYGVJm26wUDRYdgegqJLk9ZAmRoUdbK4UKRAUnZcVo99K/a6f237Z4YdJTuIuxUoAUPn38fxjkcwKaWwbjD4e+3GJzc7n95a/6z7+RdffvX1xu1vnltdGQ5nXEttRgmzIIWCMyechFFpgBWJhBfJxU+BfzEFY4VWp25ewnnBJkpkgjOH0HjD0JlIIWeOHPdnOxQc2yY/DMkxTcSkP6M61Y704+8bImDbpDskNDOMx08ItVUx9mIY16/wD7Kyn/Uvx+LH94zbu/OxCLzZfrVPxhu9wd7g4OjBYUwwaBYGR/eODg8ekniB9KLFejq+ffMZTTWvClCOS2bty3hQunPPjBNcQt2llYWS8Qs2Ad8cSU22EEpJpg3+lCMNuqIrWNlKV1BQVSEcFKsoK6ydF8kOfkM2i+kL5nL7viiAO0nxv3RjLIHXW01VvwgpyTOmUJRCxirpgiAD5ioD1h+LSRsNT2GE+YjTJGdTIC4HwiqncQPBLZFLHdEZQWmXWnAGWRXS+VNRIPUEZuQkYHVDW9y1Ya9qQNyAghnXBapSTzNWCDlfFFZ7arNlfL2nl0o/flzKcx+6LdlrHQpVmoSQJBJFiWYmJYmW6YqvctnRuReqrBwo3g4rq2RoMtxRkgoD3Mk5YZzjzCvmcJY8Z3jrHN7l1YldOmaMnoUmsEg6Esr1Yk9DTTzXgkOXEOLpKBA0FbaUbG7dXAJ16G0i34vreuu67oqilhtRfkh2jVzGHyH/OBO9S9Gz7Kptq7ffu+d9GGKiLwdDmuPHb2K7gcA5BcNmHdzon4azA+OXKpRs1jW9AKN29w7pLB3Ui+wpszlafXN4u3V3q4XB0ykzUFohtVpKxbQ9W122X1N4xOplKluV5X8EAbxSnPgWTxJyssQKnYaZScjcm15MjZjk7g1yEnBkdmVqrQELmjQqvBSl1qrCzQzIhBn6uxYKQ7pK190uLZkRKsVDIfh8dPEZWr415MPB8/29+P7e/d8Oeo9+XjxI69G30XdRP4qjB9Gj6NfoaXQW8eiv6J+1W2vrnXHnbedd549WemNt4bkTrazOn/8CqmgFVQ==</latexit> b L(w, h) := L w (1 + h) = 1 N N Â i=1 f (xi ; w (1 + h)) yi 2
  12. Example: Dropout (DropConnect) <latexit sha1_base64="WU3ObPMeoEQRhkPASjp8mK3USDw=">AAAFynicjVRfbxw1EN+WHpSjQAqPvBjuIqVSGt2maouEIlUFBA9RVdKkPalOT17v7K0br23Z3tsczr7x1fgQfAZe4Z3x7l2Ua4pUS6ed+/0Zz3gsZ0YK5yeTv27c/OjW4ONPbn86/OzO5198uXX3q5dO15bDCddS22nGHEih4MQLL2FqLLAqk/AqO/sx8q8WYJ3Q6tgvDZxWbK5EITjzCM22ZmOaO/oTSM9mFDwjtBE5lMyTw51md3KPHBBaWMbTZ4S6upoFcZC2b/BPJub0oiFU59oTqlgm2awhxc75TPzQ3OvpN/vj2dZosjfpFrkepKtglKzW89ndWy9ornldgfJcMudepxPjTwOzXnAJ7ZDWDgzjZ2wOoeu/JdsI5aTQFn/Kkw7d0FXM9NINFFRdCQ/VJsoq55ZVtovfmM1h+or50r0riuBuVr2X7owGeLvdVfWLkJK8YApFORSslj4KCmC+tuDCoZj30cExTDEf8ZqUbAHEl0BY7TVuILgjcq0juiAoHVIH3iKrYrpwLCqknkFDjiLWdrTDXTv2sgbELShouK5QlQdasErI5aqwNlBXrOOrPb1W+ulTI09D7Naw33UsVGkSQ4LD52eZZjYnmZb5hq/2xfenQShTe1C8H1ZRy9hkvJAkFxa4l0vCOMeZ18zjLHnJ8NJ5vLibEzv3zFrdxCawSDoVyo/SQGNNvNSCw5AQEug0EjQXzki2dH4pgXr0dlEYpW27fVV3SVHHrTD/J7tCruMPkH+YiX5L0bPuqm9rtD96EEIcYqbPJwe0xE8YY7uRwDlFw7iNbvQv4tmBDWsVSsZtS8/Aqvt7D2mTT9pV9py5Eq2hO7z77XC7hyHQBbNgnJBaraVi0Z+tNv3XVgGxdp3K1cZcE0TwUnEUejzLyNEaq3QeZyah8BejlFoxL/0FchJ8fIuuTq03YEHzToWXwmitatzMgsyYpW+1UBjSTbodDqlhVqgcD4Xg8zHEZyh999G5Hrzc30sf7T36bX/05OfVg3Q7+Sb5LtlJ0uRx8iT5NXmenCQ8+TP5O/kn+XdwOLCD5SD00ps3Vp6vk401+OM/jZkB9g==</latexit> Dh b L(w, 0) =

    1 N N Â i=1 w rw f (xi ; w) 2 <latexit sha1_base64="Rn707XvUUMEUdjvlyjXo/bN3tZw=">AAAFpHicjVRLjxw1EO48BsLw2pAjF8PMoiA2q+lFJIAUKQpC4RBFYR/JSPFmVe2unjHrth3bPZvB6d/Cr+EKZ/4N5e6Z1U4WpFhqdel7lKtclgurpA+TyT9Xrl67Pnjv/RsfDD/86ONPPt26+dkzbxon8EgYZdy0AI9KajwKMiicWodQFwqfF6c/Jf75Ap2XRh+GpcXjGmZaVlJAIOhk64dH0HgvQf/I+KsGSjZmvPQs/4ZjAMa9rBmvIcwFKPbkdr5DyKyGl3tfj0+2RpPdSbfY5SBfBaNstZ6e3Lx+wEsjmhp1EAq8f5FPbDiO4IIUCtshbzxaEKcww9h11rJtgkpWGUefDqxDN3Q12F66gaJuahmw3kSh9n5ZFzv0T9k8pU+t+bdFCdwp6v+kO6NF0W53VT2SSrED0CQqsYJGhSSoEELj0MfHctZH9w9xSvlYMGwOC2RhjgyaYGgDKTxTax0zFSPpkHsMjlid0sVDWRP1BM/YfsLajva0a8ee10C4Q41nwtSkKiOvoJZquSqsjdxX6/hiTy+0efjQquOYurXwu0mFasNSyApFosKAK1lhVLnha0L1/XGU2jYBteiHVTUqNZmuGiulQxHUkoEQNPMGAs1SzMGBCHQlNyf2OoBz5iw1QUXyqdRhlMf+7s2NFDhkjEU+TQQvpbcKlj4sFfJA3i6Ko7xtty/qzinuhZP2/2QXyHX8DvJ3M/EvOHnWXfVtjfZG38aYhliY15P7fE6/OKZ2E0FzSoZxm9zkX6SzQxfXKpKM25afotN3dr/jZ+WkXWUvwc/JGrvDu9MOt3sYI1+AQ+ulMnotlYv+bI3t/66OhLXrVL6x9pIggeeK/djjRcH211htyjQzhVV4M8q5k7N5eEOcQhqZ35hab6CCZp2KLoU1Rje0mUNVgOO/Gakp5Jt0OxxyC07qkg6F0fMxpGcof/vRuRw829vN7+7e/XVv9ODn1YN0I/s8+zK7neXZvexB9kv2NDvKRPZH9mf2V/b34KvB48HB4KiXXr2y8tzKNtbg5b+4w/KD</latexit> Gaussian: 1 + h ⇠ N (1, s2) <latexit sha1_base64="pyqZIGh3MbCfVPR0bpdkivFZZhE=">AAAF3XicjVTdb9s2EFe6es28r3R93As7J4CDJYGVJm26wUDRYdgegqJLk9ZAmRoUdbK4UKRAUnZcVo99K/a6f237Z4YdJTuIuxUoAUPn38fxjkcwKaWwbjD4e+3GJzc7n95a/6z7+RdffvX1xu1vnltdGQ5nXEttRgmzIIWCMyechFFpgBWJhBfJxU+BfzEFY4VWp25ewnnBJkpkgjOH0HjD0JlIIWeOHPdnOxQc2yY/DMkxTcSkP6M61Y704+8bImDbpDskNDOMx08ItVUx9mIY16/wD7Kyn/Uvx+LH94zbu/OxCLzZfrVPxhu9wd7g4OjBYUwwaBYGR/eODg8ekniB9KLFejq+ffMZTTWvClCOS2bty3hQunPPjBNcQt2llYWS8Qs2Ad8cSU22EEpJpg3+lCMNuqIrWNlKV1BQVSEcFKsoK6ydF8kOfkM2i+kL5nL7viiAO0nxv3RjLIHXW01VvwgpyTOmUJRCxirpgiAD5ioD1h+LSRsNT2GE+YjTJGdTIC4HwiqncQPBLZFLHdEZQWmXWnAGWRXS+VNRIPUEZuQkYHVDW9y1Ya9qQNyAghnXBapSTzNWCDlfFFZ7arNlfL2nl0o/flzKcx+6LdlrHQpVmoSQJBJFiWYmJYmW6YqvctnRuReqrBwo3g4rq2RoMtxRkgoD3Mk5YZzjzCvmcJY8Z3jrHN7l1YldOmaMnoUmsEg6Esr1Yk9DTTzXgkOXEOLpKBA0FbaUbG7dXAJ16G0i34vreuu67oqilhtRfkh2jVzGHyH/OBO9S9Gz7Kptq7ffu+d9GGKiLwdDmuPHb2K7gcA5BcNmHdzon4azA+OXKpRs1jW9AKN29w7pLB3Ui+wpszlafXN4u3V3q4XB0ykzUFohtVpKxbQ9W122X1N4xOplKluV5X8EAbxSnPgWTxJyssQKnYaZScjcm15MjZjk7g1yEnBkdmVqrQELmjQqvBSl1qrCzQzIhBn6uxYKQ7pK190uLZkRKsVDIfh8dPEZWr415MPB8/29+P7e/d8Oeo9+XjxI69G30XdRP4qjB9Gj6NfoaXQW8eiv6J+1W2vrnXHnbedd549WemNt4bkTrazOn/8CqmgFVQ==</latexit> b L(w, h) := L w (1 + h) = 1 N N Â i=1 f (xi ; w (1 + h)) yi 2 <latexit sha1_base64="YqByuGEhWoxlwA18ZopV6tAYRW4=">AAAGHHicjVTdbtxEFPa2XWjNT9Ny0QtuBjapEimJ1gEKQlqpCohwUaGQJs1KnWg1Hh+vh4xnrJnxbpepX4V34B24Q9wi8TacsXdDtilSR7J89H3fOT4/45NWUlg3HP7Tu3X7Tv+99+/eiz/48KOP7288ePjC6tpwOONaajNOmQUpFJw54SSMKwOsTCWcp5ffBf58BsYKrU7dooKLkk2VyAVnDqHJxu80halQnkkxVZA18WNaMeMEkxNHzrfdDhmRPUJL5gpTkuMJPWJlyQhVLJVs4ucNobuEprIGvxT5E5g2zXbw3UGKxo/fys93yLcjQnPDeHJA6PcgHZtQcBh7LjIomCPPtue7wx0SU1DZfwlONgbD/WF7yE0jWRqDaHmOJw/uPKeZ5nUJynHJrH2ZDCt34UOZXEIT09pCxfglm4JvO9qQLYQykmuDj3KkRdd0Jas66RoKqi6Fg3IdZaW1izLdxXeIZjF8aIV9UxTA3bR8K906VsCbrTarIyElec4UijLIWS1dEOTAXG3A+mdi2lmjUxhjPOI0KdgMiCuAsNpp/IDglsiVjuicoDSmFpxBVoVw/lSUSP0Ec3ISsKalLX61Za9yQNyAgjnXeDVwVDRnpZCLZWKNpzZf2ddreqn04WElL3yotmK/6pCo0iSYBG8Xv0w1MxlJtczW/GqXf3PhhapqB4p3w8prGYoMV5xkwgB3ckEY5zjzmjmcJS8YXjWHv8L6xF45ZoyehyIwSToWyg2S7q7yQgsOMSHE03EgaCZsJdnCuoUE6tC3tfwgaZqt67orilpuRPV/smvkyn4H+bs50c8o+qyq6soaHAy+8D4MMdWvhiNa4MtvYrmBwDkFh80meKP/LPQOjF+pULLZNPQSjNrb/4rOs2GzjJ4xW6Crb5u318RbHYx//IwZqKyQWq2kYtb1VldX+wCxZhXK1lV1QxDAK8WJ7/A0JScrrNRZmJmE3L0eJNSIaeFeIycBR2bXptY5YELTVoWXotJa1fgxAzJlhv6ihUKTrtNNHIe9KFSGTSG4PsIaSt5cOjeNFwf7yZP9Jz9/OXh6uFxId6NPo8+j7SiJvo6eRj9Gx9FZxHuPeqPeD72j/m/9P/p/9v/qpLd6S59PorXT//tfKk0dkw==</latexit> ∂tW(t) = PGrw Reg(W(t)), Reg(w) := 1 2 Dh b L(w, 0) <latexit sha1_base64="K+9xE/vokodOIRoociToAGw7C4o=">AAAFjXicjVRNbxMxEF0+AiV8tIUjF9OkEgdaJUUtHAAhEIIDQqW0EAlXldc7m5h6bcv2JgSzd34NV/gr/BvGu0nVUJCwtPLovTfjGY93UiOF873er3PnL1xsXbq8dKV99dr1G8srqzffO11aDgdcS20HKXMghYIDL7yEgbHAilTCh/T4eeQ/jME6odW+nxo4LNhQiVxw5hE6Wlnr0rSUEnyXUOI4k7Ah1JhZwZQnQpHupHu00ult9upFzhr9mdFJZmv3aPXiO5ppXhagPJfMuY/9nvGHgVkvuISqTUsHhvFjNoRQl1CRdYQykmuLH55cowu6gplGuoCCKgvhoVhEWeHctEjv4R6jOQxfMD9yf4oieC8t/krXjgZ4tV5n9VJISd4xhaIMclZKHwU5MF9acOG1GDbW430YYDziNRmxMRA/AsJKr/EAwR2Rcx3ROUFpmzrwFlkVw4V9USD1BiZkL2JVTTs8tWZPckDcgoIJ1wWqskBzVgg5nSVWBeryuX26po9KP3tm5GGI1Rr2RcdElSbRJKlEUaqZzUiqZbbgV/r84WEQypQeFG+alZcyFhnfFMmEBe7llDDOsecl89hLPmKWcY9vb7Fjnz2zVk9iEZgkHQjlO/1AY058pAWHNiEk0EEkaCackWzq/FQC9ehbW6HTr6r107oTijpuhfmX7BQ5t/9D/n9O9A5Fn3lVTVmdrc79EGITU/2595iOcAtdLDcS2Kfo0K2iN/qP492BDXMVSrpVRY/Bqo3NbTrJetUsesbcCF1DfXkbVXu9gSFQ/HPBOCG1mkvFuLlbbZrdFgGxah7KlcacEUTwRLEXGjxNyd4cK3QWeyYh9187fWrFcOS/IodzhGZuoWuNAyY0rFX4KIzWqsTDLMiUWfpJC4UmXaSrdpsaHEMqw0shOD7aOIb6fw6ds8b7rc3+zubO263O0xezgbSU3E7WkrtJP3mQPE1eJbvJQcKTb8n35Efys7Xc2m49aj1ppOfPzXxuJQur9fI3lRfqnw==</latexit> • scale-invariant in w <latexit sha1_base64="7eWByixdRFv8CJfWC1wWHSodQfs=">AAAFkHicjVTbbhMxEF0ugRJuBR55MWQj8QBVUsTlpeImBEIIQWkhEi6V1zubmHpty/amBLNfwNfwCl/C3zDeTaqGgoSl1Y7OZTzjsZwZKZwfDH4dO37iZOfU6ZUz3bPnzl+4uHrp8junK8thm2up7ShjDqRQsO2FlzAyFliZSXif7T2J/PspWCe02vIzAzslGytRCM48Qrur/ZRmlZTgU0KJYyWQtPi4nhLHGaYcE+ZI+jLdXe0N1gbNIkeD4TzoJfP1evfSybc017wqQXkumXMfhgPjdwKzXnAJdZdWDgzje2wMoemiJn2EclJoi5/ypEGXdCUzrXQJBVWVwkO5jLLSuVmZ3cR/zOYwfcn8xP0piuDNrPwr3RgN8LrfVPVMSEneMoWiHApWSR8FBTBfWXDhpRi30cYWjDAf8ZpM2BSInwBhlde4geCOyIWO6IKgtEsdeIusiunCliiRegX7ZDNidUM73LVhD2pA3IKCfa5LVOWBFqwUcjYvrA7UFYv4cE8flH782MidELs17IuOhSpNYkgyiaJMM5uTTMt8yVf54v5OEMpUHhRvh1VUMjYZrxXJhQXu5YwwznHmFfM4Sz5hlnGP1295Yp89s1bvxyawSDoSyveGgcaa+EQLDl1CSKCjSNBcOCPZzPmZBOrR20ShN6zr/mHdAUUdt8L8S3aIXMT/If8/E71G0bPoqm2rt967HUIcYqY/DzboBH8hxXYjgXOKhrSObvRP49mBDQsVStK6pntg1a21O3Q/H9Tz7DlzE7SG5vBu1d1+C0OgU2bBOCG1WkjFtD1bbdq/LQNi9SKVq4w5IojggWIztHiWkc0FVuo8zkxC4b/2htSK8cR/RQ6fEpq7pam1Bixo3KjwUhitVYWbWZAZs/STFgpDukzX3S41zAqV46EQfD66+AwN/3x0jgbv1teGd9fuvlnvPXw6f5BWkqvJ9eRGMkzuJQ+T58nrZDvhybfke/Ij+dm53LnfedB51EqPH5t7riRLq/PiN0/i6pM=</latexit> • same f2 scaling as L <latexit sha1_base64="rBJZ6Ybmf4/Doe2qDQ8w6DuD0eA=">AAAFmHicjVRbbxM5FB4u4ZJd2AJv8GI2U4kHWiVFXF6QKi5ikRCC0kIk3K08njMZU49t2Z6EYOZX8Gt4ZX8F/4bjmaRqKCsxUuKj73J8jo/lzEjh/HD4/dTpM2d7585fuNj/489Ll/9au3L1rdO15bDHtdR2nDEHUijY88JLGBsLrMokvMsOH0f+3RSsE1rt+rmB/YpNlCgEZx6hg7WNlGa1lOBTQonjTIIjzJF09m/YetKkZCZ8SXIw+J8+SQ/WBsPNYfuRk8FoEQySxffq4MrZNzTXvK5AeS6Zc+9HQ+P3A7NecAlNn9YODOOHbAKh7aYh6wjlpNAWf8qTFl3RVcx00hUUVF0JD9Uqyirn5lV2G9eYzWH6ivnS/SyK4O2s+iXdGg3wZr2t6pmQkrxhCkU5FKyWPgoKYL624MILMemih7swxnzEa1KyKRBfAmG117iB4I7IpY7ogqC0Tx14i6yK6cKuqJB6CTOyE7GmpR3u2rJHNSBuQcGM6wpVeaAFq4ScLwprAnXFMj7e03ulHz0ycj/Ebg37pGOhSpMYkkyiKNPM5iTTMl/x1b54sB+EMrUHxbthFbWMTcbrRXJhgXs5J4xznHnNPM6Sl8wy7vEark7so2fW6llsAoukY6H8YBRorImXWnDoE0ICHUeC5sIZyebOzyVQj942CoNR06wf1x1R1HErzP/JjpHL+Dfkv2eiNyl6ll11bQ22BndCiEPM9MfhQ1riElJsNxI4p2hIm+hG/zSeHdiwVKEkbRp6CFZtbN6ls3zYLLLnzJVoDe3hbTT99Q6GQKfMgnFCarWUiml3ttp0q60CYs0ylauNOSGI4JFiJ3R4lpGdJVbpPM5MQuE/D0bUiknpPyOHTwrN3crUOgMWNGlVeCmM1qrGzSzIjFn6QQuFIV2lm36fGmaFyvFQCD4ffXyGRj8/OieDt1ubo3ub915vDbafLh6kC8mN5O/kVjJK7ifbyT/Jq2Qv4cmX5GvyLfmvd7233XvWe95JT59aeK4lK19v5wfw0+4N</latexit> • scales as w2D with depth D <latexit sha1_base64="iItNWnwcqAbKY9Kft26zGJBY/Lc=">AAAFt3icjVTbbhQ5EG0uw2Vg2QCPvBhmot0HGE0HcXtAQiAEDwiFkMBIOMq63dXTJm7b2O5JBtM/xdfAI/slW+6eiTKElbDUcumcU+W6tCozUjg/Hn8/dfrM2d658xcu9i9d/uPKn2tXr71zurYcdriW2k4y5kAKBTteeAkTY4FVmYT32f6zyL+fgXVCq20/N7BbsakSheDMI7S39mpIs1pK8ENCyZAyY6w+HJJ/LEhUzIAUeCtw7i+iC0KnFkCFTfCf9xmhUJkygCdMjhqykTZ7a4PxaNwectJIF8YgWZzNvatn39Jc87oC5blkzn1Ix8bvBma94BKaPq0dGMb32RRCW2xD1hHKSaEtfsqTFl3RVcx00hUUVF0JD9Uqyirn5lV2G+8YzWH4ivnS/SyK4O2s+iXdOhrgzXqb1QshJXnLFIpyKFgtfRQUwHxtwYVXYtpZj7dhgvGI16Rk2GhfAmG11/iA4I7IpS72HaV96sBbZFUMF7ZFhdRrOCBbEWta2uGrLXuUA+IWFBxwXaEqD7RglZDzRWJNoK5Y2sdr+qD006dG7oZYrWGfdUxUaRJNkkkUZZrZnGRa5it+tS8e7gahTO1B8W5YRS1jkfHvI7mwwL2cE8Y5zrxmHmfJS2YZ9/iXrk7s0DNr9UEsApOkE6H8IA005sRLLTj0CSGBTiJBc+GMZHPn5xKoR9/WCoO0adaP644o6rgV5v9kx8il/Rvy33OiNyn6LKvqyhpsDO6GEIeY6cPxY1riFYZYbiRwTtFh2ERv9J/F3oENSxVKhk1D98GqO6N79CAfN4voOXMluoa2eXea/noHQ6AzZsE4IbVaSsWs66023W2rgFizDOVqY04IInik2AodnmVka4lVOo8zk1D4L4OUWjEt/RfkcOPQ3K1MrXPAhKatCn8Ko7Wq8TFcRhmz9KMWCk26Sjf9PjXMCpVjUwiujz6uofTnpXPSeLcxSu+P7r/ZGDx5vlhIF5Ibya3k7yRNHiRPkpfJZrKT8ORr8i35kfzbe9Tb6xW9spOePrXwuZ6snN6n/wDg0vur</latexit> • ⇡ ‘relative flatness’ of Petzka et al. 21
  13. Example: Dropout (DropConnect) <latexit sha1_base64="2GP4lW/rSROU/YJDZLMxLEvwpPI=">AAAGL3icjVTdbhw1FJ5tu1CGv7RccmPYDSrQRDtBFIQUqQpCcIFQSJM2UEeRx3Nm18Rju7Znt4vrh+IdeAfEDeKWt+B4ZjfKNiDV0miOvvOdz+dPLo0Uzk8mfw5u3Lw1fO3122/kb7719jvvbt25+9jp1nI44Vpqe1oyB1IoOPHCSzg1FlhTSnhSXnyd/E/mYJ3Q6tgvDZw1bKpELTjzCJ1v/X4AVulWSvEVoc9aVpExoZUjxacUPCP7hJYwFSpwvMPFfEI+ItTDcx8WhkRiCDUzprxuQohhJxaRPutEVqR7hlnWgAdLLoSUUH0cKc1pVVvGSRGKHRM3BQsUQdFe5JqGBcdZp5JTUNU6qfH51miyO+kOuW4UK2OUrc7h+Z1bj2ileduA8lwy554WE+PPArNecAko3zowjF+wKYSuyZFsI1SRWlv8lCcdusFrmOmpGyiothEemk2UNc4tm/I+/pOaQ/mG+Zl7mZTA+2Xzn+4u0ACP211W32KDySOmkFRBzVrpE6EG5lvsW/heTHtr/xhOUY94TWZsDsTPgLAWR4gbwR2Rax7RNUFqTh14i16V5MKxaND1AyzIUcJi53Z4a+e9zAFxCwoWXDfIqgKtWSPkcpVYDNTVa/tqTU+VPjgw8iykag37VadElSbJJKVEUqmZrUipZbUR1/r6y7MglGk9KN4Pq25lKjJtPamEBe7lkjDOceYt8zhLPsPN4rhYm33FpWPW6kUqApOkp0L5URFoyonPtOCQE0ICPU0OWglnJFs6v5TQLWxnhVER4/ZV3qWLOm6F+T/aFefafgX6qwXRDyjGrKvqyxrtjT4LIQ2x1M8n+3SGvzDGcpMD55QCxjFFY/w89Q5sWLOQMo6RXuATsrP7OV1Uk7hSr5ibYWjomrcT8+0ehkDnzIJxQmq1pop531tt+r9tAmJxLeVaY64REnjJOAo9XpbkaI01ukozk1D7F6OCWjGd+Rfok4AjcxtT6wMwoWnHwqUwWqsWL7MgS2bpL1ooNOmmO+Y5xadJqAqbQvD5yPEZKl5+dK4bj/d2iwe7D37cGz38ZvUg3c7ezz7M7mVF9kX2MPsuO8xOMj74ZHA4+Gnw8/C34R/Dv4Z/99Qbg1XMe9nGGf7zLx33Jzw=</latexit> Bernoulli: 1 + h =

    8 < : 0 wp p (parameter killed) 1 1 p wp 1 p (parameter rescaled) <latexit sha1_base64="Rn707XvUUMEUdjvlyjXo/bN3tZw=">AAAFpHicjVRLjxw1EO48BsLw2pAjF8PMoiA2q+lFJIAUKQpC4RBFYR/JSPFmVe2unjHrth3bPZvB6d/Cr+EKZ/4N5e6Z1U4WpFhqdel7lKtclgurpA+TyT9Xrl67Pnjv/RsfDD/86ONPPt26+dkzbxon8EgYZdy0AI9KajwKMiicWodQFwqfF6c/Jf75Ap2XRh+GpcXjGmZaVlJAIOhk64dH0HgvQf/I+KsGSjZmvPQs/4ZjAMa9rBmvIcwFKPbkdr5DyKyGl3tfj0+2RpPdSbfY5SBfBaNstZ6e3Lx+wEsjmhp1EAq8f5FPbDiO4IIUCtshbzxaEKcww9h11rJtgkpWGUefDqxDN3Q12F66gaJuahmw3kSh9n5ZFzv0T9k8pU+t+bdFCdwp6v+kO6NF0W53VT2SSrED0CQqsYJGhSSoEELj0MfHctZH9w9xSvlYMGwOC2RhjgyaYGgDKTxTax0zFSPpkHsMjlid0sVDWRP1BM/YfsLajva0a8ee10C4Q41nwtSkKiOvoJZquSqsjdxX6/hiTy+0efjQquOYurXwu0mFasNSyApFosKAK1lhVLnha0L1/XGU2jYBteiHVTUqNZmuGiulQxHUkoEQNPMGAs1SzMGBCHQlNyf2OoBz5iw1QUXyqdRhlMf+7s2NFDhkjEU+TQQvpbcKlj4sFfJA3i6Ko7xtty/qzinuhZP2/2QXyHX8DvJ3M/EvOHnWXfVtjfZG38aYhliY15P7fE6/OKZ2E0FzSoZxm9zkX6SzQxfXKpKM25afotN3dr/jZ+WkXWUvwc/JGrvDu9MOt3sYI1+AQ+ulMnotlYv+bI3t/66OhLXrVL6x9pIggeeK/djjRcH211htyjQzhVV4M8q5k7N5eEOcQhqZ35hab6CCZp2KLoU1Rje0mUNVgOO/Gakp5Jt0OxxyC07qkg6F0fMxpGcof/vRuRw829vN7+7e/XVv9ODn1YN0I/s8+zK7neXZvexB9kv2NDvKRPZH9mf2V/b34KvB48HB4KiXXr2y8tzKNtbg5b+4w/KD</latexit> Gaussian: 1 + h ⇠ N (1, s2) <latexit sha1_base64="m3ClBQRBrlqP+dPkQgzpa+pK8Go=">AAAFhXicjVRLbxMxEF4eoRBeBW5wMSSVONAqW6BwqUAgBAeEoLQQCZfK651NTL22ZXtTglmJX8MV/g7/hvFuUjUUJCytdvQ9xjMey5mRwvnB4NeJk6dOd84snT3XPX/h4qXLy1euvnO6shx2uJbaDjPmQAoFO154CUNjgZWZhPfZ/tPIv5+AdUKrbT81sFuykRKF4MwjtLd8vU+dGJXs4zrZJDQvLOPEhHTV1P295d5gbdAscjxIZ0Evma3Xe1dOv6W55lUJynPJnPuQDozfDcx6wSXUXVo5MIzvsxGEpvSarCCUk0Jb/JQnDbqgK5lppQsoqKoUHspFlJXOTcvsDv5jNofpS+bH7k9RBO9k5V/pxmiA1ytNVc+FlOQtUyjKoWCV9FFQAPOVBRdeilEbbW7DEPMRr8mYTYD4MRBWeY0bCO6InOuILghKu9SBt8iqmC5sixKpV3BAtiJWN7TDXRv2sAbELSg44LpEVR5owUohp7PC6kBdMY+P9vRB6SdPjNwNsVvDvuhYqNIkhiSTKMo0sznJtMwXfJUvHu4GoUzlQfF2WEUlY5PxLpFcWOBeTgnjHGdeMY+z5GOGl8jjnVuc2GfPrNUHsQkskg6F8r000FgTH2vBoUsICXQYCZoLZySbOj+VQD16myj00rpeOao7pKjjVph/yY6Q8/g/5P9nojcpeuZdtW311nt3Q4hDzPTnwSYd4y/0sd1I4JyioV9HN/on8ezAhrkKJf26pvtg1erafXqQD+pZ9py5MVpDc3irdXelhSHQCbNgnJBazaVi0p6tNu3flgGxep7KVcYcE0TwULEVWjzLyNYcK3UeZyah8F97KbViNPZfkZOAI3MLU2sNWNCoUeGlMFqrCjezIDNm6SctFIZ0ka67XWqYFSrHQyH4fHTxGUr/fHSOB+/W19KNtY03673Hz2YP0tnkRnIruZ2kyYPkcfIieZ3sJDz5lnxPfiQ/O0ud1c69zkYrPXli5rmWLKzOo98Lvuck</latexit> s2 = p 1 p <latexit sha1_base64="l6cdkZXq3zqff1jK+wFxXO8Jr7w=">AAAFhnicjVRLbxMxEF4egRJeBY5cDEklDhBtigpckKoiBAeEoLQQCZfK651NTL22ZXvTBrMHfg1X+Dn8G8a7SdVQkLC02tH3GM94LGdGCufT9NeZs+fOdy5cXLnUvXzl6rXrqzduvne6shx2uZbajjLmQAoFu154CSNjgZWZhA/ZwbPIf5iCdUKrHT8zsFeysRKF4MwjtL96u0+dGJfs0zqhXpOUUFEUxGCc9vdXe+kgbRY5HQznQS+Zrzf7N86/o7nmVQnKc8mc+zhMjd8LzHrBJdRdWjkwjB+wMYSm9pqsIZSTQlv8lCcNuqQrmWmlSyioqhQeymWUlc7Nyuw+/mM2h+lL5ifuT1EE72flX+nGaIDXa01VL4SU5B1TKMqhYJX0UVAA85UFF16JcRs93YER5iN4iBM2BeInQFjlNW4guCNyoSO6ICjtUgfeIqtiurAjSqRewyHZjljd0A53bdjjGhC3oOCQ6xJVeaAFK4WczQurA3XFIj7Z00elt7aM3AuxW8O+6Fio0iSGJJMoyjSzOcm0zJd8lS+e7AWhTOVB8XZYRSVjk/EykVxY4F7OCOMcZ14xj7PkE2YZ93jplid25Jm1+jA2gUXSkVC+Nww01sQnWnDoEkICHUWC5sIZyWbOzyRQj94mCr1hXa+d1B1T1HErzL9kJ8hF/B/y/zPROxQ9i67atnrrvYchxCFm+ih9Sif4C31sNxI4p2jo19GN/mk8O7BhoUJJv67pAVj1YLBBD/O0nmfPmZugNTSH96DurrUwBDplFowTUquFVEzbs9Wm/dsyIFYvUrnKmFOCCB4rtkOLZxnZXmClzuPMJBT+a29IrRhP/FfkJODI3NLUWgMWNG5UeCmM1qrCzSzIjFn6WQuFIV2m626XGmaFyvFQCD4fXXyGhn8+OqeD9+uD4aPBo7frvc3n8wdpJbmd3E3uJcPkcbKZvEzeJLsJT74l35Mfyc/OSmfQ2eg8bqVnz8w9t5Kl1dn8DS47514=</latexit> s2 ! 0 () p ! 0 <latexit sha1_base64="pyqZIGh3MbCfVPR0bpdkivFZZhE=">AAAF3XicjVTdb9s2EFe6es28r3R93As7J4CDJYGVJm26wUDRYdgegqJLk9ZAmRoUdbK4UKRAUnZcVo99K/a6f237Z4YdJTuIuxUoAUPn38fxjkcwKaWwbjD4e+3GJzc7n95a/6z7+RdffvX1xu1vnltdGQ5nXEttRgmzIIWCMyechFFpgBWJhBfJxU+BfzEFY4VWp25ewnnBJkpkgjOH0HjD0JlIIWeOHPdnOxQc2yY/DMkxTcSkP6M61Y704+8bImDbpDskNDOMx08ItVUx9mIY16/wD7Kyn/Uvx+LH94zbu/OxCLzZfrVPxhu9wd7g4OjBYUwwaBYGR/eODg8ekniB9KLFejq+ffMZTTWvClCOS2bty3hQunPPjBNcQt2llYWS8Qs2Ad8cSU22EEpJpg3+lCMNuqIrWNlKV1BQVSEcFKsoK6ydF8kOfkM2i+kL5nL7viiAO0nxv3RjLIHXW01VvwgpyTOmUJRCxirpgiAD5ioD1h+LSRsNT2GE+YjTJGdTIC4HwiqncQPBLZFLHdEZQWmXWnAGWRXS+VNRIPUEZuQkYHVDW9y1Ya9qQNyAghnXBapSTzNWCDlfFFZ7arNlfL2nl0o/flzKcx+6LdlrHQpVmoSQJBJFiWYmJYmW6YqvctnRuReqrBwo3g4rq2RoMtxRkgoD3Mk5YZzjzCvmcJY8Z3jrHN7l1YldOmaMnoUmsEg6Esr1Yk9DTTzXgkOXEOLpKBA0FbaUbG7dXAJ16G0i34vreuu67oqilhtRfkh2jVzGHyH/OBO9S9Gz7Kptq7ffu+d9GGKiLwdDmuPHb2K7gcA5BcNmHdzon4azA+OXKpRs1jW9AKN29w7pLB3Ui+wpszlafXN4u3V3q4XB0ykzUFohtVpKxbQ9W122X1N4xOplKluV5X8EAbxSnPgWTxJyssQKnYaZScjcm15MjZjk7g1yEnBkdmVqrQELmjQqvBSl1qrCzQzIhBn6uxYKQ7pK190uLZkRKsVDIfh8dPEZWr415MPB8/29+P7e/d8Oeo9+XjxI69G30XdRP4qjB9Gj6NfoaXQW8eiv6J+1W2vrnXHnbedd549WemNt4bkTrazOn/8CqmgFVQ==</latexit> b L(w, h) := L w (1 + h) = 1 N N Â i=1 f (xi ; w (1 + h)) yi 2
  14. Example: Dropout (DropConnect) <latexit sha1_base64="2GP4lW/rSROU/YJDZLMxLEvwpPI=">AAAGL3icjVTdbhw1FJ5tu1CGv7RccmPYDSrQRDtBFIQUqQpCcIFQSJM2UEeRx3Nm18Rju7Znt4vrh+IdeAfEDeKWt+B4ZjfKNiDV0miOvvOdz+dPLo0Uzk8mfw5u3Lw1fO3122/kb7719jvvbt25+9jp1nI44Vpqe1oyB1IoOPHCSzg1FlhTSnhSXnyd/E/mYJ3Q6tgvDZw1bKpELTjzCJ1v/X4AVulWSvEVoc9aVpExoZUjxacUPCP7hJYwFSpwvMPFfEI+ItTDcx8WhkRiCDUzprxuQohhJxaRPutEVqR7hlnWgAdLLoSUUH0cKc1pVVvGSRGKHRM3BQsUQdFe5JqGBcdZp5JTUNU6qfH51miyO+kOuW4UK2OUrc7h+Z1bj2ileduA8lwy554WE+PPArNecAko3zowjF+wKYSuyZFsI1SRWlv8lCcdusFrmOmpGyiothEemk2UNc4tm/I+/pOaQ/mG+Zl7mZTA+2Xzn+4u0ACP211W32KDySOmkFRBzVrpE6EG5lvsW/heTHtr/xhOUY94TWZsDsTPgLAWR4gbwR2Rax7RNUFqTh14i16V5MKxaND1AyzIUcJi53Z4a+e9zAFxCwoWXDfIqgKtWSPkcpVYDNTVa/tqTU+VPjgw8iykag37VadElSbJJKVEUqmZrUipZbUR1/r6y7MglGk9KN4Pq25lKjJtPamEBe7lkjDOceYt8zhLPsPN4rhYm33FpWPW6kUqApOkp0L5URFoyonPtOCQE0ICPU0OWglnJFs6v5TQLWxnhVER4/ZV3qWLOm6F+T/aFefafgX6qwXRDyjGrKvqyxrtjT4LIQ2x1M8n+3SGvzDGcpMD55QCxjFFY/w89Q5sWLOQMo6RXuATsrP7OV1Uk7hSr5ibYWjomrcT8+0ehkDnzIJxQmq1pop531tt+r9tAmJxLeVaY64REnjJOAo9XpbkaI01ukozk1D7F6OCWjGd+Rfok4AjcxtT6wMwoWnHwqUwWqsWL7MgS2bpL1ooNOmmO+Y5xadJqAqbQvD5yPEZKl5+dK4bj/d2iwe7D37cGz38ZvUg3c7ezz7M7mVF9kX2MPsuO8xOMj74ZHA4+Gnw8/C34R/Dv4Z/99Qbg1XMe9nGGf7zLx33Jzw=</latexit> Bernoulli: 1 + h =

    8 < : 0 wp p (parameter killed) 1 1 p wp 1 p (parameter rescaled) <latexit sha1_base64="m3ClBQRBrlqP+dPkQgzpa+pK8Go=">AAAFhXicjVRLbxMxEF4eoRBeBW5wMSSVONAqW6BwqUAgBAeEoLQQCZfK651NTL22ZXtTglmJX8MV/g7/hvFuUjUUJCytdvQ9xjMey5mRwvnB4NeJk6dOd84snT3XPX/h4qXLy1euvnO6shx2uJbaDjPmQAoFO154CUNjgZWZhPfZ/tPIv5+AdUKrbT81sFuykRKF4MwjtLd8vU+dGJXs4zrZJDQvLOPEhHTV1P295d5gbdAscjxIZ0Evma3Xe1dOv6W55lUJynPJnPuQDozfDcx6wSXUXVo5MIzvsxGEpvSarCCUk0Jb/JQnDbqgK5lppQsoqKoUHspFlJXOTcvsDv5jNofpS+bH7k9RBO9k5V/pxmiA1ytNVc+FlOQtUyjKoWCV9FFQAPOVBRdeilEbbW7DEPMRr8mYTYD4MRBWeY0bCO6InOuILghKu9SBt8iqmC5sixKpV3BAtiJWN7TDXRv2sAbELSg44LpEVR5owUohp7PC6kBdMY+P9vRB6SdPjNwNsVvDvuhYqNIkhiSTKMo0sznJtMwXfJUvHu4GoUzlQfF2WEUlY5PxLpFcWOBeTgnjHGdeMY+z5GOGl8jjnVuc2GfPrNUHsQkskg6F8r000FgTH2vBoUsICXQYCZoLZySbOj+VQD16myj00rpeOao7pKjjVph/yY6Q8/g/5P9nojcpeuZdtW311nt3Q4hDzPTnwSYd4y/0sd1I4JyioV9HN/on8ezAhrkKJf26pvtg1erafXqQD+pZ9py5MVpDc3irdXelhSHQCbNgnJBazaVi0p6tNu3flgGxep7KVcYcE0TwULEVWjzLyNYcK3UeZyah8F97KbViNPZfkZOAI3MLU2sNWNCoUeGlMFqrCjezIDNm6SctFIZ0ka67XWqYFSrHQyH4fHTxGUr/fHSOB+/W19KNtY03673Hz2YP0tnkRnIruZ2kyYPkcfIieZ3sJDz5lnxPfiQ/O0ud1c69zkYrPXli5rmWLKzOo98Lvuck</latexit> s2 = p 1 p <latexit sha1_base64="l6cdkZXq3zqff1jK+wFxXO8Jr7w=">AAAFhnicjVRLbxMxEF4egRJeBY5cDEklDhBtigpckKoiBAeEoLQQCZfK651NTL22ZXvTBrMHfg1X+Dn8G8a7SdVQkLC02tH3GM94LGdGCufT9NeZs+fOdy5cXLnUvXzl6rXrqzduvne6shx2uZbajjLmQAoFu154CSNjgZWZhA/ZwbPIf5iCdUKrHT8zsFeysRKF4MwjtL96u0+dGJfs0zqhXpOUUFEUxGCc9vdXe+kgbRY5HQznQS+Zrzf7N86/o7nmVQnKc8mc+zhMjd8LzHrBJdRdWjkwjB+wMYSm9pqsIZSTQlv8lCcNuqQrmWmlSyioqhQeymWUlc7Nyuw+/mM2h+lL5ifuT1EE72flX+nGaIDXa01VL4SU5B1TKMqhYJX0UVAA85UFF16JcRs93YER5iN4iBM2BeInQFjlNW4guCNyoSO6ICjtUgfeIqtiurAjSqRewyHZjljd0A53bdjjGhC3oOCQ6xJVeaAFK4WczQurA3XFIj7Z00elt7aM3AuxW8O+6Fio0iSGJJMoyjSzOcm0zJd8lS+e7AWhTOVB8XZYRSVjk/EykVxY4F7OCOMcZ14xj7PkE2YZ93jplid25Jm1+jA2gUXSkVC+Nww01sQnWnDoEkICHUWC5sIZyWbOzyRQj94mCr1hXa+d1B1T1HErzL9kJ8hF/B/y/zPROxQ9i67atnrrvYchxCFm+ih9Sif4C31sNxI4p2jo19GN/mk8O7BhoUJJv67pAVj1YLBBD/O0nmfPmZugNTSH96DurrUwBDplFowTUquFVEzbs9Wm/dsyIFYvUrnKmFOCCB4rtkOLZxnZXmClzuPMJBT+a29IrRhP/FfkJODI3NLUWgMWNG5UeCmM1qrCzSzIjFn6WQuFIV2m626XGmaFyvFQCD4fXXyGhn8+OqeD9+uD4aPBo7frvc3n8wdpJbmd3E3uJcPkcbKZvEzeJLsJT74l35Mfyc/OSmfQ2eg8bqVnz8w9t5Kl1dn8DS47514=</latexit> s2 ! 0 () p ! 0 <latexit sha1_base64="pyqZIGh3MbCfVPR0bpdkivFZZhE=">AAAF3XicjVTdb9s2EFe6es28r3R93As7J4CDJYGVJm26wUDRYdgegqJLk9ZAmRoUdbK4UKRAUnZcVo99K/a6f237Z4YdJTuIuxUoAUPn38fxjkcwKaWwbjD4e+3GJzc7n95a/6z7+RdffvX1xu1vnltdGQ5nXEttRgmzIIWCMyechFFpgBWJhBfJxU+BfzEFY4VWp25ewnnBJkpkgjOH0HjD0JlIIWeOHPdnOxQc2yY/DMkxTcSkP6M61Y704+8bImDbpDskNDOMx08ItVUx9mIY16/wD7Kyn/Uvx+LH94zbu/OxCLzZfrVPxhu9wd7g4OjBYUwwaBYGR/eODg8ekniB9KLFejq+ffMZTTWvClCOS2bty3hQunPPjBNcQt2llYWS8Qs2Ad8cSU22EEpJpg3+lCMNuqIrWNlKV1BQVSEcFKsoK6ydF8kOfkM2i+kL5nL7viiAO0nxv3RjLIHXW01VvwgpyTOmUJRCxirpgiAD5ioD1h+LSRsNT2GE+YjTJGdTIC4HwiqncQPBLZFLHdEZQWmXWnAGWRXS+VNRIPUEZuQkYHVDW9y1Ya9qQNyAghnXBapSTzNWCDlfFFZ7arNlfL2nl0o/flzKcx+6LdlrHQpVmoSQJBJFiWYmJYmW6YqvctnRuReqrBwo3g4rq2RoMtxRkgoD3Mk5YZzjzCvmcJY8Z3jrHN7l1YldOmaMnoUmsEg6Esr1Yk9DTTzXgkOXEOLpKBA0FbaUbG7dXAJ16G0i34vreuu67oqilhtRfkh2jVzGHyH/OBO9S9Gz7Kptq7ffu+d9GGKiLwdDmuPHb2K7gcA5BcNmHdzon4azA+OXKpRs1jW9AKN29w7pLB3Ui+wpszlafXN4u3V3q4XB0ykzUFohtVpKxbQ9W122X1N4xOplKluV5X8EAbxSnPgWTxJyssQKnYaZScjcm15MjZjk7g1yEnBkdmVqrQELmjQqvBSl1qrCzQzIhBn6uxYKQ7pK190uLZkRKsVDIfh8dPEZWr415MPB8/29+P7e/d8Oeo9+XjxI69G30XdRP4qjB9Gj6NfoaXQW8eiv6J+1W2vrnXHnbedd549WemNt4bkTrazOn/8CqmgFVQ==</latexit> b L(w, h) := L w (1 + h) = 1 N N Â i=1 f (xi ; w (1 + h)) yi 2 <latexit sha1_base64="YqByuGEhWoxlwA18ZopV6tAYRW4=">AAAGHHicjVTdbtxEFPa2XWjNT9Ny0QtuBjapEimJ1gEKQlqpCohwUaGQJs1KnWg1Hh+vh4xnrJnxbpepX4V34B24Q9wi8TacsXdDtilSR7J89H3fOT4/45NWUlg3HP7Tu3X7Tv+99+/eiz/48KOP7288ePjC6tpwOONaajNOmQUpFJw54SSMKwOsTCWcp5ffBf58BsYKrU7dooKLkk2VyAVnDqHJxu80halQnkkxVZA18WNaMeMEkxNHzrfdDhmRPUJL5gpTkuMJPWJlyQhVLJVs4ucNobuEprIGvxT5E5g2zXbw3UGKxo/fys93yLcjQnPDeHJA6PcgHZtQcBh7LjIomCPPtue7wx0SU1DZfwlONgbD/WF7yE0jWRqDaHmOJw/uPKeZ5nUJynHJrH2ZDCt34UOZXEIT09pCxfglm4JvO9qQLYQykmuDj3KkRdd0Jas66RoKqi6Fg3IdZaW1izLdxXeIZjF8aIV9UxTA3bR8K906VsCbrTarIyElec4UijLIWS1dEOTAXG3A+mdi2lmjUxhjPOI0KdgMiCuAsNpp/IDglsiVjuicoDSmFpxBVoVw/lSUSP0Ec3ISsKalLX61Za9yQNyAgjnXeDVwVDRnpZCLZWKNpzZf2ddreqn04WElL3yotmK/6pCo0iSYBG8Xv0w1MxlJtczW/GqXf3PhhapqB4p3w8prGYoMV5xkwgB3ckEY5zjzmjmcJS8YXjWHv8L6xF45ZoyehyIwSToWyg2S7q7yQgsOMSHE03EgaCZsJdnCuoUE6tC3tfwgaZqt67orilpuRPV/smvkyn4H+bs50c8o+qyq6soaHAy+8D4MMdWvhiNa4MtvYrmBwDkFh80meKP/LPQOjF+pULLZNPQSjNrb/4rOs2GzjJ4xW6Crb5u318RbHYx//IwZqKyQWq2kYtb1VldX+wCxZhXK1lV1QxDAK8WJ7/A0JScrrNRZmJmE3L0eJNSIaeFeIycBR2bXptY5YELTVoWXotJa1fgxAzJlhv6ihUKTrtNNHIe9KFSGTSG4PsIaSt5cOjeNFwf7yZP9Jz9/OXh6uFxId6NPo8+j7SiJvo6eRj9Gx9FZxHuPeqPeD72j/m/9P/p/9v/qpLd6S59PorXT//tfKk0dkw==</latexit> ∂tW(t) = PGrw Reg(W(t)), Reg(w) := 1 2 Dh b L(w, 0) <latexit sha1_base64="rRjeSgNOIhGZPvgnpUU/E/oZa8w=">AAAF03icjVTPbxw1FJ6WLpThR1M4cnHZREpEE+0UKIgqUhUOcKiqkibtSnWy8nje7Djx2Jbt2e3WnUvVK/8ad/4PriCeZ3ajbANSLY3m6fu+9/x+WC83Ujg/Gv157foHNwYffnTz4/STTz/7/NbG7S+eOd1YDsdcS23HOXMghYJjL7yEsbHA6lzC8/z858g/n4F1QqsjvzBwUrOpEqXgzCM02ahSmssGAq2Zr2wdDmHattvzHUIf/LRPH5D5KfXaEKpYLtlkTh713DdIUdfUk3C2n7WnjwnNxVRud+xuFFFdaE/y07OdyNiddLIxHO2NukOuGtnSGCbL82Ry+8ZTWmje1KA8l8y5F9nI+JPArBdcQpvSxoFh/JxNIXSNaMkWQgUptcVPedKha7qamV66hoJqauGhXkdZ7dyizu/iP0ZzGD42yb0riuDdvP5PunM0wNutLqtfhJTkKVMoKqBkjfRRUALzjQUXHolpb+0fwRjjEa9JxWZAfAWENV7jBYI7Ilc6okuC0pQ68BZZFcOFI1Ej9Rjm5DBibUc7vLVjL3JA3IKCOdc1qopAS1YLuVgm1gbqypV9uaYXSh8cGHkSYrWGvdIxUaVJNAk+En6ea2YLkmtZrPk1vvzxJAhlGg+K98MqGxmLjC+TFMIC93JBGOc484Z5nCWvmGXc4wten9hLz6zV81gEJknHQvlh1r9iXmnBISWEBDqOBC2EM5ItnF9IoB59OysMs7bduqy7oKjjVpj/k10iV/Z7yN/Pid6h6LOqqi9reG/4bQhxiLl+OdqnFf7CJpYbCZxTdNhsozf6z2LvwIaVCiWbbUvPwardve/pvBi1y+gFcxW6hq55u2261cO4C2bMgnFCarWSilnfW20uNgVi7SqUa4y5IojgheIw9Hiek8MVVusizkxC6V8PM2rFtPKvkZOAI3NrU+sdMKFpp8JHYbRWDV5mQebM0jMtFJp0nW7TlBpmhSqwKQTXR1xD2btL56rx7N5edn/v/m/fDR8eLBfSzeSr5OtkO8mSH5KHya/Jk+Q44ckfyV/J38k/g+NBGLwZvO2l168tfb5M1s7g938B5o8ECw==</latexit> Reg(w) := w>rwL(w) + N Â j=1 L(w) L(w bj) <latexit sha1_base64="USxEtFvvsm7WmQKARryi7zy2AlU=">AAAF4HicjVRPbxQ3FJ8FtoSBlgBHLm43kai0RDvQBoQUCYUDPVQVDQmshNPI43mz68RjW7Znw2Lmzg1x7Ter+DI8z85GWUIlLI3m6ffH9vOzX26kcH40+q936fKV/g9X166l12/8+NPN9Vu3XzldWw4HXEttxzlzIIWCAy+8hLGxwKpcwuv85FnkX8/AOqHVvp8bOKzYRIlScOYROlqv83+OyZMdci8b0kJ7R4YkGxJaMT/VJtBprt+GjdFG01ApKuHdUaCziNETsCqjx9p3GlobZq0+3Wha6v557hj9zbBbARf49Wh9MNoatYNcDLIuGCTdeHF068pLNPO6AuW5ZM69yUbGHwZmveASmpTWDgzjJ2wCoT2VhmwiVJBSW/yUJy26oquYWUhXUFA1JgrVKsoq5+ZVPsR/nM3h9PGM3NeiCA7z6pt0azTAm812V8+FlOQlUygqoGS19FFQAvO1BRf+FJNFtLMPY5yPeE2mbAbET4Gw2mtcQHBH5FJHdElQmlIH3iKr4nRhX1RI/QWnZC9iTUs7XLVlz/aAuAUFp1xXqCoCLVkl5LzbWBOoK5fx+ZzeKL27a+RhiNka9k7HjSpNYkhyiaJcM1uQXMtixVf78vFhEMrUHhRfFKusZUwyXlNSCAvcyzlhnGPNa+axlnzKLOMer/Nqxd769urFJHCTdCyUH2ShvcR8qgWHlBAS6DgStBDOSDZ3fi6BevS2URhkTbN5XndGUcetMP8nO0cu4++Qf5+J/kzRs8xqkdbgweBhCLGI+LRGO90Lw3QjgXWKho0mutE/i2cHdvmOoyQ+5faBbv1OT4tR081eMDdFa2gP736Tbi5gwOfOLBgnpFZLqZiFswYR/7YKiDXLqVxtzAVBBM8Ue2GB5znZW2KVLmLNJJT+/SCjVkym/j1yErBkbqVqCwNuaNKq8FIYrVWNi1mQObPYeYTCkK7STZpS7FFCFXgoBNtHim0o+7rpXAxePdjKtre2//5t8HS3a0hryd3kl+RekiWPkqfJH8mL5CDhyeder5f2rvfz/of+x/6nhfRSr/PcSVZG/98vakYJFA==</latexit> bj := (1, . . . , 1, 0 " j , 1, . . . , 1) <latexit sha1_base64="K+9xE/vokodOIRoociToAGw7C4o=">AAAFjXicjVRNbxMxEF0+AiV8tIUjF9OkEgdaJUUtHAAhEIIDQqW0EAlXldc7m5h6bcv2JgSzd34NV/gr/BvGu0nVUJCwtPLovTfjGY93UiOF873er3PnL1xsXbq8dKV99dr1G8srqzffO11aDgdcS20HKXMghYIDL7yEgbHAilTCh/T4eeQ/jME6odW+nxo4LNhQiVxw5hE6Wlnr0rSUEnyXUOI4k7Ah1JhZwZQnQpHupHu00ult9upFzhr9mdFJZmv3aPXiO5ppXhagPJfMuY/9nvGHgVkvuISqTUsHhvFjNoRQl1CRdYQykmuLH55cowu6gplGuoCCKgvhoVhEWeHctEjv4R6jOQxfMD9yf4oieC8t/krXjgZ4tV5n9VJISd4xhaIMclZKHwU5MF9acOG1GDbW430YYDziNRmxMRA/AsJKr/EAwR2Rcx3ROUFpmzrwFlkVw4V9USD1BiZkL2JVTTs8tWZPckDcgoIJ1wWqskBzVgg5nSVWBeryuX26po9KP3tm5GGI1Rr2RcdElSbRJKlEUaqZzUiqZbbgV/r84WEQypQeFG+alZcyFhnfFMmEBe7llDDOsecl89hLPmKWcY9vb7Fjnz2zVk9iEZgkHQjlO/1AY058pAWHNiEk0EEkaCackWzq/FQC9ehbW6HTr6r107oTijpuhfmX7BQ5t/9D/n9O9A5Fn3lVTVmdrc79EGITU/2595iOcAtdLDcS2Kfo0K2iN/qP492BDXMVSrpVRY/Bqo3NbTrJetUsesbcCF1DfXkbVXu9gSFQ/HPBOCG1mkvFuLlbbZrdFgGxah7KlcacEUTwRLEXGjxNyd4cK3QWeyYh9187fWrFcOS/IodzhGZuoWuNAyY0rFX4KIzWqsTDLMiUWfpJC4UmXaSrdpsaHEMqw0shOD7aOIb6fw6ds8b7rc3+zubO263O0xezgbSU3E7WkrtJP3mQPE1eJbvJQcKTb8n35Efys7Xc2m49aj1ppOfPzXxuJQur9fI3lRfqnw==</latexit> • scale-invariant in w <latexit sha1_base64="7eWByixdRFv8CJfWC1wWHSodQfs=">AAAFkHicjVTbbhMxEF0ugRJuBR55MWQj8QBVUsTlpeImBEIIQWkhEi6V1zubmHpty/amBLNfwNfwCl/C3zDeTaqGgoSl1Y7OZTzjsZwZKZwfDH4dO37iZOfU6ZUz3bPnzl+4uHrp8junK8thm2up7ShjDqRQsO2FlzAyFliZSXif7T2J/PspWCe02vIzAzslGytRCM48Qrur/ZRmlZTgU0KJYyWQtPi4nhLHGaYcE+ZI+jLdXe0N1gbNIkeD4TzoJfP1evfSybc017wqQXkumXMfhgPjdwKzXnAJdZdWDgzje2wMoemiJn2EclJoi5/ypEGXdCUzrXQJBVWVwkO5jLLSuVmZ3cR/zOYwfcn8xP0piuDNrPwr3RgN8LrfVPVMSEneMoWiHApWSR8FBTBfWXDhpRi30cYWjDAf8ZpM2BSInwBhlde4geCOyIWO6IKgtEsdeIusiunCliiRegX7ZDNidUM73LVhD2pA3IKCfa5LVOWBFqwUcjYvrA7UFYv4cE8flH782MidELs17IuOhSpNYkgyiaJMM5uTTMt8yVf54v5OEMpUHhRvh1VUMjYZrxXJhQXu5YwwznHmFfM4Sz5hlnGP1295Yp89s1bvxyawSDoSyveGgcaa+EQLDl1CSKCjSNBcOCPZzPmZBOrR20ShN6zr/mHdAUUdt8L8S3aIXMT/If8/E71G0bPoqm2rt967HUIcYqY/DzboBH8hxXYjgXOKhrSObvRP49mBDQsVStK6pntg1a21O3Q/H9Tz7DlzE7SG5vBu1d1+C0OgU2bBOCG1WkjFtD1bbdq/LQNi9SKVq4w5IojggWIztHiWkc0FVuo8zkxC4b/2htSK8cR/RQ6fEpq7pam1Bixo3KjwUhitVYWbWZAZs/STFgpDukzX3S41zAqV46EQfD66+AwN/3x0jgbv1teGd9fuvlnvPXw6f5BWkqvJ9eRGMkzuJQ+T58nrZDvhybfke/Ij+dm53LnfedB51EqPH5t7riRLq/PiN0/i6pM=</latexit> • same f2 scaling as L
  15. What have we seen so far? • Noisy gradient descent

    follows slow constrained gradient fl ow on Γ, at time scale • Driving functional (‘regulariser’) characterised in terms of (generalized) 𝜂 -derivatives of ασ2 ̂ L But: Well-known noises have zero regulariser . . .
  16. <latexit sha1_base64="mEvOs+SNOOoWqgBurlufV+6IPT0=">AAAFsnicjVRfbxw1EN+WHpTjX1oeeTFcIhWRRHelLbxEqopQeahQSZM2ok5Ps97ZW3Ne27K9dz3c/Uh8Gp6Q4Lsw3r2Lcg1IXWnl0e/PeMZjObdK+jAe/3Xt+ns3Bu9/cPPD4Ucff/LpZzu3bj/3pnECT4VRxp3l4FFJjadBBoVn1iHUucIX+fyHxL9YoPPS6JOwsnhew0zLUgoIBE13HvNcNRiX0zj/ZtK+kuyALadzWo8o4qBsBYxryBVM4/KVbBlfygIrCOzJHRLucwwwnX/dTndG48Nx97GrwWQdjLL193R668YzXhjR1KiDUOD9y8nYhvMILkihsB3yxqMFMYcZxq7Plu0RVLDSOPp1YB26pavB9tItFHVTy4D1Ngq196s636c1ZfOUvoZQ+bdFCdzP6/+kO6NF0e51VT2WSrFnoElUYAmNCklQIoTGoY9P5KyPjk7wjPKxYFgFC2ShQgZNMLSBFJ6pjY6ZkpF0yD0GR6xO6eKJrIn6GZfsOGFtR3vatWMvaiDcocalMDWpishLqKVarQtrI/flJr7c00ttHj2y6jymbi38blKh2rAUMroGYp4bcAXLjSq2fE0ovz+PUtsmoBb9sMpGpSbTxWOFdCiCWjEQgmbeQKBZigociEAXdHtirwM4Z5apCSqSn0kdRpPIU02iMlLgkDEW+VkieCG9VbDyYaWQB/J2URxN2nbvsu6C4l44af9PdoncxO8gfzcT/5KTZ9NV39bo7ujbGNMQc/N6fMQrWuIutZsImlMy7LbJTf5FOjt0caMiyW7b8jk6fXB4ny+LcbvOXoCvyBq7wztoh3s9jJEvwKH1Uhm9kcpFf7bG9qurI2HtJpVvrL0iSOCF4jj2eJ6z4w1WmyLNTGEZ3owm3MlZFd4Qp5BG5rem1huooFmnokthjdENbeZQ5eD4b0ZqCvk23Q6H3IKTuqBDYfR8DOkZmrz96FwNnt89nDw4fPDLvdHDH9cP0s3si+yr7E42yb7LHmY/ZU+z00xkf2R/Zn9n/wzuDX4dwED00uvX1p7Ps61voP4FH2P5aA==</latexit> wi k+1 wi k = arwi b L(wk

    , hk) <latexit sha1_base64="h8F/D9RYSTmhPgmmJ/9OMRM+MQM=">AAAF7XicjVRPjxs1FJ+0DZQplG05gMTFJbtSkXZXSWkLl0pVEYJDhcp2t420TqM3njcZE49tbE+ywZ0L34Eb4spn4hvwMWrPJKtNl0q1NJqn35/n9/wsZ1pw64bDf3tXrl7rf/Dh9Y/SGx9/cvPTnVu3X1hVG4YnTAllxhlYFFziieNO4FgbhCoT+DKbfx/5lws0lit57FYaJxXMJC84Axeg6c4fNBM1Eu8bClobdUYOKAhdAqESMgH0ztQvX/GG0CXPsQRHnt5dTuf7w69T+lsNOTkg79BrMI6DmFJ0cNl9GuHpfJI26XRnMDwctotcDkbrYJCs17PprWvPaa5YXaF0TIC1p6OhdhMfN2QCm5TWFjWwOczQt0fUkL0A5aRQJnzSkRbd0lWgO+kWirKuuMNqG4XK2lWV7Yd/zGZD+gpcad8WRXA/q/6Xbo0aWbPXVvUjF4I8BxlEORZQCxcFBYKrDVr/lM+66NExjkM+4hQpYYHElUigdipswJklYqMjqiBBmlKLzgRWxnT+mFeB+hmX5ChiTUvbsGvLntcQcIMSl0xVQZV7WkDFxWpdWOOpLTbxxZ5OpXryRIuJj91q+F3FQqUiMSThbrB5psDkJFMi3/LVrvhu4rnUtUPJumEVtYhNxjtLcm6QObEiwFiYeQ0uzJKVYIC5cLe3J3bmwBi1jE2EIumYSzcYeRprYqXiDFNCiKfjSNCcWy1gZd1KIHXB20Z+MGqavYu6c4paZrh+l+wCuYnfQ/5+JnqHBs+mq66twb3BN97HIWbqbPiIluHnd0O7kQhziobdJrqDfxHPDo3fqIJkt2noHI08OHxAl/mwWWfPwZbB6tvDO2jSvQ5GTxdgUFsulNxI+aI7W6W7v6l8wJpNKltrfUkQwXPFke/wLCNHG6xSeZyZwMK9Hoyo4bPSvQ6cwDAyuzW1zhAKmrWqcCm0UrIOmxkUGRj6q+IyhHSbbtI0vlBc5uFQSHg+4jM0evvRuRy8uHc4enj48Jf7g8c/rB+k68mXyVfJ3WSUfJs8Tn5KniUnCUv+693sfd77oq/6f/b/6v/dSa/01p7Pkq3V/+cNwNsNhw==</latexit> ⇡ arwi b L(wk , 0) arwi ∂h b L(wk , 0)[hk] <latexit sha1_base64="I9O5rqdPtNZTLxsrXRBp/iP9u4U=">AAAF53icjVRdjxs1FJ2UBpbwtVseeXHJrlSkdJUEKLysVIoqeKhQ2e5uI9W7kcdzJ2PisY3tSRqc+Q28IV75WbzyS7ieZFabLki1FM3Nuedc3w/rpkYK54fDvzt33rnbffe9vfd7H3z40cef7B/cu3C6shzOuZbaTlLmQAoF5154CRNjgZWphJfp/Pvof7kA64RWZ35l4LJkMyVywZlHaLq/piXzRZqSp+SEDAeE/lqxjDSgLcMFszXilElTsKsxoU7MysZYfzed0/V0fDUekN5GhEjkKpZKRu9Pw/JK1Nu/UwqeEboUGRTMk2cPltP5YPjFdL8/PB42h9w2Rlujn2zP8+nB3Rc007wqQXkumXOvRkPjLwOzXnAJdY9WDgzjczaD0DSnJkcIZSTXFn/Kkwbd4ZXMbKg7KKiqFB7KXZSVzq3KdIDfGM1h+Ngq9yYpgoO0/E93IzTA66Mmqx+ElOQFU0jKIGeV9JGQA/OVBReeidnGOjmDCcYjXpOCLYD4AgirvMYLBHdEtjyic4LUHnXgLXpVDBfORImun2BJTiNWN26Htzbe6xwQt6BgyXWJrCzQnJVCrraJ1YG6vLVv1vRK6SdPjLwMsVrDftMxUaVJNAnOn89TzWxGUi2zHV3l828vg1Cm8qD4Zlh5JWOR8bWSTFjgXq4I4xxnXjGPs+QFs4x7fNW7E3vtmbV6GYvAJOlEKN8fheYt80ILDj1CSKCT6KCZcEaylfMrCdSjtrFCf1TXRzd51y7quBXm/2g3nK39FvS3E9H7FDVtVZuy+uP+lyHEIab69fCEFvgJh1hudOCcouCwjmrUL2LvwIaWhZTDuqZzsOrh8dd0mQ3rbfSMuQKloWnew7p3tIEh0AWzYJyQWrVUsdj0VpvQ7gvE6jaUq4y5RYjgNeM0hHb7nLZYqbM4Mwm5X/dH1IpZ4dfok4AjcztT2wgwoVnDwkdhtFYVXmZBpszSX7RQaNJdd93rUcOsUBk2heD66OEaGr25dG4bF+Pj0aPjRz9/1X/8dLuQ9pLPks+TB8ko+SZ5nPyYPE/OE57809nrHHTudUX39+4f3T831DudrebTZOd0//oXsQEJEQ==</latexit> E = 0, Var = a2 s2kAkk2 2 , Ak = rwi rh b L(wk , 0) <latexit sha1_base64="2ikxvJu2am14lBK63p63rQeJDX0=">AAAF4XicjVRRbxtFED6HGopbIIVHXrY4kYqURHaAwkukqlDBQ4VKmqSRuqk1tzfnW7y3e93ds2O29wN4Q7zyy5D4McyefVHcgNSVrBt93zezMzvjSSslnR+N/u5tvXer//4Htz8c3Ln70cefbN/79MyZ2go8FUYZe56CQyU1nnrpFZ5XFqFMFb5IZ99H/sUcrZNGn/hlhRclTLXMpQBP0GR7zkvwRZqyJ+yI7fPcguCgqgLYIeNOTkt4RYaGVAHj9ydh8Uo2jP+AysOEoydwITMswLOnDxaT2d7oS7bH+OvXNWSsDW3LcAa2oejc46UPrgSlmsn2cHQwag+7aYzXxjBZn2eTe7ee88yIukTthQLnXo5Hlb8IYL0UCpsBrx1WIGYwxdA+S8N2CcpYbiz9tGctuqEroVpJN1DUdSk9lpsolM4ty3SPvjGao/CxPPe2KIJ7afmfdOtYoWh226x+lEqx56BJlGEOtfJRkCP42qILT+V0ZR2d4DnFY96wAubIfIEMam/oAikcU52OmZyRdMAdekusjuHCiSyJ+hkX7DhiTUs7urVlr3Ig3KLGhTAlqbLAcyilWq4TawJ3eWdfr+mlNo8fV+oixGor+M3ERLVh0WQ0NGKWGrAZS43KNvxqn393EaSuao9arJqV1yoWGeeUZdKi8GrJQAjqeQ2eeikKoAH1NM+bHbv0YK1ZxCIoSX4utR+OQzt/ojBS4IAxFvh5JHgmXaVg6fxSYTuTrRWG46bZva67orgTVlb/J7tGdvY7yN/Nid/n5NNVtSpreDj8KoTYxNRcjo54QZ+wQ+VGgvoUHXaa6E3+8/h2aEOnIslO0/AZWr1/8A1fZKNmHT0DV5BraB9vvxnsrmAMfA4WKyeV0Z1Uzldva6rQ/ccJa7pQrq6qG4IIXimOQ+j2znGHlSaLPVOY+zfDMbdyWvg3xCmklrmNrq0cKKFpq6KhqIzRNV1mUaVg+a9GajL5Jt0MBrwCK3VGj8JofQxoDY3fXjo3jbPDg/HDg4e/fD189GS9kG4nnydfJA+ScfJt8ij5KXmWnCYi+ae31bvTu9sX/d/7f/T/XEm3emufz5KN0//rXyeaCHo=</latexit> E = a 2 s2rwi Dh b L(wk , 0), Var = small <latexit sha1_base64="4eCcIZWwQxyB/tiHUSqBEi26akw=">AAAFi3icjVRfjxM3EDeUFEhLOcojL4bcSVTiTgm0FFBPOlFV9AFV9LiDSPiIvN7ZrBuvbdnehNTsaz9NX9vvwrdhvJucLlwrYWm1o9+f8YzHcmaV9GE4/HDh4heXel9evnK1/9XX1765vnXj21fe1E7AsTDKuHHGPSip4TjIoGBsHfAqU/A6m/2c+NdzcF4afRSWFk4qPtWykIIHhCZbdJ/uMq5sySnTPFOc3Z7ExVvZ0Od3F5PZd3SyNRjuDdtFzwejVTA4eEza9WJy49JLlhtRV6CDUNz7N6OhDSeRuyCFgqbPag+WixmfQmwbaOgOQjktjMNPB9qiG7qK2066gYKuKxmg2kR55f2yyu7hP2XzmL7iofSfihJ4L6v+k26NFkSz01b1TCpFX3KNohwKXquQBAXwUDvw8bmcdtH+EYwxHw2GlnwONJRAeR0MbiCFp2qto6agKO0zD8Ehq1O6eCQrpH6DBT1MWNPSHndt2dMaEHegYSFMhao8soJXUi1XhTWR+WIdn+3pjTZPn1p1ElO3lv9pUqHa0BRSHLyYZYa7nGZG5Ru+OhSPTqLUtg6gRTesolapyXSjaC4diKCWlAuBM695wFmKkjsuAt68zYm9C9w5s0hNYJFsLHUYjCJLNYnSSAF9Smlk40SwXHqr+NKHpQIW0NtGcTBqmp2zulOKeeGk/T/ZGXIdf4b880zsNkPPuquurcH9wYMY0xAz8264z0r8xW1sNxE4p2TYbpIb/fN0duDiWoWS7aZhM3B6d+8HtsiHzSp7zn2J1tge3m7T3+lgiGzOHVgvldFrqZx3Z2ts93dVRKxZp/K1tecECTxVHMYOzzJ6uMYqk6eZKSjC+8GIOTktw3vkFODI/MbUOgMWNG1VeCmsMbrGzRyojDv2h5EaQ7ZJN/0+s9xJneOhUHw++vgMjT59dM4Hr+7vjR7uPfz9+8HBT917RK6QW+QOuUtG5EdyQH4lL8gxEeQv8jf5h/zbu9Z70HvSW2kvXlh5bpKN1fvlIzzo6fQ=</latexit> = arwi L(wk) <latexit sha1_base64="huaKS7ThmeGsOhKxGm+6GDa0o8c=">AAAFf3icjVRfbxQ3EHcoR+FoaSi88eJyF6UPcNoLKgWJSBE80AeEaEjgJBwFr3f2zsRrW7b3wmH2u/AK34hvw3g3F+VIK2FptaPfn/GMx3JulfQhy76uXfjpYu/Sz5ev9K/+8uu139av//7Km9oJ2BdGGTfJuQclNewHGRRMrANe5Qpe50dPEv96Ds5Lo/fCwsJBxadallLwgNDh+s23b4fb2ZAaTYfsKa8qPtzcPFwfZKOsXfR8MD4JBjsPSbteHF6/+JIVRtQV6CAU9/7NOLPhIHIXpFDQ9FntwXJxxKcQ26IbuoFQQUvj8NOBtuiKruK2k66goOtKBqhWUV55v6jyO/hP2Tymr3iY+e9FCbyTV/9Jt0YLotloq3oqlaIvuUZRASWvVUiCEnioHfj4TE67aHsPJpiPBkNnfA40zIDyOhjcQApP1VJHTUlR2mcegkNWp3RxT1ZIPYdjupuwpqU97tqypzUg7kDDsTA4IV1EVvJKqsVJYU1kvlzGZ3t6o83jx1YdxNSt5R9MKlQbmkKaKxTlhruC5kYVK746lA8OotS2DqBFN6yyVqnJdItoIR2IoBaUC4Ezr3nAWYoZd1wEvG2rE3sfuHPmODWBRbKJ1GEwjizVJGZGCuhTSiObJIIV0lvFFz4sFLCA3jaKg3HTbJzVnVLMCyft/8nOkMv4B+Q/ZmJ/MPQsu+raGmwN7sWYhpib99k2m+EvDrHdROCckmHYJDf65+nswMWlCiXDpmFH4PTd0V/suMiak+wF9zO0xvbw7jb9jQ6GyObcgfVSGb2Uynl3tsZ2f1dFxJplKl9be06QwFPFbuzwPKe7S6wyRZqZgjJ8HIyZk9NZ+IicAhyZX5laZ8CCpq0KL4U1Rte4mQOVc8feGakxZKt00+8zy53UBR4Kxeejj8/Q+PtH53zwams0vj+6/+/WYOdR9x6Ry+QWuU3+JGPyN9kh/5AXZJ8I8oF8Ip/Jl95ab7M36mWd9MLaiecGWVm9h98A1q/kSw==</latexit> “= 0 on G” <latexit sha1_base64="GDFdex3JZQXfuOpeVeMRYxKOUW0=">AAAF3nicjVRPbxQ3FJ9QloYppaE99mK6iQRqEu2mLe0FCVFV9IAqGhJYKV62Hs+bHXc9trE9u2zNXHuruPar8Wna55ndKEtAYqSRn35/nt/zs5wZKZwfDN5uXfnkau/ap9vX089ufH7zi51bXz5zurYcTrmW2o4y5kAKBadeeAkjY4FVmYTn2eznyD+fg3VCqxO/NDCu2FSJQnDmEZrsOJrJGkgIDTkgtLCMUyZNyY4IVSyTjN6ehMUL0RBqmPWCyRdHEwqeEboQOZTMk8d3FpPZ/uDuWYQx6pZxSl++rFlOVsu3hPJce0fSJp3s9AeHg/Yjl4PhKugnq+/J5NbVpzTXvK5AeS6Zc2fDgfHjECviEpqU1g4M4zM2hdCeSUP2EMpJoS3+ypMW3dBVzHTSDRRUXQkP1SbKKueWVbaPa8zmMH3FfOneFUVwP6veS7dGA7zZa6t6JKQkT5lCUQ4Fq6WPggKYry248FhMu+j+CYwwH/GalGwOxJdAWO01biC4I3KtI7ogKE2pA2+RVTFdOBEVUr/BghxHrGlph7u27HkNiFtQsOC6QlUeaMEqIZerwppAXbGOL/Z0pvTDh0aOQ+zWsL90LFRpEkOCl4fPMs1sTjIt8w1f7YufxkEoU3tQvBtWUcvYZLykJBcWuJdLwjjHmdfM4yx5yfB6erzMmxN75Zm1ehGbwCLpSCjfHwYaa+KlFhxSQkigo0jQXDgj2dL5pQTq0dtGoT9smr2LunOKOm6F+ZDsArmOP0L+cSZ6m6Jn3VXXVv+o/10IcYiZfjW4T0tcwi62GwmcUzTsNtGN/nk8O7BhrULJbtPQGVh1cPgDXeSDZpU9Z65Ea2gP76BJ9zoYAp0zC8YJqdVaKubd2WrTrbYKiDXrVK425pIggueK49DhWUaO11il8zgzCYV/3R9SK6alf42cBByZ25haZ8CCpq0KL4XRWtW4mQWZMUv/1EJhSDfpJk3jEyZUjodC8PmIz9Dw3UfncvDs6HB47/De79/3H/yyepC2k6+Tb5I7yTD5MXmQ/Jo8SU4TnrxN/tva3rre+6P3d++f3ptOemVr5fkq2fh6//4PuFEIVg==</latexit> a 2 rwi ∂2 h b L(wk , 0)[hk , hk] + · · · leading order Degenerate case Suggests a limit evolution on Γ at time scale α2σ2
  17. Example <latexit sha1_base64="ty7mNR9YH7n/CsUdG3hh/JXtIY4=">AAAFqHicjVRfbxw1EN+WHpTjXwqPvBgukQpqot0DQoIUqSogeKhQSJPmUB1FXu/srYnXtmzvXQ53Pw2fhld44tsw3r2NckClWlrt6PdnPOOxnBspnE/Tv+/cfePe6M237r89fufd997/YOvBh8+dbiyHM66ltrOcOZBCwZkXXsLMWGB1LuE8v/o28ucLsE5odepXBi5qNleiFJx5hC63jqiHax9olynkkvGrVmm1W8D8G9JSQr8D6dklBc8IXYoCKubJU3JEpuzh8rPLrUm6l+La3ycxyA7SDIPDw4Pp9JBkHZWmk2S9ji8f3HtGC82bGpTnkjn3IkuNvwjMesEltGPaODBYBJtD6GpqyQ5CBSm1xU950qEbupqZXrqBgmpq4aHeRFnt3KrOH+E/ZnOYvma+cv8WRfBRXv8v3RkN8Hanq+oHISV5xhSKCihZI30UlMB8Y8GFp2LeR0enMMN8xGtSsQUQXwFhjde4geCOyEFHdElQOqYOvEVWxXThVNRI/QRLchKxtqMd7tqxNzUgbkHBkusaVUWgJauFXK0LawN15RDf7umF0k+eGHkRYreG/aZjoUqTGJLuVuSa2YLkWhYbvsaXBxdBKNN4ULwfVtnI2GS8bqQQFriXK8I4x5k3zOMsecUs4x6v5ebErj2zVi9jE1gknQnlJ1mgsSZeacFhTAgJdBYJWghnJFs5v5LQXeIuCpOsbXdu624o6rgV5lWyW+QQv4b89Uz0E4qeoau+rcl08kUIcYi5vk6PaIW/sI3tRgLnFA3bbXSjfxHPDmwYVCjZblt6BVbt7n1Fl0XarrMXzFVoDd3h7bbjnR6GQBfMgnFCajVIxaI/W236v60DYu2QyjXG/EcQwRvFSejxPCcnA1brIs5MQulfTjJqxbzyL5GTgCNzG1PrDVjQvFPhpTBaqwY3syBzZumvWigM6SbdjsfUMCtUgYdC8PkY4zM0vDXk1cHz6V62v7f/85eTx9+vH6T7ycfJp8nDJEu+Th4nPybHyVnCk9+TP5I/k79Gn4+OR+ejX3rp3Ttrz0fJxhrl/wDEEvY1</latexit> non-deg: Dh b L = 2a(w) <latexit

    sha1_base64="46K02/009PUDU4mlax6y6OMe0uE=">AAAFnHicjVTdbhQ3FB4oW2D7Q2gvkSrTTaReQDSzLWlygYTSSq3UH9GQhJVwiDyeMzsmHtuyPZtszTwHT8Nt+wy8TY9ndqJsWyQsjebo+zk+x8dybqRwPk3fXbv+0Y3Rxzdv3R5/8ulnn9/ZuPvFsdON5XDEtdR2ljMHUig48sJLmBkLrM4lPM/Pfoj88wVYJ7Q69EsDJzWbK1EKzjxCpxsZ9XDhA+0yhVwyftZ6UQNxnEkgLSWUSVMxQp2Y1+zllJxuTNLtFNfODolBtptmGOzt7U6neyTrqDSdJKv19PTujWe00LypQXkumXMvstT4k8CsF1xCO6aNA4MbszmEro6WbCFUkFJb/JQnHbqmq5nppWsoqKYWHup1lNXOLev8Af5jNofpa+Yr929RBB/k9f/SndEAb7e6qn4SUpJnTKGogJI10kdBCcw3Flz4Vcz76PEhzDAf8ZpUbAHEV0BY4zVuILgjctARXRKUjqkDb5FVMV04xEE48juck4OItR3tcNeOvawBcQsKzrmuUVUEWrJayOWqsDZQVw7x1Z5eKL2/b+RJiN0a9qeOhSpNYki6m5BrZguSa1ms+Rpf7p4EoUzjQfF+WGUjY5PxipFCWOBeLgnjHGfeMI+z5BWzjHu8iusTu/DMWn0em8Ai6UwoP8kCjTXxSgsOY0JIoLNI0EI4I9nS+aWE7uJ2UZhkbbt1VXdJUcetMO+TXSGH+APkH2ai9yl6hq76tibTybchxCHm+iJ9TCv8hU1sNxI4p2jYbKMb/Yt4dmDDoELJZtvSM7Dq4fYjel6k7Sp7wVyF1tAd3sN2vNXDEOiCWTBOSK0GqVj0Z6tN/7d1QKwdUrnGmP8IInipOAg9nufkYMBqXcSZSSj960lGrZhX/jVyEnBkbm1qvQELmncqvBRGa9XgZhZkzix9pYXCkK7T7XhMDbNCFXgoBJ+PMT5Dw1tD3h8cT7ezne2dP76bPNlfPUi3knvJ18k3SZZ8nzxJfk6eJkcJT94kb5O/kr9HX41+HP0y+q2XXr+28nyZrK3R8T9Ub/Jt</latexit> time scale as2 <latexit sha1_base64="zQXX4M+MmhK5eZQQOyQx1JRYXUg=">AAAFknicjVTdbhw1FJ6WLpTlLwXuuDFsIlqRRjMLhAQpUmmF4CJCJU3aleoQeTxndkw8tmV7dru48wg8DbfwILwNxzM7URaoVEsjH30/x+f4jJwbKZxP079v3Hzj1ujNt26/PX7n3ffe/2DrzodPnW4shzOupbaznDmQQsGZF17CzFhgdS7hWX75KPLPFmCd0OrUrwyc12yuRCk48whdbH1Ol6KAinlyfHe5S8Gze+QoxvfIF4TFLWK/TC+2Juleimt/n8QgO0gzDA4PD6bTQ5J1VJpOkvV6fHHn1hNaaN7UoDyXzLnnWWr8eWDWCy6hHdPGgWH8ks0hdH20ZAehgpTa4qc86dANXc1ML91AQTW18FBvoqx2blXnu7jHbA7T18xX7t+iCO7m9f/SndEAb3e6qn4QUpInTKGogJI10kdBCcw3Flw4FvM+OjqFGeYjXpOKLYD4CghrvMYDBHdEDjqiS4LSMXXgLbIqpgunokbqJ1iSk4i1He3w1I69qgFxCwqWXNeoKgItWS3kal1YG6grh/h6T8+VfvjQyPMQuzXsNx0LVZrEkOQSRblmtiC5lsWGr/HlwXkQyjQeFO+HVTYyNhl/LFIIC9zLFWGc48wb5nGWvGKWcY8/4ObEXnhmrV7GJrBIOhPKT7JAY0280oLDmBAS6CwStBDOSLZyfiWBevR2UZhkbbtzXXdFUcetMK+SXSOH+DXkr2ein1L0DF31bU2mky9DiEPM9Yv0iFa4hW1sNxI4p2jYbqMb/Yt4d2DDoELJdtvSS7Dq/t7XdFmk7Tp7wVyF1tBd3v12vNPDEOiCWTBOSK0GqVj0d6tNv9s6INYOqVxjzH8EEbxSnIQez3NyMmC1LuLMJJT+5SSjVswr/xI5CTgytzG13oAFzTsV/hRGa9XgYRZkziz9VQuFId2k2/GYGmaFKvBSCD4fY3yGhreGvDp4Ot3L9vf2f/5q8uD79YN0O/kk+Sy5m2TJN8mD5MfkcXKW8OT35I/kz+Sv0cejb0ffjR710ps31p6Pko01Ov4Hyt/rFA==</latexit> b L(w, h) = L(w) + a(w)h2 <latexit sha1_base64="d/ItkzxdexVI7x2P/vQ+SqOp15s=">AAAFoHicjVTNbtw2EFZ+tk23P3GaYy9s1gZ6SAzJbVy7gIHALdAcgiZx7GTR0DAoarRiTZEESe16y+hJ+jS9tk+Qt8lQWhnetAFCQNDg+xnOcAjmRgrn0/Tttes3bo4++fTWZ+PPv/jyq9sbd75+6XRjOZxwLbWd5syBFApOvPASpsYCq3MJr/LznyP/ag7WCa2O/dLAac1mSpSCM4/Q2cZD6uHCB9plCrlk/LwtYPYTaSmhv4D07IyCZ4QuRAEV8+QJOSDp2cYk3U5x7e6SGGR7aYbB/v7ezs4+yToqTSfJaj07u3PzBS00b2pQnkvm3OssNf40MOsFl9COaePA4OZsBqGrpSVbCBWk1BY/5UmHrulqZnrpGgqqqYWHeh1ltXPLOr+P/5jNYfqa+cq9L4rg/bz+X7ozGuDtVlfVr0JK8oIpFBVQskb6KCiB+caCC0/ErI8OjmGK+YjXpGJzIL4CwhqvcQPBHZGDjuiSoHRMHXiLrIrpwrGokfoNFuQoYm1HO9y1Yy9rQNyCggXXNaqKQEtWC7lcFdYG6sohvtrTa6UPD408DbFbw/7UsVClSQxJdxtyzWxBci2LNV/jy73TIJRpPCjeD6tsZGwyXjNSCAvcyyVhnOPMG+ZxlrxilnGP13F9YheeWasXsQkskk6F8pMs0FgTr7TgMCaEBDqNBC2EM5ItnV9K6C5vF4VJ1rZbV3WXFHXcCvMh2RVyiD9C/nEm+i1Fz9BV39ZkZ/J9CHGIub5ID2iFv7CJ7UYC5xQNm210o38ezw5sGFQo2Wxbeg5WPdh+SBdF2q6yF8xVaA3d4T1ox1s9DIHOmQXjhNRqkIp5f7ba9H9bB8TaIZVrjPmPIIKXiqPQ43lOjgas1kWcmYTSv5lk1IpZ5d8gJwFH5tam1huwoFmnwkthtFYNbmZB5szSP7RQGNJ1uh2PqWFWqAIPheDzMcZnaHhryIeDlzvb2e727vMfJo8OVw/SreSb5F7yXZIlPyaPksfJs+Qk4clfyd/JP8m/o3ujx6Ono+e99Pq1ledusrZGv78DkjzzPw==</latexit> deg: Dh b L = 0 <latexit sha1_base64="UWeD5s/KGOyjWi1NH5ACNakPF8M=">AAAFmnicjVTdbhw1FJ6WLm2Xn6Zw2V4YNpFakYaZLaTJRaQqCAGiQiVN2pXqEHk8Z3ZMPLZle3a7OPMYPA238BC8Dcczu1EWqFRLozn6fo7P8bGcGymcT9O/r11/78bg/Zu3bg8/+PCjj+9s3P3kpdON5XDCtdR2kjMHUig48cJLmBgLrM4lvMrPv4n8qxlYJ7Q69gsDpzWbKlEKzjxCZxtf0rkooGKePHsw36bg2UNyEOOH5AtCS8t4NiYX84tfxiSS5GxjlO6kuHZ3SQyyvTTDYH9/bzzeJ1lHpekoWa7nZ3dvvKCF5k0NynPJnHudpcafBma94BLaIW0cGMbP2RRC109LthAqSKktfsqTDl3T1cz00jUUVFMLD/U6ymrnFnW+jf+YzWH6mvnK/VsUwe28/l+6Mxrg7VZX1XdCSvKCKRQVULJG+igogfnGggvPxLSPDo5hgvmI16RiMyC+AsIar3EDwR2RKx3RJUHpkDrwFlkV04VjUSP1E8zJUcTajna4a8de1oC4BQVzrmtUFYGWrBZysSysDdSVq/hqT6+VPjw08jTEbg37TcdClSYxJLlEUa6ZLUiuZbHma3y5dxqEMo0HxfthlY2MTcYLRgphgXu5IIxznHnDPM6SVwyvkseLuD6xN55Zq+exCSySToTyoyzQWBOvtOAwJIQEOokELYQzki2cX0igHr1dFEZZ225d1V1S1HErzNtkV8hV/A7ydzPRzyh6Vl31bY3Go8chxCHm+k16QCv8hU1sNxI4p2jYbKMb/bN4dmDDSoWSzbal52DVo52v6bxI22X2grkKraE7vEftcKuHIdAZs2CckFqtpGLWn602/d/WAbF2lco1xvxHEMFLxVHo8TwnRyus1kWcmYTSX4wyasW08hfIScCRubWp9QYsaNqp8FIYrVWDm1mQObP0Vy0UhnSdbodDapgVqsBDIfh8DPEZWr015O3By/FOtruz+/NXo6ffLh+kW8m95PPkQZIlT5KnyffJ8+Qk4cnvyR/Jn8lfg/uDw8EPgx976fVrS8+nydoaHP8D9KnuRQ==</latexit> b L(w, h) = L(w) + 1 2 |w|2 h <latexit sha1_base64="8KsMlnBYj7Nqzn9aI3yB8d4xGRs=">AAAFnXicjVTRbhQ3FB0oW+jSltA+9qGGTaQ+QDSzhTR5QEKpqvYBIQgJWQmHyOO5s+PGY1u2Z5Otmf/o1/QVfqF/0+uZnSjbFglLozk659zre30t50YK59P072vXP7sx+vzmrS/Gt7/86us7G3e/ee10YzkccS21neXMgRQKjrzwEmbGAqtzCcf52c9RP16AdUKrQ780cFKzuRKl4MwjdboxpR4ufKBdppBLxs9aL2ogjjMJpKWEMmkq9nZKqBPzGsHpxiTdTnHt7JAIst00Q7C3tzud7pGsk9J0kqzWi9O7N17RQvOmBuW5ZM69yVLjTwKzXnAJ7Zg2DgzuzOYQukJasoVUQUpt8VOedOyar2amt66xoJpaeKjXWVY7t6zzB/iP2Rymr5mv3L9NkXyQ1/8rd4EGeLvVVfWrkJK8YgpNBZSskT4aSmC+seDCMzHv0ZNDmGE+4jWp2AKIr4CwxmvcQHBH5OAjuiRoHVMH3qKqYrpwiJNw5Dmck4PItZ3scNdOvawBeQsKzrmu0VUEWrJayOWqsDZQVw74ak9vlN7fN/IkxG4N+0PHQpUmEZLuKuSa2YLkWhZrcY0vd0+CUKbxoHg/rLKRscl4x0ghLHAvl4RxjjNvmMdZ8opZxj3exfWJXXhmrT6PTWCRdCaUn2SBxpp4pQWHMSEk0FkUaCGckWzp/FJCd3M7FCZZ225d9V1K1HErzMdsV8QBf4L904LoPYoxQ1d9W5Pp5McQ4hBzfZE+oRX+wia2GwWcUwzYbGM0xi/i2YENgwstm21Lz8Cqh9uP6XmRtqvsBXMVhobu8B62462ehkAXzIJxQmo1WMWiP1tt+r+tA3LtkMo1xvzHEMlLx0Ho+TwnBwNX6yLOTELp300yasW88u9Qk4Ajc2tT6wOwoHnnwkthtFYNbmZB5szS37VQCOm63I7H1DArVIGHQvD5GOMzNLw15OPg9XQ729neeflo8nR/9SDdSr5L7ic/JFnyU/I0+S15kRwlPPkz+St5n3wYfT/6ZfRs9Ly3Xr+2ivk2WVuj438AwcHy5w==</latexit> time scale a2 s2 <latexit sha1_base64="2HZNWC+nFfmIbsdDjsmoNvdxCA0=">AAAFg3icjVRfbxQ3EN9QjsKVP0n7UokX00skJJLTbQi0Lyeh9AEeUEVDAifhKPJ6Z2/deG3L9l44nOXT9LX9Pnwbxrt3UY6AhKXTzv3+jGc8ljMjhfOj0ae1az9c79348eat/k+379y9t77x8xuna8vhiGup7SRjDqRQcOSFlzAxFliVSXibnf4Z+bczsE5odejnBo4rNlWiEJx5hE7Wf6VMmpKNyWiYblNCnZhWbIx/TtYHo+GoXeRqkC6CQbJYr042rr+mueZ1BcpzyZx7l46MPw7MesElNH1aOzCMn7IphLbwhmwhlJNCW/wpT1p0RVcx00lXUFB1JTxUqyirnJtX2TZ+YzaH6SvmS/elKILbWfVVujUa4M1WW9VzISV5zRSKcihYLX0UFMB8bcGFl2LaReNDmGA+4jUp2QyIL4Gw2mvcQHBH5FJHdEFQ2qcOvEVWxXThUFRI/QVn5CBiTUs73LVlL2pA3IKCM64rVOWBFqwScr4orAnUFcv4ck/vlN7fN/I4xG4N+6BjoUqTGJJMoijTzOYk0zJf8dW++OM4CGVqD4p3wypqGZuMN4nkwgL3ck4Y5zjzmnmcJS+ZZdzjjVud2HvPrNVnsQkskk6E8oM00FgTL7Xg0CeEBDqJBM2FM5LNnZ9LoB69bRQGadNsXdZdUNRxK8y3ZJfIZfwd8u8z0QcUPcuuurYGu4PHIcQhZvr9aExL/IRNbDcSOKdo2GyiG/2zeHZgw1KFks2moadg1c7wCT3LR80ie85cidbQHt5O09/qYAh0xiwYJ6RWS6mYdWerTfe1VUCsWaZytTFXBBG8UByEDs8ycrDEKp3HmUko/PkgpVZMS3+OnAQcmVuZWmfAgqatCi+F0VrVuJkFmTFL/9FCYUhX6abfp4ZZoXI8FILPRx+fofTLR+dq8GZ3mD4dPv17b/Bsf/Eg3UzuJ78lD5M0+T15lrxIXiVHCU8+Jv8m/yX/93q9R73d3l4nvba28PySrKze+DNFxeXK</latexit> a = 0.1, s = 0.1
  18. <latexit sha1_base64="mEvOs+SNOOoWqgBurlufV+6IPT0=">AAAFsnicjVRfbxw1EN+WHpTjX1oeeTFcIhWRRHelLbxEqopQeahQSZM2ok5Ps97ZW3Ne27K9dz3c/Uh8Gp6Q4Lsw3r2Lcg1IXWnl0e/PeMZjObdK+jAe/3Xt+ns3Bu9/cPPD4Ucff/LpZzu3bj/3pnECT4VRxp3l4FFJjadBBoVn1iHUucIX+fyHxL9YoPPS6JOwsnhew0zLUgoIBE13HvNcNRiX0zj/ZtK+kuyALadzWo8o4qBsBYxryBVM4/KVbBlfygIrCOzJHRLucwwwnX/dTndG48Nx97GrwWQdjLL193R668YzXhjR1KiDUOD9y8nYhvMILkihsB3yxqMFMYcZxq7Plu0RVLDSOPp1YB26pavB9tItFHVTy4D1Ngq196s636c1ZfOUvoZQ+bdFCdzP6/+kO6NF0e51VT2WSrFnoElUYAmNCklQIoTGoY9P5KyPjk7wjPKxYFgFC2ShQgZNMLSBFJ6pjY6ZkpF0yD0GR6xO6eKJrIn6GZfsOGFtR3vatWMvaiDcocalMDWpishLqKVarQtrI/flJr7c00ttHj2y6jymbi38blKh2rAUMroGYp4bcAXLjSq2fE0ovz+PUtsmoBb9sMpGpSbTxWOFdCiCWjEQgmbeQKBZigociEAXdHtirwM4Z5apCSqSn0kdRpPIU02iMlLgkDEW+VkieCG9VbDyYaWQB/J2URxN2nbvsu6C4l44af9PdoncxO8gfzcT/5KTZ9NV39bo7ujbGNMQc/N6fMQrWuIutZsImlMy7LbJTf5FOjt0caMiyW7b8jk6fXB4ny+LcbvOXoCvyBq7wztoh3s9jJEvwKH1Uhm9kcpFf7bG9qurI2HtJpVvrL0iSOCF4jj2eJ6z4w1WmyLNTGEZ3owm3MlZFd4Qp5BG5rem1huooFmnokthjdENbeZQ5eD4b0ZqCvk23Q6H3IKTuqBDYfR8DOkZmrz96FwNnt89nDw4fPDLvdHDH9cP0s3si+yr7E42yb7LHmY/ZU+z00xkf2R/Zn9n/wzuDX4dwED00uvX1p7Ps61voP4FH2P5aA==</latexit> wi k+1 wi k = arwi b L(wk

    , hk) <latexit sha1_base64="h8F/D9RYSTmhPgmmJ/9OMRM+MQM=">AAAF7XicjVRPjxs1FJ+0DZQplG05gMTFJbtSkXZXSWkLl0pVEYJDhcp2t420TqM3njcZE49tbE+ywZ0L34Eb4spn4hvwMWrPJKtNl0q1NJqn35/n9/wsZ1pw64bDf3tXrl7rf/Dh9Y/SGx9/cvPTnVu3X1hVG4YnTAllxhlYFFziieNO4FgbhCoT+DKbfx/5lws0lit57FYaJxXMJC84Axeg6c4fNBM1Eu8bClobdUYOKAhdAqESMgH0ztQvX/GG0CXPsQRHnt5dTuf7w69T+lsNOTkg79BrMI6DmFJ0cNl9GuHpfJI26XRnMDwctotcDkbrYJCs17PprWvPaa5YXaF0TIC1p6OhdhMfN2QCm5TWFjWwOczQt0fUkL0A5aRQJnzSkRbd0lWgO+kWirKuuMNqG4XK2lWV7Yd/zGZD+gpcad8WRXA/q/6Xbo0aWbPXVvUjF4I8BxlEORZQCxcFBYKrDVr/lM+66NExjkM+4hQpYYHElUigdipswJklYqMjqiBBmlKLzgRWxnT+mFeB+hmX5ChiTUvbsGvLntcQcIMSl0xVQZV7WkDFxWpdWOOpLTbxxZ5OpXryRIuJj91q+F3FQqUiMSThbrB5psDkJFMi3/LVrvhu4rnUtUPJumEVtYhNxjtLcm6QObEiwFiYeQ0uzJKVYIC5cLe3J3bmwBi1jE2EIumYSzcYeRprYqXiDFNCiKfjSNCcWy1gZd1KIHXB20Z+MGqavYu6c4paZrh+l+wCuYnfQ/5+JnqHBs+mq66twb3BN97HIWbqbPiIluHnd0O7kQhziobdJrqDfxHPDo3fqIJkt2noHI08OHxAl/mwWWfPwZbB6tvDO2jSvQ5GTxdgUFsulNxI+aI7W6W7v6l8wJpNKltrfUkQwXPFke/wLCNHG6xSeZyZwMK9Hoyo4bPSvQ6cwDAyuzW1zhAKmrWqcCm0UrIOmxkUGRj6q+IyhHSbbtI0vlBc5uFQSHg+4jM0evvRuRy8uHc4enj48Jf7g8c/rB+k68mXyVfJ3WSUfJs8Tn5KniUnCUv+693sfd77oq/6f/b/6v/dSa/01p7Pkq3V/+cNwNsNhw==</latexit> ⇡ arwi b L(wk , 0) arwi ∂h b L(wk , 0)[hk] <latexit sha1_base64="I9O5rqdPtNZTLxsrXRBp/iP9u4U=">AAAF53icjVRdjxs1FJ2UBpbwtVseeXHJrlSkdJUEKLysVIoqeKhQ2e5uI9W7kcdzJ2PisY3tSRqc+Q28IV75WbzyS7ieZFabLki1FM3Nuedc3w/rpkYK54fDvzt33rnbffe9vfd7H3z40cef7B/cu3C6shzOuZbaTlLmQAoF5154CRNjgZWphJfp/Pvof7kA64RWZ35l4LJkMyVywZlHaLq/piXzRZqSp+SEDAeE/lqxjDSgLcMFszXilElTsKsxoU7MysZYfzed0/V0fDUekN5GhEjkKpZKRu9Pw/JK1Nu/UwqeEboUGRTMk2cPltP5YPjFdL8/PB42h9w2Rlujn2zP8+nB3Rc007wqQXkumXOvRkPjLwOzXnAJdY9WDgzjczaD0DSnJkcIZSTXFn/Kkwbd4ZXMbKg7KKiqFB7KXZSVzq3KdIDfGM1h+Ngq9yYpgoO0/E93IzTA66Mmqx+ElOQFU0jKIGeV9JGQA/OVBReeidnGOjmDCcYjXpOCLYD4AgirvMYLBHdEtjyic4LUHnXgLXpVDBfORImun2BJTiNWN26Htzbe6xwQt6BgyXWJrCzQnJVCrraJ1YG6vLVv1vRK6SdPjLwMsVrDftMxUaVJNAnOn89TzWxGUi2zHV3l828vg1Cm8qD4Zlh5JWOR8bWSTFjgXq4I4xxnXjGPs+QFs4x7fNW7E3vtmbV6GYvAJOlEKN8fheYt80ILDj1CSKCT6KCZcEaylfMrCdSjtrFCf1TXRzd51y7quBXm/2g3nK39FvS3E9H7FDVtVZuy+uP+lyHEIab69fCEFvgJh1hudOCcouCwjmrUL2LvwIaWhZTDuqZzsOrh8dd0mQ3rbfSMuQKloWnew7p3tIEh0AWzYJyQWrVUsdj0VpvQ7gvE6jaUq4y5RYjgNeM0hHb7nLZYqbM4Mwm5X/dH1IpZ4dfok4AjcztT2wgwoVnDwkdhtFYVXmZBpszSX7RQaNJdd93rUcOsUBk2heD66OEaGr25dG4bF+Pj0aPjRz9/1X/8dLuQ9pLPks+TB8ko+SZ5nPyYPE/OE57809nrHHTudUX39+4f3T831DudrebTZOd0//oXsQEJEQ==</latexit> E = 0, Var = a2 s2kAkk2 2 , Ak = rwi rh b L(wk , 0) <latexit sha1_base64="2ikxvJu2am14lBK63p63rQeJDX0=">AAAF4XicjVRRbxtFED6HGopbIIVHXrY4kYqURHaAwkukqlDBQ4VKmqSRuqk1tzfnW7y3e93ds2O29wN4Q7zyy5D4McyefVHcgNSVrBt93zezMzvjSSslnR+N/u5tvXer//4Htz8c3Ln70cefbN/79MyZ2go8FUYZe56CQyU1nnrpFZ5XFqFMFb5IZ99H/sUcrZNGn/hlhRclTLXMpQBP0GR7zkvwRZqyJ+yI7fPcguCgqgLYIeNOTkt4RYaGVAHj9ydh8Uo2jP+AysOEoydwITMswLOnDxaT2d7oS7bH+OvXNWSsDW3LcAa2oejc46UPrgSlmsn2cHQwag+7aYzXxjBZn2eTe7ee88yIukTthQLnXo5Hlb8IYL0UCpsBrx1WIGYwxdA+S8N2CcpYbiz9tGctuqEroVpJN1DUdSk9lpsolM4ty3SPvjGao/CxPPe2KIJ7afmfdOtYoWh226x+lEqx56BJlGEOtfJRkCP42qILT+V0ZR2d4DnFY96wAubIfIEMam/oAikcU52OmZyRdMAdekusjuHCiSyJ+hkX7DhiTUs7urVlr3Ig3KLGhTAlqbLAcyilWq4TawJ3eWdfr+mlNo8fV+oixGor+M3ERLVh0WQ0NGKWGrAZS43KNvxqn393EaSuao9arJqV1yoWGeeUZdKi8GrJQAjqeQ2eeikKoAH1NM+bHbv0YK1ZxCIoSX4utR+OQzt/ojBS4IAxFvh5JHgmXaVg6fxSYTuTrRWG46bZva67orgTVlb/J7tGdvY7yN/Nid/n5NNVtSpreDj8KoTYxNRcjo54QZ+wQ+VGgvoUHXaa6E3+8/h2aEOnIslO0/AZWr1/8A1fZKNmHT0DV5BraB9vvxnsrmAMfA4WKyeV0Z1Uzldva6rQ/ccJa7pQrq6qG4IIXimOQ+j2znGHlSaLPVOY+zfDMbdyWvg3xCmklrmNrq0cKKFpq6KhqIzRNV1mUaVg+a9GajL5Jt0MBrwCK3VGj8JofQxoDY3fXjo3jbPDg/HDg4e/fD189GS9kG4nnydfJA+ScfJt8ij5KXmWnCYi+ae31bvTu9sX/d/7f/T/XEm3emufz5KN0//rXyeaCHo=</latexit> E = a 2 s2rwi Dh b L(wk , 0), Var = small <latexit sha1_base64="4eCcIZWwQxyB/tiHUSqBEi26akw=">AAAFi3icjVRfjxM3EDeUFEhLOcojL4bcSVTiTgm0FFBPOlFV9AFV9LiDSPiIvN7ZrBuvbdnehNTsaz9NX9vvwrdhvJucLlwrYWm1o9+f8YzHcmaV9GE4/HDh4heXel9evnK1/9XX1765vnXj21fe1E7AsTDKuHHGPSip4TjIoGBsHfAqU/A6m/2c+NdzcF4afRSWFk4qPtWykIIHhCZbdJ/uMq5sySnTPFOc3Z7ExVvZ0Od3F5PZd3SyNRjuDdtFzwejVTA4eEza9WJy49JLlhtRV6CDUNz7N6OhDSeRuyCFgqbPag+WixmfQmwbaOgOQjktjMNPB9qiG7qK2066gYKuKxmg2kR55f2yyu7hP2XzmL7iofSfihJ4L6v+k26NFkSz01b1TCpFX3KNohwKXquQBAXwUDvw8bmcdtH+EYwxHw2GlnwONJRAeR0MbiCFp2qto6agKO0zD8Ehq1O6eCQrpH6DBT1MWNPSHndt2dMaEHegYSFMhao8soJXUi1XhTWR+WIdn+3pjTZPn1p1ElO3lv9pUqHa0BRSHLyYZYa7nGZG5Ru+OhSPTqLUtg6gRTesolapyXSjaC4diKCWlAuBM695wFmKkjsuAt68zYm9C9w5s0hNYJFsLHUYjCJLNYnSSAF9Smlk40SwXHqr+NKHpQIW0NtGcTBqmp2zulOKeeGk/T/ZGXIdf4b880zsNkPPuquurcH9wYMY0xAz8264z0r8xW1sNxE4p2TYbpIb/fN0duDiWoWS7aZhM3B6d+8HtsiHzSp7zn2J1tge3m7T3+lgiGzOHVgvldFrqZx3Z2ts93dVRKxZp/K1tecECTxVHMYOzzJ6uMYqk6eZKSjC+8GIOTktw3vkFODI/MbUOgMWNG1VeCmsMbrGzRyojDv2h5EaQ7ZJN/0+s9xJneOhUHw++vgMjT59dM4Hr+7vjR7uPfz9+8HBT917RK6QW+QOuUtG5EdyQH4lL8gxEeQv8jf5h/zbu9Z70HvSW2kvXlh5bpKN1fvlIzzo6fQ=</latexit> = arwi L(wk) <latexit sha1_base64="huaKS7ThmeGsOhKxGm+6GDa0o8c=">AAAFf3icjVRfbxQ3EHcoR+FoaSi88eJyF6UPcNoLKgWJSBE80AeEaEjgJBwFr3f2zsRrW7b3wmH2u/AK34hvw3g3F+VIK2FptaPfn/GMx3JulfQhy76uXfjpYu/Sz5ev9K/+8uu139av//7Km9oJ2BdGGTfJuQclNewHGRRMrANe5Qpe50dPEv96Ds5Lo/fCwsJBxadallLwgNDh+s23b4fb2ZAaTYfsKa8qPtzcPFwfZKOsXfR8MD4JBjsPSbteHF6/+JIVRtQV6CAU9/7NOLPhIHIXpFDQ9FntwXJxxKcQ26IbuoFQQUvj8NOBtuiKruK2k66goOtKBqhWUV55v6jyO/hP2Tymr3iY+e9FCbyTV/9Jt0YLotloq3oqlaIvuUZRASWvVUiCEnioHfj4TE67aHsPJpiPBkNnfA40zIDyOhjcQApP1VJHTUlR2mcegkNWp3RxT1ZIPYdjupuwpqU97tqypzUg7kDDsTA4IV1EVvJKqsVJYU1kvlzGZ3t6o83jx1YdxNSt5R9MKlQbmkKaKxTlhruC5kYVK746lA8OotS2DqBFN6yyVqnJdItoIR2IoBaUC4Ezr3nAWYoZd1wEvG2rE3sfuHPmODWBRbKJ1GEwjizVJGZGCuhTSiObJIIV0lvFFz4sFLCA3jaKg3HTbJzVnVLMCyft/8nOkMv4B+Q/ZmJ/MPQsu+raGmwN7sWYhpib99k2m+EvDrHdROCckmHYJDf65+nswMWlCiXDpmFH4PTd0V/suMiak+wF9zO0xvbw7jb9jQ6GyObcgfVSGb2Uynl3tsZ2f1dFxJplKl9be06QwFPFbuzwPKe7S6wyRZqZgjJ8HIyZk9NZ+IicAhyZX5laZ8CCpq0KL4U1Rte4mQOVc8feGakxZKt00+8zy53UBR4Kxeejj8/Q+PtH53zwams0vj+6/+/WYOdR9x6Ry+QWuU3+JGPyN9kh/5AXZJ8I8oF8Ip/Jl95ab7M36mWd9MLaiecGWVm9h98A1q/kSw==</latexit> “= 0 on G” <latexit sha1_base64="GDFdex3JZQXfuOpeVeMRYxKOUW0=">AAAF3nicjVRPbxQ3FJ9QloYppaE99mK6iQRqEu2mLe0FCVFV9IAqGhJYKV62Hs+bHXc9trE9u2zNXHuruPar8Wna55ndKEtAYqSRn35/nt/zs5wZKZwfDN5uXfnkau/ap9vX089ufH7zi51bXz5zurYcTrmW2o4y5kAKBadeeAkjY4FVmYTn2eznyD+fg3VCqxO/NDCu2FSJQnDmEZrsOJrJGkgIDTkgtLCMUyZNyY4IVSyTjN6ehMUL0RBqmPWCyRdHEwqeEboQOZTMk8d3FpPZ/uDuWYQx6pZxSl++rFlOVsu3hPJce0fSJp3s9AeHg/Yjl4PhKugnq+/J5NbVpzTXvK5AeS6Zc2fDgfHjECviEpqU1g4M4zM2hdCeSUP2EMpJoS3+ypMW3dBVzHTSDRRUXQkP1SbKKueWVbaPa8zmMH3FfOneFUVwP6veS7dGA7zZa6t6JKQkT5lCUQ4Fq6WPggKYry248FhMu+j+CYwwH/GalGwOxJdAWO01biC4I3KtI7ogKE2pA2+RVTFdOBEVUr/BghxHrGlph7u27HkNiFtQsOC6QlUeaMEqIZerwppAXbGOL/Z0pvTDh0aOQ+zWsL90LFRpEkOCl4fPMs1sTjIt8w1f7YufxkEoU3tQvBtWUcvYZLykJBcWuJdLwjjHmdfM4yx5yfB6erzMmxN75Zm1ehGbwCLpSCjfHwYaa+KlFhxSQkigo0jQXDgj2dL5pQTq0dtGoT9smr2LunOKOm6F+ZDsArmOP0L+cSZ6m6Jn3VXXVv+o/10IcYiZfjW4T0tcwi62GwmcUzTsNtGN/nk8O7BhrULJbtPQGVh1cPgDXeSDZpU9Z65Ea2gP76BJ9zoYAp0zC8YJqdVaKubd2WrTrbYKiDXrVK425pIggueK49DhWUaO11il8zgzCYV/3R9SK6alf42cBByZ25haZ8CCpq0KL4XRWtW4mQWZMUv/1EJhSDfpJk3jEyZUjodC8PmIz9Dw3UfncvDs6HB47/De79/3H/yyepC2k6+Tb5I7yTD5MXmQ/Jo8SU4TnrxN/tva3rre+6P3d++f3ptOemVr5fkq2fh6//4PuFEIVg==</latexit> a 2 rwi ∂2 h b L(wk , 0)[hk , hk] + · · · leading order Degenerate case Suggests a limit evolution on Γ at time scale α2σ2
  19. <latexit sha1_base64="mEvOs+SNOOoWqgBurlufV+6IPT0=">AAAFsnicjVRfbxw1EN+WHpTjX1oeeTFcIhWRRHelLbxEqopQeahQSZM2ok5Ps97ZW3Ne27K9dz3c/Uh8Gp6Q4Lsw3r2Lcg1IXWnl0e/PeMZjObdK+jAe/3Xt+ns3Bu9/cPPD4Ucff/LpZzu3bj/3pnECT4VRxp3l4FFJjadBBoVn1iHUucIX+fyHxL9YoPPS6JOwsnhew0zLUgoIBE13HvNcNRiX0zj/ZtK+kuyALadzWo8o4qBsBYxryBVM4/KVbBlfygIrCOzJHRLucwwwnX/dTndG48Nx97GrwWQdjLL193R668YzXhjR1KiDUOD9y8nYhvMILkihsB3yxqMFMYcZxq7Plu0RVLDSOPp1YB26pavB9tItFHVTy4D1Ngq196s636c1ZfOUvoZQ+bdFCdzP6/+kO6NF0e51VT2WSrFnoElUYAmNCklQIoTGoY9P5KyPjk7wjPKxYFgFC2ShQgZNMLSBFJ6pjY6ZkpF0yD0GR6xO6eKJrIn6GZfsOGFtR3vatWMvaiDcocalMDWpishLqKVarQtrI/flJr7c00ttHj2y6jymbi38blKh2rAUMroGYp4bcAXLjSq2fE0ovz+PUtsmoBb9sMpGpSbTxWOFdCiCWjEQgmbeQKBZigociEAXdHtirwM4Z5apCSqSn0kdRpPIU02iMlLgkDEW+VkieCG9VbDyYaWQB/J2URxN2nbvsu6C4l44af9PdoncxO8gfzcT/5KTZ9NV39bo7ujbGNMQc/N6fMQrWuIutZsImlMy7LbJTf5FOjt0caMiyW7b8jk6fXB4ny+LcbvOXoCvyBq7wztoh3s9jJEvwKH1Uhm9kcpFf7bG9qurI2HtJpVvrL0iSOCF4jj2eJ6z4w1WmyLNTGEZ3owm3MlZFd4Qp5BG5rem1huooFmnokthjdENbeZQ5eD4b0ZqCvk23Q6H3IKTuqBDYfR8DOkZmrz96FwNnt89nDw4fPDLvdHDH9cP0s3si+yr7E42yb7LHmY/ZU+z00xkf2R/Zn9n/wzuDX4dwED00uvX1p7Ps61voP4FH2P5aA==</latexit> wi k+1 wi k = arwi b L(wk

    , hk) <latexit sha1_base64="h8F/D9RYSTmhPgmmJ/9OMRM+MQM=">AAAF7XicjVRPjxs1FJ+0DZQplG05gMTFJbtSkXZXSWkLl0pVEYJDhcp2t420TqM3njcZE49tbE+ywZ0L34Eb4spn4hvwMWrPJKtNl0q1NJqn35/n9/wsZ1pw64bDf3tXrl7rf/Dh9Y/SGx9/cvPTnVu3X1hVG4YnTAllxhlYFFziieNO4FgbhCoT+DKbfx/5lws0lit57FYaJxXMJC84Axeg6c4fNBM1Eu8bClobdUYOKAhdAqESMgH0ztQvX/GG0CXPsQRHnt5dTuf7w69T+lsNOTkg79BrMI6DmFJ0cNl9GuHpfJI26XRnMDwctotcDkbrYJCs17PprWvPaa5YXaF0TIC1p6OhdhMfN2QCm5TWFjWwOczQt0fUkL0A5aRQJnzSkRbd0lWgO+kWirKuuMNqG4XK2lWV7Yd/zGZD+gpcad8WRXA/q/6Xbo0aWbPXVvUjF4I8BxlEORZQCxcFBYKrDVr/lM+66NExjkM+4hQpYYHElUigdipswJklYqMjqiBBmlKLzgRWxnT+mFeB+hmX5ChiTUvbsGvLntcQcIMSl0xVQZV7WkDFxWpdWOOpLTbxxZ5OpXryRIuJj91q+F3FQqUiMSThbrB5psDkJFMi3/LVrvhu4rnUtUPJumEVtYhNxjtLcm6QObEiwFiYeQ0uzJKVYIC5cLe3J3bmwBi1jE2EIumYSzcYeRprYqXiDFNCiKfjSNCcWy1gZd1KIHXB20Z+MGqavYu6c4paZrh+l+wCuYnfQ/5+JnqHBs+mq66twb3BN97HIWbqbPiIluHnd0O7kQhziobdJrqDfxHPDo3fqIJkt2noHI08OHxAl/mwWWfPwZbB6tvDO2jSvQ5GTxdgUFsulNxI+aI7W6W7v6l8wJpNKltrfUkQwXPFke/wLCNHG6xSeZyZwMK9Hoyo4bPSvQ6cwDAyuzW1zhAKmrWqcCm0UrIOmxkUGRj6q+IyhHSbbtI0vlBc5uFQSHg+4jM0evvRuRy8uHc4enj48Jf7g8c/rB+k68mXyVfJ3WSUfJs8Tn5KniUnCUv+693sfd77oq/6f/b/6v/dSa/01p7Pkq3V/+cNwNsNhw==</latexit> ⇡ arwi b L(wk , 0) arwi ∂h b L(wk , 0)[hk] <latexit sha1_base64="I9O5rqdPtNZTLxsrXRBp/iP9u4U=">AAAF53icjVRdjxs1FJ2UBpbwtVseeXHJrlSkdJUEKLysVIoqeKhQ2e5uI9W7kcdzJ2PisY3tSRqc+Q28IV75WbzyS7ieZFabLki1FM3Nuedc3w/rpkYK54fDvzt33rnbffe9vfd7H3z40cef7B/cu3C6shzOuZbaTlLmQAoF5154CRNjgZWphJfp/Pvof7kA64RWZ35l4LJkMyVywZlHaLq/piXzRZqSp+SEDAeE/lqxjDSgLcMFszXilElTsKsxoU7MysZYfzed0/V0fDUekN5GhEjkKpZKRu9Pw/JK1Nu/UwqeEboUGRTMk2cPltP5YPjFdL8/PB42h9w2Rlujn2zP8+nB3Rc007wqQXkumXOvRkPjLwOzXnAJdY9WDgzjczaD0DSnJkcIZSTXFn/Kkwbd4ZXMbKg7KKiqFB7KXZSVzq3KdIDfGM1h+Ngq9yYpgoO0/E93IzTA66Mmqx+ElOQFU0jKIGeV9JGQA/OVBReeidnGOjmDCcYjXpOCLYD4AgirvMYLBHdEtjyic4LUHnXgLXpVDBfORImun2BJTiNWN26Htzbe6xwQt6BgyXWJrCzQnJVCrraJ1YG6vLVv1vRK6SdPjLwMsVrDftMxUaVJNAnOn89TzWxGUi2zHV3l828vg1Cm8qD4Zlh5JWOR8bWSTFjgXq4I4xxnXjGPs+QFs4x7fNW7E3vtmbV6GYvAJOlEKN8fheYt80ILDj1CSKCT6KCZcEaylfMrCdSjtrFCf1TXRzd51y7quBXm/2g3nK39FvS3E9H7FDVtVZuy+uP+lyHEIab69fCEFvgJh1hudOCcouCwjmrUL2LvwIaWhZTDuqZzsOrh8dd0mQ3rbfSMuQKloWnew7p3tIEh0AWzYJyQWrVUsdj0VpvQ7gvE6jaUq4y5RYjgNeM0hHb7nLZYqbM4Mwm5X/dH1IpZ4dfok4AjcztT2wgwoVnDwkdhtFYVXmZBpszSX7RQaNJdd93rUcOsUBk2heD66OEaGr25dG4bF+Pj0aPjRz9/1X/8dLuQ9pLPks+TB8ko+SZ5nPyYPE/OE57809nrHHTudUX39+4f3T831DudrebTZOd0//oXsQEJEQ==</latexit> E = 0, Var = a2 s2kAkk2 2 , Ak = rwi rh b L(wk , 0) <latexit sha1_base64="4eCcIZWwQxyB/tiHUSqBEi26akw=">AAAFi3icjVRfjxM3EDeUFEhLOcojL4bcSVTiTgm0FFBPOlFV9AFV9LiDSPiIvN7ZrBuvbdnehNTsaz9NX9vvwrdhvJucLlwrYWm1o9+f8YzHcmaV9GE4/HDh4heXel9evnK1/9XX1765vnXj21fe1E7AsTDKuHHGPSip4TjIoGBsHfAqU/A6m/2c+NdzcF4afRSWFk4qPtWykIIHhCZbdJ/uMq5sySnTPFOc3Z7ExVvZ0Od3F5PZd3SyNRjuDdtFzwejVTA4eEza9WJy49JLlhtRV6CDUNz7N6OhDSeRuyCFgqbPag+WixmfQmwbaOgOQjktjMNPB9qiG7qK2066gYKuKxmg2kR55f2yyu7hP2XzmL7iofSfihJ4L6v+k26NFkSz01b1TCpFX3KNohwKXquQBAXwUDvw8bmcdtH+EYwxHw2GlnwONJRAeR0MbiCFp2qto6agKO0zD8Ehq1O6eCQrpH6DBT1MWNPSHndt2dMaEHegYSFMhao8soJXUi1XhTWR+WIdn+3pjTZPn1p1ElO3lv9pUqHa0BRSHLyYZYa7nGZG5Ru+OhSPTqLUtg6gRTesolapyXSjaC4diKCWlAuBM695wFmKkjsuAt68zYm9C9w5s0hNYJFsLHUYjCJLNYnSSAF9Smlk40SwXHqr+NKHpQIW0NtGcTBqmp2zulOKeeGk/T/ZGXIdf4b880zsNkPPuquurcH9wYMY0xAz8264z0r8xW1sNxE4p2TYbpIb/fN0duDiWoWS7aZhM3B6d+8HtsiHzSp7zn2J1tge3m7T3+lgiGzOHVgvldFrqZx3Z2ts93dVRKxZp/K1tecECTxVHMYOzzJ6uMYqk6eZKSjC+8GIOTktw3vkFODI/MbUOgMWNG1VeCmsMbrGzRyojDv2h5EaQ7ZJN/0+s9xJneOhUHw++vgMjT59dM4Hr+7vjR7uPfz9+8HBT917RK6QW+QOuUtG5EdyQH4lL8gxEeQv8jf5h/zbu9Z70HvSW2kvXlh5bpKN1fvlIzzo6fQ=</latexit> = arwi L(wk) <latexit sha1_base64="huaKS7ThmeGsOhKxGm+6GDa0o8c=">AAAFf3icjVRfbxQ3EHcoR+FoaSi88eJyF6UPcNoLKgWJSBE80AeEaEjgJBwFr3f2zsRrW7b3wmH2u/AK34hvw3g3F+VIK2FptaPfn/GMx3JulfQhy76uXfjpYu/Sz5ev9K/+8uu139av//7Km9oJ2BdGGTfJuQclNewHGRRMrANe5Qpe50dPEv96Ds5Lo/fCwsJBxadallLwgNDh+s23b4fb2ZAaTYfsKa8qPtzcPFwfZKOsXfR8MD4JBjsPSbteHF6/+JIVRtQV6CAU9/7NOLPhIHIXpFDQ9FntwXJxxKcQ26IbuoFQQUvj8NOBtuiKruK2k66goOtKBqhWUV55v6jyO/hP2Tymr3iY+e9FCbyTV/9Jt0YLotloq3oqlaIvuUZRASWvVUiCEnioHfj4TE67aHsPJpiPBkNnfA40zIDyOhjcQApP1VJHTUlR2mcegkNWp3RxT1ZIPYdjupuwpqU97tqypzUg7kDDsTA4IV1EVvJKqsVJYU1kvlzGZ3t6o83jx1YdxNSt5R9MKlQbmkKaKxTlhruC5kYVK746lA8OotS2DqBFN6yyVqnJdItoIR2IoBaUC4Ezr3nAWYoZd1wEvG2rE3sfuHPmODWBRbKJ1GEwjizVJGZGCuhTSiObJIIV0lvFFz4sFLCA3jaKg3HTbJzVnVLMCyft/8nOkMv4B+Q/ZmJ/MPQsu+raGmwN7sWYhpib99k2m+EvDrHdROCckmHYJDf65+nswMWlCiXDpmFH4PTd0V/suMiak+wF9zO0xvbw7jb9jQ6GyObcgfVSGb2Uynl3tsZ2f1dFxJplKl9be06QwFPFbuzwPKe7S6wyRZqZgjJ8HIyZk9NZ+IicAhyZX5laZ8CCpq0KL4U1Rte4mQOVc8feGakxZKt00+8zy53UBR4Kxeejj8/Q+PtH53zwams0vj+6/+/WYOdR9x6Ry+QWuU3+JGPyN9kh/5AXZJ8I8oF8Ip/Jl95ab7M36mWd9MLaiecGWVm9h98A1q/kSw==</latexit> “= 0 on G” <latexit sha1_base64="GDFdex3JZQXfuOpeVeMRYxKOUW0=">AAAF3nicjVRPbxQ3FJ9QloYppaE99mK6iQRqEu2mLe0FCVFV9IAqGhJYKV62Hs+bHXc9trE9u2zNXHuruPar8Wna55ndKEtAYqSRn35/nt/zs5wZKZwfDN5uXfnkau/ap9vX089ufH7zi51bXz5zurYcTrmW2o4y5kAKBadeeAkjY4FVmYTn2eznyD+fg3VCqxO/NDCu2FSJQnDmEZrsOJrJGkgIDTkgtLCMUyZNyY4IVSyTjN6ehMUL0RBqmPWCyRdHEwqeEboQOZTMk8d3FpPZ/uDuWYQx6pZxSl++rFlOVsu3hPJce0fSJp3s9AeHg/Yjl4PhKugnq+/J5NbVpzTXvK5AeS6Zc2fDgfHjECviEpqU1g4M4zM2hdCeSUP2EMpJoS3+ypMW3dBVzHTSDRRUXQkP1SbKKueWVbaPa8zmMH3FfOneFUVwP6veS7dGA7zZa6t6JKQkT5lCUQ4Fq6WPggKYry248FhMu+j+CYwwH/GalGwOxJdAWO01biC4I3KtI7ogKE2pA2+RVTFdOBEVUr/BghxHrGlph7u27HkNiFtQsOC6QlUeaMEqIZerwppAXbGOL/Z0pvTDh0aOQ+zWsL90LFRpEkOCl4fPMs1sTjIt8w1f7YufxkEoU3tQvBtWUcvYZLykJBcWuJdLwjjHmdfM4yx5yfB6erzMmxN75Zm1ehGbwCLpSCjfHwYaa+KlFhxSQkigo0jQXDgj2dL5pQTq0dtGoT9smr2LunOKOm6F+ZDsArmOP0L+cSZ6m6Jn3VXXVv+o/10IcYiZfjW4T0tcwi62GwmcUzTsNtGN/nk8O7BhrULJbtPQGVh1cPgDXeSDZpU9Z65Ea2gP76BJ9zoYAp0zC8YJqdVaKubd2WrTrbYKiDXrVK425pIggueK49DhWUaO11il8zgzCYV/3R9SK6alf42cBByZ25haZ8CCpq0KL4XRWtW4mQWZMUv/1EJhSDfpJk3jEyZUjodC8PmIz9Dw3UfncvDs6HB47/De79/3H/yyepC2k6+Tb5I7yTD5MXmQ/Jo8SU4TnrxN/tva3rre+6P3d++f3ptOemVr5fkq2fh6//4PuFEIVg==</latexit> a 2 rwi ∂2 h b L(wk , 0)[hk , hk] + · · · leading order <latexit sha1_base64="z0Bov1GY9TpjxbRr0qLKuYfMD20=">AAAHGXicjVRfb+NEEE/DUQ7z7wq88TIQV+qJtEqKOFCrSkev3PFQodLrXSt102i9nsRL1l7f7jq54PMn4TPwIXhDvPLEt2HWTkpzBeks2R7N7zez828nypW0rtf7e6391p2319+5+27w3vsffPjRvY2Pn1tdGIHPhFbaXETcopIZPnPSKbzIDfI0UngeTR55/HyKxkqdnbl5joOUjzM5koI7Ug031n5jBuOSOXzpolF5lqRVxYABbB3hGDM03CEIOuA+KY/RAeuySBVYhoyrPOHMaeiFFesCz2L6LkErxykHjzbisMfG+IKY3VvUjEeKD2fsCJXjQ4aO7GYyxoQ7OAaGLwo59WfsAGOX7GftBsF3DpxMEazgCiF0wwnsHSwcNnFd7VbNwVe7MAm7QNGHsxCEzqgaY7RAkYXnYTeYJVIkYLWaog3YZQAxnG85ynb/gO3DNrCUu8SkcDJkT3jqc4oowi3PuU9euxAfDh0EbP9Loj9e6Gsjne815PpzRbXIYYmTmQvYIAhrrPs4BG4QYrTCyAhjkBk4NKkFPYLw32qEw3ud3k6vfuC20F8IndbiORlu3HnKYi2KFDMnFLf2st/L3aDkxkmhsApYYTHnYsLHWNbjVMEmqWIYaUNv5qDWrvBSnjfUFS1mRSodpqtanlo7T6Mu/b03S+59cezrJK/sRul/wrVhjqLarKN6IpWCpzwjUowjXijnCSPkrjBoy2M5bqSDM7wgf77VCZ8iuASBF07TAVJYUEueLzJRA2bRGUIz7648owGz8CPO4NTrqhq2dGqNXsdAekP3ZCY0DUdGN2nEU6nmi8CqktnRUr6Z02WmDw9zNSh9tjn/RftAMw1eBLoPYhJpbmKItIpX7Ao3+nZQyiwvHGaiadaoUD5Jf78hlgaFU3PgQlDPC7q/MYiEGy5oolbrSneeG6NnPgkKkl3IzHX6ZT29ItFSYAAAJbvwAIulzRWfWzdXWO+LWio7/aravMm7hpif5vz/aDfApfwG9DczYp8zsllm1aTV2e18VZa+iZF+2TtgCf3KkNL1APXJG4SVtyb7qa8dmnLJIkpIi3GCJtve+ZrN4l618B5zm5BpWRdvuwo2GzXtoSld6dxKpbMlVU7LxWYoF2ulJF21dGWLPL9F8MprxmnZ6KMITpe6VMe+ZwpH7lWnz4wcJ+4VYQqpZXala40BBTSuWTQUudZZQYcZVBE3tFxlRiJbhasgYDk3MoupKEDrI6A11H996dwWnu/u9B/sPPhpt/Pw+8VCutv6rPVFa6vVb33Tetj6oXXSetYS7U/be+1H7aP1X9d/X/9j/c+G2l5b2HzSWnnW//oHIeJs7Q==</latexit> Thm (Degenerate case) Let a ! 0 and s ! s0 0, and rwDh b L ⌘ 0. At time scale tk := a2s2k, w converges to W, which solves dW(t) = PGb(W(t)) dBt + F(W(t)) : bb>(W(t)) dt b, F are described in terms of b L
  20. <latexit sha1_base64="d/ItkzxdexVI7x2P/vQ+SqOp15s=">AAAFoHicjVTNbtw2EFZ+tk23P3GaYy9s1gZ6SAzJbVy7gIHALdAcgiZx7GTR0DAoarRiTZEESe16y+hJ+jS9tk+Qt8lQWhnetAFCQNDg+xnOcAjmRgrn0/Tttes3bo4++fTWZ+PPv/jyq9sbd75+6XRjOZxwLbWd5syBFApOvPASpsYCq3MJr/LznyP/ag7WCa2O/dLAac1mSpSCM4/Q2cZD6uHCB9plCrlk/LwtYPYTaSmhv4D07IyCZ4QuRAEV8+QJOSDp2cYk3U5x7e6SGGR7aYbB/v7ezs4+yToqTSfJaj07u3PzBS00b2pQnkvm3OssNf40MOsFl9COaePA4OZsBqGrpSVbCBWk1BY/5UmHrulqZnrpGgqqqYWHeh1ltXPLOr+P/5jNYfqa+cq9L4rg/bz+X7ozGuDtVlfVr0JK8oIpFBVQskb6KCiB+caCC0/ErI8OjmGK+YjXpGJzIL4CwhqvcQPBHZGDjuiSoHRMHXiLrIrpwrGokfoNFuQoYm1HO9y1Yy9rQNyCggXXNaqKQEtWC7lcFdYG6sohvtrTa6UPD408DbFbw/7UsVClSQxJdxtyzWxBci2LNV/jy73TIJRpPCjeD6tsZGwyXjNSCAvcyyVhnOPMG+ZxlrxilnGP13F9YheeWasXsQkskk6F8pMs0FgTr7TgMCaEBDqNBC2EM5ItnV9K6C5vF4VJ1rZbV3WXFHXcCvMh2RVyiD9C/nEm+i1Fz9BV39ZkZ/J9CHGIub5ID2iFv7CJ7UYC5xQNm210o38ezw5sGFQo2Wxbeg5WPdh+SBdF2q6yF8xVaA3d4T1ox1s9DIHOmQXjhNRqkIp5f7ba9H9bB8TaIZVrjPmPIIKXiqPQ43lOjgas1kWcmYTSv5lk1IpZ5d8gJwFH5tam1huwoFmnwkthtFYNbmZB5szSP7RQGNJ1uh2PqWFWqAIPheDzMcZnaHhryIeDlzvb2e727vMfJo8OVw/SreSb5F7yXZIlPyaPksfJs+Qk4clfyd/JP8m/o3ujx6Ono+e99Pq1ledusrZGv78DkjzzPw==</latexit> deg: Dh b L = 0 <latexit sha1_base64="UWeD5s/KGOyjWi1NH5ACNakPF8M=">AAAFmnicjVTdbhw1FJ6WLm2Xn6Zw2V4YNpFakYaZLaTJRaQqCAGiQiVN2pXqEHk8Z3ZMPLZle3a7OPMYPA238BC8Dcczu1EWqFRLozn6fo7P8bGcGymcT9O/r11/78bg/Zu3bg8/+PCjj+9s3P3kpdON5XDCtdR2kjMHUig48cJLmBgLrM4lvMrPv4n8qxlYJ7Q69gsDpzWbKlEKzjxCZxtf0rkooGKePHsw36bg2UNyEOOH5AtCS8t4NiYX84tfxiSS5GxjlO6kuHZ3SQyyvTTDYH9/bzzeJ1lHpekoWa7nZ3dvvKCF5k0NynPJnHudpcafBma94BLaIW0cGMbP2RRC109LthAqSKktfsqTDl3T1cz00jUUVFMLD/U6ymrnFnW+jf+YzWH6mvnK/VsUwe28/l+6Mxrg7VZX1XdCSvKCKRQVULJG+igogfnGggvPxLSPDo5hgvmI16RiMyC+AsIar3EDwR2RKx3RJUHpkDrwFlkV04VjUSP1E8zJUcTajna4a8de1oC4BQVzrmtUFYGWrBZysSysDdSVq/hqT6+VPjw08jTEbg37TcdClSYxJLlEUa6ZLUiuZbHma3y5dxqEMo0HxfthlY2MTcYLRgphgXu5IIxznHnDPM6SVwyvkseLuD6xN55Zq+exCSySToTyoyzQWBOvtOAwJIQEOokELYQzki2cX0igHr1dFEZZ225d1V1S1HErzNtkV8hV/A7ydzPRzyh6Vl31bY3Go8chxCHm+k16QCv8hU1sNxI4p2jYbKMb/bN4dmDDSoWSzbal52DVo52v6bxI22X2grkKraE7vEftcKuHIdAZs2CckFqtpGLWn602/d/WAbF2lco1xvxHEMFLxVHo8TwnRyus1kWcmYTSX4wyasW08hfIScCRubWp9QYsaNqp8FIYrVWDm1mQObP0Vy0UhnSdbodDapgVqsBDIfh8DPEZWr015O3By/FOtruz+/NXo6ffLh+kW8m95PPkQZIlT5KnyffJ8+Qk4cnvyR/Jn8lfg/uDw8EPgx976fVrS8+nydoaHP8D9KnuRQ==</latexit>

    b L(w, h) = L(w) + 1 2 |w|2 h <latexit sha1_base64="8KsMlnBYj7Nqzn9aI3yB8d4xGRs=">AAAFnXicjVTRbhQ3FB0oW+jSltA+9qGGTaQ+QDSzhTR5QEKpqvYBIQgJWQmHyOO5s+PGY1u2Z5Otmf/o1/QVfqF/0+uZnSjbFglLozk659zre30t50YK59P072vXP7sx+vzmrS/Gt7/86us7G3e/ee10YzkccS21neXMgRQKjrzwEmbGAqtzCcf52c9RP16AdUKrQ780cFKzuRKl4MwjdboxpR4ufKBdppBLxs9aL2ogjjMJpKWEMmkq9nZKqBPzGsHpxiTdTnHt7JAIst00Q7C3tzud7pGsk9J0kqzWi9O7N17RQvOmBuW5ZM69yVLjTwKzXnAJ7Zg2DgzuzOYQukJasoVUQUpt8VOedOyar2amt66xoJpaeKjXWVY7t6zzB/iP2Rymr5mv3L9NkXyQ1/8rd4EGeLvVVfWrkJK8YgpNBZSskT4aSmC+seDCMzHv0ZNDmGE+4jWp2AKIr4CwxmvcQHBH5OAjuiRoHVMH3qKqYrpwiJNw5Dmck4PItZ3scNdOvawBeQsKzrmu0VUEWrJayOWqsDZQVw74ak9vlN7fN/IkxG4N+0PHQpUmEZLuKuSa2YLkWhZrcY0vd0+CUKbxoHg/rLKRscl4x0ghLHAvl4RxjjNvmMdZ8opZxj3exfWJXXhmrT6PTWCRdCaUn2SBxpp4pQWHMSEk0FkUaCGckWzp/FJCd3M7FCZZ225d9V1K1HErzMdsV8QBf4L904LoPYoxQ1d9W5Pp5McQ4hBzfZE+oRX+wia2GwWcUwzYbGM0xi/i2YENgwstm21Lz8Cqh9uP6XmRtqvsBXMVhobu8B62462ehkAXzIJxQmo1WMWiP1tt+r+tA3LtkMo1xvzHEMlLx0Ho+TwnBwNX6yLOTELp300yasW88u9Qk4Ajc2tT6wOwoHnnwkthtFYNbmZB5szS37VQCOm63I7H1DArVIGHQvD5GOMzNLw15OPg9XQ729neeflo8nR/9SDdSr5L7ic/JFnyU/I0+S15kRwlPPkz+St5n3wYfT/6ZfRs9Ly3Xr+2ivk2WVuj438AwcHy5w==</latexit> time scale a2 s2 <latexit sha1_base64="zL0MVrhT4Kfc7c6iIXKMClHbTvM=">AAAFinicjVRfbxM5EF/+5I7LwV2Bx3vAkFbiAaJNCi0IIaEeEjygU6+0EAlXldc7m5h6bWN7U4LZx/s0vMKH4dsw3s1Wzd0hsVKyo9+f8YzH68xI4Xyafj13/sLF3k8/X/ql/+vlK7/9vnb12iunK8vhgGup7SRjDqRQcOCFlzAxFliZSXidHf8Z+ddzsE5ote8XBg5LNlWiEJx5hI7WbtCS+Zktye4RfcbKkhGagcd/eFeJOUmP1gbp8OGDrfH9MUmHabo93tyKwXj73niTjBCJzyBZPrtHVy++pLnmVQnKc8mcezNKjT8MzHrBJdR9WjkwjB+zKYSm/ppsIJSTQlv8KU8adEVXMtNKV1BQVSk8lKsoK51blNkdfMdsDtPHFt2/RRG8k5X/SzdGA7zeaKp6JqQkL5lCUQ4Fq6SPggKYryy48EJM2+jxPkwwH/GazNgciJ8BYZXXuIDgjshOR3RBUNqnDrxFVsV0YV+USP0FJ2QvYnVDO1y1YU9rQNyCghOucVoqD7RgpZCLZWF1oK7o4rM9vVF6Z8fIwxC7NeyDjoUqTWJIMomiTDObk0zLfMVX+eLBYRDKVB4Ub4dVVDI2GQ8UyYUF7uWCMM5x5hXzOEs+Y5ZxjwdvdWLvPbNWn8QmsEg6EcoPRqE5g3ymBYc+ISTQSSRoLpyRbOH8QgL16G2iMBjV9cZZ3SlFHbfCfE92huziH5D/mInepOjpumrbGowHmyHEIWb6ffqYzvAV1rHdSOCcomG9jm70z+PegQ2dCiXrdU2Pwaq7w/v0JE/rZfacuRlaQ7N5d+v+RgtDoHNmwTghteqkYt7urTZh+Z0HxOoulauM+Y8ggqeKvdDiWUb2OqzUeZyZhMJ/HIyoFdOZ/4icBByZW5laa8CCpo0KD4XRWlW4mAWZMUvfaqEwpKt03e9Tw6xQOW4Kweujj9dQd9eQ7wevxsPR1nDr73uDJzvLC+lS8kdyK7mdjJLt5EnyPNlNDhKe/JN8Sj4nX3qXe+Pew96jVnr+3NJzPVl5ek+/ATmB6iY=</latexit> PGb ⌘ 0 <latexit sha1_base64="gfbkc/Vc1eNYyoitACZmg9/XEyw=">AAAFx3icjVRfbxQ3EF8oR+nxL5THvhgukYJEortAKS+RUPpAkRAKISEn4ejk9c7eufHalu29y9X4oV+t36LfoK/lEzDevYtyBCQsrXb0+/1mPOMZOzdSON/v/3vl6g/XOtd/vPFT9+at23furt37+b3TteVwxLXUdpgzB1IoOPLCSxgaC6zKJRznp78n/ngK1gmtDv3cwEnFxkqUgjOP0GiNUg9nPlgoag6OeE0iJdQw6wWTI0+ON/0jsku2aMX8xFZkf0RfsqpihCqWS0YfjGaE5rKGsFCEAxjHuJn8Ho3Wev3tfrPIZWOwMHrZYu2P7l17RwvN6wqU55I592HQN/4kpHS4hNiltQPD+CkbQ2iKj2QDoYKU2uKnPGnQFV3FTCtdQUHVlfBQraKscm5e5Y/xn6I5DJ/Kcl+KEvg4r75KN44GeNxosnoppCTvmEJRASWrpU+CEpivLbjwWoxba/cQhhgvtWDCpkD8BAirvcYNBHdELnVElwSlXerAW2RVChcORYXUG5iRg4TFhna4a8Oe54C4BQUzrrGHqgi0ZJWQ80ViMVBXLu2LNX1Qem/PyJOQqjXsL50SVZokk+AY8NNcM1uQXMtixa/25fOTIJSpPSjeNqusZSoyTSMphAXu5ZwwzrHnNfPYSz5hlnGPU7vasTPPrNWzVAQmSYdC+d6gnTs+0YJDlxAS6DARtBDOSDZ3fi6hmfHGCr1BjBsXdecUddwK8y3ZBXJpf4f8+5zoA4o+y6rasno7vSchpCbm+qy/Syf4C+tYbiKwT8lhPSZv9J+mswMbliqUrMdIT8Gqre1f6azox0X0grkJuobm8LZid6OF8fZOmQXjhNRqKRXT9my1Ob/biMVlKFcbc0mQwHPFQWjxPCcHS6zSReqZhNJ/7A2oFeOJ/4icBGyZW+la64AJjRsVDoXRWtW4mQWZM0v/1EKhSVfp2O2m90uoAg+F4PPRxWdo8OWjc9l4v7M9eLb97O3T3ou9xYN0I/sle5htZoPst+xF9ke2nx1lPPsn+y/7P/vUedXRnWnnrJVevbLwuZ+trM7fnwECdQI7</latexit> reduces to ∂tW(t) = PGrwReg(W(t)) <latexit sha1_base64="UeVb3Qq8yz7Bs7/V8W4+pxU8kRU=">AAAFl3icjVRbb9MwFM6Acim3DZ4QL4ZuEkjb1I7rywQaCHhACMYGlfA0Oc5Ja+bYlu20FC9/gl/DK/wL/g3HSTutDCQsRTn6Lifn+DhOjRTOd7u/Fk6dPtM6e+78hfbFS5evXF1cuvbB6dJy2OVaattPmQMpFOx64SX0jQVWpBI+pgfPIv9xBNYJrXb8xMBewQZK5IIzj9D+4iqhqSwh0IL5oS3CNgyq6s74LtkkVOoBoc9BerY/Jq8R3F/sdNe79SIng9406CTT9XZ/6cx7mmleFqA8l8y5T72u8XuBWS+4hKpNSweG8QM2gFA3U5EVhDKSa4uP8qRG53QFM410DgVVFsJDMY+ywrlJka7iO2ZzmD426v4URXA1Lf5K10YDvFqpq3oppCTvmUJRBjkrpY+CHJgvLbjwWgyaaHMH+piPeE2GbATED4Gw0mv8gOCOyJmO6JygtE0deIusiunCjiiQegNjsh2xqqYdfrVmj2pA3IKCMdcFqrJAc1YIOZkWVgXq8ll8vKdPSm9tGbkXYreGfdWxUKVJDEkqUZRqZjOSapnN+UqfP94LQpnSg+LNsPJSxibj6SKZsMC9nBDGOc68ZB5nyYfMMu7xFM5P7Itn1upxbAKLpH2hfKfXnEQ+1IJDmxASaD8SNBPOSDZxfiKBevTWUej0qmrluO6Ioo5bYf4lO0bO4v+Q/5+J3qLomXXVtNXZ6NwLIQ4x1V+6m3SIr7CM7UYC5xQNy1V0o38U9w5smKlQslxV9ACsWlt/QMdZt5pmz5gbojXUm7dWtVcaGP/nEbNgnJBazaRi1OytNkd/O2LVLJUrjTkhiOCRYjs0eJqS7RlW6CzOTELuDzs9asVg6A+Rk4Ajc3NTawxY0KBW4aEwWqsSP2ZBpszSz1ooDOk8XbXb1DArVIabQvD6aOM11Pvz0jkZfNhY7z1cf/jufufp1vRCOp/cTG4nd5Je8ih5mrxK3ia7CU++Jd+TH8nP1o3Wk9aL1qtGemph6rmezK3Wu9/VO+7N</latexit> Reg(w) = log DwL(w) <latexit sha1_base64="z0Bov1GY9TpjxbRr0qLKuYfMD20=">AAAHGXicjVRfb+NEEE/DUQ7z7wq88TIQV+qJtEqKOFCrSkev3PFQodLrXSt102i9nsRL1l7f7jq54PMn4TPwIXhDvPLEt2HWTkpzBeks2R7N7zez828nypW0rtf7e6391p2319+5+27w3vsffPjRvY2Pn1tdGIHPhFbaXETcopIZPnPSKbzIDfI0UngeTR55/HyKxkqdnbl5joOUjzM5koI7Ug031n5jBuOSOXzpolF5lqRVxYABbB3hGDM03CEIOuA+KY/RAeuySBVYhoyrPOHMaeiFFesCz2L6LkErxykHjzbisMfG+IKY3VvUjEeKD2fsCJXjQ4aO7GYyxoQ7OAaGLwo59WfsAGOX7GftBsF3DpxMEazgCiF0wwnsHSwcNnFd7VbNwVe7MAm7QNGHsxCEzqgaY7RAkYXnYTeYJVIkYLWaog3YZQAxnG85ynb/gO3DNrCUu8SkcDJkT3jqc4oowi3PuU9euxAfDh0EbP9Loj9e6Gsjne815PpzRbXIYYmTmQvYIAhrrPs4BG4QYrTCyAhjkBk4NKkFPYLw32qEw3ud3k6vfuC20F8IndbiORlu3HnKYi2KFDMnFLf2st/L3aDkxkmhsApYYTHnYsLHWNbjVMEmqWIYaUNv5qDWrvBSnjfUFS1mRSodpqtanlo7T6Mu/b03S+59cezrJK/sRul/wrVhjqLarKN6IpWCpzwjUowjXijnCSPkrjBoy2M5bqSDM7wgf77VCZ8iuASBF07TAVJYUEueLzJRA2bRGUIz7648owGz8CPO4NTrqhq2dGqNXsdAekP3ZCY0DUdGN2nEU6nmi8CqktnRUr6Z02WmDw9zNSh9tjn/RftAMw1eBLoPYhJpbmKItIpX7Ao3+nZQyiwvHGaiadaoUD5Jf78hlgaFU3PgQlDPC7q/MYiEGy5oolbrSneeG6NnPgkKkl3IzHX6ZT29ItFSYAAAJbvwAIulzRWfWzdXWO+LWio7/aravMm7hpif5vz/aDfApfwG9DczYp8zsllm1aTV2e18VZa+iZF+2TtgCf3KkNL1APXJG4SVtyb7qa8dmnLJIkpIi3GCJtve+ZrN4l618B5zm5BpWRdvuwo2GzXtoSld6dxKpbMlVU7LxWYoF2ulJF21dGWLPL9F8MprxmnZ6KMITpe6VMe+ZwpH7lWnz4wcJ+4VYQqpZXala40BBTSuWTQUudZZQYcZVBE3tFxlRiJbhasgYDk3MoupKEDrI6A11H996dwWnu/u9B/sPPhpt/Pw+8VCutv6rPVFa6vVb33Tetj6oXXSetYS7U/be+1H7aP1X9d/X/9j/c+G2l5b2HzSWnnW//oHIeJs7Q==</latexit> Thm (Degenerate case) Let a ! 0 and s ! s0 0, and rwDh b L ⌘ 0. At time scale tk := a2s2k, w converges to W, which solves dW(t) = PGb(W(t)) dBt + F(W(t)) : bb>(W(t)) dt b, F are described in terms of b L
  21. <latexit sha1_base64="fTjuJZhVp4BiQT4JyEr1o32G+Yc=">AAAFgHicjVRdb9Q4FE1Lh4/hYwuIJ14M00o8QHcCLRSkSogVggeE2NLCSLiqHOdmxtSxLduZdjD5MbzCL9p/w3UyqTq7i4SlKFfnnHt9r4/lzEjh/HD4z9LyuZXe+QsXL/UvX7l67Y/V6zc+OF1ZDvtcS21HGXMghYJ9L7yEkbHAykzCx+zor8h/nIJ1Qqs9PzNwULKxEoXgzCN0uHqL7AwJ9XDiA9GK1PQVK0t2uDoYbgw3t59spQSDZmGw/Wh7a/MpSefIIJmvd4fXV97TXPOqBOW5ZM59SofGHwRmveAS6j6tHBjGj9gYQtN1TdYRykmhLX7KkwZd0JXMtNIFFFRVCg/lIspK52Zldh//sZrD8iXzE/dvUQTvZ+X/0k2iAV6vN129ElKS90yhKIeCVdJHQQHMVxZceCPGbbSzByOsR7wmEzYF4idAWOU1biC4I7LTEV0QlPapA2+RVbFc2BMlUm/hmOxGrG5oh7s27GkPiFtQcMw12qPyQAtWCjmbN1YH6oouPjvTJ6VfvDDyIMRpDfuiY6NKkxiSTKIo08zmJNMyX8irfLF9EIQylQfFW7OKSsYh4zUiubDAvZwRxjl6XjGPXvIJs4x7vG6Ljp14Zq0+jkNgk3QklB+kgcae+EQLDn1CSKCjSNBcOCPZzPmZhOZiNlEYpHW9flZ3SlHHrTC/kp0hu/g35L+XRO9QzOmmascaPBw8CiGamOmT4Q6d4C+s4biRQJ9iwlodszF/Gs8ObOhUKFmra3oEVj3Y2KLH+bCeV8+Zm2BqaA7vQd1fb2EIdMosGCekVp1UTNuz1ab92zIgVnelXGXMfwQRPFXshhbPMrLbYaXOo2cSCv91kFIrxhP/FTkJaJlbcK1NwIbGjQovhdFaVbiZBZkxSz9roTCki3Td71PDrFA5HgrB56OPz1D31pBfBx8ebqSPNx7/vTl4/nL+IF1Mbid3k3tJmjxJnievk3fJfsKTkHxLvic/esu9e70/e2krXV6a59xMFlbv2U+oTuXg</latexit> = 0 on G Minibatching: <latexit sha1_base64="TgZGu6Nu6k2NHU2hYa/BaDGvjrE=">AAAF5nicjVTfb9s2EJa7eu28H022x71wcwLYWBJYadJmKwwUHYbtoQi6NGkDlIlBUSeLC0UKJGXHY/Uv7G3Y6/6tPe4/2VGyg3hbgRIQePi+7453POqSUgrrRqO/Onc+uNv98N79j3off/LpZw82Nj9/ZXVlOJxxLbU5T5gFKRScOeEknJcGWJFIeJ1cfR/41zMwVmh16hYlXBRsqkQmOHMITTY8nYsUcubI88F8h4JjQzImNDOMx37/uCbUVsXEi3FcXx6TQfxNkFyKIaGVSsEkqANPEzGVg2xwPRFP5sPdaTCGATTDy/16grxk/MqPv6NPCAUpJ2IwH9b1ZKM/2hsdHD0+jAkazULj6OHR4cG3JF4i/Wi5Xkw2776kqeZVAcpxyax9E49Kd+GZcYJLqHu0slDiUWwKvrmbmmwjlJJMG/yUIw26pitY2UrXUFBVIRwU6ygrrF0UyQ7uIZrF8AVzuf23KIA7SfG/dONYAq+3m6x+FFKSl0yhKIWMVdIFQQbMVQasfy6mrTU+hXOMR5wmOZsBcTkQVjmNBwhuiVzpiM4ISnvUgjPIqhDOn4oCqWOYk5OA1Q1t8dSGvckBcQMK5lwXqEo9zVgh5GKZWO2pzVb27ZreKP3sWSkvfKi2ZL/qkKjSJJik6X2imUlJomW65le57OjCC1VWDhRvm5VVMhQZHitJhQHu5IIwzrHnFXPYS54zfHUOH/V6x64dM0bPQxGYJD0XyvVjT0NOPNeCQ48Q4ul5IGgqbCnZwrqFBOrQt7F8P67r7du6G4pabkT5LtktcmW/h/z9nOhXFH1WVbVl9ff7D70PTUz09WhMc9z8FpYbCOxTcNiqgzf6z8LdgfErFUq26ppegVG7e4d0no7qZfSU2RxdfXN5u3Vvu4XxD58xA6UVUquVVMzau9Vlu5vCI1avQtmqLP8jCOCN4sS3eJKQkxVW6DT0TELm3vZjasQ0d2+Rk4Ats2tdax0woWmjwkdRaq0qPMyATJihv2ih0KTrdN3r0ZIZgdMLxwGOjx6OodWsIe82Xu3vxY/2Hv180H/6w3Ig3Y++jL6OBlEcPY6eRj9FL6KziEd/d+51Njqb3bz7W/f37h+t9E5n6fNFtLa6f/4DXOYKLg==</latexit> b

    L(w, h) = 1 2N N Â i=1 (1 + hi) f (xi ; w) g(xi) 2 | {z } =: `i(w) <latexit sha1_base64="o6TcBJp1g9Jq8C5FTQivbZ8VSCk=">AAAF2XicjVTPbxw1FJ60WWiHXykcuRg2izh0o51SCpdIVSsEBxSVNGlXipfI43mza+KxLduzm607h94QV/41LvwrnHie2Y0yDUi1NJqn7/ve8/thvdxI4fxk8vfOrdu7g/fev3M3/eDDjz7+ZO/epy+cri2HU66lttOcOZBCwakXXsLUWGBVLuFlfvE08i+XYJ3Q6sSvDcwqNleiFJx5hM73FAXPfhXkkNAc5kIFjsFcQ9JxRr6iHi59ICtDGpKNaVFaxkl1RCg9e0h/036WbrBwNK6aqu9wJU8pqGIT93xvODmYtIfcNLKNMUw259n5vd3ntNC8rkB5LplzZ9nE+Flg1gsuoUlp7cAwfsHmENp2NGSEUEFKbfFTnrRoT1cx00l7KKi6Eh6qPsoq59ZVfh//MRq2ZlQxv3BviyJ4P6/+k24dDfBm1Gb1o5CSPGcKRQWUrJY+Ckpgvrbgws9i3lmHJzDFeMRrsmBLIH4BhNVe4wWCOyK3OqJLgtKUOvAWWRXDhRNRIXUEK3IcsaalHd7aslc5IG5BwYrrClVFoCWrhFxvEmsCdeXWvl7TmdJPnhg5C7Faw17pmKjSJJoklyjKNbMFybUsen61L7+fBaFM7UHxblhlLWOR8X2SQljgXq4J4xxnXjOPs+QLhm/J4zvuT+zSM2v1KhaBSdKpUH6YBRpz4gstOKSEkECnkaCFcEaytfNrCe1Dba0wzJpmdF13RVHHrTD/J7tGbu13kL+bE/2Cos+2qq6s4YPhNyHEIeb6cnJIF/gL+1huJHBO0WG/id7ov4y9Axu2KpTsNw29AKvGB9/SVTFpNtEL5hboGtrmjZt01MEQ6JJZME5IrbZSsex6q033t1VArNmGcrUxNwQRvFIchw7Pc3K8xSpdxJlJKP3rYUatmC/8a+Qk4Mhcb2qdAyY0b1X4KIzWqsbLLMicWdxJQqFJ+3STptQwK1SBTSG4PlJcQ9nbS+em8eLBQfbo4NEvD4ePf9gspDvJ58mXyddJlnyXPE5+Sp4lpwlP/kr+2bm9szs4G7wZ/D74o5Pe2tn4fJb0zuDPfwH8iAXR</latexit> hi = 8 > > < > > : 1 wp 1 m N N m m wp m N <latexit sha1_base64="t80poQXiRICnPpaCBvfPMdiWgbQ=">AAAF1HicjVTNbxw1FJ+WLpTloykcubhsIhWpjXbSpA2HSFURgkNVlTRJV6rTlcfzZseJxx7Znt0ujk+IK/8aZ/4QrsDzzE7IFirV0mre/j6e3/OznNVSWDce/3Ht+gc3Bh9+dPPj4Seffvb5rY3bX5xY3RgOx1xLbSYZsyCFgmMnnIRJbYBVmYSX2fl3kX85B2OFVkduWcNpxWZKFIIzh9B0Q9BMMn7uaQaO3V18M/VnIpADQhVDgt6Z+sXrs/DvP4qy16igC5FDyRx5GsWFYTz1O8/CWz5CQcqpwLyBTDdG4+3x7v6jvZRg0C4M9h/s7+1+S9IVMkpW6/n09o0XNNe8qUA5Lpm1r9Jx7U49M05wCWFIGws1Fs9m4NuTCGQLoZwU2uBPOdKia7qK1Z10DQXVVMJBtY6yytplld3Db8xmMX3FXGnfFkXwXlb9L90aa+Bhq63qByElecEUinIoWCNdFBTAXGPA+qdi1kUHRzDBfMRpUrI5EFcCYY3TuIHglsheR3RBUDqkFpxBVsV0/khUSD2DBTmMWGhpi7u27GUNiBtQsOC6QlXuacEqIZerwoKntujjqz29UvrJk1qe+thtzX7WsVClSQxJe5syzUxOMi3zNV/jiv1TL1TdOFC8G1bRyNhkvJokFwa4k0vCOMeZN8zhLHnJ8HI5vMLrE3vjmDF6EZvAIulEKDdKPY018VILDkNCiKeTSNBc2FqypXVLCdSht438KA1h66rukqKWG1G/S3aF7OP3kL+fid6h6Om76toa7YweeB+HmOk34wNa4sdvYruRwDlFw2aIbvTP49mB8b0KJZsh0HMw6v72Hl3k47DKnjNbotW3h3c/DLc6GDydMwO1FVKrXirm3dnquvuayiMW+lS2qev/CCJ4qTj0HZ5l5LDHKp3HmUko3MUopUbMSneBnAQcmV2bWmfAgmatCi9FrbVqcDMDMmOGnmmhMKTrdBgOac2MUDkeCsHnY4jPUP/WkHcHJzvb6cPthz/tjh5/v3qQbiZfJV8nd5M0eZQ8Tn5MnifHCU9+T/5M/kr+HpwMLga/DH7tpNevrTxfJmtr8Ns/kPAHpw==</latexit> b(w)ji = rwj r hi b L = 1 2N rwj `i(w) <latexit sha1_base64="Ywds2ANd2SkukRL5c9CQe65AXpQ=">AAAGAHicjVTNbhs3EF6lUZuqP3HSYy90JQNJYRsrB0l6MRC4h+bgFKkrxwJCw+ByZ7WsuSRLcmWrzF76DH2I3ope+yZ9lZ463JUMy26BEFhw8M33DWc43MmMFM6n6d+9Ox/c7X/40b2PB598+tnn9zcePHzjdG05HHMttZ1mzIEUCo698BKmxgKrMgkn2fm30X8yB+uEVhO/MHBasZkSheDMI3S28RvNoUBtGylkVvBzC3kTXk5eHTbh4On4eZo2g8GhqIQn8HPdyohwhBKayRrCiBpmvWDyzJOTR/4x2SfpqKHbm8h4JZTImOelULNWA5UpA71xlqsN2J0cZqDAMg+bzdnGMN1N20VuG+OlMUyW6/XZg7sTmmteV6A8l8y5t+PU+NMQM+MSmgGtHRjGz9kMwmV7fkO2EMtJoS1+ypMWXSNWzHTUNRRUjXcB1TrKKucWVbaNe4zmMHzFfOlukiK4nVX/6W6FBniz1Wb1nZCS/MgUkrBHrJY+EgpgvrbgwqGYddb+BKYYj3hNSjYH4ksgrPYaDxDcEbniEV0QpA6oA2/Rq2K4MBEVur6HC3IUsaZ1Ozy19V7lgLjF/lxwXSErD7RglZCLZWJNoK5Y2ddreqv0wYGRpyFWa9gvOiaqNIkmySSSMs1sTjIt8zVd7YtvToNQpvageNesopaxyPiMSS4scC8XhHGOTcd3ib3kJbOMe3zu6x279MxafRGLwCTpVCg/HAcac+KlFhwGhJBAp9FBc+GMZAvnFxKoR21rheG4abau865c1HErzP/RrjlX9nvQ309ENylqVlV1ZQ33hk9CiE3M9GW6T0vcwgjLjQ7sUxSMmqhG/TzeHdiwYiFl1DT0HKza2X1KL/K0WUbPmcO/2If28naawVYHQ6BzZsE4IbVaUcW8u1ttut1WAbFmFQr/d3OLEMErxlHo8CwjRyus0nnsmYTCvxuOqRWz0r9DnwRsmVvrWifAhGYtCx+F0VrVeJgFmTFLf9JCoUnX3Tjm4iwTKsdLITg/BjiHxjenzm3jzd7u+Nnusx/2hi8OlhPpXvJl8lXyKBknz5MXycvkdXKc8OSfHuk97n3d/7X/e/+P/p8d9U5vqfkiWVv9v/4FxL8WeQ==</latexit> Limit equation is ∂tW(t) = 0 ! Minibatching is super-degenerate! <latexit sha1_base64="z0Bov1GY9TpjxbRr0qLKuYfMD20=">AAAHGXicjVRfb+NEEE/DUQ7z7wq88TIQV+qJtEqKOFCrSkev3PFQodLrXSt102i9nsRL1l7f7jq54PMn4TPwIXhDvPLEt2HWTkpzBeks2R7N7zez828nypW0rtf7e6391p2319+5+27w3vsffPjRvY2Pn1tdGIHPhFbaXETcopIZPnPSKbzIDfI0UngeTR55/HyKxkqdnbl5joOUjzM5koI7Ug031n5jBuOSOXzpolF5lqRVxYABbB3hGDM03CEIOuA+KY/RAeuySBVYhoyrPOHMaeiFFesCz2L6LkErxykHjzbisMfG+IKY3VvUjEeKD2fsCJXjQ4aO7GYyxoQ7OAaGLwo59WfsAGOX7GftBsF3DpxMEazgCiF0wwnsHSwcNnFd7VbNwVe7MAm7QNGHsxCEzqgaY7RAkYXnYTeYJVIkYLWaog3YZQAxnG85ynb/gO3DNrCUu8SkcDJkT3jqc4oowi3PuU9euxAfDh0EbP9Loj9e6Gsjne815PpzRbXIYYmTmQvYIAhrrPs4BG4QYrTCyAhjkBk4NKkFPYLw32qEw3ud3k6vfuC20F8IndbiORlu3HnKYi2KFDMnFLf2st/L3aDkxkmhsApYYTHnYsLHWNbjVMEmqWIYaUNv5qDWrvBSnjfUFS1mRSodpqtanlo7T6Mu/b03S+59cezrJK/sRul/wrVhjqLarKN6IpWCpzwjUowjXijnCSPkrjBoy2M5bqSDM7wgf77VCZ8iuASBF07TAVJYUEueLzJRA2bRGUIz7648owGz8CPO4NTrqhq2dGqNXsdAekP3ZCY0DUdGN2nEU6nmi8CqktnRUr6Z02WmDw9zNSh9tjn/RftAMw1eBLoPYhJpbmKItIpX7Ao3+nZQyiwvHGaiadaoUD5Jf78hlgaFU3PgQlDPC7q/MYiEGy5oolbrSneeG6NnPgkKkl3IzHX6ZT29ItFSYAAAJbvwAIulzRWfWzdXWO+LWio7/aravMm7hpif5vz/aDfApfwG9DczYp8zsllm1aTV2e18VZa+iZF+2TtgCf3KkNL1APXJG4SVtyb7qa8dmnLJIkpIi3GCJtve+ZrN4l618B5zm5BpWRdvuwo2GzXtoSld6dxKpbMlVU7LxWYoF2ulJF21dGWLPL9F8MprxmnZ6KMITpe6VMe+ZwpH7lWnz4wcJ+4VYQqpZXala40BBTSuWTQUudZZQYcZVBE3tFxlRiJbhasgYDk3MoupKEDrI6A11H996dwWnu/u9B/sPPhpt/Pw+8VCutv6rPVFa6vVb33Tetj6oXXSetYS7U/be+1H7aP1X9d/X/9j/c+G2l5b2HzSWnnW//oHIeJs7Q==</latexit> Thm (Degenerate case) Let a ! 0 and s ! s0 0, and rwDh b L ⌘ 0. At time scale tk := a2s2k, w converges to W, which solves dW(t) = PGb(W(t)) dBt + F(W(t)) : bb>(W(t)) dt b, F are described in terms of b L
  22. <latexit sha1_base64="+RjQsAHf3p0EqLhNcZ5TACZEtkw=">AAAFvHicjVRfb9s2EFe7emu9f2n32BduToAESAIrTdoMQ4aiw7A9FEGbJq2BMjEo6mSxoUiCpOx4rD7XPsse9rp9jR0lO4i3FSgBQYffn+Mdj2BmpHB+OPzj1u1P7vQ+/ezuvf7nX3z51ddr9x+8drq2HM64ltqOMuZACgVnXngJI2OBVZmEN9nlT5F/MwXrhFanfm7gvGITJQrBmUdovPaSzkQOJfPk+eZsm4JnW+SI0MIynoa944ZQV1fjII7S5uKY0ExM5GaxeTUWP8y2diYx2NqJrgsRObt1sTdeGwx3h/uHTw5SgkG7MDh8dHiw/z1JF8ggWawX4/t3XtFc87oC5blkzr1Nh8afB2a94BKaPq0dGMYv2QRC23FDNhDKSaEtfsqTFl3RVcx00hUUVF0JD9Uqyirn5lW2jf+YzWH6ivnS/VsUwe2s+l+6NRrgzUZb1S9CSvKKKRTlULBa+igogPnaggvPxaSLjk5hhPmI16RkUyC+BMJqr3EDwR2RSx3RBUFpnzrwFlkV04VTUSF1DDNyErGmpR3u2rLXNSBuQcGM6wpVeaAFq4ScLwprAnXFMr7Z01ulnz0z8jzEbg37TcdClSYxJJlEUaaZzUmmZb7iq31xeB6EMrUHxbthFbWMTcYrSHJhgXs5J4xznHnNPM6SlwzvnMerujqxK8+s1bPYBBZJR0L5QRporImXWnDoE0ICHUWC5sIZyebOzyVQj942CoO0aTZu6q4p6rgV5kOyG+Qy/gj5x5notxQ9y666tgZ7g0chxCFm+mp4REv8hXVsNxI4p2hYb6Ib/dN4dmDDUoWS9aahl2DVzu4BneXDZpE9Z65Ea2gPb6fpb3QwBDplFowTUqulVEy7s9Wm+9sqINYsU7namP8IInitOAkdnmXkZIlVOo8zk1D494OUWjEp/XvkJODI3MrUOgMWNGlVeCmM1qrGzSzIjFn6TguFIV2lm36fGmaFyvFQCD4ffXyGlm8N+XDwem83fbz7+OX+4OnPiwfpbvIw+S7ZTNLkSfI0+TV5kZwlPPk9+TP5K/m792Mv7132qk56+9bC802ysnrTfwC+Xvxq</latexit> b L(w, h) = 1 2N N Â

    i=1 f (xi ; w) g(xi) hi 2 Label noise: limit equation <latexit sha1_base64="qE+3q4Y7l7hpE6SQURI0H1zZrjA=">AAAFuHicjVRfb9s2EFe7euu8f2n7uBd2ToAUaAK7a7tiQ4Ciw9A9bEOWJo2BMvBO1MlmQ5EESdn1WH2qfZq+bl9kR8kO4mYDKkDQ4ffneMejmFslfRgO3127/tGN3sef3Py0/9nnX3z51dat2y+9qZ3AE2GUceMcPCqp8STIoHBsHUKVKzzNz39M/OkcnZdGH4elxbMKplqWUkAgaLL1K7fgggQ1Cex0N9xj/IcDetkeryDMXMUOJ/w5VBUwriFXwO9OFoznqsa4UsQjnDbNbjLfm2wNhvvD9mFXg9EqGGSr53By68YLXhhRV6iDUOD9q9HQhrOYahIKmz6vPVoQ5zDF2HbbsB2CClYaR68OrEU3dBXYTrqBoq4rGbDaRKHyflnl9+mbsnlKn9ry74sSeD+v/pNujRZFs9NW9VwqxV6AJlGBJdQqJEGJEGqHPv4ip110cIxjyseCYTOYIwszZFAHQwtI4Zla65gpGUn73GNwxOqULh7LiqjfcMGOEta0tKdVW/aiBsIdalwIQzPUReQlVFItV4U1kftyHV/u6ZU2z55ZdRZTtxb+NKlQbVgKGR0DcZ4bcAXLjSo2fHUon5xFqW0dUItuWGWtUpPp+LFCOhRBLRkIQTOvIdAsxQwciEDHdHNibwI4ZxapCSqSj6UOg1F37sTMSIF9xljk40TwQnqrYOnDUiEP5G2jOBg1zc5l3QXFvXDS/p/sErmOP0D+YSZ+l5Nn3VXX1uDB4NsY0xBz82Z4wGf0idvUbiJoTsmw3SQ3+edp79DFtYok203Dz9Hpvf1HfFEMm1X2AvyMrLHdvL2mv9PB9PfOwaH1Uhm9lsp5t7fGXvzbhDXrVL629ooggReKo9jhec6O1lhlijQzhWV4OxhxJ6ez8JY4hTQyvzG1zkAFTVsVHQprjK5pMYcqB8dfG6kp5Jt00++nS0zqgjaF0fXRp2to9P6lczV4+WB/9Hj/8e8PB09/Wl1IN7Ovs2+y3WyUfZc9zX7ODrOTTGR/Ze+yv7N/et/3/uhNe7KTXr+28tzJNp6e+xe7ovtE</latexit> ∂tW(t) = PGrwReg(W(t)) <latexit sha1_base64="qUTwMOrHXhsjzyDEeNd5BjD4/zs=">AAAFoHicjVTdbtQ4FE6BWSD7Q4FLbkynlboSVJPysytQJVRWggsEtLQwWlxVjnMyY+rYlu3MMJg8CU+zt7tPwNvscTJTdeiuhKUoR9/PyTk+jnMjhfODwdeVCxcv9X64fOVq+uNPP/9ybfX6jbdO15bDIddS22HOHEih4NALL2FoLLAql/AuP3ka+XcTsE5odeBnBo4qNlKiFJx5hI5XH6Q0lzUEWjE/tlXYh1HTbE5/JfTxox36mNDSMp5tE/oHSM+Op+QFkunxan+wNWgXOR9k86CfzNfr4+uX3tBC87oC5blkzr3PBsYfBWa94BKalNYODOMnbASh7aohGwgVpNQWH+VJiy7pKmY66RIKqq6Eh2oZZZVzsyq/g++YzWH62LH7VhTBO3n1n3RrNMCbjbaqZ0JK8oYpFBVQslr6KCiB+dqCCy/EqIt2DmCI+YjXZMwmQPwYCKu9xg8I7ohc6IguCUpT6sBbZFVMFw5EhdRLmJL9iDUt7fCrLXtaA+IWFEy5rlBVBFqySsjZvLAmUFcu4rM9vVd6d9fIoxC7NeyTjoUqTWJIcomiXDNbkFzLYslX+/L3oyCUqT0o3g2rrGVsMh4zUggL3MsZYZzjzGvmcZZ8zPA0eTyOyxP76Jm1ehqbwCLpUCjfz7ojycdacEgJIYEOI0EL4YxkM+dnEqhHbxuFftY0G2d1pxR13Arzf7Iz5CL+Dvn3mehtip5FV11b/e3+vRDiEHP9cbBDx/gK69huJHBO0bDeRDf6J3HvwIaFCiXrTUNPwKq7Ww/otBg08+wFc2O0hnbz7jbpRgfjjz1hFowTUquFVEy6vdXm9LdHrFmkcrUx5wQRPFXshw7Pc7K/wCpdxJlJKP3nfkatGI39Z+Qk4Mjc0tQ6AxY0alV4KIzWqsaPWZA5s/SDFgpDukw3aUoNs0IVuCkEr494DWXfXjrng7fbW9nDrYd79/tPducX0pXkVrKWbCZZ8lvyJHmevE4OE558Sf5K/k7+6a31nvde9fY66YWVuedmsrR6f/4LWOTxQA==</latexit> Reg(w) := 1 2 DwL(w) bias towards wide valleys! Li-Wang-Arora 21 <latexit sha1_base64="z0Bov1GY9TpjxbRr0qLKuYfMD20=">AAAHGXicjVRfb+NEEE/DUQ7z7wq88TIQV+qJtEqKOFCrSkev3PFQodLrXSt102i9nsRL1l7f7jq54PMn4TPwIXhDvPLEt2HWTkpzBeks2R7N7zez828nypW0rtf7e6391p2319+5+27w3vsffPjRvY2Pn1tdGIHPhFbaXETcopIZPnPSKbzIDfI0UngeTR55/HyKxkqdnbl5joOUjzM5koI7Ug031n5jBuOSOXzpolF5lqRVxYABbB3hGDM03CEIOuA+KY/RAeuySBVYhoyrPOHMaeiFFesCz2L6LkErxykHjzbisMfG+IKY3VvUjEeKD2fsCJXjQ4aO7GYyxoQ7OAaGLwo59WfsAGOX7GftBsF3DpxMEazgCiF0wwnsHSwcNnFd7VbNwVe7MAm7QNGHsxCEzqgaY7RAkYXnYTeYJVIkYLWaog3YZQAxnG85ynb/gO3DNrCUu8SkcDJkT3jqc4oowi3PuU9euxAfDh0EbP9Loj9e6Gsjne815PpzRbXIYYmTmQvYIAhrrPs4BG4QYrTCyAhjkBk4NKkFPYLw32qEw3ud3k6vfuC20F8IndbiORlu3HnKYi2KFDMnFLf2st/L3aDkxkmhsApYYTHnYsLHWNbjVMEmqWIYaUNv5qDWrvBSnjfUFS1mRSodpqtanlo7T6Mu/b03S+59cezrJK/sRul/wrVhjqLarKN6IpWCpzwjUowjXijnCSPkrjBoy2M5bqSDM7wgf77VCZ8iuASBF07TAVJYUEueLzJRA2bRGUIz7648owGz8CPO4NTrqhq2dGqNXsdAekP3ZCY0DUdGN2nEU6nmi8CqktnRUr6Z02WmDw9zNSh9tjn/RftAMw1eBLoPYhJpbmKItIpX7Ao3+nZQyiwvHGaiadaoUD5Jf78hlgaFU3PgQlDPC7q/MYiEGy5oolbrSneeG6NnPgkKkl3IzHX6ZT29ItFSYAAAJbvwAIulzRWfWzdXWO+LWio7/aravMm7hpif5vz/aDfApfwG9DczYp8zsllm1aTV2e18VZa+iZF+2TtgCf3KkNL1APXJG4SVtyb7qa8dmnLJIkpIi3GCJtve+ZrN4l618B5zm5BpWRdvuwo2GzXtoSld6dxKpbMlVU7LxWYoF2ulJF21dGWLPL9F8MprxmnZ6KMITpe6VMe+ZwpH7lWnz4wcJ+4VYQqpZXala40BBTSuWTQUudZZQYcZVBE3tFxlRiJbhasgYDk3MoupKEDrI6A11H996dwWnu/u9B/sPPhpt/Pw+8VCutv6rPVFa6vVb33Tetj6oXXSetYS7U/be+1H7aP1X9d/X/9j/c+G2l5b2HzSWnnW//oHIeJs7Q==</latexit> Thm (Degenerate case) Let a ! 0 and s ! s0 0, and rwDh b L ⌘ 0. At time scale tk := a2s2k, w converges to W, which solves dW(t) = PGb(W(t)) dBt + F(W(t)) : bb>(W(t)) dt b, F are described in terms of b L
  23. Conclusions and outlook • We can characterise on-Γ behaviour of

    noisy gradient descent, in limit of small step size (and in some cases small noise) • Time scale depends on degeneracy of noise • Results are applicable to any function and any form of (small) noise • Application to more ‘serious’ systems • Use the characterisation to design tailor-made forms of noise injection • What happens when one adds momentum? • Investigate correlated noise • …