Breandan Considine
December 07, 2019
53

# Derivatives. Important Concept. Simple to grasp in Kotlin.

Differentiation tells us how to make a small change to the inputs of a function, so as to produce the largest change in output. At first, this idea may not seem very important for software engineers, but differential equations can be found at the heart of every other engineering discipline and nearly every major contribution to the physical sciences in the last three centuries. As digital computers begin to interface with the physical world, derivatives will begin to play an increasingly significant role in computing.

Contrary to popular belief, #derivatives are surprisingly simple to understand and compute. In this talk, we will see how to implement automatic differentiation in Kotlin, using functional programming, and see some applications for physical simulation, machine learning and automatic testing. No prior experience or mathematical background is assumed or required.

## Breandan Considine

December 07, 2019

## Transcript

1. ### Copenhagen Denmark Derivatives. Important concept. Simple to grasp in Kotlin.

Breandan Considine @breandan
2. ### 1. What are derivatives? 2. How do derivatives work? 3.

What can be derived? 4. How can I derive in Kotlin? 5. What’s the difference?

4. ### https://en.wikipedia.org/wiki/Gottfried_Wilhelm_Leibniz “It is unworthy of excellent [minds] to lose hours

like slaves in the labor of calculation which could safely be relegated to anyone else if machines were used.” -Gottfried Wilhelm Leibniz Leibniz: derivatives as rate of change

8. ### tailrec fun <I, O : Comparable<O>> minimize( fn: (I) ->

(O), min: I, budget: Int): I = 1. An algorithm for minimizing any function
9. ### tailrec fun <I, O : Comparable<O>> minimize( fn: (I) ->

(O), min: I, budget: Int): I = if (budget <= 0) min else minimize(fn, sample<I>().let { input -> if (fn(input) < fn(min)) input else min }, budget - 1) 1. An algorithm for minimizing any function
10. ### tailrec fun <I, O : Comparable<O>> minimize( fn: (I) ->

(O), min: I, budget: Int): I = if (budget <= 0) min else minimize(fn, sample<I>().let { input -> if (fn(input) < fn(min)) input else min }, budget - 1) fun <I> sample(): I = TODO() 1. An algorithm for minimizing any function

12. ### 2. Better algorithm (but more restrictive) interface Metric<T : Metric<T>>

: Comparable<T> { operator fun plus(t: T): T operator fun minus(t: T): T }
13. ### 2. Better algorithm (but more restrictive) interface Metric<T : Metric<T>>

: Comparable<T> { operator fun plus(t: T): T operator fun minus(t: T): T } tailrec fun <I, O : Metric<O>> minimizeMetric( fn: (I) -> (O), min: I, budget: Int): I = if (budget <= 0) min else minimizeMetric(fn, wiggle(min).filter { fn(it) < fn(min) } .maxBy { fn(min) - fn(it) } ?: min, budget - 1)
14. ### interface Metric<T : Metric<T>> : Comparable<T> { operator fun plus(t:

T): T operator fun minus(t: T): T } tailrec fun <I, O : Metric<O>> minimizeMetric( fn: (I) -> (O), min: I, budget: Int): I = if (budget <= 0) min else minimizeMetric(fn, wiggle(min).filter { fn(it) < fn(min) } .maxBy { fn(min) - fn(it) } ?: min, budget - 1) fun <I> wiggle(min: I): Sequence<I> = TODO() 2. Better algorithm (but more restrictive)

16. ### interface Field<T : Field<T>> : Metric<T> { operator fun times(t:

T): T operator fun div(t: T): T } 3. Still better algorithm (even more restrictive)
17. ### interface Field<T : Field<T>> : Metric<T> { operator fun times(t:

T): T operator fun div(t: T): T } tailrec fun <T: Field<T>> fieldMinimize( fn: (T) -> (T), a: T, min: T, budget: Int): T = if(budget <= 0) min else minimizeField(fn, a, min - (fn(min + a) - fn(min)) / a, budget – 1) 3. Still better algorithm (even more restrictive)
18. ### interface Field<T : Field<T>> : Metric<T> { operator fun times(t:

T): T operator fun div(t: T): T } tailrec fun <T: Field<T>> fieldMinimize( fn: (T) -> (T), a: T, min: T, budget: Int): T = if(budget <= 0) min else minimizeField(fn, a, min - (fn(min + a) - fn(min)) / a, budget – 1) 3. Still better algorithm (even more restrictive) d dx f(x) = df dx = lim a!0 f(x + a) f(x) a <latexit sha1_base64="urfHwtn74c2LNBAddruoAv7AtsA=">AAACRnicbVBNbxMxFHybAi3ho6EcuTwRIbVCRJs2kOaAVMGFYyuRtlJ2Fb31ehOr3vXK9tJE1v66Xjhz4ydw4QBCXOt8gIAykqXxzBs9e5JSCmPD8HPQ2Lh1+87m1t3mvfsPHm63Hu2cGlVpxodMSaXPEzJcioIPrbCSn5eaU55IfpZcvF34Zx+4NkIV7+285HFOk0JkgpH10rgVR4ma8dRhlGliLq1dOqsx253t4etfWrYS/V2KfOwo0mIytaS1usSwXk/5CD5H2sMXy3TtqMZ63GqHncGgN3jZQ08O++GrPnY74RK/SRvWOB63PkWpYlXOC8skGTPqhqWNHWkrmOR1M6oML4ld0ISPPC0o5yZ2yxpqfOaVFDOl/SksLtU/E45yY+Z54idzslPzr7cQ/+eNKpsdxk4UZWV5wVaLskqiVbjoFFOhObNy7gkxLfxbkU3Jt2J9801fwo0v3ySn+53uQWf/pNc+erOuYwuewFPYhS704QjewTEMgcEVfIFv8D34GHwNfgQ/V6ONYJ15DH+hAdecaLCQ</latexit>
19. ### 4. So what’s the problem? f(x + a) f(x) a

<latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
20. ### fn: (T) -> (T) 4. So what’s the problem? f(x

+ a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
21. ### fn: (T) -> (T) fn: (T, T) -> (T) 4.

So what’s the problem? f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
22. ### fn: (T) -> (T) fn: (T, T) -> (T) fn:

(T, T, T) -> (T) 4. So what’s the problem? f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
23. ### fn: (T) -> (T) fn: (T, T) -> (T) fn:

(T, T, T) -> (T) 4. So what’s the problem? O(n) <latexit sha1_base64="acO24dpBaZe9AJXh30rtHH8bH7A=">AAAB9XicbVBNTwIxFHyLX4hfqEcvjcQEL2RBFLgRvXgTEwETWEm3dKGh2920XQ3Z8D+8eNAYr/4Xb/4bu0CMipM0mcy8lzcdN+RMadv+tFJLyyura+n1zMbm1vZOdnevpYJIEtokAQ/krYsV5UzQpmaa09tQUuy7nLbd0UXit++pVCwQN3ocUsfHA8E8RrA20l3Xx3pIMI+vJnlx3Mvm7EKtVq6dlpEh1Yp9VkHFgj3FN8nBHI1e9qPbD0jkU6EJx0p1inaonRhLzQink0w3UjTEZIQHtGOowD5VTjxNPUFHRukjL5DmCY2m6s+NGPtKjX3XTCYp1V8vEf/zOpH2qk7MRBhpKsjskBdxpAOUVID6TFKi+dgQTCQzWREZYomJNkVlTAkLX14krVKheFIoXZdz9fN5HWk4gEPIQxEqUIdLaEATCEh4hGd4sR6sJ+vVepuNpqz5zj78gvX+BZgBkpQ=</latexit> f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
24. ### fn: (T) -> (T) fn: (T, T) -> (T) fn:

(T, T, T) -> (T) fn: (T) -> Pair<T, T> fn: (T, T) -> Triple<T, T> 4. So what’s the problem? O(n) <latexit sha1_base64="acO24dpBaZe9AJXh30rtHH8bH7A=">AAAB9XicbVBNTwIxFHyLX4hfqEcvjcQEL2RBFLgRvXgTEwETWEm3dKGh2920XQ3Z8D+8eNAYr/4Xb/4bu0CMipM0mcy8lzcdN+RMadv+tFJLyyura+n1zMbm1vZOdnevpYJIEtokAQ/krYsV5UzQpmaa09tQUuy7nLbd0UXit++pVCwQN3ocUsfHA8E8RrA20l3Xx3pIMI+vJnlx3Mvm7EKtVq6dlpEh1Yp9VkHFgj3FN8nBHI1e9qPbD0jkU6EJx0p1inaonRhLzQink0w3UjTEZIQHtGOowD5VTjxNPUFHRukjL5DmCY2m6s+NGPtKjX3XTCYp1V8vEf/zOpH2qk7MRBhpKsjskBdxpAOUVID6TFKi+dgQTCQzWREZYomJNkVlTAkLX14krVKheFIoXZdz9fN5HWk4gEPIQxEqUIdLaEATCEh4hGd4sR6sJ+vVepuNpqz5zj78gvX+BZgBkpQ=</latexit> f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
25. ### fn: (T) -> (T) fn: (T, T) -> (T) fn:

(T, T, T) -> (T) fn: (T) -> Pair<T, T> fn: (T, T) -> Triple<T, T> 4. So what’s the problem? O(n) <latexit sha1_base64="acO24dpBaZe9AJXh30rtHH8bH7A=">AAAB9XicbVBNTwIxFHyLX4hfqEcvjcQEL2RBFLgRvXgTEwETWEm3dKGh2920XQ3Z8D+8eNAYr/4Xb/4bu0CMipM0mcy8lzcdN+RMadv+tFJLyyura+n1zMbm1vZOdnevpYJIEtokAQ/krYsV5UzQpmaa09tQUuy7nLbd0UXit++pVCwQN3ocUsfHA8E8RrA20l3Xx3pIMI+vJnlx3Mvm7EKtVq6dlpEh1Yp9VkHFgj3FN8nBHI1e9qPbD0jkU6EJx0p1inaonRhLzQink0w3UjTEZIQHtGOowD5VTjxNPUFHRukjL5DmCY2m6s+NGPtKjX3XTCYp1V8vEf/zOpH2qk7MRBhpKsjskBdxpAOUVID6TFKi+dgQTCQzWREZYomJNkVlTAkLX14krVKheFIoXZdz9fN5HWk4gEPIQxEqUIdLaEATCEh4hGd4sR6sJ+vVepuNpqz5zj78gvX+BZgBkpQ=</latexit> O(nm) <latexit sha1_base64="nYuOvkYbWnBGULx7fv6BFYlf9aA=">AAAB+HicbVDLSgMxFM34rPXRUZdugkWom2GmVtvuim7cWcE+oB1KJs20oUlmSDJCHfolblwo4tZPceffmD4QtR4IHM65l3tygphRpV3301pZXVvf2MxsZbd3dvdy9v5BU0WJxKSBIxbJdoAUYVSQhqaakXYsCeIBI61gdDX1W/dEKhqJOz2Oic/RQNCQYqSN1LNzXY70ECOW3kwKgp/27LzrVKul6nkJGlIpuxdl6DnuDN8kDxao9+yPbj/CCSdCY4aU6nhurP0USU0xI5NsN1EkRniEBqRjqECcKD+dBZ/AE6P0YRhJ84SGM/XnRoq4UmMemMlpTPXXm4r/eZ1EhxU/pSJONBF4fihMGNQRnLYA+1QSrNnYEIQlNVkhHiKJsDZdZU0JS19eJs2i4505xdtSvna5qCMDjsAxKAAPlEENXIM6aAAMEvAInsGL9WA9Wa/W23x0xVrsHIJfsN6/AN93kzw=</latexit> f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>

27. ### d dx (l + r) = dl dx + dr

dx <latexit sha1_base64="sp2EEuZ7A99i2OOn0HWOb1TODlA=">AAACH3icbZBLSwMxEMez9VXXV9WDiJdgESpC2dbax0EoevFYwT6gLSWbzbah2QdJVixLv4kXv4oXD4oUb/Xi1Y9hui2i1oHAf34zw2T+ps+okIYx1mILi0vLK/FVfW19Y3Mrsb1TE17AMalij3m8YSJBGHVJVVLJSMPnBDkmI3Wzfzmp128JF9Rzb+TAJ20HdV1qU4ykQp1EvmVzhENrGFp3wxSDJ5Afw3M4oyzCCs5yHuW6rncSSSNdKuVKZzmoRLFg5Aswkzai+BbJcvFjtPf+uV/pJN5alocDh7gSMyREM2P4sh0iLilmZKi3AkF8hPuoS5pKusghoh1G9w3hkSIWtD2unithRH9OhMgRYuCYqtNBsif+1ibwv1ozkHaxHVLXDyRx8XSRHTAoPTgxC1qUEyzZQAmEOVV/hbiHlBVSWToxYe7keVHLpjOn6ey1cuMCTCMODsAhSIEMKIAyuAIVUAUY3INH8AxetAftSXvVRtPWmDab2QW/Qht/ARKCpiQ=</latexit> 1. Sum rule
28. ### d dx (l · r) = dl dx · r

+ l · dr dx <latexit sha1_base64="dT1Clq6leve2PLTo4GR7EgsFg4A=">AAACL3icbZBbSwJBFMdn7WZ2s3qI6GVIAiOQXbPUh0AKokeDvICKzM7O6uDshZnZSBa/US99k/AlulA91rdoXC26HRj48/+dw5nzN31GhdT1ey02NT0zOxefTywsLi2vJFfXqsILOCYV7DGP100kCKMuqUgqGan7nCDHZKRm9k5GvHZJuKCeeyH7Pmk5qONSm2IkldVOnjZtjnBoDULrapBmsIktT0K+C4/ghLAITXy4Bz97JphHuJ1M6ZliMVc8yEElCnn9MA+NjB7Vl0iVCi+3Gw9vm+V2cti0PBw4xJWYISEahu7LVoi4pJiRQaIZCOIj3EMd0lDSRQ4RrTC6dwB3lGNB2+PquRJG7veJEDlC9B1TdTpIdsVvNjL/Y41A2oVWSF0/kMTF40V2wKD04Cg8aFFOsGR9JRDmVP0V4i5SOUgVcUKF8Ofkv6KazRj7mey5SuMYjCsOtsA2SAMD5EEJnIEyqAAMrsEQPIIn7Ua7056113FrTJvMrIMfpb1/AIeVrcs=</latexit> 2. Product rule
29. ### d dx (l r) = dl dr · dr dx

<latexit sha1_base64="/6RQkpZx19m0jk8qpzWM2vqNUBw=">AAACJHicbVDLSgMxFM3UV62vqgsRN8Ei1E2ZqdW2iFB047KCfUBbSiaTaUMzD5KMWIb5GDf+ihsXVXGhC936GaYPRa0HAodzzuXmHtNnVEhdf9FiM7Nz8wvxxcTS8srqWnJ9oyq8gGNSwR7zeN1EgjDqkoqkkpG6zwlyTEZqZu9s6NeuCBfUcy9l3yctB3VcalOMpJLayeOmzREOrSi0rqM0g01MOYZ8H57AicOUxSNlWJ780vgo3k6m9EyxmCse5qAihbx+lIdGRh/hm6RKhffB1uvHdrmdfGxaHg4c4krMkBANQ/dlK0RcUsxIlGgGgvgI91CHNBR1kUNEKxwdGcE9pVjQ9rh6roQj9edEiBwh+o6pkg6SXfHXG4r/eY1A2oVWSF0/kMTF40V2wKD04LAxaFFOsGR9RRDmVP0V4i5SPUjVa0KVMHXyNKlmM8ZBJnuh2jgFY8TBDtgFaWCAPCiBc1AGFYDBDbgDA/Cg3Wr32pP2PI7GtMnMJvgF7e0TpHKp3w==</latexit> 2. Chain rule
30. ### 2. Chain rule, revisited Pk(x) = ( p1 x =

x if k = 1 pk Pk 1 x if k > 1 <latexit sha1_base64="UNaDT9DlVhDQhoUwMCujqFY5XjA=">AAACV3icbVFNb9NAEF27paThK4VjLyMiUDkQ2SWQ5tCqggvHIJG2UhxZ6804XXm9tnbHKJHlP4m49K/0ApsPEKWMtNLTe29mdt8mpZKWguDG83d2H+w9bO23Hz1+8vRZ5+D5hS0qI3AsClWYq4RbVFLjmCQpvCoN8jxReJlkn1b65Tc0Vhb6Ky1LnOZ8rmUqBSdHxR09irOjxRs4hSjBudS1cMNsA2UcQiSkEbBw2gJeR4QLqmUKDWSnYRQ5R7YxjOI6exs2v+13nHAGbg7q2XZu3OkGveGwP3zfBwdOBsGHAYS9YF1/QJdtaxR3vkezQlQ5ahKKWzsJg5KmNTckhcKmHVUWSy4yPseJg5rnaKf1OpcGXjlmBmlh3NEEa/bvjprn1i7zxDlzTtf2X21F/k+bVJSeTGupy4pQi82itFJABaxChpk0KEgtHeDCSHdXENfccEHuK9ouhHtPvg8ujnvhu97xl373/OM2jhY7ZC/ZEQvZgJ2zz2zExkywH+zW2/F2vRvvp7/ntzZW39v2vGB3yj/4BdYzsZ0=</latexit> f = l r = l(r(x)) <latexit sha1_base64="nkPZBWmrknfJELHcRhv+09I3JUE=">AAACAHicbZDLSsNAFIYnXmu9RV24cDNYhHYTklptuxCKblxWsBdoQ5lMJ+3QySTMTMQSuvFV3LhQxK2P4c63cdIWUesPAx//OYcz5/ciRqWy7U9jaXlldW09s5Hd3Nre2TX39psyjAUmDRyyULQ9JAmjnDQUVYy0I0FQ4DHS8kZXab11R4SkIb9V44i4ARpw6lOMlLZ65qEPLyCDXUwFhiLlvMjfFwo9M2db1WqpelaCGipl+7wMHcue6htyYK56z/zo9kMcB4QrzJCUHceOlJsgoShmZJLtxpJECI/QgHQ0chQQ6SbTAybwRDt96IdCP67g1P05kaBAynHg6c4AqaH8W0vN/2qdWPkVN6E8ihXheLbIjxlUIUzTgH0qCFZsrAFhQfVfIR4igbDSmWV1CAsnL0KzaDmnVvGmlKtdzuPIgCNwDPLAAWVQA9egDhoAgwl4BM/gxXgwnoxX423WumTMZw7ALxnvX1mYlFY=</latexit> df dx = dl dr · dr dx <latexit sha1_base64="xFN/reyOMTMYc0BqJoUqohqlBfg=">AAACGnicbVDLSsNAFJ34rPUVdelmsAiuSlKrbRdC0Y3LCvYBTSiTyaQdOnkwMxFLyHe48VfcuFDEnbjxb5ymUdR6YODcc+7lzj1OxKiQhvGhLSwuLa+sFtaK6xubW9v6zm5HhDHHpI1DFvKegwRhNCBtSSUjvYgT5DuMdJ3xxdTv3hAuaBhcy0lEbB8NA+pRjKSSBrppeRzhxPXSxL1N4RnMa6ZqnkILu6H80njWM9BLRrnRqDZOqlCRes04rUGzbGT4JiWQozXQ3yw3xLFPAokZEqJvGpG0E8QlxYykRSsWJEJ4jIakr2iAfCLsJDsthYdKcaEXcvUCCTP150SCfCEmvqM6fSRH4q83Ff/z+rH06nZCgyiWJMCzRV7MoAzhNCfoUk6wZBNFEOZU/RXiEVI5SJVmUYUwd/I86VTK5nG5clUtNc/zOApgHxyAI2CCGmiCS9ACbYDBHXgAT+BZu9cetRftdda6oOUze+AXtPdPZTiiYA==</latexit> dP dp1 = dpk dpk 1 dpk 1 dpk 2 dpk 2 dpk 3 . . . dp2 dp1 <latexit sha1_base64="tk1VNbcDvJXBcNFPYk6ku+8MR7I=">AAACWXicbVHPS8MwGE2rm3P+mnr0EhyCF0fbTbcdBNGLxwnODbZR0jTdQtM2JKkwSv9JD4L4r3gw28pQ54PAy3vf9yV58TijUlnWh2FubZfKO5Xd6t7+weFR7fjkRSapwKSPE5aIoYckYTQmfUUVI0MuCIo8RgZe+LDwB69ESJrEz2rOySRC05gGFCOlJbfGx4FAOPN7eeZz187hLSwU7oZLLQuv7Dxfi6ttwZzfhrM2mtrwEyXXrlPMr7q1utXodlvd6xbUpNO2btrQblhLrEkdFOi5tTc9CacRiRVmSMqRbXE1yZBQFDOSV8epJBzhEE3JSNMYRUROsmUyObzQig+DROgVK7hUf3ZkKJJyHnm6MkJqJv96C/E/b5SqoDPJaMxTRWK8OihIGVQJXMQMfSoIVmyuCcKC6rtCPEM6DKU/YxHCxpM3yYvTsJsN56lVv7sv4qiAM3AOLoEN2uAOPIIe6AMM3sGXUTLKxqdpmBWzuio1jaLnFPyCefoN2qK2zw==</latexit> dP dp1 = dpk dpk 1 ✓ dpk 1 dpk 2 ✓ dpk 2 dpk 3 . . . dp2 dp1 ◆◆ <latexit sha1_base64="5Hf2lOUg+IsmPFYHgZi0yY9UU2s=">AAACcnicbZFbS8MwGIbTep6nqXijoNEhKOho63TuQhC98XKCU2EbJU3TLSw9kHwVRukP8O9556/wxh9gNsvw9EHIy/vky+GNlwiuwLLeDHNqemZ2bn6htLi0vLJaXlt/UHEqKWvRWMTyySOKCR6xFnAQ7CmRjISeYI/e4GbEH5+ZVDyO7mGYsG5IehEPOCWgLbf80gkkoZnfzDM/ce0cX+LCSdzB2MsGJ3aedwQL4HCCvsxCOf9hZ4JPNfZjUBPqFGd1JO/14aiY3HLFqjYatcZZDWtxUbfO69iuWuOaiAoqqumWX/W2NA1ZBFQQpdq2lUA3IxI4FSwvdVLFEkIHpMfaWkYkZKqbjSPL8YF2fBzEUo8I8Nj93pGRUKlh6OmVIYG++s1G5n+snUJw0c14lKTAIvp1UJAKDDEe5Y99LhkFMdSCUMn1XTHtE50M6F8q6RD+PPmveHCq9mnVuatVrq6LOObRNtpHh8hGdXSFblETtRBF78amsWPsGh/mlrlnFtmZRtGzgX6UefwJkbbAZA==</latexit> dP dp1 = ✓✓ dpk dpk 1 dpk 1 dpk 2 ◆ dpk 2 dpk 3 ◆ . . . dp2 dp1 <latexit sha1_base64="jTakLDXIlC1Q4eVTqc8mpDQ6B10=">AAACc3icbZHLSgMxFIYz473eqoIbFwaroAvLzLRauxBENy4rWBXaMmQymTY0cyE5I5RhXsDHc+dbuHFv2g71eiDhz/+dk8uJlwiuwLLeDHNufmFxaXmltLq2vrFZ3tp+UHEqKWvTWMTyySOKCR6xNnAQ7CmRjISeYI/e8GbMH5+ZVDyO7mGUsF5I+hEPOCWgLbf80g0koZnfyjM/ce0cX+KuYAEcF/OUJu5wwrPhqZ3nM3O6LJSjgeT9AZx8586M1764H4OaJTnFySW3XLGqzWa9eVbHWlw0rPMGtqvWJGaigopoueVXvRNNQxYBFUSpjm0l0MuIBE4Fy0vdVLGE0CHps46WEQmZ6mWTnuX4SDs+DmKpRwR44n6vyEio1Cj0dGZIYKB+s7H5H+ukEFz0Mh4lKbCITg8KUoEhxuMPwD6XjIIYaUGo5PqumA6Ibgbobxo34c+T/4oHp2rXqs5dvXJ1XbRjGe2hA3SMbNRAV+gWtVAbUfRu7Br7BjY+zD3zwDycpppGUbODfoR5+gnj7sB4</latexit> “Forward accumulation”: “Reverse accumulation”:

33. ### class D<X: Fun<X>>(val f: Fun<X>): Fun<X>(f) { fun Fun<X>.df(): Fun<X>

= when (this) { is Sum -> left.df() + right.df() is Prod -> left.df() * right + left * right.df() is Comp -> left.df()(right) * right.df() } } Differentiation rules in Kotlin
34. ### Differentiation rules in Kotlin class D<X: Fun<X>>(val f: Fun<X>): Fun<X>(f)

{ fun Fun<X>.df(): Fun<X> = when (this) { is Const -> Zero() is Var -> One() is Sum -> left.df() + right.df() is Prod -> left.df() * right + left * right.df() is Comp -> left.df()(right) * right.df() is D -> f.df() } }
35. ### Functions in Kotlin sealed class Fun<X : Fun<X>>(open val sVars:

Set<Var<X>> = emptySet()) : Field<Fun<X>>, (Bindings<X>) -> Fun<X> { constructor(fn: Fun<X>) : this(fn.sVars) constructor(vararg fns: Fun<X>) : this(fns.flatMap { it.sVars }.toSet()) override operator fun plus(addend: Fun<X>): Fun<X> = Sum(this, addend) override operator fun times(multiplicand: Fun<X>): Fun<X> = Prod(this, multiplicand) override operator fun div(divisor: Fun<X>): Fun<X> = this * divisor.pow(-One<X>()) override operator fun invoke(bnds: Bindings<X>): Fun<X> = Composition(this, bnds).run { if (bnds.isReassignmentFree) evaluate else this } open operator fun invoke(): Fun<X> = invoke(Bindings()) open fun d(v1: Var<X>): Fun<X> = Derivative(this, v1) open fun d(v1: Var<X>, v2: Var<X>): Vec<X, D2> = Vec(Derivative(this, v1), Derivative(this, v2)) open fun d(vararg vars: Var<X>): Map<Var<X>, Fun<X>> = vars.map { it to Derivative(this, it) }.toMap() }

37. ### Lazy Composition class Composition<X : Fun<X>>(val fn: Fun<X>, val bindings:

Bindings<X>) : Fun<X>() { val evaluate by lazy { apply() } override val sVars: Set<Var<X>> by lazy { evaluate.sVars } fun Fun<X>.apply(): Fun<X> = bindings.sMap.getOrElse(this) { when (this) { is Var -> this is Const -> this is Prod -> left.apply() * right.apply() is Sum -> left.apply() + right.apply() is Power -> base.apply() pow exponent.apply() is Derivative -> df().apply() is Composition -> fn.apply().apply() } } }

40. ### Optimization is just one perspective “With four parameters I can

fit an elephant, and with five I can make him wiggle his trunk.” –John Von Neumann

discrete
43. ### Physicists World is differentiable dp dt = @H @q ,

dq dt = @H @p <latexit sha1_base64="mvt81qlkGmhRnX0oT+ghEaP0p30=">AAACh3icnVHJTsMwFHTCXrYCRy4WFRJCUNKFLgcklgtHkCggNVHlOE6xcJbaDqKy/Ct8FDf+BictiO3EkyyNZuY9P4/9lFEhHefNsmdm5+YXFpdKyyura+vljc1bkWQckx5OWMLvfSQIozHpSSoZuU85QZHPyJ3/eJHrd0+EC5rEN3KcEi9Cw5iGFCNpqEH5xfWTZxIoN+QIq8CNkHzwQ5VqrQKp4Qk8nChuirikiBUGjJi6NI4PUn20jbTWB/DHrNH/ZpkVtB6UK0612212j5vQgE7babVhreoU9QkqYFpXg/KrGyQ4i0gsMUNC9GtOKj2Vz8aM6JKbCZIi/IiGpG9gjCIiPFXkqOGuYQIYJtycWMKC/dqhUCTEOPKNM19S/NRy8i+tn8mw4ykap5kkMZ5cFGYMygTmnwIDygmWbGwAwpyaXSF+QCYrab6uZEL49eTf4LZerTWq9etm5fR8Gsci2AY7YA/UQBucgktwBXoAW7PWvtWwmvaSfWS37M7EalvTni3wreyzd1o4yio=</latexit> Hamiltonian mechanics Wave equation Pendulum with friction @2u @t2 = c2 ✓ @2u @x2 + @2u @y2 + @2u @z2 ◆ <latexit sha1_base64="Uzb8tUvoA5hhpp36ioX3nITwlnw=">AAACoXicjVFbSxtBGJ3dWi/xFttHoQwGwVIImzUa81AQ24f2QYliVMgmYXbybTJk9sLMt8V02f/l7+ib/8bZJIiXgn4wcOZ858zlfH4ihUbHubfsDwsfF5eWV0qra+sbm+WtT1c6ThWHNo9lrG58pkGKCNooUMJNooCFvoRrf/yj6F//AaVFHF3iJIFuyIaRCARnaKh++c7z41sYZF6gGKeZlzCFgsmeS9P8cUexR92cfqe853oSAtx7qae9zM2fOm4LIs+/vSmcvFf4dyb0lBiO8GveL1ecarNZbx7UqQFHDeewQWtVZ1qPoELm1eqX/3mDmKchRMgl07pTcxLsZsXpXEJe8lINCeNjNoSOgRELQXezacI53TXMgAaxMitCOmWfOjIWaj0JfaMMGY70y15B/q/XSTE46mYiSlKEiM8uClKTeEyLcdGBUMBRTgxgXAnzVspHzCSFZqglE8KrL78GV261tl91z+uV45N5HMtkm+yQPVIjDXJMfpEWaRNufbF+WqfWmV2xf9st+2Imta255zN5VnbnAXTQ0ac=</latexit> d2x dt2 + 2r! dx dt + !2x = 0 <latexit sha1_base64="xCCOcd4qYvmT9/noOiBJXCmIH6w=">AAACX3icbVHRSiMxFM2Mu9tau+6s+7T4EiyCsFCmtbttHwRxX3x0YatCp5ZM5k4bmkyGJCOWMD/pm+DL/omZWmStXggczrnn5uYkzjnTJgwfPH/rw8dPtfp2Y6f5efdL8HXvUstCURhRyaW6jokGzjIYGWY4XOcKiIg5XMWL35V+dQtKM5n9NcscJoLMMpYySoyjpsFtFMs7SKyNUkUotpEgZq4EtkmJb2y3vCtfUabiyvJHV+FICpgR/J5x0+QM6+7VyJOwnAatsD0c9oY/e9iBQT/81ceddriqF9BC67qYBvdRImkhIDOUE63HnTA3E0uUYZRD2YgKDTmhCzKDsYMZEaAndpVPiQ8dk+BUKncyg1fs/w5LhNZLEbvOam29qVXke9q4MOlgYlmWFwYy+nxRWnBsJK7CxglTQA1fOkCoYm5XTOfE5WXclzRcCG+e/BZcdtud43b3T691eraOo4720QE6Qh3UR6foHF2gEaLo0fO9Ha/p/fNr/q4fPLf63trzDb0q//sTwKC3gw==</latexit>
44. ### Nonlinear dynamics: no closed form solution Three Body Orbit https://en.wikipedia.org/wiki/Three-body_problem

https://en.wikipedia.org/wiki/Double_pendulum Double Pendulum

53. ### Differentiable Programming “Neural networks are not just another classifier, they

represent the beginning of a fundamental shift in how we write software.” -Andrej Karpathy