Derivatives. Important Concept. Simple to grasp in Kotlin.

Differentiation tells us how to make a small change to the inputs of a function, so as to produce the largest change in output. At first, this idea may not seem very important for software engineers, but differential equations can be found at the heart of every other engineering discipline and nearly every major contribution to the physical sciences in the last three centuries. As digital computers begin to interface with the physical world, derivatives will begin to play an increasingly significant role in computing.

Contrary to popular belief, #derivatives are surprisingly simple to understand and compute. In this talk, we will see how to implement automatic differentiation in Kotlin, using functional programming, and see some applications for physical simulation, machine learning and automatic testing. No prior experience or mathematical background is assumed or required.

Transcript

1. Copenhagen Denmark Derivatives. Important concept. Simple to grasp in Kotlin. Breandan Considine @breandan
2. 1. What are derivatives? 2. How do derivatives work? 3. What can be derived? 4. How can I derive in Kotlin? 5. What’s the difference?
3. What are derivatives?
4. https://en.wikipedia.org/wiki/Gottfried_Wilhelm_Leibniz “It is unworthy of excellent [minds] to lose hours like slaves in the labor of calculation which could safely be relegated to anyone else if machines were used.” -Gottfried Wilhelm Leibniz Leibniz: derivatives as rate of change
5. http://www.helbig.dk/neucc/index.htm Linnainmaa: Reverse mode differentiation Seppo Linnainmaa
6. Griewank: Algorithmic differentiation Andreas Griewank
7. 1. An algorithm for minimizing any function
8. tailrec fun <I, O : Comparable<O>> minimize( fn: (I) -> (O), min: I, budget: Int): I = 1. An algorithm for minimizing any function
9. tailrec fun <I, O : Comparable<O>> minimize( fn: (I) -> (O), min: I, budget: Int): I = if (budget <= 0) min else minimize(fn, sample<I>().let { input -> if (fn(input) < fn(min)) input else min }, budget - 1) 1. An algorithm for minimizing any function
10. tailrec fun <I, O : Comparable<O>> minimize( fn: (I) -> (O), min: I, budget: Int): I = if (budget <= 0) min else minimize(fn, sample<I>().let { input -> if (fn(input) < fn(min)) input else min }, budget - 1) fun <I> sample(): I = TODO() 1. An algorithm for minimizing any function
11. 2. Better algorithm (but more restrictive)
12. 2. Better algorithm (but more restrictive) interface Metric<T : Metric<T>> : Comparable<T> { operator fun plus(t: T): T operator fun minus(t: T): T }
13. 2. Better algorithm (but more restrictive) interface Metric<T : Metric<T>> : Comparable<T> { operator fun plus(t: T): T operator fun minus(t: T): T } tailrec fun <I, O : Metric<O>> minimizeMetric( fn: (I) -> (O), min: I, budget: Int): I = if (budget <= 0) min else minimizeMetric(fn, wiggle(min).filter { fn(it) < fn(min) } .maxBy { fn(min) - fn(it) } ?: min, budget - 1)
14. interface Metric<T : Metric<T>> : Comparable<T> { operator fun plus(t: T): T operator fun minus(t: T): T } tailrec fun <I, O : Metric<O>> minimizeMetric( fn: (I) -> (O), min: I, budget: Int): I = if (budget <= 0) min else minimizeMetric(fn, wiggle(min).filter { fn(it) < fn(min) } .maxBy { fn(min) - fn(it) } ?: min, budget - 1) fun <I> wiggle(min: I): Sequence<I> = TODO() 2. Better algorithm (but more restrictive)
15. 3. Still better algorithm (even more restrictive)
16. interface Field<T : Field<T>> : Metric<T> { operator fun times(t: T): T operator fun div(t: T): T } 3. Still better algorithm (even more restrictive)
17. interface Field<T : Field<T>> : Metric<T> { operator fun times(t: T): T operator fun div(t: T): T } tailrec fun <T: Field<T>> fieldMinimize( fn: (T) -> (T), a: T, min: T, budget: Int): T = if(budget <= 0) min else minimizeField(fn, a, min - (fn(min + a) - fn(min)) / a, budget – 1) 3. Still better algorithm (even more restrictive)
18. interface Field<T : Field<T>> : Metric<T> { operator fun times(t: T): T operator fun div(t: T): T } tailrec fun <T: Field<T>> fieldMinimize( fn: (T) -> (T), a: T, min: T, budget: Int): T = if(budget <= 0) min else minimizeField(fn, a, min - (fn(min + a) - fn(min)) / a, budget – 1) 3. Still better algorithm (even more restrictive) d dx f(x) = df dx = lim a!0 f(x + a) f(x) a <latexit sha1_base64="urfHwtn74c2LNBAddruoAv7AtsA=">AAACRnicbVBNbxMxFHybAi3ho6EcuTwRIbVCRJs2kOaAVMGFYyuRtlJ2Fb31ehOr3vXK9tJE1v66Xjhz4ydw4QBCXOt8gIAykqXxzBs9e5JSCmPD8HPQ2Lh1+87m1t3mvfsPHm63Hu2cGlVpxodMSaXPEzJcioIPrbCSn5eaU55IfpZcvF34Zx+4NkIV7+285HFOk0JkgpH10rgVR4ma8dRhlGliLq1dOqsx253t4etfWrYS/V2KfOwo0mIytaS1usSwXk/5CD5H2sMXy3TtqMZ63GqHncGgN3jZQ08O++GrPnY74RK/SRvWOB63PkWpYlXOC8skGTPqhqWNHWkrmOR1M6oML4ld0ISPPC0o5yZ2yxpqfOaVFDOl/SksLtU/E45yY+Z54idzslPzr7cQ/+eNKpsdxk4UZWV5wVaLskqiVbjoFFOhObNy7gkxLfxbkU3Jt2J9801fwo0v3ySn+53uQWf/pNc+erOuYwuewFPYhS704QjewTEMgcEVfIFv8D34GHwNfgQ/V6ONYJ15DH+hAdecaLCQ</latexit>
19. 4. So what’s the problem? f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
20. fn: (T) -> (T) 4. So what’s the problem? f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
21. fn: (T) -> (T) fn: (T, T) -> (T) 4. So what’s the problem? f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
22. fn: (T) -> (T) fn: (T, T) -> (T) fn: (T, T, T) -> (T) 4. So what’s the problem? f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
23. fn: (T) -> (T) fn: (T, T) -> (T) fn: (T, T, T) -> (T) 4. So what’s the problem? O(n) <latexit sha1_base64="acO24dpBaZe9AJXh30rtHH8bH7A=">AAAB9XicbVBNTwIxFHyLX4hfqEcvjcQEL2RBFLgRvXgTEwETWEm3dKGh2920XQ3Z8D+8eNAYr/4Xb/4bu0CMipM0mcy8lzcdN+RMadv+tFJLyyura+n1zMbm1vZOdnevpYJIEtokAQ/krYsV5UzQpmaa09tQUuy7nLbd0UXit++pVCwQN3ocUsfHA8E8RrA20l3Xx3pIMI+vJnlx3Mvm7EKtVq6dlpEh1Yp9VkHFgj3FN8nBHI1e9qPbD0jkU6EJx0p1inaonRhLzQink0w3UjTEZIQHtGOowD5VTjxNPUFHRukjL5DmCY2m6s+NGPtKjX3XTCYp1V8vEf/zOpH2qk7MRBhpKsjskBdxpAOUVID6TFKi+dgQTCQzWREZYomJNkVlTAkLX14krVKheFIoXZdz9fN5HWk4gEPIQxEqUIdLaEATCEh4hGd4sR6sJ+vVepuNpqz5zj78gvX+BZgBkpQ=</latexit> f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
24. fn: (T) -> (T) fn: (T, T) -> (T) fn: (T, T, T) -> (T) fn: (T) -> Pair<T, T> fn: (T, T) -> Triple<T, T> 4. So what’s the problem? O(n) <latexit sha1_base64="acO24dpBaZe9AJXh30rtHH8bH7A=">AAAB9XicbVBNTwIxFHyLX4hfqEcvjcQEL2RBFLgRvXgTEwETWEm3dKGh2920XQ3Z8D+8eNAYr/4Xb/4bu0CMipM0mcy8lzcdN+RMadv+tFJLyyura+n1zMbm1vZOdnevpYJIEtokAQ/krYsV5UzQpmaa09tQUuy7nLbd0UXit++pVCwQN3ocUsfHA8E8RrA20l3Xx3pIMI+vJnlx3Mvm7EKtVq6dlpEh1Yp9VkHFgj3FN8nBHI1e9qPbD0jkU6EJx0p1inaonRhLzQink0w3UjTEZIQHtGOowD5VTjxNPUFHRukjL5DmCY2m6s+NGPtKjX3XTCYp1V8vEf/zOpH2qk7MRBhpKsjskBdxpAOUVID6TFKi+dgQTCQzWREZYomJNkVlTAkLX14krVKheFIoXZdz9fN5HWk4gEPIQxEqUIdLaEATCEh4hGd4sR6sJ+vVepuNpqz5zj78gvX+BZgBkpQ=</latexit> f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
25. fn: (T) -> (T) fn: (T, T) -> (T) fn: (T, T, T) -> (T) fn: (T) -> Pair<T, T> fn: (T, T) -> Triple<T, T> 4. So what’s the problem? O(n) <latexit sha1_base64="acO24dpBaZe9AJXh30rtHH8bH7A=">AAAB9XicbVBNTwIxFHyLX4hfqEcvjcQEL2RBFLgRvXgTEwETWEm3dKGh2920XQ3Z8D+8eNAYr/4Xb/4bu0CMipM0mcy8lzcdN+RMadv+tFJLyyura+n1zMbm1vZOdnevpYJIEtokAQ/krYsV5UzQpmaa09tQUuy7nLbd0UXit++pVCwQN3ocUsfHA8E8RrA20l3Xx3pIMI+vJnlx3Mvm7EKtVq6dlpEh1Yp9VkHFgj3FN8nBHI1e9qPbD0jkU6EJx0p1inaonRhLzQink0w3UjTEZIQHtGOowD5VTjxNPUFHRukjL5DmCY2m6s+NGPtKjX3XTCYp1V8vEf/zOpH2qk7MRBhpKsjskBdxpAOUVID6TFKi+dgQTCQzWREZYomJNkVlTAkLX14krVKheFIoXZdz9fN5HWk4gEPIQxEqUIdLaEATCEh4hGd4sR6sJ+vVepuNpqz5zj78gvX+BZgBkpQ=</latexit> O(nm) <latexit sha1_base64="nYuOvkYbWnBGULx7fv6BFYlf9aA=">AAAB+HicbVDLSgMxFM34rPXRUZdugkWom2GmVtvuim7cWcE+oB1KJs20oUlmSDJCHfolblwo4tZPceffmD4QtR4IHM65l3tygphRpV3301pZXVvf2MxsZbd3dvdy9v5BU0WJxKSBIxbJdoAUYVSQhqaakXYsCeIBI61gdDX1W/dEKhqJOz2Oic/RQNCQYqSN1LNzXY70ECOW3kwKgp/27LzrVKul6nkJGlIpuxdl6DnuDN8kDxao9+yPbj/CCSdCY4aU6nhurP0USU0xI5NsN1EkRniEBqRjqECcKD+dBZ/AE6P0YRhJ84SGM/XnRoq4UmMemMlpTPXXm4r/eZ1EhxU/pSJONBF4fihMGNQRnLYA+1QSrNnYEIQlNVkhHiKJsDZdZU0JS19eJs2i4505xdtSvna5qCMDjsAxKAAPlEENXIM6aAAMEvAInsGL9WA9Wa/W23x0xVrsHIJfsN6/AN93kzw=</latexit> f(x + a) f(x) a <latexit sha1_base64="wndWpLry/iyHZGen5rF9ioi7WSw=">AAACDHicbVDLSgMxFM3UV62vqks3wSK0iMNMrbbdFd24rGAf0BlKJpNpQzMPkoy0DPMBbvwVNy4UcesHuPNvTB+IrwOBwznncnOPEzEqpGF8aJml5ZXVtex6bmNza3snv7vXFmHMMWnhkIW86yBBGA1IS1LJSDfiBPkOIx1ndDn1O7eECxoGN3ISEdtHg4B6FCOppH6+YDnhmLgJtDyOcOIVx/AYohI8gYqW0gSlMFUpQ6/XK/WzClSkVjXOq9DUjRm+SAEs0Ozn3y03xLFPAokZEqJnGpG0E8QlxYykOSsWJEJ4hAakp2iAfCLsZHZMCo+U4kIv5OoFEs7U7xMJ8oWY+I5K+kgOxW9vKv7n9WLp1eyEBlEsSYDni7yYQRnCaTPQpZxgySaKIMyp+ivEQ6Rakaq/nCrhz8l/Sbusm6d6+bpSaFws6siCA3AIisAEVdAAV6AJWgCDO/AAnsCzdq89ai/a6zya0RYz++AHtLdPTbKZ4Q==</latexit>
26. How do derivatives work?
27. d dx (l + r) = dl dx + dr dx <latexit sha1_base64="sp2EEuZ7A99i2OOn0HWOb1TODlA=">AAACH3icbZBLSwMxEMez9VXXV9WDiJdgESpC2dbax0EoevFYwT6gLSWbzbah2QdJVixLv4kXv4oXD4oUb/Xi1Y9hui2i1oHAf34zw2T+ps+okIYx1mILi0vLK/FVfW19Y3Mrsb1TE17AMalij3m8YSJBGHVJVVLJSMPnBDkmI3Wzfzmp128JF9Rzb+TAJ20HdV1qU4ykQp1EvmVzhENrGFp3wxSDJ5Afw3M4oyzCCs5yHuW6rncSSSNdKuVKZzmoRLFg5Aswkzai+BbJcvFjtPf+uV/pJN5alocDh7gSMyREM2P4sh0iLilmZKi3AkF8hPuoS5pKusghoh1G9w3hkSIWtD2unithRH9OhMgRYuCYqtNBsif+1ibwv1ozkHaxHVLXDyRx8XSRHTAoPTgxC1qUEyzZQAmEOVV/hbiHlBVSWToxYe7keVHLpjOn6ey1cuMCTCMODsAhSIEMKIAyuAIVUAUY3INH8AxetAftSXvVRtPWmDab2QW/Qht/ARKCpiQ=</latexit> 1. Sum rule
28. d dx (l · r) = dl dx · r + l · dr dx <latexit sha1_base64="dT1Clq6leve2PLTo4GR7EgsFg4A=">AAACL3icbZBbSwJBFMdn7WZ2s3qI6GVIAiOQXbPUh0AKokeDvICKzM7O6uDshZnZSBa/US99k/AlulA91rdoXC26HRj48/+dw5nzN31GhdT1ey02NT0zOxefTywsLi2vJFfXqsILOCYV7DGP100kCKMuqUgqGan7nCDHZKRm9k5GvHZJuKCeeyH7Pmk5qONSm2IkldVOnjZtjnBoDULrapBmsIktT0K+C4/ghLAITXy4Bz97JphHuJ1M6ZliMVc8yEElCnn9MA+NjB7Vl0iVCi+3Gw9vm+V2cti0PBw4xJWYISEahu7LVoi4pJiRQaIZCOIj3EMd0lDSRQ4RrTC6dwB3lGNB2+PquRJG7veJEDlC9B1TdTpIdsVvNjL/Y41A2oVWSF0/kMTF40V2wKD04Cg8aFFOsGR9JRDmVP0V4i5SOUgVcUKF8Ofkv6KazRj7mey5SuMYjCsOtsA2SAMD5EEJnIEyqAAMrsEQPIIn7Ua7056113FrTJvMrIMfpb1/AIeVrcs=</latexit> 2. Product rule
29. d dx (l r) = dl dr · dr dx <latexit sha1_base64="/6RQkpZx19m0jk8qpzWM2vqNUBw=">AAACJHicbVDLSgMxFM3UV62vqgsRN8Ei1E2ZqdW2iFB047KCfUBbSiaTaUMzD5KMWIb5GDf+ihsXVXGhC936GaYPRa0HAodzzuXmHtNnVEhdf9FiM7Nz8wvxxcTS8srqWnJ9oyq8gGNSwR7zeN1EgjDqkoqkkpG6zwlyTEZqZu9s6NeuCBfUcy9l3yctB3VcalOMpJLayeOmzREOrSi0rqM0g01MOYZ8H57AicOUxSNlWJ780vgo3k6m9EyxmCse5qAihbx+lIdGRh/hm6RKhffB1uvHdrmdfGxaHg4c4krMkBANQ/dlK0RcUsxIlGgGgvgI91CHNBR1kUNEKxwdGcE9pVjQ9rh6roQj9edEiBwh+o6pkg6SXfHXG4r/eY1A2oVWSF0/kMTF40V2wKD04LAxaFFOsGR9RRDmVP0V4i5SPUjVa0KVMHXyNKlmM8ZBJnuh2jgFY8TBDtgFaWCAPCiBc1AGFYDBDbgDA/Cg3Wr32pP2PI7GtMnMJvgF7e0TpHKp3w==</latexit> 2. Chain rule
30. 2. Chain rule, revisited Pk(x) = ( p1 x = x if k = 1 pk Pk 1 x if k > 1 <latexit sha1_base64="UNaDT9DlVhDQhoUwMCujqFY5XjA=">AAACV3icbVFNb9NAEF27paThK4VjLyMiUDkQ2SWQ5tCqggvHIJG2UhxZ6804XXm9tnbHKJHlP4m49K/0ApsPEKWMtNLTe29mdt8mpZKWguDG83d2H+w9bO23Hz1+8vRZ5+D5hS0qI3AsClWYq4RbVFLjmCQpvCoN8jxReJlkn1b65Tc0Vhb6Ky1LnOZ8rmUqBSdHxR09irOjxRs4hSjBudS1cMNsA2UcQiSkEbBw2gJeR4QLqmUKDWSnYRQ5R7YxjOI6exs2v+13nHAGbg7q2XZu3OkGveGwP3zfBwdOBsGHAYS9YF1/QJdtaxR3vkezQlQ5ahKKWzsJg5KmNTckhcKmHVUWSy4yPseJg5rnaKf1OpcGXjlmBmlh3NEEa/bvjprn1i7zxDlzTtf2X21F/k+bVJSeTGupy4pQi82itFJABaxChpk0KEgtHeDCSHdXENfccEHuK9ouhHtPvg8ujnvhu97xl373/OM2jhY7ZC/ZEQvZgJ2zz2zExkywH+zW2/F2vRvvp7/ntzZW39v2vGB3yj/4BdYzsZ0=</latexit> f = l r = l(r(x)) <latexit sha1_base64="nkPZBWmrknfJELHcRhv+09I3JUE=">AAACAHicbZDLSsNAFIYnXmu9RV24cDNYhHYTklptuxCKblxWsBdoQ5lMJ+3QySTMTMQSuvFV3LhQxK2P4c63cdIWUesPAx//OYcz5/ciRqWy7U9jaXlldW09s5Hd3Nre2TX39psyjAUmDRyyULQ9JAmjnDQUVYy0I0FQ4DHS8kZXab11R4SkIb9V44i4ARpw6lOMlLZ65qEPLyCDXUwFhiLlvMjfFwo9M2db1WqpelaCGipl+7wMHcue6htyYK56z/zo9kMcB4QrzJCUHceOlJsgoShmZJLtxpJECI/QgHQ0chQQ6SbTAybwRDt96IdCP67g1P05kaBAynHg6c4AqaH8W0vN/2qdWPkVN6E8ihXheLbIjxlUIUzTgH0qCFZsrAFhQfVfIR4igbDSmWV1CAsnL0KzaDmnVvGmlKtdzuPIgCNwDPLAAWVQA9egDhoAgwl4BM/gxXgwnoxX423WumTMZw7ALxnvX1mYlFY=</latexit> df dx = dl dr · dr dx <latexit sha1_base64="xFN/reyOMTMYc0BqJoUqohqlBfg=">AAACGnicbVDLSsNAFJ34rPUVdelmsAiuSlKrbRdC0Y3LCvYBTSiTyaQdOnkwMxFLyHe48VfcuFDEnbjxb5ymUdR6YODcc+7lzj1OxKiQhvGhLSwuLa+sFtaK6xubW9v6zm5HhDHHpI1DFvKegwRhNCBtSSUjvYgT5DuMdJ3xxdTv3hAuaBhcy0lEbB8NA+pRjKSSBrppeRzhxPXSxL1N4RnMa6ZqnkILu6H80njWM9BLRrnRqDZOqlCRes04rUGzbGT4JiWQozXQ3yw3xLFPAokZEqJvGpG0E8QlxYykRSsWJEJ4jIakr2iAfCLsJDsthYdKcaEXcvUCCTP150SCfCEmvqM6fSRH4q83Ff/z+rH06nZCgyiWJMCzRV7MoAzhNCfoUk6wZBNFEOZU/RXiEVI5SJVmUYUwd/I86VTK5nG5clUtNc/zOApgHxyAI2CCGmiCS9ACbYDBHXgAT+BZu9cetRftdda6oOUze+AXtPdPZTiiYA==</latexit> dP dp1 = dpk dpk 1 dpk 1 dpk 2 dpk 2 dpk 3 . . . dp2 dp1 <latexit sha1_base64="tk1VNbcDvJXBcNFPYk6ku+8MR7I=">AAACWXicbVHPS8MwGE2rm3P+mnr0EhyCF0fbTbcdBNGLxwnODbZR0jTdQtM2JKkwSv9JD4L4r3gw28pQ54PAy3vf9yV58TijUlnWh2FubZfKO5Xd6t7+weFR7fjkRSapwKSPE5aIoYckYTQmfUUVI0MuCIo8RgZe+LDwB69ESJrEz2rOySRC05gGFCOlJbfGx4FAOPN7eeZz187hLSwU7oZLLQuv7Dxfi6ttwZzfhrM2mtrwEyXXrlPMr7q1utXodlvd6xbUpNO2btrQblhLrEkdFOi5tTc9CacRiRVmSMqRbXE1yZBQFDOSV8epJBzhEE3JSNMYRUROsmUyObzQig+DROgVK7hUf3ZkKJJyHnm6MkJqJv96C/E/b5SqoDPJaMxTRWK8OihIGVQJXMQMfSoIVmyuCcKC6rtCPEM6DKU/YxHCxpM3yYvTsJsN56lVv7sv4qiAM3AOLoEN2uAOPIIe6AMM3sGXUTLKxqdpmBWzuio1jaLnFPyCefoN2qK2zw==</latexit> dP dp1 = dpk dpk 1 ✓ dpk 1 dpk 2 ✓ dpk 2 dpk 3 . . . dp2 dp1 ◆◆ <latexit sha1_base64="5Hf2lOUg+IsmPFYHgZi0yY9UU2s=">AAACcnicbZFbS8MwGIbTep6nqXijoNEhKOho63TuQhC98XKCU2EbJU3TLSw9kHwVRukP8O9556/wxh9gNsvw9EHIy/vky+GNlwiuwLLeDHNqemZ2bn6htLi0vLJaXlt/UHEqKWvRWMTyySOKCR6xFnAQ7CmRjISeYI/e4GbEH5+ZVDyO7mGYsG5IehEPOCWgLbf80gkkoZnfzDM/ce0cX+LCSdzB2MsGJ3aedwQL4HCCvsxCOf9hZ4JPNfZjUBPqFGd1JO/14aiY3HLFqjYatcZZDWtxUbfO69iuWuOaiAoqqumWX/W2NA1ZBFQQpdq2lUA3IxI4FSwvdVLFEkIHpMfaWkYkZKqbjSPL8YF2fBzEUo8I8Nj93pGRUKlh6OmVIYG++s1G5n+snUJw0c14lKTAIvp1UJAKDDEe5Y99LhkFMdSCUMn1XTHtE50M6F8q6RD+PPmveHCq9mnVuatVrq6LOObRNtpHh8hGdXSFblETtRBF78amsWPsGh/mlrlnFtmZRtGzgX6UefwJkbbAZA==</latexit> dP dp1 = ✓✓ dpk dpk 1 dpk 1 dpk 2 ◆ dpk 2 dpk 3 ◆ . . . dp2 dp1 <latexit sha1_base64="jTakLDXIlC1Q4eVTqc8mpDQ6B10=">AAACc3icbZHLSgMxFIYz473eqoIbFwaroAvLzLRauxBENy4rWBXaMmQymTY0cyE5I5RhXsDHc+dbuHFv2g71eiDhz/+dk8uJlwiuwLLeDHNufmFxaXmltLq2vrFZ3tp+UHEqKWvTWMTyySOKCR6xNnAQ7CmRjISeYI/e8GbMH5+ZVDyO7mGUsF5I+hEPOCWgLbf80g0koZnfyjM/ce0cX+KuYAEcF/OUJu5wwrPhqZ3nM3O6LJSjgeT9AZx8586M1764H4OaJTnFySW3XLGqzWa9eVbHWlw0rPMGtqvWJGaigopoueVXvRNNQxYBFUSpjm0l0MuIBE4Fy0vdVLGE0CHps46WEQmZ6mWTnuX4SDs+DmKpRwR44n6vyEio1Cj0dGZIYKB+s7H5H+ukEFz0Mh4lKbCITg8KUoEhxuMPwD6XjIIYaUGo5PqumA6Ibgbobxo34c+T/4oHp2rXqs5dvXJ1XbRjGe2hA3SMbNRAV+gWtVAbUfRu7Br7BjY+zD3zwDycpppGUbODfoR5+gnj7sB4</latexit> “Forward accumulation”: “Reverse accumulation”:
31. Forward vs. Reverse mode accumulation cf. Breleux et al., https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9244-deep-learning-with-myia.pdf
32. Forward vs. Reverse mode accumulation cf. Breleux et al., https://developer.download.nvidia.com/video/gputechconf/gtc/2019/presentation/s9244-deep-learning-with-myia.pdf
33. class D<X: Fun<X>>(val f: Fun<X>): Fun<X>(f) { fun Fun<X>.df(): Fun<X> = when (this) { is Sum -> left.df() + right.df() is Prod -> left.df() * right + left * right.df() is Comp -> left.df()(right) * right.df() } } Differentiation rules in Kotlin
34. Differentiation rules in Kotlin class D<X: Fun<X>>(val f: Fun<X>): Fun<X>(f) { fun Fun<X>.df(): Fun<X> = when (this) { is Const -> Zero() is Var -> One() is Sum -> left.df() + right.df() is Prod -> left.df() * right + left * right.df() is Comp -> left.df()(right) * right.df() is D -> f.df() } }
35. Functions in Kotlin sealed class Fun<X : Fun<X>>(open val sVars: Set<Var<X>> = emptySet()) : Field<Fun<X>>, (Bindings<X>) -> Fun<X> { constructor(fn: Fun<X>) : this(fn.sVars) constructor(vararg fns: Fun<X>) : this(fns.flatMap { it.sVars }.toSet()) override operator fun plus(addend: Fun<X>): Fun<X> = Sum(this, addend) override operator fun times(multiplicand: Fun<X>): Fun<X> = Prod(this, multiplicand) override operator fun div(divisor: Fun<X>): Fun<X> = this * divisor.pow(-One<X>()) override operator fun invoke(bnds: Bindings<X>): Fun<X> = Composition(this, bnds).run { if (bnds.isReassignmentFree) evaluate else this } open operator fun invoke(): Fun<X> = invoke(Bindings()) open fun d(v1: Var<X>): Fun<X> = Derivative(this, v1) open fun d(v1: Var<X>, v2: Var<X>): Vec<X, D2> = Vec(Derivative(this, v1), Derivative(this, v2)) open fun d(vararg vars: Var<X>): Map<Var<X>, Fun<X>> = vars.map { it to Derivative(this, it) }.toMap() }
36. Compile-time shape safety
37. Lazy Composition class Composition<X : Fun<X>>(val fn: Fun<X>, val bindings: Bindings<X>) : Fun<X>() { val evaluate by lazy { apply() } override val sVars: Set<Var<X>> by lazy { evaluate.sVars } fun Fun<X>.apply(): Fun<X> = bindings.sMap.getOrElse(this) { when (this) { is Var -> this is Const -> this is Prod -> left.apply() * right.apply() is Sum -> left.apply() + right.apply() is Power -> base.apply() pow exponent.apply() is Derivative -> df().apply() is Composition -> fn.apply().apply() } } }
38. Dataflow graphs Kraphviz by Stefan Niederhauser: https://github.com/nidi3/graphviz-java#kotlin-dsl
39. What can be derived?
40. Optimization is just one perspective “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk.” –John Von Neumann
41. Continuity: bounded differences
42. Computer science Point Cloud Voxelization (Tommy Hinks) The world is discrete
43. Physicists World is differentiable dp dt = @H @q , dq dt = @H @p <latexit sha1_base64="mvt81qlkGmhRnX0oT+ghEaP0p30=">AAACh3icnVHJTsMwFHTCXrYCRy4WFRJCUNKFLgcklgtHkCggNVHlOE6xcJbaDqKy/Ct8FDf+BictiO3EkyyNZuY9P4/9lFEhHefNsmdm5+YXFpdKyyura+vljc1bkWQckx5OWMLvfSQIozHpSSoZuU85QZHPyJ3/eJHrd0+EC5rEN3KcEi9Cw5iGFCNpqEH5xfWTZxIoN+QIq8CNkHzwQ5VqrQKp4Qk8nChuirikiBUGjJi6NI4PUn20jbTWB/DHrNH/ZpkVtB6UK0612212j5vQgE7babVhreoU9QkqYFpXg/KrGyQ4i0gsMUNC9GtOKj2Vz8aM6JKbCZIi/IiGpG9gjCIiPFXkqOGuYQIYJtycWMKC/dqhUCTEOPKNM19S/NRy8i+tn8mw4ykap5kkMZ5cFGYMygTmnwIDygmWbGwAwpyaXSF+QCYrab6uZEL49eTf4LZerTWq9etm5fR8Gsci2AY7YA/UQBucgktwBXoAW7PWvtWwmvaSfWS37M7EalvTni3wreyzd1o4yio=</latexit> Hamiltonian mechanics Wave equation Pendulum with friction @2u @t2 = c2 ✓ @2u @x2 + @2u @y2 + @2u @z2 ◆ <latexit sha1_base64="Uzb8tUvoA5hhpp36ioX3nITwlnw=">AAACoXicjVFbSxtBGJ3dWi/xFttHoQwGwVIImzUa81AQ24f2QYliVMgmYXbybTJk9sLMt8V02f/l7+ib/8bZJIiXgn4wcOZ858zlfH4ihUbHubfsDwsfF5eWV0qra+sbm+WtT1c6ThWHNo9lrG58pkGKCNooUMJNooCFvoRrf/yj6F//AaVFHF3iJIFuyIaRCARnaKh++c7z41sYZF6gGKeZlzCFgsmeS9P8cUexR92cfqe853oSAtx7qae9zM2fOm4LIs+/vSmcvFf4dyb0lBiO8GveL1ecarNZbx7UqQFHDeewQWtVZ1qPoELm1eqX/3mDmKchRMgl07pTcxLsZsXpXEJe8lINCeNjNoSOgRELQXezacI53TXMgAaxMitCOmWfOjIWaj0JfaMMGY70y15B/q/XSTE46mYiSlKEiM8uClKTeEyLcdGBUMBRTgxgXAnzVspHzCSFZqglE8KrL78GV261tl91z+uV45N5HMtkm+yQPVIjDXJMfpEWaRNufbF+WqfWmV2xf9st+2Imta255zN5VnbnAXTQ0ac=</latexit> d2x dt2 + 2r! dx dt + !2x = 0 <latexit sha1_base64="xCCOcd4qYvmT9/noOiBJXCmIH6w=">AAACX3icbVHRSiMxFM2Mu9tau+6s+7T4EiyCsFCmtbttHwRxX3x0YatCp5ZM5k4bmkyGJCOWMD/pm+DL/omZWmStXggczrnn5uYkzjnTJgwfPH/rw8dPtfp2Y6f5efdL8HXvUstCURhRyaW6jokGzjIYGWY4XOcKiIg5XMWL35V+dQtKM5n9NcscJoLMMpYySoyjpsFtFMs7SKyNUkUotpEgZq4EtkmJb2y3vCtfUabiyvJHV+FICpgR/J5x0+QM6+7VyJOwnAatsD0c9oY/e9iBQT/81ceddriqF9BC67qYBvdRImkhIDOUE63HnTA3E0uUYZRD2YgKDTmhCzKDsYMZEaAndpVPiQ8dk+BUKncyg1fs/w5LhNZLEbvOam29qVXke9q4MOlgYlmWFwYy+nxRWnBsJK7CxglTQA1fOkCoYm5XTOfE5WXclzRcCG+e/BZcdtud43b3T691eraOo4720QE6Qh3UR6foHF2gEaLo0fO9Ha/p/fNr/q4fPLf63trzDb0q//sTwKC3gw==</latexit>
44. Nonlinear dynamics: no closed form solution Three Body Orbit https://en.wikipedia.org/wiki/Three-body_problem https://en.wikipedia.org/wiki/Double_pendulum Double Pendulum
45. How do I derive in Kotlin? Gottfried Wilhelm Leibniz
46. Arbitrarily high order differentiation
47. Arbitrarily high order differentiation
48. Type safe vector manipulation
49. Type safe matrix manipulation
50. Type safe currying (experimental)
51. What’s the difference?
52. Code is the interface
53. Differentiable Programming “Neural networks are not just another classifier, they represent the beginning of a fundamental shift in how we write software.” -Andrej Karpathy
54. Why do derivatives matter?
55. Learn more at: kg.ndan.co Kotlin∇
56. Interested in mathematical abstractions? github.com/mipt-npm/kmath KMath
57. Special Thanks Liam Paull Michalis Famelis Jin Guo
58. #KotlinConf THANK YOU AND REMEMBER TO VOTE @breandan