An Intellectual History of Automatic Differentiation traces the research surrounding a collection of techniques for computing derivatives without using either approximation or the manipulation of subscript-filled equations used to terrorize high school students. While its simplicity gives this method the mystery of "deep magic," it has its roots in work on differential equations in the late 19th century; inspired Alonzo Church's discovery of the untyped lambda calculus; influenced the development of functional programming, concurrency, and Unix in the 1970s; and has been recently rediscovered with applications to type theory, modelling stochastic processes, and training recurrent neural networks.

References

- Computer Aided Manipulation of Symbols, Fred McBride 1971

- Coroutines and Networks of Parallel Processes, Gilles Kahn & David MacQueen, 1977

- Squinting at Power Series, Doug McIlroy, 1989

- Generating Power of Lazy Semantics, Jerzy Karczmarczuk,1997

- Power Series, Power Serious, Doug McIlroy, 1998

- Calculus in Coinductive Form, Pavlovic & Escardo 1998

- Functional Differentiation of Computer Programs, Jerzy Karczmarczuk, 2000

- Adjoint Codes in Functional Framework, Jerzy Karczmarczuk, 2000

- Perturbation Confusion and Referential Transparency: Correct Functional Implementation of Forward-Mode AD, Pearlmutter & Siskind, 2005

- Reverse-Mode AD in a Functional Framework: Lambda the Ultimate Backpropagator, Pearlmutter & Siskind, 2008

- The Differential Lambda-Calculus, Ehrhard & Regnier, 2001

- Efficient Implementation of a Higher-Order Language with Built-In AD, Pearlmutter & Siskind, 2016

May 24, 2017