Fast Higher-order Automatic Differentiation for Physical Models

Taking higher-order derivatives is crucial for physical models like ODEs and PDEs, and it would be great to get it done by automatic differentiation. Yet, existing packages in Julia either has exponential scaling w.r.t. order (nesting first-order AD) or has exponential scaling w.r.t. dimension (nested Taylor polynomial in TaylorSeries.jl). The author presents TaylorDiff.jl (https://github.com/JuliaDiff/TaylorDiff.jl) which is specifically optimized for fast higher-order directional derivatives.

The author will be presenting efficient higher-order automatic differentiation (AD) algorithms and its potential application in scientific models where higher-order derivatives are required to calculated efficiently, such as solving ODEs and PDEs with neural functions. Existing methods to achieve higher-order AD often suffer from one or more of the following problems: (1) nesting first-order AD would result in exponential scaling w.r.t. order; (2) ad-hoc hand-written higher-order rules which is hard to maintain and not utilizing existing first-order AD infrastructures; (3) inefficient data representation and manipulation that causes "penalty of abstraction" problem at first- or second-order when compared to highly-optimized first-order AD libraries. By combining advanced techniques in computational science, i.e. aggressive type specializing, metaprogramming and symbolic computing, TaylorDiff.jl will address all three problems.

TaylorDiff.jl is currently capable for the followings: (1) Taylor-mode AD algorithms which can compute the n-th order derivative of scalar functions in O(n) time; (2) automatic generation of higher-order rules from first-order rules in ChainRules.jl; (3) achieved comparable performance with ForwardDiff.jl at lower order.