Introduction:

Less than two months ago I had an exchange with Barak Pearlmutter, a neuroscientist who has made seminal contributions to the field of automatic differentiation, on the relation between Cauchy’s Integral Formula for derivatives and forward mode automatic differentiation. In particular, he mentioned that if we collapse the contour to a point we obtain the complex-step method which is in some sense equivalent to forward-mode algorithmic differentiation [3].

Therefore the objective of this article is to derive the complex-step method from the Cauchy Integral Formula for derivatives and thereby establish a correspondence between the latter and forward-mode algorithmic differentiation.

Problem:

The complex derivative for a holomorphic function in one complex variable is given by:

\begin{equation} \forall z_0 \in A,f’(z_0)=\lim_{z \to z_0} \frac{f(z)-f(z_0)}{z-z_0} \end{equation}

and using Cauchy’s Integral Formula we also have:

\begin{equation} \forall z_0 \in A,f’(z_0)= \frac{1}{2\pi i} \int_{\gamma} \frac{f(z)}{(z-z_0)^2} dz \end{equation}

where is a simple closed piecewise smooth and positively oriented curve in where .

It is possible to derive (2) from (1) using Cauchy’s Integral Formula without too much difficulty and we can also derive the complex-step derivative approximation from (1) as follows:

Let’s suppose . Due to path-independence we have:

\begin{equation} f’(x) = \lim_{z \to x} \frac{f(z)-f(x)}{z-x} = \lim_{h \to 0} \frac{f(x+ih)-f(x)}{(x+ih)-x} = \lim_{h \to 0} \frac{f(x+ih)-f(x)}{ih} \end{equation}

Now, if we compute the Taylor series of we have:

\begin{equation} f(x+ih)=f(x) + f’(x)(ih)-\frac{h^2 f^{(2)}(x)}{2} + … \end{equation}

and so (3) simplifies to:

\begin{equation} f’(x) = \lim_{h \to 0} \frac{\text{Im}(f(x+ih))}{h} \end{equation}

which is the complex-step method as introduced in [3]. Now, the question is how to reach (5) from (2) by ‘collapsing’ the contour to a point.

Solution:

By defining the contour in (2) as a disk with radius centred at :

\begin{equation} \gamma = x+ U_r = \{x+r\cdot e^{i \theta}: r >0, \theta \in [0,2\pi]\} \subset A \end{equation}

the Cauchy Integral Formula for derivatives, i.e. equation (2), simplifies to:

\begin{equation} f’(x)= \frac{1}{2\pi}\int_{0}^{2\pi} \frac{f(x+r\cdot e^{i\theta})}{r\cdot e^{i\theta}} d\theta \end{equation}

Due the symmetries of , so if we take the limit as and use the Taylor series expansion of at , (7) becomes:

\begin{equation} f’(x)= \lim_{r \to 0} \frac{1}{2\pi}\int_{0}^{2\pi} \frac{f(x) + f’(x) \cdot (r\cdot e^{i\theta})+…}{r\cdot e^{i\theta}} d\theta = \lim_{r \to 0} \frac{\text{Im}(f(x+ir))}{r} \end{equation}

where to obtain the RHS we implicitly used the cancellation of terms and path-independence to choose so .

References:

  1. J.N. Lyness & C.B. Moler. NUMERICAL DIFFERENTIATION OF ANALYTIC FUNCTIONS. SIAM Journal of Numerical Analysis. 1967.
  2. William Squire & George Trapp. USING COMPLEX VARIABLES TO ESTIMATE DERIVATIVES OF REAL FUNCTIONS. SIAM review. 1998.
  3. Joaquim Martins, Peter Sturdza, and Juan J. Alonso. THE CONNECTION BETWEEN THE COMPLEX-STEP DERIVATIVE APPROXIMATION AND ALGORITHMIC DIFFERENTIATION. American Institute of Aeronautics and Astronautics. 2001.