autodiff | Rudimentary automatic differentiation framework | Machine Learning library

TensorFlow 2 does not directly computes the derivative of a function of complex variables. It seems that it computes the derivative of a function of a complex variable as the function of the real part and the imaginary part, using Wirtinger calculus. You can also find an explanation here.

I found the problem: in my mat_fun the type of the return had to be "Real" in order for it to propgate through. Before it was Float64, which was not consistent with the fact I guess all types have to be Real with the autodifferentiate. Even though a Float64 is clearly Real, it looks like the inheritence isn't perserved i.e you have to make sure everything that is returned and inputed are type Real.

Functions of interest are to be converted into templates that may accept either double or boost fvar arguments. Note that boost provides custom implementations of trigonometric functions from standard library (such as sin, cos) suitable for fvar:

Could figure out a way, sharing it here.

For a given function foo, Zygote.pullback(foo, args...) returns foo(args...) and the backward pass (which allows for gradients computations).

My goal is to tell Zygote to use Enzyme for the backward pass.

This can be done by means of Zygote.@adjoint (see more here).

In case of array-valued functions, Enzyme requires a mutating version that returns nothing and its result to be in args (see more here).

The function f! in the question post is an Enzyme-compatible version of a sum of two arrays.

Since f! returns nothing, Zygote would simply return nothing when the backward pass is called on some gradient passed to us.

A solution is to place f! inside a wrapper (say f) that returns the array s

and to define Zygote.@adjoint for f, rather than f!.

Hence,

The current recommended answer is to use symbolic::Expression instead of AutoDiffXd when you need more than one derivative. While all of our C++ code should work if it was compiled with AutoDiffX to provide second derivatives, we currently don't build those as one of our default scalar types in libdrake.so.

In the context you ask about you can think that closure is a function that references to some variables that are defined in its outer scope (for other cases see the answer by @phipsgabler). Here is a minimal example:

I suppose you write your constraint using a python function. I would suggest to write this python function to handle both float and autodiffxd, so something like this

While I'm not familiar with Drake/PyDrake, any autodiffing program requires functions be implemented in a way that their derivatives are known. It seems that PyDrake is inspecting your code, identifying functions it knows autodiff versions of (e.g., np.arctan2) and replacing them with those versions. It looks like this is the list of functions PyDrake has implemented, so you may want to refer to this list rather than use trial and error. Oddly enough, arctan is there as well as arctan2. I think there may be an additional problem here, specifically that arctan(y/x) is not differentiable everywhere, whereas arctan2(x, y) is designed to fix that. See these plots of arctan(y/x) and arctan2(x, y) as examples.

Regardless, for mathematical reasons you probably want to be using arctan2 to find that angle, unless you know it's restricted to a certain range.

Here's a way using the numderiv function in madness: