backprop | Goodbye get and set | Frontend Framework library

Backpropagation is a special case of reverse mode automatic differentiation (AD). In contrast to the forward mode, the reverse mode has the major advantage that you can compute the derivative of an output w.r.t. all inputs of a computation in one pass.

However, the downside is that you need to store all intermediate results of the algorithm you want to differentiate in a suitable data structure (like a graph or a Wengert tape) for as long as you are computing its Jacobian with reverse mode AD, because you're basically "working your way backwards" through the algorithm.

Forward mode AD does not have this disadvantage, but you need to repeat its calculation for every input, so it only makes sense if your algorithm has a lot more output variables than input variables.

Finally, I have found no other solution than re-creating a layer object from the class tf.keras.layers.Layer and re-defining its methods build and call, giving for a dense layer for example:

You should use Tensorflow operations when calculating the loss of your model. Try replacing np.max with tf.math.reduce_max:

For a measure to be used as a loss function, it must be differentiable, with non-trivial gradients.

For instance, in image classification, accuracy is the most common measure of success. However, if you try to differentiate accuracy, you'll see that the gradients are zero almost everywhere and therefore one cannot train a model with accuracy as a loss function.
Similarly, IoU, in its native form, also has meaningless gradients and cannot be used as a loss function. However, extensions to IoU that preserve gradients exist and can be effectively used as a loss function for training.

I managed to sidestep these numerical errors by doing the following:

1. Rescale my dependent variables and parameters so everything is within a few orders of magnitude of each other.
2. Increase the number of independent variable values at which the ODE solver returns dependent variable values. I only need 20 evenly spaced points in time to calculate my loss function, but if I want to calculate the gradient of that loss function wrt my parameters the ODE solver must return at least 60 evenly spaced points in time. I just throw out the extra points.

From the pytorch docs, you're basically on the right track. You can loop over all the parameters in each layer and then add to them directly

For anyone to stumble upon this: I found the solution after some hours of searching and trying. The custom component is only recognized as HTMLElement if the tooltipProps are passed to the main node. Guess I wanted to have the modal too simple

They are not equivalent. The top one will sum nablas over minibatch. The bottom one will only keep values from the last sample.

• Inference

  By default, an inference on your model will allocate memory to store the activations of each layer (activation as in intermediate layer inputs). This is needed for backpropagation where those tensors are used to compute the gradients. A simple but effective example is a function defined by f: x -> x². Here, df/dx = 2x, i.e. in order to compute df/dx you are required to keep x in memory.

  If you use the torch.no_grad() context manager, you will allow PyTorch to not save those values thus saving memory. This is particularly useful when evaluating or testing your model, i.e. when backpropagation is performed. Of course, you won't be able to use this during training!

• Backward propagation

  The backward pass call will allocate additional memory on the device to store each parameter's gradient value. Only leaf tensor nodes (model parameters and inputs) get their gradient stored in the grad attribute. This is why the memory usage is only increasing between the inference and backward calls.

• Model parameter update

  Since you are using a stateful optimizer (Adam), some additional memory is required to save some parameters. Read related PyTorch forum post on that. If you try with a stateless optimizer (for instance SGD) you should not have any memory overhead on the step call.

All three steps can have memory needs. In summary, the memory allocated on your device will effectively depend on three elements:

1. The size of your neural network: the bigger the model, the more layer activations and gradients will be saved in memory.

2. Whether you are under the torch.no_grad context: in this case, only the state of your model needs to be in memory (no activations or gradients necessary).

3. The type of optimizer used: whether it is stateful (saves some running estimates during parameter update, or stateless (doesn't require to).

whether you require to do back