Tensor | A library and extension that provides objects for scientific computing in PHP | Machine Learning library

It happened the same to me last friday. I think it has something to do with Cuda instalation in Google Colab but I don't know exactly the reason

This seems to be a bug in Tensorflow 2.7 when using model.save combined with the parameter save_format="tf", which is set by default. The layers RandomFlip, RandomRotation, RandomZoom, and RandomContrast are causing the problems, since they are not serializable. Interestingly, the Rescaling layer can be saved without any problems. A workaround would be to simply save your model with the older Keras H5 format model.save("test", save_format='h5'):

TLDR; Both are two different interfaces to perform gradient computation: torch.autograd.grad is non-mutable while torch.autograd.backward is.

Descriptions

The torch.autograd module is the automatic differentiation package for PyTorch. As described in the documentation it only requires minimal change to code base in order to be used:

you only need to declare Tensors for which gradients should be computed with the requires_grad=True keyword.

The two main functions torch.autograd provides for gradient computation are torch.autograd.backward and torch.autograd.grad:

torch.autograd.backward (source) torch.autograd.grad (source) Description Computes the sum of gradients of given tensors with respect to graph leaves. Computes and returns the sum of gradients of outputs with respect to the inputs. Header torch.autograd.backward(
tensors,
grad_tensors=None,
retain_graph=None,
create_graph=False,
grad_variables=None,
inputs=None) torch.autograd.grad(
outputs,
inputs,
grad_outputs=None,
retain_graph=None,
create_graph=False,
only_inputs=True,
allow_unused=False) Parameters - tensors – Tensors of which the derivative will be computed.
- grad_tensors – The "vector" in the Jacobian-vector product, usually gradients w.r.t. each element of corresponding tensors.
- retain_graph – If False, the graph used to compute the grad will be freed. [...]
- inputs – Inputs w.r.t. which the gradient be will be accumulated into .grad. All other Tensors will be ignored. If not provided, the gradient is accumulated into all the leaf Tensors that were used [...]. - outputs – outputs of the differentiated function.
- inputs – Inputs w.r.t. which the gradient will be returned (and not accumulated into .grad).
- grad_tensors – The "vector" in the Jacobian-vector product, usually gradients w.r.t. each element of corresponding tensors.
- retain_graph – If False, the graph used to compute the grad will be freed. [...]. Usage examples

In terms of high-level usage, you can look at torch.autograd.grad as a non-mutable function. As mentioned in the documentation table above, it will not accumulate the gradients on the grad attribute but instead return the computed partial derivatives. In contrast torch.autograd.backward will be able to mutate the tensors by updating the grad attribute of leaf nodes, the function won't return any value. In other words, the latter is more suitable when computing gradients for a large number of parameters.

In the following, we will take two inputs (x1 and, x2), calculate a tensor y with them, and then compute the partial derivatives of the result w.r.t both inputs, i.e. dL/dx1 and dL/dx2:

You do not have to use tf.map_fn. Maybe try something like this:

So if you look at the source code for tensorly linked here you can see that the documentation for the function in question partial_tucker says:

Your issue is not related to 'transformer_transducer/transducer_encoder/inputs_embedding/ convolution_stack/conv2d/kernel:0'.
The error code tells you this element is referring to a non trackable element. It seems the non-trackable object is not directly assigned to an attribute of this conv2d/kernel:0.

To solve your issue, we need to localize Tensor("77040:0", shape=(), dtype=resource) from this error code:

The Reformer model was proposed in the paper Reformer: The Efficient Transformer by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya. The paper contains a method for factorization gigantic matrix which is resulted of working with very long sequences! This factorization is relying on 2 assumptions

1. the parameter config.axial_pos_embds_dim is set to a tuple (d1,d2) which sum has to be equal to config.hidden_size
2. config.axial_pos_shape is set to a tuple (n1s,n2s) which product has to be equal to config.max_embedding_size (more on these here!)

Finally your question ;)

• I'm almost sure your session crushed duo to ram overflow
• you can change any config parameter during model instantiation like the official documentation!

As the name suggests, a "type parameter" is a type. Not 'a length of a string'.

To be specific: One can imagine the concept of the type fixed length string, and one can imagine this concept has a parameter, whose type is int; one could have FixedString<5> myID = "HELLO"; and that would compile, but FixedString<5> myID = "GOODBYE"; would be an error, hopefully a compile-time one.

Java does not support this concept whatsoever. If that's what you're looking for, hack it together; you can of course make this work with code, but it means all the errors and checking occurs at runtime, nothing special would occur at compile time.

Instead, generics are to give types the ability to parameterize themselves, but only with a type. If you want to convey the notion of 'A List... but not just any list, nono, a list that stores Strings' - you can do that, that's what generics are for. That concept applies only to types and not to anything else though (such as lengths).

Furthermore, javac will be taking care of applying the parameter. So you can't hack it together by making some faux hierarchy such as:

TLDR; the derivative of a sum of functions is the sum of their derivatives

Let x be your input vector made of x_i (where i in [0,n]), y = x**2 and L = sum(y_i). You are looking to compute dL/dx, a vector of the same size as x whose components are the dL/dx_j (where j in [0,n]).

For j in [0,n], dL/dx_j is simply dy_j/dx_j (derivative of the sum is the sum of derivates and only one of them is different to zero), which is d(x_j**2)/dx_j, i.e. 2*x_j. Therefore, dL/dx = [2*x_j where j in [0,n]].

This is the result you get in x.grad when either computing the gradient of x as: