training | Custom Object Detection and Classification Training | Computer Vision library

This seems to be a bug in Tensorflow 2.7 when using model.save combined with the parameter save_format="tf", which is set by default. The layers RandomFlip, RandomRotation, RandomZoom, and RandomContrast are causing the problems, since they are not serializable. Interestingly, the Rescaling layer can be saved without any problems. A workaround would be to simply save your model with the older Keras H5 format model.save("test", save_format='h5'):

You end up with the error with precision because some of your penalization is too strong for this model, if you check the results, you get 0 for f1 score when C = 0.001 and C = 0.01

Your issue is not related to 'transformer_transducer/transducer_encoder/inputs_embedding/ convolution_stack/conv2d/kernel:0'.
The error code tells you this element is referring to a non trackable element. It seems the non-trackable object is not directly assigned to an attribute of this conv2d/kernel:0.

To solve your issue, we need to localize Tensor("77040:0", shape=(), dtype=resource) from this error code:

I had this issue a few weeks ago, what a pain! In my case I had one dependency that was requiring background location without me noticing. Secondly, I had a wrong permission declaration on Google Play so my builds kept being rejected.

To do this I used the Merged Manifest inspector in Android Studio. This shows you what your manifest looks like after all project dependencies have been taken into account. Find ACCESS_BACKGROUND_LOCATION and double click on it, this will bring you to the actual manifest where it's requested. Scroll to the top of this file and the package=some.package.name should help you identify what it is. In my case the permission was requested by an old dependency I didn't use anymore so I just uninstalled it.

Note: if you're often working on different branches, make sure you have the correct dependencies installed and make a clean build before checking the merged manifest:

TensorFlow has 2 execution modes: eager execution, and graph mode. TensorFlow default behavior, since version 2, is to default to eager execution. Eager execution is great as it enables you to write code close to how you would write standard python. It's easier to write, and it's easier to debug. Unfortunately, it's really not as fast as graph mode.

So the idea is, once the function is prototyped in eager mode, to make TensorFlow execute it in graph mode. For that you can use tf.function. tf.function compiles a callable into a TensorFlow graph. Once the function is compiled into a graph, the performance gain is usually quite important. The recommended approach when developing in TensorFlow is the following:

• Debug in eager mode, then decorate with @tf.function.
• Don't rely on Python side effects like object mutation or list appends.
• tf.function works best with TensorFlow ops; NumPy and Python calls are converted to constants.

I would add: think about the critical parts of your program, and which ones should be converted first into graph mode. It's usually the parts where you call a model to get a result. It's where you will see the best improvements.

You can find more information in the following guides:

• Better performance with tf.function
• Introduction to graphs and tf.function

So, there are at least two things you can change in your code to make it run quite faster:

1. The first one is to not use model.predict on a small amount of data. The function is made to work on a huge dataset or on a generator. (See this comment on Github). Instead, you should call the model directly, and for performance enhancement, you can wrap the call to the model in a tf.function.

Model.predict is a top-level API designed for batch-predicting outside of any loops, with the fully-features of the Keras APIs.

1. The second one is to make your training step a separate function, and to decorate that function with @tf.function.

So, I would declare the following things before your training loop:

Remove roc_auc if it is multi class. They do not play well together. Use default scoring or choose something else.

The way you are loading your data is not the recommended way to load your parameters because you're overwriting the graph connections (or something along those lines...). You even save the model state_dict, so why not use it!

I changed the load function to:

There is some issue with your code. I will try to answer the main issue here and discard some side questions such as whether you should use Dropout or BatchNormalization layers in your model or not because that's totally out of the scope of your main question and also irrelevant.

If you try to build your model, using m = define_model(), I'm pretty sure you will encounter the following error:

Thanks to @Mr.For Example for his convenient answer.

Usually, I also observed that using Gradient Accumulation, won't speed up training since we are doing n_gradients times forward pass and compute all the gradients. But it will speed up the convergence of our model. And I found that using the mixed_precision technique here can be really helpful here. Details here.