oneDNN | oneAPI Deep Neural Network Library | Machine Learning library

I eventually caught the pod in the act. For a brief moment tensorflow-predictor reported itself as "Killed", before silently regenerating. Turns out the pod did not have enough memory, so the container was killing off tensorflow-predictor as described here whenever an actual query triggered it.

I think there is something wrong with tensorflow in my mac installation.

I now run it with Docker so I don't have the environment problem.

I have the following Dockerfile

I don't think the issue is the small dataset, since transfer learning is used to deal with smaller datasets.

The issue is that you are freezing all the layers of the pre-trained model (VGG), without adding any new Dense Layer. Then you call model.fit, but none of the layers are trainable. Therefore, nothing is allowed to change. In fact, your problem is not that you are getting very low accuracy, but that the accuracy doesn't change at all among epochs. This should be a red flag meaning something in your code is broken!

Try to add at least another Dense layer before the last.

EDIT:

You are also compiling and calling fit() on model instead of new_model.

I hope I've been helpful

Since I am working in a conda venv and using Poetry for handling dependencies, I needed to re-install torch - a dependency for Hugging Face 🤗 Transformers.

First, install torch: PyTorch's website lets you chose your exact setup/ specification for install. I my case, the command was

It seems like the suggestions from this post helped - I've copied the following files from the zipped cudnn bin sub folder (cudnn-11.2-windows-x64-v8.1.1.33\cuda\bin) into my cuda bin folder (C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\bin)

run with CUDA_VISIBLE_DEVICES="-1" ./your_code.py if using python script or import os; os.environ['CUDA_VISIBLE_DEVICES'] = '-1'` in the code.

If you experienced significant change in nvidia-smi and/or speed/duration of the training, then you were using GPU in the first place. ( having `CUDA_VISIBLE_DEVICES="0" ( or "0,1,2" if on multi-gpu setting)

• Make sure you are importing and using tf.keras.
• Make sure you have installed tensorflow-gpu
• Watch GPU utilization with watch -n 1 nvidia-smi while .fit is running.
• Check version compatibility table. This is important.
• Ignore the cuda version shown in nvidia-smi, as it is the version of cuda, your driver came with. The installed cuda version is shown with nvcc -V.

The model is getting loaded to GPU. So, it is not related to your GPU utilization issue. It is possible that your train_gen and val_gen takes time or they are buggy. Try without performing any specific augmentation to make sure the problem is not related to *_gen.

After a long time and didn't get an answer, I asked the question in the TensorFlow's forum and get the answer. It seems that the current version of TensorFlow has a problem with loading decision forests with C_API. So we can use the Yggdrasil library as discussed in the answer.

According to the documentation, the argument labels must be a batch_size vector with values in [0, num_classes) From your logs:

The equivalent of this model using the Functional API: