API_Docs

Why do batch_normalization produce all-zero output when training = True

It's because your batch size = 1 here.

Batch normalization layer normalizes its input by using batch mean and batch standard deviation for each channel.

When the batch size is 1 and after flatten, there is only one single value in each channel, so that the batch mean(for that channel) will be the single value itself, thus outputting a zero tensor after the batch normalization layer.

but produce non-zero output when training = False?

During inference, batch normalization layer normalizes inputs by using moving average of batch mean and SD instead of current batch mean and SD.

The moving mean and SD are initialized as zero and one respectively and updated gradually. Therefore, the moving mean doesn't equal that single value in each channel at the beginning, therefore the layer will not output a zero tensor.

In conclusion: use batch size > 1 and input tensor with random values/realistic data values rather than tf.ones() in which all elements are the same.

That's because holding all elements of your dataset in the buffer is expensive. Unless you absolutely need perfect randomness, you should use a smaller buffer_size. All elements will eventually be taken, but in a more deterministic manner.

This is what's going to happen with a smaller buffer_size, say 3. The buffer is the brackets, and Tensorflow samples a random value in this bracket. The one randomly picked is ^

You can try this

Both tf.keras.callbacks.LearningRateScheduler() and tf.keras.optimizers.schedules.LearningRateSchedule() provide the same functionality i.e to implement a learning rate decay while training the model.

A visible difference could be that tf.keras.callbacks.LearningRateScheduler takes in a function in its constructor, as mentioned in the docs,

The reason you're getting this error is that the indices in tf.tensor_scatter_nd_update requires at least two axes, or tf.rank(indices) > = 2 need to be fullfilled. The reason for indices in 2D (in scaler update) is to hold two information, one is the length of the updates (num_updates) and the length of the index vector. For a detailed overview of this, check the following answer regarding this: Tensorflow 2 - what is 'index depth' in tensor_scatter_nd_update?.

Here is the correct implementation of SPL loss in tensorflow.

Just as the official docs for tf.keras.Sequential suggest, no batch_size needs to be provided when inputs are instances of tf.data.Dataset while calling tf.keras.Sequential.fit(),

Integer or None. Number of samples per gradient update. If unspecified, batch_size will default to 32. Do not specify the batch_size if your data is in the form of datasets, generators, or keras.utils.Sequence instances (since they generate batches).

In case of tf.data.Dataset, the fit() method expects a batched dataset.

To batch the tf.data.Dataset, use the batch() method,

I found the answer to this particular problem. Amongst other issues, "class_mode" has to be set to None for this kind of model. With that set, the second array in both X and y is not written by the ImageDataGenerator. As a result, X and y are interpreted as the data and the mask (which is what we want) in the combined ImageDataGenerator. Otherwise, X_val_gen already produces the tuple shown in the screenshot, where the second entry is interpreted as the class, which would make sense in a classification problem with images spread out in various folders each labeled with a class ID.

About the tf.tensor_scatter_nd_update, you're right that it returns a new tf.tensor (and not tf.Variable). But about the assign which is an attribute of tf.Variable, I think you somewhat misread the document; the value is just the new item that you want to assign in particular indices of your old variable.

AFAIK, in tensorflow all tensors are immutable like python numbers and strings; you can never update the contents of a tensor, only create a new one, source. And directly updating or manipulating of tf.tensor or tf.Variable such as numpy like item assignment is still not supported. Check the following Github issues to follow up the discussions: #33131, #14132.

In numpy, we can do an in-place item assignment that you showed in the comment box.