batch_normalization | Batch Normalization Layer for Caffe | Machine Learning library

Try using tf.tensor_scatter_nd_update to make the top-k probabilities 1 and the rest 0:

The input shape of CIRFAR is (32, 32, 3) but your model's input isn't taking that shape. You can try as follows for your model input.

Apparently the tutorial I was following was inaccurate. Since my NN ha s5 branches as it needs to make 5 predictions, my model.fit function needed to be changed and each Nn branch needed to be mapped to the corresponding labels.

code with error:

Why do batch_normalization produce all-zero output when training = True

It's because your batch size = 1 here.

Batch normalization layer normalizes its input by using batch mean and batch standard deviation for each channel.

When the batch size is 1 and after flatten, there is only one single value in each channel, so that the batch mean(for that channel) will be the single value itself, thus outputting a zero tensor after the batch normalization layer.

but produce non-zero output when training = False?

During inference, batch normalization layer normalizes inputs by using moving average of batch mean and SD instead of current batch mean and SD.

The moving mean and SD are initialized as zero and one respectively and updated gradually. Therefore, the moving mean doesn't equal that single value in each channel at the beginning, therefore the layer will not output a zero tensor.

In conclusion: use batch size > 1 and input tensor with random values/realistic data values rather than tf.ones() in which all elements are the same.

One possible solution (or an idea) is to wrap your tensorflow operation into the Lambda layer and use it to build the keras model. Something like

Solved by reworking the network and putting all the layers one after the other rather than having multiple instances of a model. So everything from beginning to end is in one single subclassed model.

It could be that the decay_rate argument in tf.keras.optimizers.schedules.ExponentialDecay is decaying your learning rate quicker than you think it is, effectively making your learning rate zero.

Use class_mode="input" at the flow_from_directory so returned Y will be same as X

https://github.com/tensorflow/tensorflow/blob/v2.4.1/tensorflow/python/keras/preprocessing/image.py#L867-L958

class_mode: One of "categorical", "binary", "sparse", "input", or None. Default: "categorical". Determines the type of label arrays that are returned: - "categorical" will be 2D one-hot encoded labels, - "binary" will be 1D binary labels, "sparse" will be 1D integer labels, - "input" will be images identical to input images (mainly used to work with autoencoders). - If None, no labels are returned (the generator will only yield batches of image data, which is useful to use with model.predict()). Please note that in case of class_mode None, the data still needs to reside in a subdirectory of directory for it to work correctly.

Code should end up like:

... also tried with sigmoid but the issue persists ...

You don't want to be "trying" out activation functions or loss functions for a well-defined problem statement. It seems you are mixing up a single-label multi-class and a multi-label multi-class architecture.

Your output is a 2 class multi-class output with softmax activation which is great, but you use binary_crossentropy which would only make sense when used in a multi-class setting for multi-label problems.

You would want to use categorical_crossentropy instead. Furthermore, I would have suggested focal loss if there was class imbalance but it seems you have a 50,50 class proportion, so that's not necessary.

Remember, accuracy is decided based on which loss is being used! Check the different classes here. When you use binary_crossentropy the accuracy used is binaryaccuracy while with categorical_crossentropy, it uses categoricalaccuracy

Check this chart for details on what to use in what type of problem statement.

Other than that, there is a bottleneck in your network at flatten() and Dense(). The number of trainable parameters is quite high relative to other layers. I would advise using another CNN layer to bring the number of filters to say 128 and the size of sequence even smaller. And reduce the number of neurons for that Dense layer as well.

98.9% (3,277,000/3,311,354) of all of your trainable parameters reside between the Flatten and Dense layer! Not a great architectural choice!

Outside the above points, the model results are totally dependent on your data itself. I wouldn't be able to help more without knowledge of the data.