cnn-lstm | CNN LSTM architecture implemented in Pytorch for Video | Machine Learning library

Problem solved ! If you encounter the same problem, please refer to this link: Combining CNN and bidirectional LSTM

I believe the input_shape is (128, 216, 1)

The issue here is that you don't have a time-axis to time distribute your CNN (DenseNet169) layer over.

In this step -

You are trying to reshape X_train to (51135, 4,32,32). It is impossible because X_train shape is (51135, 128, 12). 51135 x 128 x 12 != 51135 x 4 x 32 x 32

Since you are using return_sequences=True, this means LSTM will return the output with shape (batch_size, 84, 64). The 84 here comes due to Conv1D parameters you used. So when you apply Dense layer with 1 units, it reduces the last dimension to 1, which means (batch_size, 84, 64) will become (batch_size, 84, 1) after Dense layer application. You either should not use return_sequences=True or use another layer/layers to flatten the output to 2 dimensions before feeding it to Dense layer.

I see some problem in your code...

the last dimension output must be equal to the number of class and with multiclass tasks you need to apply a softmax activation: Dense(num_classes, activation='softmax')

you must set return_sequences=False in your last lstm cell because you need a 2D output and not a 3D

you must use categorical_crossentropy as loss function with one-hot encoded target

here a complete dummy example...

your data are in 3d format and this is all you need to feed a conv1d or an LSTM. if your target is 2D remember to set return_sequences=False in your last LSTM cell.

using a flatten before an LSTM is a mistake because you are destroying the 3D dimensionality

pay attention also on the pooling operation in order to not have a negative time dimension to reduce (I use 'same' padding in the convolution above in order to avoid this)

below is an example in a binary classification task

You can do this. However, in order to successfully implement this new model, you need to use the Functional API (https://www.tensorflow.org/guide/keras/functional) in order to achieve this.

Below I am giving you an example on how you can add new layers to a pretrained model.

I think I have found the answer...

The problem is in the convolutional layer. The kernel size is too small, which causes the dimensionality of the output layer is too high. To solve this problem, I change the kernel size from (2, 100) to (3, 100). Furthermore, I also add dropout to my model. The summary of the model I use now is given below:

Now the model could run smoothly in the Google Colab.

Hence, I think if a similar problem occurs, please check the output dimension of each layer. The Keras API may stop in the training epochs if the model creates a very high dimensional output.

You have to pass tensors between the layers as this is how the Functional API works, for all layers, using the Layer(params...)(input) notation: