Digit-Recognizer | newfound Neural Network knowledge - I used Python | Machine Learning library

x_train is a numpy array, while x_test is a pandas dataframe.

See that in your first version the x_train reshape works, while fails in the second.

Convert x_test to a numpy array instead of a dataframe, and your first version should work fine.

You know by either printing the type as you have done, or by Googling for the docs of the function you are using.

See here for the docs on the fit_transform method, which says:

Returns X_new: ndarray array of shape (n_samples, n_features_new)

As a general rule of thumb, pd is pandas and returns a pandas Dataframe, whilst np and sklearn usually use numpy arrays.

Here is how I eventually saved the augmented data with labels. I sampled 5 rows for viewing pleasure. And the for loop might not be the best way to write to array when full dataset is considered

It seems that you forgot to add the .batch() method at the end of your tf.data.Dataset objects, since your error refers to the batch dimension. From what I understand, creating a tf.data.Dataset stores the data set as something similar to a python generator rather than storing the whole data set in memory. This means that the cardinality (number of data points) of the data set is unknown. When you pass in a number to steps_per_epoch when using a tf.data.Dataset, your model uses that number to take that many batch sized samples from your data set. It is unable to calculate ahead of time the size of batches since the cardinality is unknown. Since you haven't batched your data, it will take individual samples. When creating data as numpy arrays, you have a defined number of data points, so your model will be able to calculate the size of your batches and use that.

As explained in this discussion, torch.save() saves the whole tensor, not just the slice. You need to explicitly copy the data using clone().

Don't worry, at runtime the data is only allocated once unless you explicitly create copies.

As a general advice: If the data easily fits into your memory, just load it at once. For MNIST with 130 MB that's certainly the case.

However, I would still batch the data because it converges faster. Look up the advantages of SGD for more details.

I think in ClassCNN last layer activation is 'relu' which should be 'softmax' as is the case with other models... It is just a human mistake ..... Thankyou...