tensorflow-mnist | Tensorflow MNIST | Machine Learning library

1. Units is the number of neurons, which is the dimensionality of the output for that layer. This information can be found at the documentation.

2. The number of parameters are dependent on the layer input and the number of units. For the SimpleRNN layer this is 128 * 128 + 128 * 28 + 128 = 20096 (see this answer). For the dense layer this is 128 * 10 + 10 = 1290. These 10 and 128 that are added are because of the bias weights in the layer, which is turned on by default.

3. input_shape = (28, 28) means that your network will handle inputs of size 28x28 data points. Since the first dimension is the batch dimension, it will handle 28 vectors of length 28 (as depicted in your image). Inputs of a variable length are usually split up to fit in the given input_shape. If an input is smaller than the input_shape, padding can be applied to make it fit.

You can (sort of) think of it exactly as you think of fully connected layers. Units are neurons.

The dimension of the output is the number of neurons, as with most of the well known layer types.

The difference is that in LSTMs, these neurons will not be completely independent of each other, they will intercommunicate due to the mathematical operations lying under the cover.

Before going further, it might be interesting to take a look at this very complete explanation about LSTMs, its inputs/outputs and the usage of stative = true/false: Understanding Keras LSTMs. Notice that your input shape should be input_shape=(look_back, 1). The input shape goes for (time_steps, features).

• hidden layer 1: 4 units
• hidden layer 2: 4 units
• output layer: 1 unit

Where input_shape = (batch_size, arbitrary_steps, 3)

Each LSTM layer will keep reusing the same units/neurons over and over until all the arbitrary timesteps in the input are processed.

• The output will have shape:
  • (batch, arbitrary_steps, units) if return_sequences=True.
  • (batch, units) if return_sequences=False.
• The memory states will have a size of units.
• The inputs processed from the last step will have size of units.

To be really precise, there will be two groups of units, one working on the raw inputs, the other working on already processed inputs coming from the last step. Due to the internal structure, each group will have a number of parameters 4 times bigger than the number of units (this 4 is not related to the image, it's fixed).

Flow:

• Takes an input with n steps and 3 features
• Layer 1:
  • For each time step in the inputs:
    • Uses 4 units on the inputs to get a size 4 result
    • Uses 4 recurrent units on the outputs of the previous step
  • Outputs the last (return_sequences=False) or all (return_sequences = True) steps
    • output features = 4
• Layer 2:
  • Same as layer 1
• Layer 3:
  • For each time step in the inputs:
    • Uses 1 unit on the inputs to get a size 1 result
    • Uses 1 unit on the outputs of the previous step
  • Outputs the last (return_sequences=False) or all (return_sequences = True) steps

It uses predefined MNIST dataset:

The problem is fixed, I forgot to pass the value of variable pkeep. I had to make the following changes to make it work.

You have correctly defined your labels as

I had the same issue. The solution is to run the code from within the folder where all the code resides. Don't just copy the mnist_1.0_softmax.py code to your IDE and run it. Download or clone entire repo from the link below

So after searching my self i came to a solution. I applied some tricks from this answer. I reduced my batch size from 32 to 16 which resulted in slow training but i had to make some trade off :). I also changed the structure of D and G by reducing the no. of neurons in hidden layers. And lastly i applied some tensorflow memory allocation tips from this answer above that helped me.

I hope my answer can help beginners like me.

The bias will change as the network is trained. It is just inizialitining at that value. Bias is very important in the fully connected networks, it is a value that will always be there not depeding on the input, and networks perform better with it.

I've figured out that the problem was the checkpoints files, it seems that a model trained on an x64 architecture whit the x64 Tensorflow package is incompatible and cannot be loaded on an x64ARM whit the Tensorflow package for ARM, or at least whit the versions I have of them, so I've retrained the model directly on the Raspberry and all works fine now.