udacity-deep-learning | Udacity Deep Learning class with TensorFlow | Machine Learning library

I spent a whole day to figure out a way to handle it and it turns out the solution is simpler than I thought.

Solution: I was able to solve this issue by catching the ValueError exception and continue with the reading of image files.

Here is the code:

I read it like this:

Join path and folder name which is coming from the for loop one at a time, if the joined path and folder name is a directory. If not, just skip the said path.

The purpose of the code is to build a list of sorted directories.

Biases are tuned alongside weights by learning algorithms such as gradient descent. biases differ from weights is that they are independent of the output from previous layers. Conceptually bias is caused by input from a neuron with a fixed activation of 1, and so is updated by subtracting the just the product of the delta value and learning rate.

In a large model, removing the bias inputs makes very little difference because each node can make a bias node out of the average activation of all of its inputs, which by the law of large numbers will be roughly normal. At the first layer, the ability for this to happens depends on your input distribution. For MNIST for example, the input's average activation is roughly constant. On a small network, of course you need a bias input, but on a large network, removing it makes almost no difference.

Although in a large network it has no difference, it still depends on network architecture. For instance in LSTM:

Most applications of LSTMs simply initialize the LSTMs with small random weights which works well on many problems. But this initialization effectively sets the forget gate to 0.5. This introduces a vanishing gradient with a factor of 0.5 per timestep, which can cause problems whenever the long term dependencies are particularly severe. This problem is addressed by simply initializing the forget gates bias to a large value such as 1 or 2. By doing so, the forget gate will be initialized to a value that is close to 1, enabling gradient flow.

See also:

• The rule of bias in Neural network
• What is bias in Neural network
• An Empirical Exploration of Recurrent Network Architectures

Remember the purpose of the RNN you are training is to predict the next character in a string and you have one LSTM for each character position. In the code above the cited code they have also made a mapping from a character to a number ' ' is 0, 'a' is 1, 'b' is 2 etc. And this is further translated into '1-hot' encodings that is ' ' which is 0 is encoded as [1 0 0 0 ... 0], 'a' which is 1 as [0 1 0 0 ... 0], 'b' as [0 0 1 0 0 ... 0]. In my explanation below I skip this mapping for clarity, so all my characters should really be numbers or actually 1-hot encodings.

Let me start with the simpler case, where batch_size = 1 and num_unrollings =1. Let us also say your training data is "anarchists advocate social relations based upon voluntary association of autonomous individuals mutu"

In this case your first character is the 'a' in anarchists and the expected output (label) is the 'n'. In the code this is represented by the return value of next(). batches = [ [ 'a' ], ['n' ]], where the first element of the list is the input and the last element is the label. This is then in step 0 fed into the RNN. In the next step the input is 'n' and the label is 'a' (the third letter in 'anarchi...', so the next batches = [ ['n'], ['a'] ] and the third step batches is batches = [ ['a'] , ['r']] and so on. Notice how the last element (self._last_batch) in the inner list is the first element in the inner list at the next time-step (batches = [self._last_batch]).

This is if the num_unrollings = 1. If num_unrollings = 5 then instead of only stepping one lstm unit forward every time you step num_unrolling=5 lstm units forward in every time-step. Thus the next function should provide the input for the 5 first RNN that is the 5 characters 'a','n','a','r','c' and also the corresponding labels 'n','a','r','c','h'. Notice the last four input characters are the same as the first 4 labels, so to be more memory efficient this is encoded as a list of the first 6 characters i.e.

batches = [ [ 'a'],['n'],['a'],['r'],['c'], ['h'] ],

with the understanding that the first 5 charaters are the input and the last 5 characters are the labels. and the next time next is called next returns the input and labels for the next 5 lstms

batches = [ [ 'h'], ['i'], ['s'], ['t'], ['s'], [' '] ] ], Notice the 'h' is also in this list as it before only was used as a label and now only is used as an input.

if batch_size > 1 you simoultaneously feed several sequences into the RNN update step. Notice here cursor is not a cursor but a list of cursors --- one for each sequence. consider now batch_size = 2. In the above example where the trainingdata is the 100 characters "anarchists advocate social relations based upon voluntary association of autonomous individuals mutu" the second sequence of text simply starts in the middle "luntary association of autonomous individuals mutu", so the batches in the first step contain the information [ 'a','n','a','r','c', 'h'] and ['l','u','n','t','a','r'], corresponding to the same 6 first characters as before and the first 6 characters after the middle. but it is organized as follows (tranposed) batches = [ [ 'a', 'l'], ['n', 'u'], ['a', 'n'], ['r', 't'], [c', 'a'], ['h', 'r']] the batches in the second time-step is contains the information [ 'h', 'i', 's', 't', 's', ' ' ] and ['r','y', ' ', 'a', 's', 's'] , but again tranposed batches = [[ 'h', 'r'], ['i', 'y'], ['s', ' '], ['t', 'a'], ['s', 's', [' ', 's'] ] and so on.

The above is a technical answer to what num_unrollings means for the generation of the batches. However num_unrollings is also the number of characters you go back in the backpropagation part of the updating of the weights in the RNN. This is so as in each time-step in the RNN learning algorithm you feed in num_unrolling input characters and you only calculate on the corresponding lstm's, while the (hidden) input from the previous part of the sequence is stored in variables that are not trainable. You may try to set the num_urollings to 1 and see whether it is harder to learn long range correlations. (you might need many timesteps).