neural-networks-and-deep-learning | repository contains code samples | Machine Learning library

What are all the differences between numpy.random.rand and numpy.random.randn?

From the docs, I know that the only difference among them are from the probabilistic distribution each number is drawn from, but the overall structure (dimension) and data type used (float) are the same. I have a hard time debugging a neural network because of believing this.

Specifically, I am trying to re-implement the Neural Network provided in the Neural Network and Deep Learning book by Michael Nielson. The original code can be found here. My implementation was the same as the original one, except that I defined and initialized weights and biases with numpy.random.rand in init function, rather than numpy.random.randn as in the original.

However, my code that use random.rand to initialize weights and biases doesn't work because the network won't learn and the weights and biases are will not change.

What difference(s) among two random functions cause this weirdness?

I tried porting the NN code presented here to Julia, hoping for a speed increase in training the network. On my desktop, this proved to be the case.

However, on my MacBook, Python + numpy beats Julia by miles.
Training with the same parameters, Python is more than twice as fast as Julia (4.4s vs 10.6s for one epoch). Considering that Julia is faster than Python (by ~2s) on my desktop, it seems like there's some resource that Python/numpy is utilizing on the mac that Julia isn't. Even parallelizing the code only gets me down to ~6.6s (although this might be due to me not being that experienced in writing parallel code). I thought the problem might be that Julia's BLAS was slower than the vecLib library used natively in mac, but experimenting with different builds didn't seem to get me much closer. I tried building both with USE_SYSTEM_BLAS = 1, and building with MKL, of which MKL gave the faster result (the times posted above).

I'll post my version info for the laptop as well as my Julia implementation below for reference. I don't have access to the desktop at this time, but I was running the same version of Julia on Windows, using openBLAS, comparing with a clean installation of Python 2.7 also using openBLAS.

Is there something I'm missing here?

EDIT: I know that my Julia code leaves a lot to be desired in terms of optimization, I really appreciate any tips to make it faster. However, this is not a case of Julia being slower on my laptop but rather Python being much faster. On my desktop, Python runs one epoch in ~13 seconds, on the laptop it only takes ~4.4s. What I'm interested in the most is where this difference comes from. I realize the question may be somewhat poorly formulated.

Versions on laptop:

I was studying the AdaDelta optimization algorithm so I tried to implement it in Python, but there is something wrong with my code, since I get the following error:

AttributeError: 'numpy.ndarray' object has no attribute 'sqrt'

I did not find something about what is causing that error. According to the message, it's because of this line of code:

rms_grad = np.sqrt(self.e_grad + epsilon)

This line is similar to this equation:

RMS[g]t=√E[g^2]t+ϵ

I got the core equations of the algorithm in this article: http://ruder.io/optimizing-gradient-descent/index.html#adadelta

Just one more detail: I'm initializing the E[g^2]t matrix like this:

self.e_grad = (1 - mu)*np.square(nabla)

Where nabla is the gradient. Similar to this equation:

E[g2]t = γE[g2]t−1 + (1−γ)g2t (the first term is equal to zero in the first iteration, just like the line of code above)

So I want to know if I'm initializing the E matrix the wrong way or I'm doing the square root inappropriately. I tried to use the pow() function but it doesn't work. If anyone could help me with this I would be very grateful, I'm trying this for weeks.

Additional details requested by andersource:

Here is the entire source code on github: https://github.com/pedrovbeltran/neural-networks-and-deep-learning/blob/experimental/modified-networks/network2_with_adadelta.py .

I have a list of size 50000. say a. Each element is a tuple, say b=a[0]. Each tuple consists of 2 lists, say c=b[0], d=b[1]. 1st list i.e. c is of length 784 and the second i.e. d is of length 10. From this list, I need to extract the following:
Group first 10 elements of list a. From these 10 tuples, extract their first element (c) and put them in a matrix of size 784x10. Also extract the second elements of the tuples and put them in another matrix of size 10x10. Repeat this for every batch of 10 elements in list a.
Is this possible to do in a single line using list comprehension? Or do I have to write multiple for loops? Which method is efficient and best? Note: It is okay if I get as a list or numpy.ndarray matrix.

Additional Information: I'm following this tutorial on neural networks which aims to design a neural network to recognize handwritten digits. MNIST database is used to train the network. The training data is in the above described format. I need to create a matrix of input_images and expected_output for every mini_batch.

Here is the code I've tried. I'm getting a list of size 50000. Its not getting split into mini_batches

I am trying to learn Keras. I see machine learning code for recognizing handwritten digits here (also given here). It seems to have feedforward, SGD and backpropagation methods written from a scratch. I just want to know if it is possible to write this program using Keras? A starting step in that direction will be appreciated.

I am using the practising code of mnist data for deep learning in Python 3.4

The original code is

I'm following along mnielsen's online book. I'm trying to implement momentum weight update as defined here to his code here. The overall idea is that for momentum weight update, you don't directly change weight vector with negative gradient. You have a parameter velocity which you set to zero to begin with and then you set hyperparameter mu to typically 0.9 .

I have mnist training list in the following form:

I followed a tutorial for creating a simple neural network using signoidal function "Using neural nets to recognize handwritten digits", the tutorial is very simple with theory and code examples.

The problem is that it does not give any examples of digits recognition using network.py.

For example, I have the following number and I want to recognize it as 0 from the image below What should I do next for number recognition?

To make recognition of number requires the use of other technologies such as theano or tensorflow? Good day!