neural-networks | Using Ruby and Elixir | Awesome List library

The HTML on the question's page shows that the GIF images for figure 6(a) and 6(b) are not responsive.

Here is the HTML for figure 6(a):

Long story short, your dataset is likely too complex for such a small and simple network.

When I wrote the OCR example, and I was kind of showing off a little by "compressing" all 94 chars into a single output neuron. It's not typically done this way and certainly not with complex datasets.

Usually, you would want to dedicate an output neuron for each "class" that the network needs to identify.

Put simply, its harder for the network to learn to properly increment or decrement the output value by 0.01 on a single neuron (as is the case of my OCR ANN) than to learn to associate a dedicated output neuron / pattern with a specific class.

You can find a better example of a more typical classifier implementation in the MNIST subfolder in my repo for the OCR "family" of neural networks: https://github.com/geekgirljoy/OCR_Neural_Network

My suggestion is to redesign your ANN.

Based on your code your network looks like this:

L0: IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

L1: HHHHHHHHHHHHHHHH

L2: O


Whereas it would probably operate (classify) your data better if you redesigned it like this:

First, determine the number of distinct classes types, in the example you gave I saw 0.07 listed so I will assume there are seven different classes of order types.

So, the ANN should look like this:

L0: IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

L1: A sufficient number of "hidden" neurons

L2: OOOOOOO

Where O1 represents class 1, O2 class 2 etc...

Which means that your training data would change to something like this:


60000 32 7
0 0 0 0 0 1 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 1 0 0 0 1 1 0 0 1 0
1 0 0 0 0 0 0
0 0 0 0 0 1 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 1 0 0 0 1 1 0 1 0 0
1 0 0 0 0 0 0
0 0 0 0 0 1 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 1 0 0 0 1 1 0 1 1 0
1 0 0 0 0 0 0
0 0 0 0 0 1 1 0 0 1 1 0 1 1 0 0 1 1 1 0 1 1 1 0 0 0 1 1 1 0 0 0
1 0 0 0 0 0 0
0 0 0 1 1 1 0 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 0 1 0 0 1 0 1 0
0 0 0 0 0 0 1
0 0 0 1 1 1 0 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 0 1 0 0 1 1 0 0
0 0 0 0 0 0 1
0 0 0 1 1 1 0 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 0 1 0 0 1 1 1 0
0 0 0 0 0 0 1
0 0 0 1 1 1 0 1 1 1 1 0 0 1 0 0 0 1 0 0 1 1 1 0 0 1 0 1 0 0 0 0
0 0 0 0 0 0 1

Class Output Examples:

Class 1: 1 0 0 0 0 0 0
Class 2: 0 1 0 0 0 0 0
Class 3: 0 0 1 0 0 0 0
Class 4: 0 0 0 1 0 0 0
Class 5: 0 0 0 0 1 0 0
Class 6: 0 0 0 0 0 1 0
Class 7: 0 0 0 0 0 0 1


Also, depending on your methodology, you MAY get better results using a harder negative value like -1 instead of 0, like this:

60000 32 7
-1 -1 -1 -1 -1 1 1 -1 -1 1 1 -1 1 1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 1 1 -1 -1 1 -1
1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 1 1 -1 -1 1 1 -1 1 1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 1 1 -1 1 -1 -1
1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 1 1 -1 -1 1 1 -1 1 1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 1 1 -1 1 1 -1
1 -1 -1 -1 -1 -1 -1
-1 -1 -1 -1 -1 1 1 -1 -1 1 1 -1 1 1 -1 -1 1 1 1 -1 1 1 1 -1 -1 -1 1 1 1 -1 -1 -1
1 -1 -1 -1 -1 -1 -1
-1 -1 -1 1 1 1 -1 1 1 1 1 -1 -1 1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 1 -1 -1 1 -1 1 -1
-1 -1 -1 -1 -1 -1 1
-1 -1 -1 1 1 1 -1 1 1 1 1 -1 -1 1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 1 -1 -1 1 1 -1 -1
-1 -1 -1 -1 -1 -1 1
-1 -1 -1 1 1 1 -1 1 1 1 1 -1 -1 1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 1 -1 -1 1 1 1 -1
-1 -1 -1 -1 -1 -1 1
-1 -1 -1 1 1 1 -1 1 1 1 1 -1 -1 1 -1 -1 -1 1 -1 -1 1 1 1 -1 -1 1 -1 1 -1 -1 -1 -1
-1 -1 -1 -1 -1 -1 1


This is because you are using a "symmetric" hidden/output function like FANN_SIGMOID_SYMMETRIC which is a sigmoid and so the relationship between -1 to 0 and from 0 to 1 isn't linear so you should get better/harder distinctions between classifications and potentially faster training / fewer training epochs by more strongly contrasting the inputs/outputs like this.

Anyway, once you have trained the network and run your tests, you simply take the max() output neuron as your answer.

Example:

// ANN calc inputs and store outputs in the result array
$result = fann_run($ann, $input);

// Lets say the ANN responds like this:
// [-0.9,0.1,-0.2,0.4,0.1,0.5,0.6,0.99,-0.6,0.4]

// Let's also say there are 10 outputs representing that many classes
// 0 - 9
// [0,1,2,3,4,5,6,7,8,9]
//
// Find which output contains the highest value (the prediction/classification)
$highest = max($result); // $highest now contains the value 0.99

// So to convert the highest value to a class we find the key/position in the $result array
$class = array_search($highest, $result);

var_dump($class);
// int(7)

Why? Because the 7th key (7th / 8th (depending on how you look at it) is the high value):

array(0=>0.9,
1=>0.1,
2=>-0.2,
3=>0.4,
4=>0.1,
5=>0.5,
6=>0.6,
7=>0.99,
8=>-0.6,
0=>0.4
);

In the case of multiple class types being possible at the same time, you "softmax" instead.

Hope this helps! :-)

In TensorFlow, tf.keras.layers.Conv1D takes in a tensor of shape (batch_shape + (steps, input_dim)). Which means that what is commonly known as channels appears on the last axis. For instance in 2D convolution you would have (batch, height, width, channels). This is different from PyTorch where the channel dimension is right after the batch axis: torch.nn.Conv1d takes in shapes of (batch, channel, length). So you will need to permute two axes.

For torch.nn.Conv1d:

• in_channels is the number of channels in the input tensor
• out_channels is the number of filters, i.e. the number of channels the output will have
• stride the step size of the convolution
• padding the zero-padding added to both sides

In PyTorch there is no option for padding='same', you will need to choose padding correctly. Here stride=1, so padding must equal to kernel_size//2 (i.e. padding=2) in order to maintain the length of the tensor.

In your example, since x has a shape of (256, 237, 1, 21), in TensorFlow's terminology it will be considered as an input with:

• a batch shape of (256, 237),
• steps=1, so the length of your 1D input is 1,
• 21 input channels.

Whereas in PyTorch, x of shape (256, 237, 1, 21) would be:

• batch shape of (256, 237),
• 1 input channel
• a length of 21.

Have kept the input in both examples below (TensorFlow vs. PyTorch) as x.shape=(256, 237, 21) assuming 256 is the batch size, 237 is the length of the input sequence, and 21 is the number of channels (i.e. the input dimension, what I see as the dimension on each timestep).

In TensorFlow:

Solved it! I followed this tutorial

I added in readthedocs.yml:

Since the gradients are calculated by averaging the gradients across all training examples, we first "accumulate" the gradients while looping across all the training examples. We do this by summing the gradient across all training examples. So the line you highlighted with the plus is not the gradient update step. (Notice that alpha is not there as well.) It might be somewhere else. It is most likely outside of the loop from 1 to m.

Also, I am not sure when you will learn about this (I'm sure it's somewhere in the course), but you could also vectorize the code :)

I have worked with the Keras LSTM layer in the past, and this seems like a very interesting application of it. I would like to help, but there are many things that go into formatting data for the LSTM layer and before getting it to work properly I would like to clarify the goal of this application.

The positional play data, is that where players are located on the field?

The play outcome data, is this the results of the play i.e. yards gained/lost, passing/running play, etc.?

What are the values you hope to get out of this? (Categorical or numerical)

EDIT/Answer:

Use the .append() method on a list to add to it.

Consider the filter (or kernel) in image below having 9 pixels and the image having 49 pixels.

In a fully connected layer, we'll have 9*49 = 441 weights.

While in a CNN this same filter keeps on moving (convolving) over the entire image. All pixel values in image will be multiplied with those same 9 values of filter (hence we say weights are reused). So, we need just 9 weights per filter instead of 441 in FC layer.

The job of a filter is to identify features (such as texture, lines etc), which could be anywhere in an image. So, we want to reuse this same filter over the entire image.

No, you still have one LSTM layer with four LSTM Neurons.

BTW: If you're looking for a fast way to visualize an ANN: Netron

It's probably an issue with specifying input data to Keras' fit() function. I would recommend using a tf.data.Dataset as input to fit() like so: