autoencoders | Implementation of several different types of autoencoders | SDK library

AutoEncoders is a particular network which tries to solve the identity problem expressed x' = g(h(x)), where h is the encoder block and g the decoder block.

The latent space z is the minimal expression for a given input x, and it resides in the middle of the network. It's valid to clarify that in such space resides different shapes and each one corresponds to certain instance given during the training phase. Using the CNN that you reffered to support this, it's like a feature map but instead a bunch of feature maps accross the network there's only one, and again, it holds different representations bassed on what it observed during the training.

So, the question is how does it happens to compress and decompress? Well, the data used for training has a domain and every instance has similarities (all the cats have the same abstract qualities, the same for mountains, all have something in common), therefore the network learns how to fit what describes the data into smaller pieces combined, and from smaller pieces (with ranges from 0-1) how to build bigger pieces.

Taking the same samples from cats, all of them have two ears, have a fur, have two eyes and so on, I didn't mentioned the deatils, but you can think on the shape of those ears, how the fur is and probably how big those eyes are, the colors and the bightness. Think on the listing that I put as the latent space z and the details as the x' output.

For more details see this exhaustive explanation with the different AE variants: https://wikidocs.net/3413.

Hope that this helps you.

EDIT 1:

How and which layer extract the features of images?

Its design:

AutoEncoder is a network with a design that makes it possible to compress and decompress the training data, is not an arbitrary network at all.

First, it has a shape of a sand-clock, meaning this that the next layer has fewer neurons that the previous in the encoder block and right after the "latent space layer(s)" it starts doing the opposite by increasing the neurons size in the decoder block until it reaches the size of the input layer (the reconstruction, therefore the output).

Next, each layer is a Dense layer, saying that all the neurons of each layer is fully plugged to the next, so, all the features are carried from layer to layer. The activation function of each neuron (ideally) is the tanh saying that all the possible outputs are [-1,1] being the case; finally, the loss function tends to be the Root Mean Squared Error which tries to tell how far the reconstruction is from the original.

A bonus to this is to normalize the input tensors setting the mean of each feature to zero, this helps a lot the network to learn and I'll explain next.

Words are cheap, show me the backpropagation

Remember that values in the hidden layers are [-1,1]? well, this range and the support of the weights and a bias (Wx + b) makes it possible to have on each layer a continuous combination of fewer features (values from -1 to 1, considering ALL the possible rational numbers within).

With backpropagation (supported in the loss function), the idea is to find a sweet spot of weights to turn the domain training set (say black and white MNIST digits, RGB Cats images, and so on) into a low dimension continuous set (really smalls numbers ranging between [-1,1]) in the encoding layers, then, in the decoding layers it tries to use the same weights (remember is a sand-clock shape network) to emit the higher representation of the previous [-1,1] combination.

An analogy

To put this into a kind of game, two persons are back to back, one looking trough a window and the other has a whiteboard upfront. The first looks outside and see a flower with all the details, and says sunflower (the latent space), the second person hears that and draws a sunflower with all the colors and details that that person learned in the past.

A real world sample please

Continuing with the sunflower analogy, imagine the same case, but your input image (tensor) has noise (you know, glitchy). The AutoEncoder was trained with high quality images, so it's capable of compress the sunflower concept, then reconstruct it. What happened to the glitch? The network encoded the sunflower colors, shape, and background (let's say is a blue sky), the decoder reconstruct it, the glitch was left behind as residual. And this, is a Denoise AutoEncoder, one of many application of the network.

The time-distributed dense layer as name suggested just an ordinary dense layer that apply to every temporal slice of an input, you can think it as special form of RNN cell, i.e without recurrent hidden state.

So you can using any layer that is time-distributed as your output layer for an Autoencoder that deal with time-distributed inputs, e.g RNN layer with LSTM Cell, GRU Cell, Simple RNN Cell or time-distributed dense layer; As in research paper that propose the LSTM-Autoencoder, it basic model for reconstruct sequence of vectors (image patches or features) only using one LSTM layer in both encoder and decoder, model structure is:

Following is an example to using time-distributed dense layer in decoder:

There is no official or correct way of designing the architecture of an LSTM based autoencoder... The only specifics the name provides is that the model should be an Autoencoder and that it should use an LSTM layer somewhere.

The implementations you found are each different and unique on their own even though they could be used for the same task.

Let's describe them:

• TF implementation:

  • It assumes the input has only one channel, meaning that each element in the sequence is just a number and that this is already preprocessed.
  • The default behaviour of the LSTM layer in Keras/TF is to output only the last output of the LSTM, you could set it to output all the output steps with the return_sequences parameter.
  • In this case the input data has been shrank to (batch_size, LSTM_units)
  • Consider that the last output of an LSTM is of course a function of the previous outputs (specifically if it is a stateful LSTM)
  • It applies a Dense(1) in the last layer in order to get the same shape as the input.

• PyTorch 1:

  • They apply an embedding to the input before it is fed to the LSTM.
  • This is standard practice and it helps for example to transform each input element to a vector form (see word2vec for example where in a text sequence, each word that isn't a vector is mapped into a vector space). It is only a preprocessing step so that the data has a more meaningful form.
  • This does not defeat the idea of the LSTM autoencoder, because the embedding is applied independently to each element of the input sequence, so it is not encoded when it enters the LSTM layer.

• PyTorch 2:

  • In this case the input shape is not (seq_len, 1) as in the first TF example, so the decoder doesn't need a dense after. The author used a number of units in the LSTM layer equal to the input shape.

In the end you choose the architecture of your model depending on the data you want to train on, specifically: the nature (text, audio, images), the input shape, the amount of data you have and so on...

Hidden variables z are used in VAEs as the extracted features for dimensionality reduction. Here is an example dimensionality reduction from four features in the original space ([x1,x2,x3,x4]) to two features in the reduced space ([z1,z2]) (source):

Once you have trained the model, you can pass a sample to the encoder it extracts the features. You may find a Keras implementation example on mnist data here (see the plot_label_clusters function):

Since the Anomaly Detector is usually trained unsupervised, it can be hard to incorporate labels directly into that process without loosing outlier detection properties. A simple alternative is to take the instances that were marked as anomalies, and put them into a classifier that classifies into "real anomaly" vs "not real anomaly". This classifier would be trained on prior anomalies that have been labeled. It can be either binary classification, or one-class wrt to known "not real" samples. A simple starting point would be k-Nearest-Neighbours or a domain-specific distance function. The classifier can use the latent feature vector as input, or do its own feature extraction.

This kind of system is described in Anomaly Detection with False Positive Suppression (relayr.io). The same basic idea is used in this paper to minimize False Negative Rate: SNIPER: Few-shot Learning for Anomaly Detection to Minimize False-negative Rate with Ensured True-positive Rate

From your explanation, it seems that you are approaching a Novelty detection problem. The novelty detection problems are usually a semi-supervised problem (exceptions or approaches can vary).

Now the problem with huge matrix size can be solved if you use batch processing. This can help you- https://scikit-learn.org/0.15/modules/scaling_strategies.html

Finally yes, if you could use deep learning your problem can be solved in a much better way using both unsupervised learning or semi-supervised learning(I recommend this).

On the first loop, i==0 because range(10) is [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. You can't use 0 as an index for the subplots, which causes that error. You should instead use i+1 in your plt.subplot() to get the correct axis.

The conv2d op raised an error message:

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

Looking above, we see

Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3891 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
failed to allocate 3.80G (4080218880 bytes) from device:
CUDA_ERROR_OUT_OF_MEMORY: out of memory
failed to allocate 3.42G (3672196864 bytes) from device:
CUDA_ERROR_OUT_OF_MEMORY: out of memory

So this graph would need more memory than there is available on your GeForce GTX 1650 Ti (3891 MB). Try using a smaller input image size and/or a smaller batch size.

APaul31,

Specifically in your code I suggest adding call() function to the VAE class: