LSTM-autoencoder | TensorFlow LSTM-autoencoder implementation | Machine Learning library

There is no official or correct way of designing the architecture of an LSTM based autoencoder... The only specifics the name provides is that the model should be an Autoencoder and that it should use an LSTM layer somewhere.

The implementations you found are each different and unique on their own even though they could be used for the same task.

Let's describe them:

• TF implementation:

  • It assumes the input has only one channel, meaning that each element in the sequence is just a number and that this is already preprocessed.
  • The default behaviour of the LSTM layer in Keras/TF is to output only the last output of the LSTM, you could set it to output all the output steps with the return_sequences parameter.
  • In this case the input data has been shrank to (batch_size, LSTM_units)
  • Consider that the last output of an LSTM is of course a function of the previous outputs (specifically if it is a stateful LSTM)
  • It applies a Dense(1) in the last layer in order to get the same shape as the input.

• PyTorch 1:

  • They apply an embedding to the input before it is fed to the LSTM.
  • This is standard practice and it helps for example to transform each input element to a vector form (see word2vec for example where in a text sequence, each word that isn't a vector is mapped into a vector space). It is only a preprocessing step so that the data has a more meaningful form.
  • This does not defeat the idea of the LSTM autoencoder, because the embedding is applied independently to each element of the input sequence, so it is not encoded when it enters the LSTM layer.

• PyTorch 2:

  • In this case the input shape is not (seq_len, 1) as in the first TF example, so the decoder doesn't need a dense after. The author used a number of units in the LSTM layer equal to the input shape.

In the end you choose the architecture of your model depending on the data you want to train on, specifically: the nature (text, audio, images), the input shape, the amount of data you have and so on...

First, the ReLU function is not a cure-all activation function. Specifically, it still suffers from the exploding gradient problem, since it is unbounded in the positive domain. Implying, this problem would still exist in deeper LSTM networks. Most LSTM networks become very deep, so they have a decent chance of running into the exploding gradient problem. RNNs also have exploding gradients when using the same weight matrix at each time step. There are methods, such as gradient clipping, that help reduce this problem in RNNs. However, ReLU functions themselves do not solve the exploding gradient problem.

The ReLU function does help reduce the vanishing gradient problem, but it doesn't solve the vanishing gradient completely. Methods, such as batch normalization, can help reduce the vanishing gradient problem even further.

Now, to answer your question about using a ReLU function in place of a tanh function. As far as I know, there shouldn't be much of a difference between the ReLU and tanh activation functions on their own for this particular gate. Neither of them completely solve the vanishing/exploding gradient problems in LSTM networks. For more information about how LSTMs reduce the vanishing and exploding gradient problem, please refer to this post.

I believe the activation function you are using here i.e. 'relu' might be killing the gradients. Try using other activation functions such as 'tanh' suitable for your data.

As you said it's an autoencoder. Your autoencoder tries to reconstruct your input. As you see, the output values are very close to the input values, there is not a big error. So the autoencoder is well trained.

Now if you want to detect outliers in your data, you can compute the reconstruction error (Could be Mean square Error between input and output) and set up a threshold.

If reconstruction error is superior than the threshold it's gonna be an outlier, since the autoencoder is not trained on reconstructing outlier data.

This schema reprensents better the idea:

I hope this helps ;)