lstm_autoencoder | LSTM Autoencoder that works with variable timesteps | Machine Learning library

First, the ReLU function is not a cure-all activation function. Specifically, it still suffers from the exploding gradient problem, since it is unbounded in the positive domain. Implying, this problem would still exist in deeper LSTM networks. Most LSTM networks become very deep, so they have a decent chance of running into the exploding gradient problem. RNNs also have exploding gradients when using the same weight matrix at each time step. There are methods, such as gradient clipping, that help reduce this problem in RNNs. However, ReLU functions themselves do not solve the exploding gradient problem.

The ReLU function does help reduce the vanishing gradient problem, but it doesn't solve the vanishing gradient completely. Methods, such as batch normalization, can help reduce the vanishing gradient problem even further.

Now, to answer your question about using a ReLU function in place of a tanh function. As far as I know, there shouldn't be much of a difference between the ReLU and tanh activation functions on their own for this particular gate. Neither of them completely solve the vanishing/exploding gradient problems in LSTM networks. For more information about how LSTMs reduce the vanishing and exploding gradient problem, please refer to this post.