vae | Generate new images with Variational Autoencoder | Machine Learning library

I think the main issue occurs when you try to get the output from the logits_y layer, (AFAIK), you can't do that, and instead, you need to build your encoder model with two outputs. Something like this way

If you're using tf 2.x, then import your keras modules as follows.

This a problem due to the output shape of your decoder... you can simply solve it by changing the final layer of your decoder with:

Well, apparently François Chollet has made a few changes very recently (5 days ago), including changes in how the kl_loss and reconstruction_loss are computed, see here.

Having run the previous version (that you can find at the link above), I significantly reduced the difference between the two members of the equation, even reducing with increasing epoch (from epoch 7, the difference is <.2), as compared to your values.

It seems that VAE are subject to reconstruction loss underestimation, which is an ongoing issue, and for that, I encourage you to dig a bit in the litterature, with e.g. this article (may not be the best one).

Hope that helps! At least it's a step forward.

I've find out the reason of the issue. It turns out that the decoder model derives output value in the range of 0.4 to 0.6 to stabilize the BCE loss. BCE loss can't be 0 even if the prediction is correct to answer. Also the loss value is non-linear to the range of the output. The easiest way to lower the loss is give 0.5 for the output, and my model did. To avoid this error, I standardize my data and added some outlier data to avoid BCE issue. VAE is such complicated network for sure.

I know this may not be intuitive, but when you use a kernel_size with 2-dim (e.g., (3,3)), then your Conv1d has 4-dim weights. Therefore, to solve your issue, you must change from:

It seems like eq. 2 is wrong.

It should have been something like. I did not derive it, just trying to match with the input... so, please verify.

I modified your function below.

My understanding is that to generate new data in RBM I would need to pass in real data. Is there a way to get generated data without real data? Like how VAE and GAN samples latent variable from prior distribution to generate data.

Yes, of course. This is actually the process that is happening in the negative phase of the training. You're sampling from a joint distribution, therefore letting the network "dream" of what it has been trained for. I guess this depends on your implementation, but I've been able to do that by initializing inputs as zeros and running Gibbs sampling for a few iterations. The result, as I interpret it, is that I should see "number-looking things" in the visible nodes, not necessarily numbers from your dataset.

This is an example I like, trained on MNIST, and sampled without any nodes clamped:

To your second question:

If so, in the case of labeled dataset like MNIST, how can I generate data from a specific class? Do I need to train 10 different RBM models for each digit?

What you can do when using labeled data is to use your labels as additional visible nodes. Check "Training Restricted Boltzmann Machines: An Introduction" Figure 2.

Also, for both these cases I'm thinking that using other sampling techniques that gradually lower the sampling temperature (e.g. Simulated Annealing) , will give you better results.

Yes you can definitely use a Pytorch module inside another Pytorch module. The way you are doing this in your example code is a bit unusual though, as external modules (VAE, in your case) are more often initialized in the __init__ function and then saved as attributes of the main module (integrated). Among other things, this avoids having to reload the sub-module every time you call forward.

One other thing that looks a bit funny is your for loop over repeated invocations of model(x). If there is no randomness involved in model's evaluation, then you would only need a single call to model(x), since all 100 calls will give the same value. So assuming there is some randomness, you should consider whether you can get the desired effect by batching together 100 copies of x and using a single call to model with this batched input. This ultimately depends on additional information about why you are calling this function multiple times on the same input, but either way, using a single batched evaluation will be a lot faster than using many unbatched evaluations.