CVAE | Convolutional Variational Autoencoder | Machine Learning library

The input_shape should not include the batch dimension. Use input_shape=(70,).

I've been having the same problem for a long time, but managed to figure it out. The problem is that TF only accepts loss functions which accept (input, output) parameters which are then compared. However, you are computing your (kl_)loss using also mu and sigma, which are basically dense layers. Until tensorflow v2.1 it magically knew what these parameters were and knew how to include/manipulate them, but from then on, you have to be more careful. After reading this tutorial (EDIT: also scroll to the bottom of the page for the complete VAE example) I propose these changes to your code:

1. When compiling your model, only define perceptual_loss as a loss:

CVAE.compile(optimizer = "adam", loss = perceptual_loss, metrics = [perceptual_loss])

2. Change sampling function into a class, and under call method, add your kl_loss, something like

One way of doing it would be to have a CVAE that takes as input (and output) values of all the spectral features for each of the spatial coordinates (the stacks circled in red in the picture). So, in the case of your image, you would have 2500*2500 = 6250000 input data samples, which are all vectors of length 15. And then the dimension of the middle layer would be a vector of length 5. And, instead of 2D convolutions that are normally used along the spatial domain of images, in this case it would make sense to use 1D convolution over the spectral domain (since the values of neighbouring wavelengths are also correlated). But I think using only fully-connected layers would also make sense.

As a disclaimer, I haven’t seen CVAEs used in this way before, but like this, you would also get many data samples, which is needed in order for the learning generalise well.

Another option would be indeed what you suggested -- to just generate the samples (patches) using a moving window (maybe with a stride that is the half size of the patch). Even though you wouldn't necessarily get enough data samples for the CVAE to generalise really well on all HSI images, I guess it doesn't matter (if it overfits), since you want to use it on that same image.

The expressions in the code you posted assume X is an uncorrelated multi-variate Gaussian random variable. This is apparent by the lack of cross terms in the determinant of the covariance matrix. Therefore the mean vector and covariance matrix take the forms

Using this we can quickly derive the following equivalent representations for the components of the original expression

Substituting these back into the original expression gives

Considering that the conditionals are the same for both models

Do this: