svgp | WIP - SVG Parser , DOM API provider and Serializer | Animation library

SVGP is a GPflow model for a variational approximation. Using MCMC on the q(u) distribution parameterised by q_mu and q_sqrt doesn't make sense (if you want to do MCMC on q(u) in a sparse approximation, use SGPMC).

You can still put (hyper)priors on the hyperparameters in the SVGP model; gradient-based optimisation will then lead to the maximum a-posteriori (MAP) point estimate (as opposed to pure maximum likelihood).

Yes, the SVGP (as well as VGP) model predictions crucially depend on the q(u) distribution parametrised by model.q_mu and model.q_sqrt. You can transfer all parameters (including those two) using

GPflow's gpflow.optimizers.Scipy() is a wrapper around Scipy's minimize(), and as it calls into non-TensorFlow operations, you cannot wrap it in tf.function. Moreover, the optimizers implemented in Scipy's minimize are second-order methods that assume that your gradients are not stochastic, and aren't compatible with minibatching.

If you want to do full-batch optimization with Scipy: The minimize() method of gpflow.optimizers.Scipy(), by default, does wrap the objective and gradient computation inside tf.function (see its compile argument with default True). It also does the full optimization, so you only have to call the minimize() method once (by default it runs until convergence or failure to continue optimization; you can supply a maximum number of iterations using the options=dict(maxiter=1000) argument).

If you want to use mini-batching: simply use one of the TensorFlow optimizers, such as tf.optimizers.Adam(), and then your code should run fine including the @tf.function decorator on your optimization_step() function (and in that case you do need to call it in a loop as in your example).

Here's some code that's a bit simpler.

Yes, the Coregion kernel implemented in GPflow is what you can use for your problem.

Let's set up some data from the generative model you describe, with different lengths for the timeseries:

If you want to e.g. decompose an additive Kernel, I think the easiest way for vanilla GPR would be to just switch out the Kernel to the part you're interested in, while still keeping the learned hyperparameters.

I'm not totally sure about it, but I think it could also work out for SVGP, since the approximation itself is just a standard GP using the same kernel but conditioned on the Inducing Points.

However, I'm not sure if the decomposition of the Variational approximation can be assumed to be close to the decomposition of the true posterior.

You're right, and that's by design. A constrained variable in GPflow is represented by a Parameter. The Parameter wraps the unconstrained_variable. When you access .trainable_variables on your model, this will include the unconstrained_variable of the Parameter, and so when you pass these variables to the optimizer, the optimizer will train those rather than the Parameter itself.

But your model doesn't see the unconstrained_value, it sees the Parameter interface which is a tf.Tensor-like interface related to the wrapped unconstrained_variable via an optional transformation. This transformation maps the unconstrained value to a constrained value. As such, your model will only see the constrained value. It's not a problem that your constrained value must be positive, the transform will map negative values of the unconstrained values to positive values for the constrained value.

You can see the unconstrained and constrained values of the first Parameter for your kernel, as well as the transform that relates them, with

There are several elements to your question, I'll try and address them separately:

SVGP vs SGPMC objective: In SVGP, we parametrize a closed-form posterior distribution q(u) by defining it as a normal (Gaussian) distribution with mean q_mu and covariance q_sqrt @ q_sqrt.T. In SGPMC, the distribution q(u) is implicitly represented by samples - V holds a single sample at a time. In SVGP, the ELBO has a KL term that pulls q_mu and q_sqrt towards q(u) = p(u) = N(0, Kuu) (with whitening, q_mu and q_sqrt parametrize q(v), the KL term is driving them towards q(v) = p(v) = N(0, I), and u = chol(Kuu) v). In SGPMC, the same effect comes from the prior on V in the MCMC sampling. This is still reflected when doing MAP optimisation with a stochastic optimizer, but different from the KL term. You can set q_sqrt to zero and non-trainable for the SVGP model, but they still have slightly different objectives. Stochastic optimization in the SGPMC model might give you a better data fit, but this is not a variational optimization so you might be overfitting to your training data.

training_loss: For all GPflow models, model.training_loss includes the log_prior_density. (Just by default the SVGP model parameters do not have any priors set.) The SGPMC training_loss() corresponds to the negative of eq. (7) in the SGPMC paper [1].

Inducing points: By default the inducing points Z do not have a prior, so it would just be maximum likelihood. Note that [1] suggests to keep Z fixed in the SGPMC model (and base it on the optimsed locations in a previously-fit SVGP model).

What happens in conditional() with q_sqrt=None: conditional() computes the posterior distribution of f(Xnew) given the distribution of u; this handles both the case used in (S)VGP, where we have a variational distribution q(u) = N(q_mu, q_sqrt q_sqrt^T), and the noise-free case where "u is known" which is used in (S)GPMC. q_sqrt=None is equivalent to saying "the variance is zero", like a delta spike on the mean, but saving computation.

The reason we set autograph to False in most of the tf.function wrapped objectives is because GPflow makes use a multi-dispatch Dispatcher which internally uses generators. TensorFlow, however, can not deal with generator objects in autograph mode (see Capabilities and Limitations of AutoGraph), which leads to these warning: