gumbel | Gumbel-Softmax Variational Autoencoder with Keras | Machine Learning library

I wonder whether should I take log of attn_weights before passing it into gumbel_softmax?

If attn_weights are probabilities (sum to 1; e.g., output of a softmax), then yes. Otherwise, no.

I wonder how to choose tau in gumbel_softmax, any guidelines?

Usually, it requires tuning. The references provided in the docs can help you with that.

From Categorical Reparameterizaion with Gumbel-Softmax:

• Figure 1, caption:

  ... (a) For low temperatures (τ = 0.1, τ = 0.5), the expected value of a Gumbel-Softmax random variable approaches the expected value of a categorical random variable with the same logits. As the temperature increases (τ = 1.0, τ = 10.0), the expected value converges to a uniform distribution over the categories.

• Section 2.2, 2nd paragraph (emphasis mine):

  While Gumbel-Softmax samples are differentiable, they are not identical to samples from the corresponding categorical distribution for non-zero temperature. For learning, there is a tradeoff between small temperatures, where samples are close to one-hot but the variance of the gradients is large, and large temperatures, where samples are smooth but the variance of the gradients is small (Figure 1). In practice, we start at a high temperature and anneal to a small but non-zero temperature.

• Lastly, they remind the reader that tau can be learned:

  If τ is a learned parameter (rather than annealed via a fixed schedule), this scheme can be interpreted as entropy regularization (Szegedy et al., 2015; Pereyra et al., 2016), where the Gumbel-Softmax distribution can adaptively adjust the "confidence" of proposed samples during the training process.

One simple way to keep the name is to use rlist::flatten() before the call to cbind:

have I missed any?

You actually have!

Please check https://github.com/sdv-dev/Copulas

This library has been and is actively being developed at the DAI-Lab, a research group in LIDS, MIT, and is part of The Synthetic Data Vault project, an ecosystem of Synthetic Data Generation libraries for multiple data modalities.

I cannot say that it is the "best one" (I'm actually a maintainer), but the numbers say it is the most popular (number of github stars and monthly downloads). It implements a multitude of univariate and multivariate copulas, including the three major Archimedean Copulas (Clayton, Frank and Gumbel) and Vines, and it has an extensive suite of numerical and unit tests.

So it possibly is what you are looking for!

How about function from Extreme Value Distributions package?

Matlab's evrnd generates random variates from the Gumbel distribution, also known as the Type I extreme value distribution. As explained in that link,

The version used here is suitable for modeling minima; the mirror image of this distribution can be used to model maxima by negating R.

You can use NumPy's implementation of the Gumbel distribution, but it uses the version of the distribution that models maxima, so you'll have to flip the values around the location (i.e. mu) parameter.

Here's a script containing the Python function evrnd. The plot that it generates is below.