voxceleb | VoxCeleb dataset - a large-scale speaker identification | Speech library

Notice that in your model:

• y_true = labels
• y_pred = kullback-leibler divergence

These two cannot be compared, see this example:

For correct results, when y_true == 1 (same speaker), Kullback-Leibler is y_pred == 0 (no divergence).

So it's totally expected that metrics will not work properly.

Then, either you create a custom metric, or you count only on the loss for evaluations.
This custom metric should need a few adjustments in order to be feasible, as explained below.

This might be a problem

First, notice that you're using clip in the values for the Kullback-Leibler. This may be bad because clips lose the gradients in the clipped regions. And since your activation is a PRelu, you have values lower than zero and bigger than 1. Then there are certainly zero gradient cases here and there, with the risk of having a frozen model.

So, you might not want to clip these values. And to avoid having negative values with the PRelu, you can try to use a 'softplus' activation, which is kind of a soft relu without negative values. You might also "sum" an epsilon to avoid trouble, but there is no problem in leaving values bigger than one: