Shuang | : pencil2 : Shuangpin practice | Frontend Framework library

github: https://github.com/sephiroce/tfsr/tree/exprimental

I'm trying to reproduce recognition accuracies described in the speech transformer paper [1]. The attention penalty is a technique I could not fully understand. This is the description of the attention penalty in the paper.

"In addition, we encouraged the model attending to closer positions by adding bigger penalty on the attention weights of more distant position-pairs."

I understood as it means adding smaller negative values for more away from the diagonal on scaled attention logits (before masking) except for the first multi-head attention in decoders.

This is a code snippet for computing attention weights.

I am trying to modify cells in existing df -- if I find string with no alpha characters (e.g such as "*" ) I set it to "0.0" string and when all cells are processed I try to convert a column numeric type. But setting "0.0" for some reason does not reflect in resulting df