attentions | PyTorch implementation of some attentions for Deep Learning | Machine Learning library

When paper introduce they method they said:

The attention modules aim to exploit the relationship between disease labels and (1) diagnosis-specific feature channels, (2) diagnosis-specific locations on images (i.e. the regions of thoracic abnormalities), and (3) diagnosis-specific scales of the feature maps.

(1), (2), (3) corresponding to channel-wise attention, element-wise attention, scale-wise attention

We can tell that element-wise attention is for deal with disease location & weight info, i.e: at each location on image, how likely there is a disease, as it been mention again when paper introduce the element-wise attention：

The element-wise attention learning aims to enhance the sensitivity of feature representations to thoracic abnormal regions, while suppressing the activations when there is no abnormality.

OK, we could easily get location & weight info for one disease, but we have multiple disease:

Since there are multiple thoracic diseases, we choose to estimate an element-wise attention map for each category in this work.

We could store the multiple disease location & weight info by using a tensor A with shape (height, width, number of disease):

The all-category attention map is denoted by A ∈ RH×W×C, where each element aijc is expected to represent the relative importance at location (i, j) for identifying the c-th category of thoracic abnormalities.

And we have linear classifiers for produce a tensor S with same shape as A, this can be interpret as:

At each location on feature maps X(CA), how confident those linear classifiers think there is certain disease at that location

Now we element-wise multiply S and A to get M, i.e we are:

prevent the attention maps from paying unnecessary attention to those location with non-existent labels

So after all those, we get tensor M which tells us:

location & weight info about certain disease that linear classifiers are confident about it

Then if we do global average pooling over M, we get prediction of weight for each disease, add another softmax (or sigmoid) we could get prediction of probability for each disease

Now since we have label and prediction, so, naturally we could minimizing loss function to optimize the model.

Following code is tested on colab and will show you how to implement channel-wise attention and element-wise attention, and build and training a simple model base on your code with DenseNet121 and without scale-wise attention:

You should use BertModel and not BertModelForSequenceClassification, as BertModelForSequenceClassification adds a linear layer for classification on top of BERT model and uses CrossEntropyLoss, which is meant for multiclass classification.

Hence, first use BertModel instead of BertModelForSequenceClassification:

For Code Formatting: First of all, install the prettier extension if haven't already.

Link for esbenp.prettier-vscode

• Now go to File>Preferences>Settings>

• Search format

• Set Default Formatter to esbenp.prettier-vscode

• Enable Format On Save option

After your settings should look like this:

About Overallaping Error check: this SO post

The reason is that you are using AutoModelWithLMHead which is a wrapper for the actual model. It calls the BERT model (i.e., an instance of BERTModel) and then it uses the embedding matrix as a weight matrix for the word prediction. In between the underlying model indeed returns attentions, but the wrapper does not care and only returns the logits.

You can either get the BERT model directly by calling AutoModel. Note that this model does not return the logits, but the hidden states.

In case anybody else needs help with this, it was quite a complex fix but here is what I did:

Changed from using numpy arrays to tf datasets

I don't think this is entirely necessary, so if you're using numpy arrays still then ignore this paragraph and alter the reshape functions below accordingly (from tf.reshape to np reshape methods)

From:

This worked for me

The short answer: Yes, you are correct. Indeed, they use the CLS token (and only that) for BertForSequenceClassification.

Looking at the implementation of the BertPooler reveals that it is using the first hidden state, which corresponds to the [CLS] token. I briefly checked one other model (RoBERTa) to see whether this is consistent across models. Here, too, classification only takes place based on the [CLS] token, albeit less obvious (check lines 539-542 here).

If you check the source code, specifically BertEncoder, you can see that the returned states are initialized as an empty tuple and then simply appended per iteration of each layer.

The final layer is appended as the last element after this loop, see here, so we can safely assume that hidden_states[12] is the final vectors.

You are using max function on a dictionary label_probs, which returns the alphabetically greatest key in dictionary.

To achieve the desired result, you have to,

Replace: