allennlp | An open-source NLP research library , built on PyTorch | Natural Language Processing library

As discussed on GitHub: The problem is that you are constructing a 7-way classifier on top of BERT. Even though the BERT model will be identical, the 7-way classifier on top of it is randomly initialized every time.

BERT itself does not come with a classifier. That has to be fine-tuned for your data.

If you are on the later versions of allennlp-models, you can use this archive_file instead: https://storage.googleapis.com/allennlp-public-models/structured-prediction-srl-bert.2020.12.15.tar.gz.

The latest versions of the model archive files can be found on the demo page in the Model Card tab: https://demo.allennlp.org/semantic-role-labeling

You are right, those two formats don't align.

I'm afraid there is no easy way out. I think you'll have to write a TokenEmbedder that can read and apply the output from bidirectional_language_model.jsonnet.

If you do, we'd love to have it as a contribution to AllenNLP!

The easiest way to resolve this is to patch SrlReader so that it uses PretrainedTransformerTokenizer (from AllenNLP) or AutoTokenizer (from Huggingface) instead of BertTokenizer. SrlReader is an old class, and was written against an old version of the Huggingface tokenizer API, so it's not so easy to upgrade.

If you want to submit a pull request in the AllenNLP project, I'd be happy to help you get it merged into AllenNLP!

The information you are looking for is in 'clusters', where each list corresponds to an entity. Within each entity list, you will find the mentions referring to the same entity. The number are indices that mark the beginning and ending of each coreferential mention. E.g. Paul Allen [0,1] and Allen [24, 24].

This is a very good question, and shows a common misconception about Transformers, stemming from an (unfortunate) formulation in the original Transformers paper. In particular, the authors write the following in Section 3.2.2:

In this work, we employ h = 8 parallel attention layers, or heads. For each of these we use d_k = d_v = d_(model) / h = 64. [...]

Note that the equality of d_k/d_v = d_(model) is not strictly necessary; it is only important that you do match the final hidden representation (d_(model)) after the Feed-Forward portion of each layer. Specifically for mt5-small, the authors actually use an internal dimension of 384 which is simply the product of parameters d_kv * num_heads = 64 * 6.

Now, the problem is that many libraries make a similar assumption of the enforced relation between d_kv and d_(model), because it saves some implementation effort that most people won't use anyways. I suspect (not super familiar with AllenNLP) that they have made similar assumptions here, which is why you cannot load the model.

Also, to clarify this, here is a peek at the modules of a loaded mt5-small:

During training, the trainer will call the metric (using Metric.__call__()) with the results from every batch. The metric is supposed to update its internal state when this happens. The trainer expects to get the current value(s) of the metric when it calls Metric.get_metric(). Metric.reset() has to reset the metric into a state as if it had never been called before. When get_metric() gets called with reset = True, it's expected to reset the metric as well.

From what I can tell, your code does all these things correctly. Your code will not run correctly in a distributed setting, but if you are not training on multiple GPUs, that's not a problem.

What you're doing is similar to the SQuAD metric: https://github.com/allenai/allennlp-models/blob/main/allennlp_models/rc/metrics/squad_em_and_f1.py The SQuAD metric goes out of its way to call the original SQuAD evaluation code, so it's a little more complicated than what you would want, but maybe you can adapt it? The main difference would be that you are calculating F scores across the whole dataset, while SQuAD calculates them per-document, and then averages across documents.

Finally, you can write a simple test for your metric, similar to the SQuAD test: https://github.com/allenai/allennlp-models/blob/main/tests/rc/metrics/squad_em_and_f1_test.py That might help narrow down where the problem is.

Turns out that I wasn't comparing exactly the right things. This thread: https://github.com/allenai/allennlp/discussions/5471 goes into all the detail. Briefly, because pytorch can use additional resources under the hood, my baseline test without multiprocessing wasn't taxing my computer enough when running two instances in parallel; I had to run 4 instances to see the penalty, and in that case, the total processing time was essentially the same for 4 parallel nonmultiprocessing invocations, or one multiprocessing case with 4 subprocesses.

Check out the way the BiDAF model uses ELMo: https://raw.githubusercontent.com/allenai/allennlp-models/main/training_config/rc/bidaf_elmo.jsonnet

You can steal some of the components of that config. You will need the token embedder under the name "elmo", and, I believe, both the token indexers under "tokens" and "elmo".

It should work without having to write any code.