pytorch-transformer | PyTorch implementation of the Transformer model | Machine Learning library

I have a sentence like: "I like sitting in my new chair and _____ about life".

And I have a SPECIFIC set of tokens like ["watch", "run", "think", "apple", "light"]

I would like to calculate the probability of each of those tokens to appear as the next word in that incomplete sentence. Hopefully I should get that the probability of "think" is higher that "apple" for instance.

I am working with pytorch-transformers (GPT2LMHeadModel specifically), and a possible solution is to evaluate the score of the full sentence with each of the tokens, but when number of tokens to evaluate is on the order of 100 or 1000 then the computation time starts to be too long.

It must be possible to process the sentence only once and somehow use the hidden states to calculate the probabilities of the set of tokens, but I don't know how to do it.

Any ideas? Thanks in advance

EDIT:

The actual code looks like the one below (estimating the probability for the full sentence every time). For every sentence it takes about 0.1 seconds to run the score() method, which turns into hours if I want to evaluate some thousands of words.

I started working on this about two months ago on Google Colab for a midterm project and everything worked perfectly. Now I am modifying it for a final project and keep getting the error 'RuntimeError: Trying to create tensor with negative dimension -1: [-1, 768]'. It looks like pytorch recently pushed a new version 1.5, so I downgraded to version 1.4 and still got the same error. Same with 1.3, and I know I wasn't using anything lower since that came out last year. I checked it with my midterm code and still got the same error, so I don't know what's going on. Here is the chunk of code related to downloading and using the model.

I wanted to test TextGeneration with CTRL using PyTorch-Transformers, before using it for fine-tuning. But it doesn't prompt anything like it does with GPT-2 and other similar language generation models. I'm very new for this and am stuck and can't figure out what's going on.

This is the procedure I followed in my Colab notebook,

Hi I am working on implementing a multi-classification model (5 classes) with the new SpaCy Model en_pytt_bertbaseuncased_lg. The code for the new pipe is here:

I am currently working with the spacy-pytorch-transformer package to experiment with the respective embeddings.
When reading the introductionary article (essentially the GitHub README), my understanding was that the token-level embeddings are the mean over the embeddings of all corresponding word pieces, i.e. embed(complex) would be the same as 1/2 * embed(comp#) * embed(#lex).

According to the BERT paper, this should simply utilize the last_hidden_state property of the network, but my MCVE below shows that this is not the same for Spacy 2.1.8 and spacy-pytorch-transformers 0.4.0, for at least BERT and RoBERTa (have not verified it for more models):

In relation to the previous post on stackoverflow Model() got multiple values for argument 'nr_class' - SpaCy multi-classification model (BERT integration) in which my problem partialy have beed resolved I wanted to share the issue which comes up after implementing the solution.

if I take out the nr_class argument, I get this error here: