BERT | modification of official bert for downstream task | Natural Language Processing library

You can just use the tokenizer decode function:

You have done the hard part, the results are already in the results variable.

But the results need parsing by the wiki.page() nethod, which only takes one argument.

The solution? Use a loop to parse all results one by one.

The easiest way will be using for loops, but the list comprehension method is the best.

Replace the last two lines with the following:

The error states that you do not have a variable called sentences in the scope. I believe the tutorial presumes you already have a list of sentences and are tokenizing it.

Have a look at the documentation The first argument can be either a string or list of string or list of list of strings.

You can make the entire class inherit torch.nn.Module like so:

As the error message tells you, you need to use AutoModelForMaskedLM:

This depends a lot of your task. Your task seems to be masked language modelling, that, is to predict one or more masked words:

today I ate ___ .

(pizza) or (pasta) could be equally correct, so you cannot use a metric such as accuray. But (water) should be less "correct" than the other two. So what you normally do is to check how "surprised" the language model is, on an evaluation data set. This metric is called perplexity. Therefore, before and after you finetune a model on you specific dataset, you would calculate the perplexity and you would expect it to be lower after finetuning. The model should be more used to your specific vocabulary etc. And that is how you test your model.

As you can see, they calculate the perplexity in the tutorial you mentioned:

A vector size of 1000 dimensions is very uncommon, and would require massive amounts of data to train. For example, the famous GoogleNews vectors were for 3 million words, trained on something like 100 billion corpus words - and still only 300 dimensions. Your STEMMED_TOKENS may not be enough data to justify 100-dimensional vectors, much less 300 or 1000.

A choice of min_count=1 is a bad idea. This algorithm can't learn anything valuable from words that only appear a few times. Typically people get better results by discarding rare words entirely, as the default min_count=5 will do. (If you have a lot of data, you're likely to increase this value to discard even more words.)

Are you examining the model's size or word-to-word results at all to ensure it's doing what you expect? Despite your colum being named STEMMED_TOKENS, I don't see any actual splitting-into-tokens, and the Word2Vec class expects each text to be a list-of-strings, not a string.

Finally, without seeing all your other choices for feeding word-vector-enriched data to your other classification steps, it is possible (likely even) that there are other errors there.

Given that a binary-classification model can always get at least 50% accuracy by simply classifying every example with whichever class is more common, any accuracy result less than 50% should immediately cause suspicions of major problems in your process like:

• misalignment of examples & labels
• insufficient/unrepresentative training data
• some steps not running at all due to data-prep or invocation errors

You can easily do this by using python functions.

I would suggest doing the following:

1. First pre-train BERT on the MLM objective. HuggingFace provides a script especially for training BERT on the MLM objective on your own data. You can find it here. As you can see in the run_mlm.py script, they use AutoModelForMaskedLM, and you can specify any architecture you want.

2. Second, if want to train on the next sentence prediction task, you can define a BertForPretraining model (which has both the MLM and NSP heads on top), then load in the weights from the model you trained in step 1, and then further pre-train it on a next sentence prediction task.

UPDATE: apparently the next sentence prediction task did help improve performance of BERT on some GLUE tasks. See this talk by the author of BERT.