language-modeling | machine learning model that is trained to predict | Machine Learning library

Ok, for those who want to use the pretrained tokenizer in question answering tasks it can be done for one example as follows:

The variable last_hidden_state[mask_index] is the logits for the prediction of the masked token. So to get token probabilities you can use a softmax over this, i.e.

I think you are mixing two distinct actions.

1. The first guide you posted explains how to create a model from scratch
2. The run_mlm.py script is for fine-tuning (see line 17 of the script) an already existing model

So, if you just want to create a model from scratch, step 1 should be enough. If you want to fine-tune the model you just created, you have to run step 2. Note that training a RoBERTa model from scratch already implies a MLM phase, so this step is useful only in case that you will have a different dataset in the future and you want to improve your model by further fine-tuning it.

However, you are not loading the model you just created, you are loading the roberta-base model from the Huggingface repository: --model_name_or_path roberta-base \

Coming to the warning, it tells you that you loaded a model (roberta-base, as cleared out) that was pre-trained for Masked Language Modeling (MaskedLM) task. This means you loaded a checkpoint of a model So, quoting:

If your task is similar to the task the model of the checkpoint was trained on, you can already use RobertaForMaskedLM for predictions without further training.

This means that, if you going to perform a MaskedLM task, the model is good to go. If you want to use for another task (for example, question answering), you should probably fine-tune it because the model as is would not provide satisfactory results.

Concluding, if you want to create a model from scratch to perform MLM, follow step 1. This will create a model that can perform MLM.

If you want to fine-tune in MLM an already existing model (see the Huggingface repository), follow step 2.

You have to make sure the followings are correct:

1. GPU is correctly installed on your environment

To resume training from checkpoint you use the --model_name_or_path parameter. So instead of giving the default gpt2 you direct this to your latest checkpoint folder.

So your command becomes:

I think there is some confusion about the different components - I'll try to clarify:

1. The tokenizer does not produce vectors. It's just a component that segments texts into tokens. In spaCy, it's rule-based and not trainable, and doesn't have anything to do with vectors. It looks at whitespace and punctuation to determine which are the unique tokens in a sentence.
2. An nlp model in spaCy can have predefined (static) word vectors that are accessible on the Token level. Every token with the same Lexeme gets the same vector. Some tokens/lexemes may indeed be OOV, like misspellings. If you want to redefine/extend all vectors used in a model, you can use something like init-model.
3. The tok2vec layer is a machine learning component that learns how to produce suitable (dynamic) vectors for tokens. It does this by looking at lexical attributes of the token, but may also include the static vectors of the token (cf item 2). This component is generally not used by itself, but is part of another component, such as an NER. It will be the first layer of the NER model, and it can be trained as part of training the NER, to produce vectors that are suitable for your NER task.

In spaCy v2, you can first train a tok2vec component with pretrain, and then use this component for a subsequent train command. Note that all settings need to be the same across both commands, for the layers to be compatible.

To answer your questions:

Isn't the tok2vec the part that generates the vectors?

If you mean the static vectors, then no. The tok2vec component produces new vectors (possibly with a different dimension) on top of the static vectors, but it won't change the static ones.

What does it mean loading pretrained vectors and then train a component to predict these vectors? What's the purpose of doing this?

The purpose is to get a tok2vec component that is already pretrained from external vectors data. The external vectors data already embeds some "meaning" or "similarity" of the tokens, and this is -so to say- transferred into the tok2vec component, which learns to produce the same similarities. The point is that this new tok2vec component can then be used & further fine-tuned in the subsequent train command (cf item 3)

Is there a way to still make use of this for OOV words?

It really depends on what your "use" is. As https://stackoverflow.com/a/57665799/7961860 mentions, you can set the vectors yourself, or you can implement a user hook which will decide on how to define token.vector.

I hope this helps. I can't really recommend the best approach for you to follow, without understanding why you want the OOV vectors / what your use-case is. Happy to discuss further in the comments!

The .raw only indicates that they use the raw version of the WikiText, they are regular text files containing the raw text:

We're using the raw WikiText-2 (no tokens were replaced before the tokenization).

The description of the data files options also says that they are text files. From run_language_modeling.py - L86-L88: