Flaubert | Unsupervised Language Model Pre-training | Machine Learning library

I want to re-finetuned a transformer model but I get an unknown error when I tried to train the model. I can't change the "num_labels" on loading the model. So, I tried to change it manually

I am trying to save the tokenizer in huggingface so that I can load it later from a container where I don't need access to the internet.

BERT model for Language Model and Sequence classification includes an extra projection layer between the last transformer and the classification layer (it contains a linear layer of size hidden_dim x hidden_dim, a dropout layer and a tanh activation). This was not described in the paper originally but was clarified here. This intermediate layer is pre-trained together with the rest of the transformers.

In huggingface's BertModel, this layer is called pooler.

According to the paper, FlauBERT model (XLMModel fine-tuned on French corpus) also includes this pooler layer: "The classification head is composed of the following layers, in order: dropout, linear,tanhactivation, dropout, and linear.". However, when loading a FlauBERT model with huggingface (e.g, with FlaubertModel.from_pretrained(...), or FlaubertForSequenceClassification.from_pretrained(...)), the model seem to include no such layer.

Hence the question: why is there no pooler layer in huggingfaces' FlauBERT model ?

my question concerns the example, available in the great huggingface/transformers library.

I am using a notebook, provided by library creators as a starting point for my pipeline. The notebook below presents a pipeline of finetuning a BERT for Sentence Classification on Glue dataset. https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb#scrollTo=uBzDW1FO63pK
When getting into the code, I noticed a very weird thing, which I cannot explain.

In the example, input data is introduced to the model as the instances of the InputFeatures class from here: https://github.com/huggingface/transformers/blob/011cc0be51cf2eb0a91333f1a731658361e81d89/src/transformers/data/processors/utils.py This class has 4 attributes, including the label attribute: