Hierarchical-BERT-Model-with-Limited-Labelled-Data | This repository is temporarily associated with paper Lu , J | Natural Language Processing library

FastText + SVM: We use 300-dimensional word vectors constructed by a FastText language model pre-trained with the Wikipedia corpus (Joulin et al., 2016). Averaged word embeddings are used as the representation of the document. For preprocessing, all text is converted to lowercase and we remove all punctuation and stop words. SVM is used as the classifier. We tune the hyper-parameters of the SVM classifier using a grid-search based on 5-fold cross-validation performed on the training set, after that, we re-train the classifier with optimised hyper-parameters. This hyper-parameter tuning method is applied in RoBERTa + SVM as well. RoBERTa + SVM: We use 768-dimensional word vectors generated by a pre-trained RoBERTa language model (Liu et al., 2019). We do not fine-tune the pre-trained language model and use the averaged word vectors as the representation of the document. Since all BERT-based models are configured to take as input a maximum of 512 tokens, we divided the long documents with W words into k = W/511 fractions, which is then fed into the model to infer the representation of each fraction (each fraction has a "<S>" token in front of 511 tokens, so, 512 tokens in total). Based on the approach of (Sun et al., 2020), the vector of each fraction is the average embeddings of words in that fraction, and the representation of the whole text sequence is the mean of all k fraction vectors. For preprocessing, the only operation performed is to convert all tokens to lowercase. SVM is used as the classifier. Fine-tuned RoBERTa: For the document classification task, fine-tuning RoBERTa means adding a softmax layer on top of the RoBERTa encoder output and fine-tuning all parameters in the model. In this experiment, we fine-tune the same 768-dimensional pre-trained RoBERTa model with a small training set. The settings of all hyper-parameters follow (Liu et al., 2019). we set the learning rate to 1*10-4 and the batch size to 4, and use the Adam optimizer with epsilon equals to 1*10-8 through hyperparameter tuning. However, since we assume that the amount of labelled data available for training is small, we do not have the luxury of a hold out validation set to use to implement early stopping during model fine tuning. Instead, after training for 15 epochs we roll back to the model with the lowest loss based on the training dataset. This rollback strategy is also applied to HAN and HBM due to the limited number of instances in training sets. For preprocessing, the only operation performed is to convert all tokens to lowercase. Hierarchical Attention Network: Following (Yang et al., 2016), we apply two levels of Bi-GRU with attention mechanism for document classification. All words are first converted to word vectors using GloVe (Pennington et al., 2014) (300 dimension version pre-trained using the wiki gigaword corpus) and fed into a word-level Bi-GRU with attention mechanism to form sentence vectors. After that, a sentence vector along with its context sentence vectors are input into sentence-level Bi-GRU with attention mechanism to form the document representation which is then passed to a softmax layer for final prediction. For preprocessing, the only operation performed is to convert all tokens to lowercase, and separate documents into sentences. We apply Python NLTK sent_tokenize function to split documents into sentences. Hierarchical BERT Model: For HBM, we set the number of BERT layers to 4, and the maximum number of sentences to 114, 64, 128, 128, 100, and 64 for the Movie Review, Multi-domain Customer Review, Blog Author Gender, Guardian 2013, Reuters and 20 Newsgroups datasets respectively, these values are based on the length of documents in these datasets. After some preliminary experiments, we set the attention head to 1, the learning rate to 2*10-5, dropout probability to 0.01, used 50 epochs, set the batch size to 4 and used the Adam optimizer with epsilon equals to 1*10-8. The only text preprocessing operation performed is to convert all tokens to lowercase and split documents into sentences. We apply Python NLTK sent_tokenize function to split documents into sentences.