bosen | Parallel ML System - Bosen project | Machine Learning library

1. If the memory problems still persist, you could opt for DistillGPT2, as it has a 33% reduction in the parameters of the network (the forward pass is also twice as fast). Particularly for a small GPU memory like 6GB VRAM, it could be a solution/alternative to your problem.
2. At the same time, it depends on how you preprocess the data. Indeed, the model is capable of "receiving" a maximum length of N tokens (could be for example 512/768) depending on the models you choose. I recently trained a named entity recognition model and the model had a maximum length of 768 tokens. However, when I manually set the dimension of the padded tokens in my PyTorch DataLoader() to a big number, I also got OOM memory (even on 3090 24GB VRAM). As I reduced the dimension of the tokens to a much smaller one (512 instead of 768 for example) the training started to work and I did not get any issues with the lack of memory.

TLDR: Reducing the number of tokens in the preprocessing phase, regardless of the max capacity of the network, can also help to solve your memories problem. Note that reducing the number of tokens to process in a sequence is different from the dimension of a token.