seq2seq | Seq2Seq code in PyTorch | Messaging library

Can you explain a little more about what you're trying to do with a model from every epoch/some number of steps? I think it already archives the model every time it gets a new best score, so I'm wondering what you want to do that can't be accomplished with that.

Edit:

It looks like AllenNLP already saves a model every epoch, but it only keeps a maximum of 2 by default. I believe you can change that by adding a checkpointer to your training config, e.g.:

1. Please set add_special_tokens = False.
2. Use tokenizer.convert_tokens_to_string (which takes the list of subword tokens as input), where tokenizer refers to the tokenizer used by your DatasetReader.

Please let us know if you have further questions!

I eventually found a solution. The file can be found at: https://github.com/huggingface/transformers/blob/master/examples/legacy/seq2seq/seq2seq_trainer.py

For some reason when importing the file Python picks up a commented link and throws an error. To get around this simply make a copy of the file without the comments at the top. That worked for me.

EDIT: I found a neater solution:

Use this:

The offline model of T5 - t5_base_en_2.7.1_2.4_1610133506835 - was trained on SparkNLP 2.7.1, and there was a breaking change in 2.7.2.

Solved by downloading and re-saving the new version with

The problem is that you clone the master branch of the repository and try to run the run_seq2seq.py script with a transformers version (4.3.3) that is behind that master branch.

run_seq2seq.py was updated to import is_offline_mode on the 6th of march with this merge.

All you need to do is to clone the branch that was used for your used transformers version:

The main technical limitation of the vocabulary size is the GPU memory. The word embeddings and the output projection are the biggest parameters in the model. With a too large vocabulary, you would be forced to use small training batches which would significantly slow down the training.

Also, it is not necessarily so that the bigger the vocabulary, the better the performance. Words in a natural language are distributed according to Zipf's law, which means that the frequency of words decreases exponentially with the frequency rank. With the increasing vocabulary size, you add words that are less and less common in the language. The word embeddings get updated only when the word occurs in the training data. With a very large vocabulary, the embeddings of less frequent words end up undertrained and the model cannot handle them properly anyway.

MT models typically used a vocabulary of 30k-50k tokens. These are however not words, but so-called sub-words. The text gets segmented using a statistical heuristic, such that most of the common words remain as they are and less frequent words get split into subwords, ultimately into single characters.

No, it just has to be a sequence like requirement.

Klaus Greff, et al., LSTM: A Search Space Odyssey, 2015 : Since LSTMs are effective at capturing long-term temporal dependencies without suffering from the optimization hurdles that plague simple recurrent networks (SRNs), they have been used to advance the state of the art for many difficult problems. This includes handwriting recognition and generation, language modeling and translation, acoustic modeling of speech, speech synthesis, protein secondary structure prediction, analysis of audio, and video data among others.

Felix A. Gers, et al., Learning to Forget: Continual Prediction with LSTM, 2000 : LSTM holds promise for any sequential processing task in which we suspect that a hierarchical decomposition may exist, but do not know in advance what this decomposition is.

You can upgrade the Tensorflow to latest stable version since Tensorflow 2.4 version supports new Nvidia's Ampere architecture which is of RTX 30 series and CUDA 11 support also is available.
You can check in this chart for details and follow the guide to install the same.
https://www.tensorflow.org/install/source_windows#tested_build_configurations

Regarding the memory usage on GPU, you can always set the memory growth at the start of your code like mentioned here.