corpus | Yet another CSS toolkit

Incase someone has similar question, I'll paste the reply I got when asking this question in the Gensim Disussion Group for reference:

You can try it, but I wouldn't expect it to work well for most purposes.

The build_vocab() call establishes the known vocabulary of the model, & caches some stats about the corpus.

If you then supply another corpus – & especially one with more words – then:

• You'll want your train() parameters to reflect the actual size of your training corpus. You'll want to provide a true total_examples and total_words count that are accurate for the training-corpus.
• Every word in the training corpus that's not in the know vocabulary is ignored completely, as if it wasn't even there. So you might as well filter your corpus down to just the words-of-interest first, then use that same filtered corpus for both steps. Will the example texts still make sense? Will that be enough data to train meaningful, generalizable word-vectors for just the words-of-interest, alongside other words-of-interest, without the full texts? (You could look at your pref-filtered corpus to get a sense of that.) I'm not sure - it could depend on how severely trimming to just the words-of-interest changed the corpus. In particular, to train high-dimensional dense vectors – as with vector_size=300 – you need a lot of varied data. Such pre-trimming might thin the corpus so much as to make the word-vectors for the words-of-interest far less useful.

You could certainly try it both ways – pre-filtered to just your words-of-interest, or with the full original corpus – and see which works better on downstream evaluations.

More generally, if the concern is training time with the full corpus, there are likely other ways to get an adequate model in an acceptable amount of time.

If using corpus_file mode, you can increase workers to equal the local CPU core count for a nearly-linear speedup from number of cores. (In traditional corpus_iterable mode, max throughput is usually somewhere in the 6-12 workers threads, as long as you ahve that many cores.)

min_count=1 is usually a bad idea for these algorithms: they tend to train faster, in less memory, leaving better vectors for the remaining words when you discard the lowest-frequency words, as the default min_count=5 does. (It's possible FastText can eke a little bit of benefit out of lower-frequency words via their contribution to character-n-gram-training, but I'd only ever lower the default min_count if I could confirm it was actually improving relevant results.

If your corpus is so large that training time is a concern, often a more-aggressive (smaller) sample parameter value not only speeds training (by dropping many redundant high-frequency words), but ofthen improves final word-vector quality for downstream purposes as well (by letting the rarer words have relatively more influence on the model in the absense of the downsampled words).

And again if the corpus is so large that training time is a concern, than epochs=100 is likely overkill. I believe the GoogleNews vectors were trained using only 3 passes – over a gigantic corpus. A sufficiently large & varied corpus, with plenty of examples of all words all throughout, could potentially train in 1 pass – because each word-vector can then get more total training-updates than many epochs with a small corpus. (In general larger epochs values are more often used when the corpus is thin, to eke out something – not on a corpus so large you're considering non-standard shortcuts to speed the steps.)

-- Gordon

Not much different from what you did, still quite fast:

You can use the Tokenizer __call__ method to join both sentences when encoding them.

In case you're using the PyTorch implementation, here is an example:

Like explained in the comment from @PitaJ, *acc.entry(word).or_insert(0) += 1 has the type () while the compiler expects fold()'s callback to return the next state each time (it is sometimes called reduce() in other languages, e.g. JavaScript; in Rust; fold() allows you to specify the start value while reduce() takes it from the first item in the iterator).

Because of this, I feel like it is more appropriate use case for a loop, since you don't need to return a new state but to update the map:

The vectors setting is not related to the transformer or what you're trying to do.

In the new config, you want to use the source option to load the components from the existing pipeline. You would modify the [component] blocks to contain only the source setting and no other settings:

To be certain, can you verify your current nltk_data folder structure? The correct structure is:

(Asking for software/data recommendations is off-topic for StackOverflow; but I have tried to give a more general "approach" answer.)

1. Another approach to finding related words would be one of the machine learning approaches. If you are dealing with words in isolation, look at word embeddings such as GloVe or Word2Vec. Spacy and gensim have libraries for working with them, though I'm also getting some search hits for tutorials of working with them in nltk.

2/3. One of the (in my opinion) core reasons for the success of Princeton WordNet was the liberal license they used. That means you can branch the project, add your extra data, and redistribute.

You might also find something useful at http://globalwordnet.org/resources/global-wordnet-grid/ Obviously most of them are not for English, but there are a few multilingual ones in there, that might be worth evaluating?

Another approach would be to create a wrapper function. It first searches a lookup list of fixes and additions you think should be in there. If not found then it searches WordNet as normal. This allows you to add 'succeed', 'success', 'successful', and then other sets of words as end users point out something missing.

Get rid of flex-col on #parent-div and use h-fit (height: fit-content;) on your button.

See here

You may want to read this github page: https://nicharuc.github.io/topic_modeling/

According to this example, your code could go like this: