nltk | NKTL Japanese related files

you need to update the line to read the file to

Not sure if the final format is json, yet below is an example to process the data into the print format, i.e.

I'm not sure what is the cause of the problem, but since the output of you SVM model and DecisionTreeClassfier always output 1, I suggest you try a more complex model like RandomForestClassifier and see what it comes out.

I've similar experience before, no matter how I tuned the training hyperparameters, the model always give the same performance metric -- this may cause by 2 probabilities:

1. Our data is not suitable for the model, for example all values in the vector is zero: [0, 0, 0, 0, 0, 0, 0]
2. Our model is too simple, which could only perform linear modeling, so that it could not learn too complex mapping function.

Since your SVM is built with linear kernel, could you try an more complex model and see what it comes out? And could you examine that if your X_train_vectors is all zero's in the matrix?

Where speed is a concern, SpaCy is often preferable over NLTK. It offers both batch processing as well as GPU integration.

With an iterable of strings, this is the basic procedure of how you'd perform batch processing (note that there are a lot of options to tweak, like disabling certain parts of the pipeline that you don't need and setting a batch size, all of which is explained in detail in the SpaCy docs).

First & foremost, beyond a certain size of data, & especially when working with raw text or tokenized text, you probably don't want to be using Pandas dataframes for every interim result.

They add extra overhead & complication that isn't fully 'Pythonic'. This is particularly the case for:

• Python list objects where each word is a separate string: once you've tokenized raw strings into this format, as for example to feed such texts to Gensim's Word2Vec model, trying to put those into Pandas just leads to confusing list-representation issues (as with your columns where the same text might be shown as either ['yessir', 'shit', 'is', 'real'] – which is a true Python list literal – or [yessir, shit, is, real] – which is some other mess likely to break if any tokens have challenging characters).
• the raw word-vectors (or later, text-vectors): these are more compact & natural/efficient to work with in raw Numpy arrays than Dataframes

So, by all means, if Pandas helps for loading or other non-text fields, use it there. But then use more fundamntal Python or Numpy datatypes for tokenized text & vectors - perhaps using some field (like a unique ID) in your Dataframe to correlate the two.

Especially for large text corpuses, it's more typical to get away from CSV and instead use large text files, with one text per newline-separated line, and any each line being pre-tokenized so that spaces can be fully trusted as token-separated.

That is: even if your initial text data has more complicated punctuation-sensative tokenization, or other preprocessing that combines/changes/splits other tokens, try to do that just once (especially if it involves costly regexes), writing the results to a single simple text file which then fits the simple rules: read one text per line, split each line only by spaces.

Lots of algorithms, like Gensim's Word2Vec or FastText, can either stream such files directly or via very low-overhead iterable-wrappers - so the text is never completely in memory, only read as needed, repeatedly, for multiple training iterations.

For more details on this efficient way to work with large bodies of text, see this artice: https://rare-technologies.com/data-streaming-in-python-generators-iterators-iterables/

You collect document tokens into a list of lists, list (list.append(tokenize)), and then in a for tokens in list: loop you assign the most_common() value to the most_common variable thus getting only the most common terms for the last document only.

I sugges to .extend the list of tokens to collect all tokens into a single flat list of words, and then getting the top 10 most common tokens by passing the list to the nltk.FreqDist() method:

Pass the class, not its instance:

As metatoaster has elaborated in his comment, you would prob. have to restructure your data to perform the operation in the exact way you want (without O(n)).

That being said, in the current state and with reference to your example, you could do:

By looking at the pypi site for keras-nightly library, I could see that there are no versions named 2.5.0.dev. Check which package is generating the error and try downgrading that package.