jieba | Stuttering Chinese word segmentation

Just for anyone who somehow has the same problem:

The problem for me was that it worked on my local machine but not on AWS EB but without errors. The problem was the memoryerror mentioned above. I was using a free tier hence my memory limit was at 1gb and AWS EB crashes once you exceed that limit.

There are two ways to fix it that is quite obvious but was not obvious to me in the first place:

1. Expand your tier to one with higher memory capacity
2. Make your program more memory efficient

I did the latter and the problem was solved.

Some useful commands to help you debug:

You don't want to modify the segmentation setup in the loaded model.

It's technically possible to switch the loaded model from pkuseg to jieba, but if you do that, the model components will perform terribly because they've only been trained on pkuseg segmentation.

I'd suggest if you're working with pandas and want to work on some form of parallel processing, I invite you to use dask. It's a Python package that has the same API as pandas dataframes, so in your example, if you have a csv file called file.csv, you can do something like:

You'll have to do some setup for a dask Client and tell it how many workers you want and how many cores to use.

You can use absolute path of Python for executing the script in Jenkins.

numpy makes gradient available, this function would probably be useful for solving your problem

if you add data/code to the question I can try and suggest something more sensible!

Now I got it, the reason is I have too much single labelled data.

I used some high value dataset and got the correct result.

So, the answer is: polish the dataset.

pip3 in the terminal is almost certainly installing your package into a different Python installation.

Rather than have you hunt around for the right installation, you can use Python and Jupyter themselves to ensure you are using the correct Python binary. This relies on three tricks:

• You can execute the pip command-line tool as a module by running python -m pip .... This uses the pip module installed for the python command, so you don't have to verify what python installation the pip3 command is tied to.

• You can get the path of the current Python interpreter with the sys.executable attribute.

• You can execute shell commands from Jupyter notebooks by prefixing the shell command with !, and you can insert values generated with Python code with {expression}

You can combine these to run pip (to install packages or run other commands) against the current Python installation, from Jupyter itself, by adding this into a cell:

Are you using the same python that's being used by your project interpreter? Try

Unfortunately, spaCy does not have a pretrained Chinese model yet (see here), which means you have to use the default Chinese() model which only performs tokenization, and no POS tagging or entity recognition.

There is definitely some work in progress around Chinese for spaCy though, check the issues here.