imbalanced-learn | Python Package to Tackle the Curse of Imbalanced Datasets

by scikit-learn-contrib Python Version: 0.9.0 License: MIT

Download this library from

kandi X-RAY | imbalanced-learn Summary

imbalanced-learn is a Python library typically used in Institutions, Learning, Education, Data Science, Pandas applications. imbalanced-learn has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has medium support. You can install using 'pip install imbalanced-learn' or download it from GitHub, PyPI.

A Python Package to Tackle the Curse of Imbalanced Datasets in Machine Learning

Support

Quality

Security

License

Reuse

kandi-support Support

imbalanced-learn has a medium active ecosystem.
It has 5660 star(s) with 1161 fork(s). There are 150 watchers for this library.
There were 2 major release(s) in the last 12 months.
There are 43 open issues and 442 have been closed. On average issues are closed in 75 days. There are 16 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of imbalanced-learn is 0.9.0

imbalanced-learn Support

Best in #Python

Average in #Python

imbalanced-learn Support

Best in #Python

Average in #Python

quality kandi Quality

imbalanced-learn has 0 bugs and 0 code smells.

imbalanced-learn Quality

Best in #Python

Average in #Python

imbalanced-learn Quality

Best in #Python

Average in #Python

security Security

imbalanced-learn has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
imbalanced-learn code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

imbalanced-learn Security

Best in #Python

Average in #Python

imbalanced-learn Security

Best in #Python

Average in #Python

license License

imbalanced-learn is licensed under the MIT License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

imbalanced-learn License

Best in #Python

Average in #Python

imbalanced-learn License

Best in #Python

Average in #Python

build Reuse

imbalanced-learn releases are available to install and integrate.
Deployable package is available in PyPI.
Build file is available. You can build the component from source.
imbalanced-learn saves you 6003 person hours of effort in developing the same functionality from scratch.
It has 13560 lines of code, 626 functions and 139 files.
It has high code complexity. Code complexity directly impacts maintainability of the code.

imbalanced-learn Reuse

Best in #Python

Average in #Python

imbalanced-learn Reuse

Best in #Python

Average in #Python

Top functions reviewed by kandi - BETA

kandi has reviewed imbalanced-learn and discovered the below as its top functions. This is intended to give you an instant insight into imbalanced-learn implemented functionality, and help decide if they suit your requirements.

Computes sensitivity support .
Generate a classification report .
Download and return datasets .
Compute geometric mean squared error .
Check the sampling strategy .
Decorator for scoring functions .
Compute sensitivity score .
Computes sensitivity score .
Make an imbalanced dataset .
A batch generator .

imbalanced-learn Key Features

A Python Package to Tackle the Curse of Imbalanced Datasets in Machine Learning

imbalanced-learn Examples and Code Snippets

See all related Code Snippets

Get error: unexpected keyword argument 'random_state' when using TomekLinks

Copy

Download

X, y = make_classification(n_classes=2, class_sep=2,
                           weights=[0.1, 0.9], 
                           n_informative=3, n_redundant=1, 
                           flip_y=0, n_features=20, 
                           n_clusters_per_class=1, 
                           n_samples=1000, random_state=10
                          )

feature importance bagging classifier and column names

Copy

Download

...

bunch = load_iris()
X = bunch.data
y = bunch.target
feature_names = bunch.feature_names

clf = BaggingClassifier(DecisionTreeClassifier(), random_state=42)
clf.fit(X, y)

feature_importances = np.mean([tree.feature_importances_ for tree in clf.estimators_], axis=0)

output = {fn:fi for fn,fi in zip(feature_names,feature_importances)}
print(output)

{
    'sepal length (cm)': 0.008652347823679744,
    'sepal width (cm)': 0.01945400672681583,
    'petal length (cm)': 0.539297348817521,
    'petal width (cm)': 0.43259629663198346
}

-----------------------

...

bunch = load_iris()
X = bunch.data
y = bunch.target
feature_names = bunch.feature_names

clf = BaggingClassifier(DecisionTreeClassifier(), random_state=42)
clf.fit(X, y)

feature_importances = np.mean([tree.feature_importances_ for tree in clf.estimators_], axis=0)

output = {fn:fi for fn,fi in zip(feature_names,feature_importances)}
print(output)

{
    'sepal length (cm)': 0.008652347823679744,
    'sepal width (cm)': 0.01945400672681583,
    'petal length (cm)': 0.539297348817521,
    'petal width (cm)': 0.43259629663198346
}

Cant install imbalanced-learn on an Azure ML Environment

Copy

Download

!pip install -U "imbalanced-learn < 0.9"

Download UNIX python wheel in windows

Copy

Download

$ pip download --only-binary=:all: --platform win_amd64 imbalanced_learn
[...]
Saved ./imbalanced_learn-0.9.0-py3-none-any.whl
Saved ./joblib-1.1.0-py2.py3-none-any.whl
Saved ./numpy-1.22.2-cp39-cp39-win_amd64.whl
Saved ./scikit_learn-1.0.2-cp39-cp39-win_amd64.whl
Saved ./scipy-1.8.0-cp39-cp39-win_amd64.whl
Saved ./threadpoolctl-3.1.0-py3-none-any.whl
[...]

$ pip download --only-binary=:all: --platform manylinux2014_x86_64 imbalanced_learn
[...]
Saved ./imbalanced_learn-0.9.0-py3-none-any.whl
Saved ./joblib-1.1.0-py2.py3-none-any.whl
Saved ./numpy-1.22.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./scipy-1.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./threadpoolctl-3.1.0-py3-none-any.whl
[...]

Cannot find conda info. Please verify your conda installation on EMR

Copy

Download

wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda

echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda config --set always_yes yes --set changeps1 no
conda config -f --add channels conda-forge


conda create -n zoo python=3.7 # "zoo" is conda environment name
conda init bash
source activate zoo
conda install python 3.7.0 -c conda-forge orca 
sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv

“spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
"zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
"zeppelin.python": "/home/hadoop/conda/bin/python"

-----------------------

wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda

echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda config --set always_yes yes --set changeps1 no
conda config -f --add channels conda-forge


conda create -n zoo python=3.7 # "zoo" is conda environment name
conda init bash
source activate zoo
conda install python 3.7.0 -c conda-forge orca 
sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv

“spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
"zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
"zeppelin.python": "/home/hadoop/conda/bin/python"

Multipoint(df['geometry']) key error from dataframe but key exist. KeyError: 13 geopandas

Copy

Download

# https://www.kaggle.com/new-york-state/nys-nyc-transit-subway-entrance-and-exit-data
import kaggle.cli
import sys, requests, urllib
import pandas as pd
from pathlib import Path
from zipfile import ZipFile

# fmt: off
# download data set
url = "https://www.kaggle.com/new-york-state/nys-nyc-transit-subway-entrance-and-exit-data"
sys.argv = [sys.argv[0]] + f"datasets download {urllib.parse.urlparse(url).path[1:]}".split(" ")
kaggle.cli.main()
zfile = ZipFile(f'{urllib.parse.urlparse(url).path.split("/")[-1]}.zip')
dfs = {f.filename: pd.read_csv(zfile.open(f)) for f in zfile.infolist() if Path(f.filename).suffix in [".csv"]}
# fmt: on

df_subway = dfs['nyc-transit-subway-entrance-and-exit-data.csv']

from shapely.geometry import Point, MultiPoint
from shapely.ops import nearest_points
import geopandas as gpd

geometry = [Point(xy) for xy in zip(df_subway['Station Longitude'], df_subway['Station Latitude'])]

# Coordinate reference system :
crs = {'init': 'EPSG:4326'}

# Creating a Geographic data frame 
gdf_subway_entrance_geometry = gpd.GeoDataFrame(df_subway, crs=crs, geometry=geometry).to_crs('EPSG:5234')
gdf_subway_entrance_geometry

df_yes_entry = gdf_subway_entrance_geometry
df_yes_entry = gdf_subway_entrance_geometry[gdf_subway_entrance_geometry.Entry=='YES']
df_yes_entry

# randomly select a point....
gpdPoint = gdf_subway_entrance_geometry.sample(1).geometry.tolist()[0]
pts = MultiPoint(df_yes_entry['geometry'].values) # does not work with a geopandas series, works with a numpy array
pt = Point(gpdPoint.x, gpdPoint.y)
#[o.wkt for o in nearest_points(pt, pts)]
for o in nearest_points(pt, pts):
  print(o)

How to install PyCaret in AWS Glue

Copy

Download

I [Meghana] was able to successfully replicate the error at my end as well using the below steps. Also, please find the workaround mentioned below to avoid the same.

- Create a Glue 2.0 job and configure the Job Parameters as below
===
-- additional-python-modules  pycaret==2.3.2,spacy==3.0.6 
===

On running the Glue job I see that it tried to a Pip3 install, Please find the same below 
===
 INFO   2021-07-07 05:23:15,813 0   com.amazonaws.services.glue.PythonModuleInstaller   [main]  pip3 install  --user pycaret==2.3.2 spacy==3.0.6 
===

However, pip install can be called against a pypi package, local project or wheel hosted via HTTPS we can use this functionality to install packages publicly hosted on pypi as well as those not available publicly available. This gives us the ability to install the majority of packages to use with Glue, including those which are c-based. There is however a subset that will fail (like spacy); packages that require root privileges to install or to be compiled during installation will fail. Glue does not give root access and there is no exception made for package installation. What can be done in this case is pre-compiling the binaries into a wheel compatible with Glue and installing that wheel. 

Please find the below error that I have observed while installing the above “pycaret==2.3.2,spacy==3.0.6” and import them in glue job failed as the compilation failed while replicating.
==
INFO    2021-07-07 05:23:15,813 0   com.amazonaws.services.glue.PythonModuleInstaller   [main]  pip3 install  --user pycaret==2.3.2 spacy==3.0.6 
INFO    2021-07-07 05:23:26,751 10938   com.amazonaws.services.glue.PythonModuleInstaller   [main]  Collecting pycaret==2.3.2  Downloading https://files.pythonhosted.org/packages/bc/b6/9d620a23a038b3abdc249472ffd9be217f6b1877d2d952bfb3f653622a28/pycaret-2.3.2-py3-none-any.whl  (263kB)Collecting spacy==3.0.6  Downloading https://files.pythonhosted.org/packages/6d/0d/4379e9aa35a444b6440ffe1af4c612533460e0d5ac5c7dca1f96ff6f2e23/spacy-3.0.6.tar.gz  (7.1MB)    Complete output from command python setup.py egg_info:        Error compiling Cython file:    ------------------------------------------------------------    ...    from libc.stdint cimport int64_t    from libcpp.vector cimport vector    from libcpp.set cimport set    from cymem.cymem cimport Pool    ^    ------------------------------------------------------------        spacy/strings.pxd:4:0: 'cymem/cymem.pxd' not found        Error compiling Cython file:    ------------------------------------------------------------    ...    from libc.stdint cimport int64_t    from libcpp.vector cimport vector    from libcpp.set cimport set    from cymem.cymem cimport Pool    ^    ------------------------------------------------------------        spacy/strings.pxd:4:0: 'cymem/…
…
..
INFO    2021-07-07 05:23:26,753 10940   com.amazonaws.services.glue.PythonModuleInstaller   [main]  Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ahnegkpn/spacy/
==

Thus, the importing of the above failed during compilation stage. The error that you have received below is due the python version mismatch. Here the tar file spacy-2.3.7.tar.gz  is compiled using python 3.7 and the Glue job is using 3.6 and hence it failed with the below error. However, even if you provide as mentioned above it still has c-dependencies and stills fails. Please find the work-around to avoid the same.
—
INFO    2021-07-05 17:01:53,986 17142   com.amazonaws.services.glue.PythonModuleInstaller   [main]  Collecting setuptools  Downloading …

7c5f7541eb883181b564a8c8ba15d21b2d7b8a38ae32f31763575cf8857d/spacy-2.3.7.tar.gz (5.8MB)  
  Complete output from command python setup.py egg_info:    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_e6ejo9m/blis/    Traceback (most recent call last):      File "/home/spark/.local/lib/python3.7/site-packages/setuptools/installer.py", line 128, in fetch_build_egg        subprocess.check_call(cmd)      File "/usr/lib64/python3.7/subprocess.py", line 363, in check_call        raise CalledProcessError(retcode, cmd)    subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpo5fqe7du', '--quiet', 'blis<0.8.0,>=0.4.0']' returned non-zero exit status 1.        During handling of the above exception, another exception occurred:        Traceback (most recent call last):      File "<string>", line 1, in <module>      File "/tmp/pip-build-xoyv9lar/spacy/setup.py", line 252, in <module>        setup_package()      File "/tmp/pip-build-xoyv9lar/spacy/setup.py", line 247, in setup_package        cmdclass={"build_ext": build_ext_subclass},      File "/home/spark/.local/lib/…

      File "/home/spark/.local/lib/python3.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg        raise DistutilsError(str(e))    distutils.errors.DistutilsError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpo5fqe7du', '--quiet', 'blis<0.8.0,>=0.4.0']' returned non-zero exit status 1.        ----------------------------------------
—

###Workaround 

- To compile a library in a C-based language, the compiler must know the target OS and processor architecture. 
- If the library is compiled against a different OS or processor architecture, then the wheel will fail to install in Glue. 
- Because Glue is a manages service, we do not give users cluster-access to develop these dependencies. 
- Below I will walk you through using a Docker image to prepare an environment you can use to compile wheels that will be compatible with Glue. For this example we will be compiling pycaret,spacy which requires GCC to be installed on the target device as root. 

Step-1: Launch an m5.xlarge EC2 instance with Amazon Linux (2) and enough volume space for your libs. 

Step-2:  Install Docker on the instance, set up nonsudo access, and start it
    1. sudo yum install docker -y
    2. sudo usermod -a -G docker ec2-user
    3.  sudo service docker start

Step-3: Create a Dockerfile as below 
$ vi docfile
—
# Base for Glue
FROM amazonlinux
RUN yum update -y
RUN yum install shadow-utils.x86_64 -y
RUN yum install -y java-1.8.0-openjdk.x86_64
RUN yum install -y python3
RUN yum install -y gcc autoconf automake libtool zlib-devel openssl-devel maven wget protobuf-compiler cmake make gcc-c++

# Additonal Components needed for psutil
WORKDIR /root
RUN yum install python3-devel -y
RUN yum install python-devel -y
RUN pip3 install wheel

# Install psutil
RUN pip3 install pycaret
RUN pip3 install spacy

# Create a directory for the wheel
RUN mkdir wheel_dir 

# create the wheel
RUN pip3 wheel pycaret -w wheel_dir
RUN pip3 wheel spacy -w wheel_dir
—

Step-4: Run docker build to build your Dockerfile 
==
restart the docker daemon 
[ec2-user@ip-xxx ~]$ sudo service docker restart
[ec2-user@ip-xxx ~]$ docker build -f docfile .

[ec2-user@ip-xxx ~]$ docker build -f docfile .
Sending build context to Docker daemon  16.38kB
Step 1/17 : FROM amazonlinux
 ---> 7443854fbdb0
Step 2/17 : RUN yum update -y
 ---> Using cache
…
Removing intermediate container xxx
 ---> xxx
Successfully built xx
==

[ec2-user@ip-xxx~]$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

[ec2-user@ip-xxx~]$ docker image ls
REPOSITORY          TAG       IMAGE ID       CREATED          SIZE
<none>              <none>    xxx   24 seconds ago   4.28GB
<none>              <none>    xxx   43 minutes ago   3.94GB

[ec2-user@ip-xxx ~]$ docker ps -a
CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS                      PORTS     NAMES

Xxx                  yyy        "/bin/bash"              37 minutes ago   Exited (0) 37 minutes ago             brave_meninsky


Step-5 : Extract the whl from the docker container
    - Get the container ID
docker ps (get container ID)
    - Run the container and keep it from exiting
docker run -dite <image>
    - Verify the location of the wheel file (and get it’s filename)
 docker exec -t -i <container_id> ls /root/wheel_dir/
    - Copy the wheel out of docker to EC2
 docker cp <containerID>:/root/wheel_dir/<wheelFile> .
===
[ec2-user@ip-xxx ~]$ docker run -dite e0e1f71b8fad
9f0b1aff06dd959f3744edd3804512e73b68aaeef178962f2c0c063b290dbf78

[ec2-user@ip-xxx~]$ docker ps
CONTAINER ID   IMAGE          COMMAND       CREATED          STATUS          PORTS     NAMES
9f0b1aff06dd   e0e1f71b8fad   "/bin/bash"   49 seconds ago   Up 48 seconds             quirky_bose
[ec2-user@ip-xxx ~]$ docker ps
CONTAINER ID   IMAGE          COMMAND       CREATED         STATUS         PORTS     NAMES
52ab3413c962   e0e1f71b8fad   "/bin/bash"   3 seconds ago   Up 2 seconds             fervent_williamson
9f0b1aff06dd   e0e1f71b8fad   "/bin/bash"   2 minutes ago   Up 2 minutes             quirky_bose
[ec2-user@ip-xxx~]$ docker exec -t -i 52ab3413c962 ls /root/wheel_dir/
Boruta-0.3-py3-none-any.whl
Bottleneck-1.3.2-cp37-cp37m-linux_x86_64.whl
…
[ec2-user@ip-xxx ~]$ aws s3 cp pycaret-2.3.2-py3-none-any.whl  s3://cxx/
upload: ./pycaret-2.3.2-py3-none-any.whl to s3://cxx/pycaret-2.3.2-py3-none-any.whl

===

Step-6: Upload the wheel to S3
aws s3 cp <wheelFile> s3://path/to/wheel/

Step-7: Pass the S3 URL to Glue

Edit your Job
Expand “Security configuration, script libraries, and job parameters (optional)”
In “Job parameters enter a new Key-Value pair
--
Key --additional-python-modules
Value. <s3URI>
--

How to correct Python Attribute error: 'SMOTE' object has no attribute 'fit_sample'

Copy

Download

os_data_X, os_data_y = os.fit_resample(X_train, y_train)

Imbalances-learn module base.py file syntax error coming up while importing SMOTE

Copy

Download

class SamplerMixin(BaseEstimator):
    __metaclass__ = ABC

How to store result of pip command into Pandas Datafarme

Copy

Download

import os
import pandas as pd

# create pip list txt
os.system('pip list > pip_list.txt')

# read content into pandas df
df = pd.read_csv('pip_list.txt', sep=r'\s+', skiprows=[1])

# clean up
os.remove('pip_list.txt')

             Package  Version
0            appnope    0.1.0
1        argon2-cffi   20.1.0
2            astroid    2.4.2
3    async-generator     1.10
4              attrs   20.2.0
..               ...      ...
100          urllib3  1.25.10
101          wcwidth    0.2.5
102     webencodings    0.5.1
103            wrapt   1.12.1
104             xlrd    1.2.0

[105 rows x 2 columns]

-----------------------

import os
import pandas as pd

# create pip list txt
os.system('pip list > pip_list.txt')

# read content into pandas df
df = pd.read_csv('pip_list.txt', sep=r'\s+', skiprows=[1])

# clean up
os.remove('pip_list.txt')

             Package  Version
0            appnope    0.1.0
1        argon2-cffi   20.1.0
2            astroid    2.4.2
3    async-generator     1.10
4              attrs   20.2.0
..               ...      ...
100          urllib3  1.25.10
101          wcwidth    0.2.5
102     webencodings    0.5.1
103            wrapt   1.12.1
104             xlrd    1.2.0

[105 rows x 2 columns]

-----------------------

import subprocess
from io import StringIO
command = "pip list"
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
pd.read_csv(StringIO(output.decode("utf-8")),  sep=r"\s*", skiprows=[1])

See all related Code Snippets

Community Discussions

Trending Discussions on imbalanced-learn

Get error: unexpected keyword argument 'random_state' when using TomekLinks
feature importance bagging classifier and column names
TypeError: Encoders require their input to be uniformly strings or numbers. Got ['int', 'str']
Cant install imbalanced-learn on an Azure ML Environment
Download UNIX python wheel in windows
Cannot find conda info. Please verify your conda installation on EMR
A problem in using AIF360 metrics in my code
Solving conda environment stuck
Multipoint(df['geometry']) key error from dataframe but key exist. KeyError: 13 geopandas
getting scikit-learn version warning using yml environment

Trending Discussions on imbalanced-learn

QUESTION

Get error: unexpected keyword argument 'random_state' when using TomekLinks

Asked 2022-Mar-24 at 14:44

My code is:

  undersample = TomekLinks(sampling_strategy='majority', n_jobs= -1,  random_state = 42)
  X_tl, y_tl = undersample.fit_resample(X, y)

When I run it, I get this error:

TypeError: __init__() got an unexpected keyword argument 'random_state'

My package version is:

 imbalanced-learn==0.9.0

although in the documentation this parameter exists:

random_state : int, RandomState instance or None, optional (default=None)

when I check the constructor in _tomek_links.py, I don't see the random state field:

  @_deprecate_positional_args
    def __init__(self, *, sampling_strategy="auto", n_jobs=None):
        super().__init__(sampling_strategy=sampling_strategy)
        self.n_jobs = n_jobs

ANSWER

Answered 2022-Mar-24 at 14:44

I think, you're looking at the wrong documentation. That one is for version 0.3.0-dev, so I checked: https://imbalanced-learn.org/stable/references/generated/imblearn.under_sampling.TomekLinks.html -- this parameter has been deprecated in a newer version 0.9.0.

Also as the documentation goes, seems you have to specify it in make_classification function as below:

X, y = make_classification(n_classes=2, class_sep=2,
                           weights=[0.1, 0.9], 
                           n_informative=3, n_redundant=1, 
                           flip_y=0, n_features=20, 
                           n_clusters_per_class=1, 
                           n_samples=1000, random_state=10
                          )

Source https://stackoverflow.com/questions/71604064

QUESTION

TypeError: Encoders require their input to be uniformly strings or numbers. Got ['int', 'str']

Asked 2022-Feb-20 at 14:24

I already referred the posts here, here and here. Don't mark it as duplicate.

I am working on a binary classification problem where my dataset has categorical and numerical columns.

However, some of the categorical columns has a mix of numeric and string values. Nontheless, they only indicate the category name.

For instance, I have a column called biz_category which has values like A,B,C,4,5 etc.

I guess the below error is thrown due to values like 4 and 5.

Therefore, I tried the belowm to convert them into category datatype. (but still it doesn't work)

cols=X_train.select_dtypes(exclude='int').columns.to_list()
X_train[cols]=X_train[cols].astype('category')

And my data info looks like below

<class 'pandas.core.frame.DataFrame'>
Int64Index: 683 entries, 21 to 965
Data columns (total 9 columns):
 #   Column                                           Non-Null Count  Dtype   
---  ------                                           --------------  -----   
 0   Feature_A                                        683 non-null    category
 1   Product Classification                           683 non-null    category
 2   Industry                                         683 non-null    category
 3   DIVISION                                         683 non-null    category
 4   biz_category                                     683 non-null    category
 5   Country                                          683 non-null    category
 6   Product segment                                  683 non-null    category
 7   SUBREGION                                        683 non-null    category
 8   Quantity 1st year                                683 non-null    int64   
dtypes: category(8), int64(1)

So, after dtype conversion, when I try the below SMOTENC, I get an error

print("Before OverSampling, counts of label '1': {}".format(sum(y_train == 1)))
print("Before OverSampling, counts of label '0': {} \n".format(sum(y_train == 0)))
cat_index = [0,1,2,3,4,5,6,7]
# import SMOTE module from imblearn library
# pip install imblearn (if you don't have imblearn in your system)
from imblearn.over_sampling import SMOTE, SMOTENC
sm = SMOTENC(categorical_features=cat_index,random_state = 2,sampling_strategy = 'minority')
X_train_res, y_train_res = sm.fit_resample(X_train, y_train)

This results in an error as shown below

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) ~\AppData\Roaming\Python\Python39\site-packages\sklearn\utils_encode.py in _unique_python(values, return_inverse) 134 --> 135 uniques = sorted(uniques_set) 136 uniques.extend(missing_values.to_list())

TypeError: '<' not supported between instances of 'str' and 'int'

During handling of the above exception, another exception occurred:

TypeError Traceback (most recent call last) C:\Users\SATHAP~1\AppData\Local\Temp/ipykernel_31168/1931674352.py in 6 from imblearn.over_sampling import SMOTE, SMOTENC 7 sm = SMOTENC(categorical_features=cat_index,random_state = 2,sampling_strategy = 'minority') ----> 8 X_train_res, y_train_res = sm.fit_resample(X_train, y_train) 9 10 print('After OverSampling, the shape of train_X: {}'.format(X_train_res.shape))

~\AppData\Roaming\Python\Python39\site-packages\imblearn\base.py in fit_resample(self, X, y) 81 ) 82 ---> 83 output = self.fit_resample(X, y) 84 85 y = (

~\AppData\Roaming\Python\Python39\site-packages\imblearn\over_sampling_smote\base.py in fit_resample(self, X, y) 511 512 # the input of the OneHotEncoder needs to be dense --> 513 X_ohe = self.ohe.fit_transform( 514 X_categorical.toarray() if sparse.issparse(X_categorical) else X_categorical 515 )

~\AppData\Roaming\Python\Python39\site-packages\sklearn\preprocessing_encoders.py in fit_transform(self, X, y) 486 """ 487 self._validate_keywords() --> 488 return super().fit_transform(X, y) 489 490 def transform(self, X):

~\AppData\Roaming\Python\Python39\site-packages\sklearn\base.py in fit_transform(self, X, y, **fit_params) 850 if y is None: 851 # fit method of arity 1 (unsupervised transformation) --> 852 return self.fit(X, **fit_params).transform(X) 853 else: 854 # fit method of arity 2 (supervised transformation)

~\AppData\Roaming\Python\Python39\site-packages\sklearn\preprocessing_encoders.py in fit(self, X, y) 459 """ 460 self._validate_keywords() --> 461 self.fit(X, handle_unknown=self.handle_unknown, force_all_finite="allow-nan") 462 self.drop_idx = self._compute_drop_idx() 463 return self

~\AppData\Roaming\Python\Python39\site-packages\sklearn\preprocessing_encoders.py in _fit(self, X, handle_unknown, force_all_finite) 92 Xi = X_list[i] 93 if self.categories == "auto": ---> 94 cats = _unique(Xi) 95 else: 96 cats = np.array(self.categories[i], dtype=Xi.dtype)

~\AppData\Roaming\Python\Python39\site-packages\sklearn\utils_encode.py in _unique(values, return_inverse) 29 """ 30 if values.dtype == object: ---> 31 return _unique_python(values, return_inverse=return_inverse) 32 # numerical 33 out = np.unique(values, return_inverse=return_inverse)

~\AppData\Roaming\Python\Python39\site-packages\sklearn\utils_encode.py in _unique_python(values, return_inverse) 138 except TypeError: 139 types = sorted(t.qualname for t in set(type(v) for v in values)) --> 140 raise TypeError( 141 "Encoders require their input to be uniformly " 142 f"strings or numbers. Got {types}"

TypeError: Encoders require their input to be uniformly strings or numbers. Got ['int', 'str']

Should I transform y_train into categorical as well? Currently, it is int64.

Help please

ANSWER

Answered 2022-Feb-20 at 14:22

Cause of the problem

SMOTE requires the values in each categorical/numerical column to have uniform datatype. Essentially you can not have mixed datatypes in any of the column in this case your biz_category column. Also merely casting the column to categorical type does not necessarily mean that the values in that column will have uniform datatype.

Possible solution

One possible solution to this problem is to re-encode the values in those columns which have mixed data types for example you could use lableencoder but I think in your case simply changing the dtype to string would also work.

Source https://stackoverflow.com/questions/71193740

QUESTION

Cant install imbalanced-learn on an Azure ML Environment

Asked 2022-Feb-15 at 21:26

I have an Azure ML Workspace which comes by default with some pre-installed packages.

I tried to install

!pip install -U imbalanced-learn

But I got this error

Requirement already up-to-date: scikit-learn in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (0.24.2)
Requirement already satisfied, skipping upgrade: scipy>=0.19.1 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from scikit-learn) (1.4.1)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from scikit-learn) (0.14.1)
Requirement already satisfied, skipping upgrade: numpy>=1.13.3 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from scikit-learn) (1.18.5)
Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from scikit-learn) (2.1.0)
Collecting imbalanced-learn
  Using cached imbalanced_learn-0.9.0-py3-none-any.whl (199 kB)
Requirement already satisfied, skipping upgrade: threadpoolctl>=2.0.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from imbalanced-learn) (2.1.0)
Requirement already satisfied, skipping upgrade: joblib>=0.11 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from imbalanced-learn) (0.14.1)
Requirement already satisfied, skipping upgrade: scipy>=1.1.0 in /anaconda/envs/azureml_py36/lib/python3.6/site-packages (from imbalanced-learn) (1.4.1)
ERROR: Could not find a version that satisfies the requirement scikit-learn>=1.0.1 (from imbalanced-learn) (from versions: 0.9, 0.10, 0.11, 0.12, 0.12.1, 0.13, 0.13.1, 0.14, 0.14.1, 0.15.0b1, 0.15.0b2, 0.15.0, 0.15.1, 0.15.2, 0.16b1, 0.16.0, 0.16.1, 0.17b1, 0.17, 0.17.1, 0.18, 0.18.1, 0.18.2, 0.19b2, 0.19.0, 0.19.1, 0.19.2, 0.20rc1, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.20.4, 0.21rc2, 0.21.0, 0.21.1, 0.21.2, 0.21.3, 0.22rc2.post1, 0.22rc3, 0.22, 0.22.1, 0.22.2, 0.22.2.post1, 0.23.0rc1, 0.23.0, 0.23.1, 0.23.2, 0.24.dev0, 0.24.0rc1, 0.24.0, 0.24.1, 0.24.2)
ERROR: No matching distribution found for scikit-learn>=1.0.1 (from imbalanced-

learn)

Not sure how to solve this, I have read in other posts to use conda, but that didnt work either.

ANSWER

Answered 2022-Feb-15 at 14:23

scikit-learn 1.0.1 and up require Python >= 3.7; you use Python 3.6. You need to upgrade Python or downgrade imbalanced-learn. imbalanced-learn 0.8.1 allows Python 3.6 so

!pip install -U "imbalanced-learn < 0.9"

Source https://stackoverflow.com/questions/71127858

QUESTION

Download UNIX python wheel in windows

Asked 2022-Feb-09 at 12:20

I want to download a UNIX python wheel into my windows PC, for later install it in a UNIX server with no access to the internet

I tried

python -m pip download --only-binary :all: --no-cache imbalanced_learn

But I get the wheels win_amd, but I need manylinux

Example download in windows:

Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
  Saved c:\users\c58210a\onedrive - experian services corp\documents\proyectos\migracion db\descargar librerias\test\imbalanced_learn2\threadpoolctl-3.1.0-py3-none-any.whl
Collecting scipy>=1.1.0
  Downloading scipy-1.7.3-cp37-cp37m-win_amd64.whl (34.1 MB)
     |████████████████████████████████| 34.1 MB 6.8 MB/s
  Saved c:\users\c58210a\onedrive - experian services corp\documents\proyectos\migracion db\descargar librerias\test\imbalanced_learn2\scipy-1.7.3-cp37-cp37m-win_amd64.whl
Collecting scikit-learn>=1.0.1
  Downloading scikit_learn-1.0.2-cp37-cp37m-win_amd64.whl (7.1 MB)
     |████████████████████████████████| 7.1 MB 6.1 kB/s
  Saved c:\users\c58210a\onedrive - experian services corp\documents\proyectos\migracion db\descargar librerias\test\imbalanced_learn2\scikit_learn-1.0.2-cp37-cp37m-win_amd64.whl
Collecting joblib>=0.11
  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
     |████████████████████████████████| 306 kB 242 kB/s
  Saved c:\users\c58210a\onedrive - experian services corp\documents\proyectos\migracion db\descargar librerias\test\imbalanced_learn2\joblib-1.1.0-py2.py3-none-any.whl
Collecting numpy>=1.14.6
  Downloading numpy-1.21.5-cp37-cp37m-win_amd64.whl (14.0 MB)
     |████████████████████████████████| 14.0 MB 6.4 MB/s
  Saved c:\users\c58210a\onedrive - experian services corp\documents\proyectos\migracion db\descargar librerias\test\imbalanced_learn2\numpy-1.21.5-cp37-cp37m-win_amd64.whl
Successfully downloaded imbalanced-learn threadpoolctl scipy scikit-learn joblib numpy

Example download in UNIX:

Collecting imbalanced_learn
  Downloading imbalanced_learn-0.9.0-py3-none-any.whl (199 kB)
  Saved ./s3:/experian-s3-ascend-colombia-internal-buro-data/TempJC/imbalanced_learn-0.9.0-py3-none-any.whl
Collecting numpy>=1.14.6
  Downloading numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
  Saved ./s3:/experian-s3-ascend-colombia-internal-buro-data/TempJC/numpy-1.21.5-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Collecting joblib>=0.11
  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
  Saved ./s3:/experian-s3-ascend-colombia-internal-buro-data/TempJC/joblib-1.1.0-py2.py3-none-any.whl
Collecting scikit-learn>=1.0.1
  Downloading scikit_learn-1.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (24.8 MB)
  Saved ./s3:/experian-s3-ascend-colombia-internal-buro-data/TempJC/scikit_learn-1.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
  Saved ./s3:/experian-s3-ascend-colombia-internal-buro-data/TempJC/threadpoolctl-3.1.0-py3-none-any.whl
Collecting scipy>=1.1.0
  Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
  Saved ./s3:/experian-s3-ascend-colombia-internal-buro-data/TempJC/scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Successfully downloaded imbalanced-learn numpy joblib scikit-learn threadpoolctl scipy

Regards

ANSWER

Answered 2022-Feb-09 at 12:20

Use --platform ?
On my Ubuntu x86_64 :

$ pip download --only-binary=:all: --platform win_amd64 imbalanced_learn
[...]
Saved ./imbalanced_learn-0.9.0-py3-none-any.whl
Saved ./joblib-1.1.0-py2.py3-none-any.whl
Saved ./numpy-1.22.2-cp39-cp39-win_amd64.whl
Saved ./scikit_learn-1.0.2-cp39-cp39-win_amd64.whl
Saved ./scipy-1.8.0-cp39-cp39-win_amd64.whl
Saved ./threadpoolctl-3.1.0-py3-none-any.whl
[...]

$ pip download --only-binary=:all: --platform manylinux2014_x86_64 imbalanced_learn
[...]
Saved ./imbalanced_learn-0.9.0-py3-none-any.whl
Saved ./joblib-1.1.0-py2.py3-none-any.whl
Saved ./numpy-1.22.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./scikit_learn-1.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./scipy-1.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Saved ./threadpoolctl-3.1.0-py3-none-any.whl
[...]

Or use another version for manylinux.

Source https://stackoverflow.com/questions/71034816

QUESTION

Cannot find conda info. Please verify your conda installation on EMR

Asked 2022-Feb-05 at 00:17

I am trying to install conda on EMR and below is my bootstrap script, it looks like conda is getting installed but it is not getting added to environment variable. When I manually update the $PATH variable on EMR master node, it can identify conda. I want to use conda on Zeppelin.

I also tried adding condig into configuration like below while launching my EMR instance however I still get the below mentioned error.

    "classification": "spark-env",
    "properties": {
        "conda": "/home/hadoop/conda/bin"
    }

[hadoop@ip-172-30-5-150 ~]$ PATH=/home/hadoop/conda/bin:$PATH
[hadoop@ip-172-30-5-150 ~]$ conda
usage: conda [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

#!/usr/bin/env bash


# Install conda
wget https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda


conda config --set always_yes yes --set changeps1 no
conda install conda=4.2.13
conda config -f --add channels conda-forge
rm ~/miniconda.sh
echo bootstrap_conda.sh completed. PATH now: $PATH
export PYSPARK_PYTHON="/home/hadoop/conda/bin/python3.5"

echo -e '\nexport PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda create -n zoo python=3.7 # "zoo" is conda environment name, you can use any name you like.
conda activate zoo
sudo pip3 install tensorflow
sudo pip3 install boto3
sudo pip3 install botocore
sudo pip3 install numpy
sudo pip3 install pandas
sudo pip3 install scipy
sudo pip3 install s3fs
sudo pip3 install matplotlib
sudo pip3 install -U tqdm
sudo pip3 install -U scikit-learn
sudo pip3 install -U scikit-multilearn
sudo pip3 install xlutils
sudo pip3 install natsort
sudo pip3 install pydot
sudo pip3 install python-pydot
sudo pip3 install python-pydot-ng
sudo pip3 install pydotplus
sudo pip3 install h5py
sudo pip3 install graphviz
sudo pip3 install recmetrics
sudo pip3 install openpyxl
sudo pip3 install xlrd
sudo pip3 install xlwt
sudo pip3 install tensorflow.io
sudo pip3 install Cython
sudo pip3 install ray
sudo pip3 install zoo
sudo pip3 install analytics-zoo
sudo pip3 install analytics-zoo[ray]
#sudo /usr/bin/pip-3.6 install -U imbalanced-learn

ANSWER

Answered 2022-Feb-05 at 00:17

I got the conda working by modifying the script as below, emr python versions were colliding with the conda version.:

wget https://repo.anaconda.com/miniconda/Miniconda3-py37_4.9.2-Linux-x86_64.sh  -O /home/hadoop/miniconda.sh \
    && /bin/bash ~/miniconda.sh -b -p $HOME/conda

echo -e '\n export PATH=$HOME/conda/bin:$PATH' >> $HOME/.bashrc && source $HOME/.bashrc


conda config --set always_yes yes --set changeps1 no
conda config -f --add channels conda-forge


conda create -n zoo python=3.7 # "zoo" is conda environment name
conda init bash
source activate zoo
conda install python 3.7.0 -c conda-forge orca 
sudo /home/hadoop/conda/envs/zoo/bin/python3.7 -m pip install virtualenv

and setting zeppelin python and pyspark parameters to:

“spark.pyspark.python": "/home/hadoop/conda/envs/zoo/bin/python3",
"spark.pyspark.virtualenv.enabled": "true",
"spark.pyspark.virtualenv.type":"native",
"spark.pyspark.virtualenv.bin.path":"/home/hadoop/conda/envs/zoo/bin/,
"zeppelin.pyspark.python" : "/home/hadoop/conda/bin/python",
"zeppelin.python": "/home/hadoop/conda/bin/python"

Orca only support TF upto 1.5 hence it was not working as I am using TF2.

Source https://stackoverflow.com/questions/70901724

QUESTION

A problem in using AIF360 metrics in my code

Asked 2022-Jan-29 at 15:28

I am trying to run AI Fairness 360 metrics on skit-learn (imbalanced-learn) algorithms, but I have a problem with my code. The problem is when I apply skit-learn (imbalanced-learn) algorithms like SMOTE, it return a numpy array. While AI Fairness 360 preprocessing methods return BinaryLabelDataset. Then the metrics should receive an object from BinaryLabelDataset class. I am stuck in how to convert my arrays to BinaryLabelDataset to be able to use measures.

My preprocessing algorithm needs to receive X,Y. So, I split the dataset before calling SMOTE method into X and Y. The dataset before using SMOTE was standard_dataset and it was ok to use metrics, but the problem after I used SMOTE method because it converts data to numpy array.

I got the following error after running the code :

File "adult_dataset.py", line 654, in <module>
cm = BinaryLabelDatasetMetric(y_pred, privileged_groups=p, unprivileged_groups=u)

raise TypeError("'dataset' should be a BinaryLabelDataset or a MulticlassLabelDataset")
TypeError: 'dataset' should be a BinaryLabelDataset or a MulticlassLabelDataset

Here is my code:

# dataset_orig is standard_dataset
scaler = MinMaxScaler(copy=False)
dataset_orig.features = scaler.fit_transform(dataset_orig.features)

# Spilitting data to X and Y 
X_orig = dataset_orig.features
y_orig = dataset_orig.labels

# SMOTETomek returns X_resampled, y_resampled as numpy arrays
smote_tomek = SMOTETomek(random_state=0)
X_resampled, y_resampled = smote_tomek.fit_resample(X_orig, X_orig)


#Creation of Train and Test dataset
X_train, X_test, y_train, y_test = 
train_test_split(X_resampled,y_resampled,test_size=0.2,random_state=42)


model.fit(X_train, y_train)
y_pred = model.predict(X_test)

p = [{'sex': 1.}]
u = [{'sex': 0.}]
cm = BinaryLabelDatasetMetric(y_pred, privileged_groups=p, unprivileged_groups=u)
print("Disparate_Impact", cm.disparate_impact())
print("Statistical Parity Difference", cm.statistical_parity_difference())
print("Consistency (Individual Fairness)", cm.consistency())

I think the problem with y_pred when calling BinaryLabelDatasetMetric. It should to be BinaryLabelDataset. Is there any way to use AIF360 Metrics in my code?

ANSWER

Answered 2021-Sep-21 at 17:34

You are correct that the problem is with y_pred. You can concatenate it to X_test, transform it to a StandardDataset object, and then pass that one to the BinaryLabelDatasetMetric. The output object will have the methods for calculating different fairness metrics. I do not know how your dataset looks like, but here is a complete reproducible example that you can adapt to do this process for your dataset.

Source https://stackoverflow.com/questions/69082773

QUESTION

Solving conda environment stuck

Asked 2021-Dec-22 at 18:02

I'm trying to install conda environment using the command:

conda env create -f devenv.yaml

My .yaml file is

name: myname
channels:
  - conda-forge
  - bioconda
dependencies:
  # Package creation and environment management
  - conda-build
  # Automation control (command line interface, workflow and multi-process management)
  - python-dotenv
  - click
  - snakemake-minimal
  - joblib
  - numba
  # Workspace
  - notebook
  # Visualization
  - plotly
  - plotly-orca
  - matplotlib
  - seaborn
  - shap
  - openpyxl
  - ipywidgets
  - tensorboard
  # Data manipulation
  - numpy
  - pandas
  - pyarrow
  # Functional style tools
  - more-itertools
  - toolz
  # Machine learning
  - scikit-learn
  - imbalanced-learn
  - scikit-image
  - statsmodels
  - catboost
  - hyperopt
  - tsfresh
  # Deep learning
  - pytorch
  # code checking and formatting
  - pylint
  - black
  - flake8
  - mypy
  # Python base
  - python
  - pip
  - pip:

I've tried to update conda but it doesn't help. It just stuck on solving the environment.

conda version: 4.11.0 c OS: Ubuntu 18.04.5 LTS

The exact same environment works fine on my mac, but not on that server. What could be the issue? I appreciate any suggestions. Thx.

ANSWER

Answered 2021-Dec-22 at 18:02

This solves fine (), but is indeed a complex solve mainly due to:

underspecification
lack of modularization

Underspecification

This particular environment specification ends up installing well over 300 packages. And there isn't a single one of those that are constrained by the specification. That is a huge SAT problem to solve and Conda will struggle with this. Mamba will help solve faster, but providing additional constraints can vastly reduce the solution space.

At minimum, specify a Python version (major.minor), such as python=3.9. This is the single most effective constraint.

Beyond that, putting minimum requirements on central packages (those that are dependencies of others) can help, such as minimum NumPy.

Lack of Modularization

I assume the name "devenv" means this is a development environment. So, I get that one wants all these tools immediately at hand. However, Conda environment activation is so simple, and most IDE tooling these days (Spyder, VSCode, Jupyter) encourages separation of infrastructure and the execution kernel. Being more thoughtful about how environments (emphasis on the plural) are organized and work together, can go a long way in having a sustainable and painless data science workflow.

The environment at hand has multiple red flags in my book:

conda-build should be in base and only in base
snakemake should be in a dedicated environment
notebook (i.e., Jupyter) should be in a dedicated environment, co-installed with nb_conda_kernels; all kernel environments need are ipykernel

I'd probably also have the linting/formatting packages separated, but that's less an issue. The real killer though is snakemake - it's just a massive piece of infrastructure and I'd strongly encourage keeping that separated.

Source https://stackoverflow.com/questions/70451652

QUESTION

Multipoint(df['geometry']) key error from dataframe but key exist. KeyError: 13 geopandas

Asked 2021-Oct-11 at 14:51

data source: https://catalog.data.gov/dataset/nyc-transit-subway-entrance-and-exit-data

I tried looking for a similar problem but I can't find an answer and the error does not help much. I'm kinda frustrated at this point. Thanks for the help. I'm calculating the closest distance from a point.

    df_subway = pd.read_csv('/content/drive/MyDrive/Despliegue_de_modelos/NYC_Transit_Subway_Entrance_And_Exit_Data.csv')

geometry = [Point(xy) for xy in zip(df_subway['Station Longitude'], df_subway['Station Latitude'])]

# Coordinate reference system :
crs = {'init': 'EPSG:4326'}

# Creating a Geographic data frame 
gdf_subway_entrance_geometry = gpd.GeoDataFrame(df_subway, crs=crs, geometry=geometry).to_crs('EPSG:5234')
gdf_subway_entrance_geometry

df_yes_entry = gdf_subway_entrance_geometry[gdf_subway_entrance_geometry.Entry=='YES']
df_yes_entry

from shapely.geometry import Point, MultiPoint
from shapely.ops import nearest_points
pts = MultiPoint(df_yes_entry['geometry']) #it fails in this line
pt = Point(gpdPoint.x, gpdPoint.y)
#[o.wkt for o in nearest_points(pt, pts)]
for o in nearest_points(pt, pts):
  print(o)

The problem is that if I do the same but with gdf_subway_entrance_geometry instead of df_yes_entry it works, but I need to do some filters!

This is the error:
---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2897             try:
-> 2898                 return self._engine.get_loc(casted_key)
   2899             except KeyError as err:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 13


    The above exception was the direct cause of the following exception:
    
    KeyError                                  Traceback (most recent call last)
    
    7 frames
    
    <ipython-input-480-b6733def46e0> in <module>()
          1 from shapely.geometry import Point, MultiPoint
          2 from shapely.ops import nearest_points
    ----> 3 pts = MultiPoint(df_yes_entry['geometry'])
          4 pt = Point(gpdPoint.x, gpdPoint.y)
          5 #[o.wkt for o in nearest_points(pt, pts)]
    
    /usr/local/lib/python3.7/dist-packages/shapely/geometry/multipoint.py in __init__(self, points)
         56             pass
         57         else:
    ---> 58             self._geom, self._ndim = geos_multipoint_from_py(points)
         59 
         60     def shape_factory(self, *args):
    
    /usr/local/lib/python3.7/dist-packages/shapely/geometry/multipoint.py in geos_multipoint_from_py(ob)
        169     # add to coordinate sequence
        170     for i in range(m):
    --> 171         coords = ob[i]
        172         geom, ndims = point.geos_point_from_py(coords)
        173 
    
    /usr/local/lib/python3.7/dist-packages/geopandas/geoseries.py in __getitem__(self, key)
        606 
        607     def __getitem__(self, key):
    --> 608         return self._wrapped_pandas_method("__getitem__", key)
        609 
        610     @doc(pd.Series)
    
    /usr/local/lib/python3.7/dist-packages/geopandas/geoseries.py in _wrapped_pandas_method(self, mtd, *args, **kwargs)
        599     def _wrapped_pandas_method(self, mtd, *args, **kwargs):
        600         """Wrap a generic pandas method to ensure it returns a GeoSeries"""
    --> 601         val = getattr(super(), mtd)(*args, **kwargs)
        602         if type(val) == Series:
        603             val.__class__ = GeoSeries
    
    /usr/local/lib/python3.7/dist-packages/pandas/core/series.py in __getitem__(self, key)
        880 
        881         elif key_is_scalar:
    --> 882             return self._get_value(key)
        883 
        884         if is_hashable(key):
    
    /usr/local/lib/python3.7/dist-packages/pandas/core/series.py in _get_value(self, label, takeable)
        988 
        989         # Similar to Index.get_value, but we do not fall back to positional
    --> 990         loc = self.index.get_loc(label)
        991         return self.index._get_values_for_loc(self, loc, label)
        992 
    
    /usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
       2898                 return self._engine.get_loc(casted_key)
       2899             except KeyError as err:
    -> 2900                 raise KeyError(key) from err
       2901 
       2902         if tolerance is not None:
    
    KeyError: 13

I am working with colab this are my packages:

Package                       Version
----------------------------- --------------
absl-py                       0.12.0
alabaster                     0.7.12
albumentations                0.1.12
altair                        4.1.0
appdirs                       1.4.4
argcomplete                   1.12.3
argon2-cffi                   21.1.0
arviz                         0.11.4
astor                         0.8.1
astropy                       4.3.1
astunparse                    1.6.3
atari-py                      0.2.9
atomicwrites                  1.4.0
attrs                         21.2.0
audioread                     2.1.9
autograd                      1.3
Babel                         2.9.1
backcall                      0.2.0
beautifulsoup4                4.6.3
bleach                        4.1.0
blis                          0.4.1
bokeh                         2.4.0
Bottleneck                    1.3.2
branca                        0.4.2
bs4                           0.0.1
CacheControl                  0.12.6
cached-property               1.5.2
cachetools                    4.2.4
Cartopy                       0.19.0.post1
catalogue                     1.0.0
certifi                       2021.5.30
cffi                          1.14.6
cftime                        1.5.1
chardet                       3.0.4
charset-normalizer            2.0.6
clang                         5.0
click                         7.1.2
click-plugins                 1.1.1
cligj                         0.7.2
cloudpickle                   1.3.0
cmake                         3.12.0
cmdstanpy                     0.9.5
colorcet                      2.0.6
colorlover                    0.3.0
community                     1.0.0b1
contextlib2                   0.5.5
convertdate                   2.3.2
coverage                      3.7.1
coveralls                     0.5
crcmod                        1.7
cufflinks                     0.17.3
cvxopt                        1.2.7
cvxpy                         1.0.31
cycler                        0.10.0
cymem                         2.0.5
Cython                        0.29.24
daft                          0.0.4
dask                          2.12.0
datascience                   0.10.6
debugpy                       1.0.0
decorator                     4.4.2
defusedxml                    0.7.1
descartes                     1.1.0
dill                          0.3.4
distributed                   1.25.3
dlib                          19.18.0
dm-tree                       0.1.6
docopt                        0.6.2
docutils                      0.17.1
dopamine-rl                   1.0.5
earthengine-api               0.1.284
easydict                      1.9
ecos                          2.0.7.post1
editdistance                  0.5.3
en-core-web-sm                2.2.5
entrypoints                   0.3
ephem                         4.1
et-xmlfile                    1.1.0
fa2                           0.3.5
fastai                        1.0.61
fastdtw                       0.3.4
fastprogress                  1.0.0
fastrlock                     0.6
fbprophet                     0.7.1
feather-format                0.4.1
filelock                      3.3.0
Fiona                         1.8.20
firebase-admin                4.4.0
fix-yahoo-finance             0.0.22
Flask                         1.1.4
flatbuffers                   1.12
folium                        0.8.3
future                        0.16.0
gast                          0.4.0
GDAL                          2.2.2
gdown                         3.6.4
gensim                        3.6.0
geographiclib                 1.52
geopandas                     0.10.1
geopy                         1.17.0
geoviews                      1.9.2
gin-config                    0.4.0
glob2                         0.7
google                        2.0.3
google-api-core               1.26.3
google-api-python-client      1.12.8
google-auth                   1.35.0
google-auth-httplib2          0.0.4
google-auth-oauthlib          0.4.6
google-cloud-bigquery         1.21.0
google-cloud-bigquery-storage 1.1.0
google-cloud-core             1.0.3
google-cloud-datastore        1.8.0
google-cloud-firestore        1.7.0
google-cloud-language         1.2.0
google-cloud-storage          1.18.1
google-cloud-translate        1.5.0
google-colab                  1.0.0
google-pasta                  0.2.0
google-resumable-media        0.4.1
googleapis-common-protos      1.53.0
googledrivedownloader         0.4
graphviz                      0.10.1
greenlet                      1.1.2
grpcio                        1.41.0
gspread                       3.0.1
gspread-dataframe             3.0.8
gym                           0.17.3
h5py                          3.1.0
HeapDict                      1.0.1
hijri-converter               2.2.2
holidays                      0.10.5.2
holoviews                     1.14.6
html5lib                      1.0.1
htmlmin                       0.1.12
httpimport                    0.5.18
httplib2                      0.17.4
httplib2shim                  0.0.3
humanize                      0.5.1
hyperopt                      0.1.2
ideep4py                      2.0.0.post3
idna                          2.10
ImageHash                     4.2.1
imageio                       2.4.1
imagesize                     1.2.0
imbalanced-learn              0.4.3
imblearn                      0.0
imgaug                        0.2.9
importlib-metadata            4.8.1
importlib-resources           5.2.2
imutils                       0.5.4
inflect                       2.1.0
iniconfig                     1.1.1
intel-openmp                  2021.4.0
intervaltree                  2.1.0
ipykernel                     4.10.1
ipython                       5.5.0
ipython-genutils              0.2.0
ipython-sql                   0.3.9
ipywidgets                    7.6.5
itsdangerous                  1.1.0
jax                           0.2.21
jaxlib                        0.1.71+cuda111
jdcal                         1.4.1
jedi                          0.18.0
jieba                         0.42.1
Jinja2                        2.11.3
joblib                        1.0.1
jpeg4py                       0.1.4
jsonschema                    2.6.0
jupyter                       1.0.0
jupyter-client                5.3.5
jupyter-console               5.2.0
jupyter-core                  4.8.1
jupyterlab-pygments           0.1.2
jupyterlab-widgets            1.0.2
kaggle                        1.5.12
kapre                         0.3.5
keras                         2.6.0
Keras-Preprocessing           1.1.2
keras-vis                     0.4.1
kiwisolver                    1.3.2
korean-lunar-calendar         0.2.1
librosa                       0.8.1
lightgbm                      2.2.3
llvmlite                      0.34.0
lmdb                          0.99
LunarCalendar                 0.0.9
lxml                          4.2.6
Markdown                      3.3.4
MarkupSafe                    2.0.1
matplotlib                    3.2.2
matplotlib-inline             0.1.3
matplotlib-venn               0.11.6
missingno                     0.5.0
mistune                       0.8.4
mizani                        0.6.0
mkl                           2019.0
mlxtend                       0.14.0
more-itertools                8.10.0
moviepy                       0.2.3.5
mpmath                        1.2.1
msgpack                       1.0.2
multimethod                   1.6
multiprocess                  0.70.12.2
multitasking                  0.0.9
munch                         2.5.0
murmurhash                    1.0.5
music21                       5.5.0
natsort                       5.5.0
nbclient                      0.5.4
nbconvert                     5.6.1
nbformat                      5.1.3
nest-asyncio                  1.5.1
netCDF4                       1.5.7
networkx                      2.6.3
nibabel                       3.0.2
nltk                          3.2.5
notebook                      5.3.1
numba                         0.51.2
numexpr                       2.7.3
numpy                         1.19.5
nvidia-ml-py3                 7.352.0
oauth2client                  4.1.3
oauthlib                      3.1.1
okgrade                       0.4.3
opencv-contrib-python         4.1.2.30
opencv-python                 4.1.2.30
openpyxl                      2.5.9
opt-einsum                    3.3.0
osqp                          0.6.2.post0
packaging                     21.0
palettable                    3.3.0
pandas                        1.1.5
pandas-datareader             0.9.0
pandas-gbq                    0.13.3
pandas-profiling              3.1.0
pandocfilters                 1.5.0
panel                         0.12.4
param                         1.11.1
parso                         0.8.2
pathlib                       1.0.1
patsy                         0.5.2
pep517                        0.11.0
pexpect                       4.8.0
phik                          0.12.0
pickleshare                   0.7.5
Pillow                        7.1.2
pip                           21.1.3
pip-tools                     6.2.0
plac                          1.1.3
plotly                        4.4.1
plotnine                      0.6.0
pluggy                        0.7.1
pooch                         1.5.1
portpicker                    1.3.9
prefetch-generator            1.0.1
preshed                       3.0.5
prettytable                   2.2.1
progressbar2                  3.38.0
prometheus-client             0.11.0
promise                       2.3
prompt-toolkit                1.0.18
protobuf                      3.17.3
psutil                        5.4.8
psycopg2                      2.7.6.1
ptyprocess                    0.7.0
py                            1.10.0
pyarrow                       3.0.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycocotools                   2.0.2
pycparser                     2.20
pyct                          0.4.8
pydantic                      1.8.2
pydata-google-auth            1.2.0
pydot                         1.3.0
pydot-ng                      2.0.0
pydotplus                     2.0.2
PyDrive                       1.3.1
pyemd                         0.5.1
pyerfa                        2.0.0
pyglet                        1.5.0
Pygments                      2.6.1
pygobject                     3.26.1
pymc3                         3.11.4
PyMeeus                       0.5.11
pymongo                       3.12.0
pymystem3                     0.2.0
PyOpenGL                      3.1.5
pyparsing                     2.4.7
pyproj                        3.2.1
pyrsistent                    0.18.0
pyshp                         2.1.3
pysndfile                     1.3.8
PySocks                       1.7.1
pystan                        2.19.1.1
pytest                        3.6.4
python-apt                    0.0.0
python-chess                  0.23.11
python-dateutil               2.8.2
python-louvain                0.15
python-slugify                5.0.2
python-utils                  2.5.6
pytz                          2018.9
pyviz-comms                   2.1.0
PyWavelets                    1.1.1
PyYAML                        5.4.1
pyzmq                         22.3.0
qdldl                         0.1.5.post0
qtconsole                     5.1.1
QtPy                          1.11.2
regex                         2019.12.20
requests                      2.26.0
requests-oauthlib             1.3.0
resampy                       0.2.2
retrying                      1.3.3
rpy2                          3.4.5
rsa                           4.7.2
scikit-image                  0.16.2
scikit-learn                  0.22.2.post1
scipy                         1.7.1
screen-resolution-extra       0.0.0
scs                           2.1.4
seaborn                       0.11.2
semver                        2.13.0
Send2Trash                    1.8.0
setuptools                    57.4.0
setuptools-git                1.2
Shapely                       1.7.1
simplegeneric                 0.8.1
six                           1.15.0
sklearn                       0.0
sklearn-pandas                1.8.0
smart-open                    5.2.1
snowballstemmer               2.1.0
sortedcontainers              2.4.0
SoundFile                     0.10.3.post1
spacy                         2.2.4
Sphinx                        1.8.5
sphinxcontrib-serializinghtml 1.1.5
sphinxcontrib-websupport      1.2.4
SQLAlchemy                    1.4.25
sqlparse                      0.4.2
srsly                         1.0.5
statsmodels                   0.10.2
sympy                         1.7.1
tables                        3.4.4
tabulate                      0.8.9
tangled-up-in-unicode         0.1.0
tblib                         1.7.0
tensorboard                   2.6.0
tensorboard-data-server       0.6.1
tensorboard-plugin-wit        1.8.0
tensorflow                    2.6.0
tensorflow-datasets           4.0.1
tensorflow-estimator          2.6.0
tensorflow-gcs-config         2.6.0
tensorflow-hub                0.12.0
tensorflow-metadata           1.2.0
tensorflow-probability        0.14.1
termcolor                     1.1.0
terminado                     0.12.1
testpath                      0.5.0
text-unidecode                1.3
textblob                      0.15.3
Theano-PyMC                   1.1.2
thinc                         7.4.0
tifffile                      2021.8.30
toml                          0.10.2
tomli                         1.2.1
toolz                         0.11.1
torch                         1.9.0+cu111
torchsummary                  1.5.1
torchtext                     0.10.0
torchvision                   0.10.0+cu111
tornado                       5.1.1
tqdm                          4.62.3
traitlets                     5.1.0
tweepy                        3.10.0
typeguard                     2.7.1
typing-extensions             3.10.0.2
tzlocal                       1.5.1
uritemplate                   3.0.1
urllib3                       1.24.3
vega-datasets                 0.9.0
visions                       0.7.4
wasabi                        0.8.2
wcwidth                       0.2.5
webencodings                  0.5.1
Werkzeug                      1.0.1
wheel                         0.37.0
widgetsnbextension            3.5.1
wordcloud                     1.5.0
wrapt                         1.12.1
xarray                        0.18.2
xgboost                       0.90
xkit                          0.0.0
xlrd                          1.1.0
xlwt                          1.3.0
yellowbrick                   0.9.1
zict                          2.0.0
zipp                          3.6.0

ANSWER

Answered 2021-Oct-11 at 14:21

geopandas 0.10.1

have noted that your data is on kaggle, so start by sourcing it
there really is only one issue shapely.geometry.MultiPoint() constructor does not work with a filtered series. Pass it a numpy array instead and it works.
full code below, have randomly selected a point to serve as gpdPoint

# https://www.kaggle.com/new-york-state/nys-nyc-transit-subway-entrance-and-exit-data
import kaggle.cli
import sys, requests, urllib
import pandas as pd
from pathlib import Path
from zipfile import ZipFile

# fmt: off
# download data set
url = "https://www.kaggle.com/new-york-state/nys-nyc-transit-subway-entrance-and-exit-data"
sys.argv = [sys.argv[0]] + f"datasets download {urllib.parse.urlparse(url).path[1:]}".split(" ")
kaggle.cli.main()
zfile = ZipFile(f'{urllib.parse.urlparse(url).path.split("/")[-1]}.zip')
dfs = {f.filename: pd.read_csv(zfile.open(f)) for f in zfile.infolist() if Path(f.filename).suffix in [".csv"]}
# fmt: on

df_subway = dfs['nyc-transit-subway-entrance-and-exit-data.csv']

from shapely.geometry import Point, MultiPoint
from shapely.ops import nearest_points
import geopandas as gpd

geometry = [Point(xy) for xy in zip(df_subway['Station Longitude'], df_subway['Station Latitude'])]

# Coordinate reference system :
crs = {'init': 'EPSG:4326'}

# Creating a Geographic data frame 
gdf_subway_entrance_geometry = gpd.GeoDataFrame(df_subway, crs=crs, geometry=geometry).to_crs('EPSG:5234')
gdf_subway_entrance_geometry

df_yes_entry = gdf_subway_entrance_geometry
df_yes_entry = gdf_subway_entrance_geometry[gdf_subway_entrance_geometry.Entry=='YES']
df_yes_entry

# randomly select a point....
gpdPoint = gdf_subway_entrance_geometry.sample(1).geometry.tolist()[0]
pts = MultiPoint(df_yes_entry['geometry'].values) # does not work with a geopandas series, works with a numpy array
pt = Point(gpdPoint.x, gpdPoint.y)
#[o.wkt for o in nearest_points(pt, pts)]
for o in nearest_points(pt, pts):
  print(o)

Source https://stackoverflow.com/questions/69521034

QUESTION

getting scikit-learn version warning using yml environment

Asked 2021-Aug-13 at 16:38

I want to deploy a machine learning model and have the environment yml file and the model pickle file. When I include scikit-learn=0.23.2 to the dependencies, conda automatically uninstall this scikit-learn version and install scikit-learn-0.24.2 . Therefore, I get the following warning when I load the pickle file.

UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.23.2 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.

Here is the environment:

name: environment
channels:
  - defaults
dependencies:
  - blas=1.0
  - ca-certificates=2021.5.25
  - certifi=2021.5.30
  - icc_rt=2019.0.0
  - intel-openmp=2021.2.0
  - joblib=0.17.0
  - mkl=2020.2
  - mkl-service=2.3.0
  - mkl_fft=1.3.0
  - mkl_random=1.1.1
  - numpy=1.19.2
  - numpy-base=1.19.2
  - openssl=1.1.1k
  - pandas=1.2.4
  - patsy=0.5.1
  - pickleshare=0.7.5
  - pip=21.1.1
  - pyodbc=4.0.30
  - python=3.7.4
  - python-dateutil
  - pytz=2021.1
  - scipy=1.6.2
  - setuptools=52.0.0
  - six=1.15.0
  - sqlite=3.35.4
  - statsmodels=0.12.0
  - threadpoolctl=2.1.0
  - vc=14.2
  - vs2015_runtime=14.27.29016
  - wheel=0.36.2
  - wincertstore=0.2
  - scikit-learn=0.23.2
  - pip:
    - imblearn==0.0
prefix: C:\Users

And the result of conda env create -f environment.yml is:

Installing collected packages: scikit-learn, imbalanced-learn, imblearn
  Attempting uninstall: scikit-learn
    Found existing installation: scikit-learn 0.23.2
    Uninstalling scikit-learn-0.23.2:
      Successfully uninstalled scikit-learn-0.23.2
Successfully installed imbalanced-learn-0.8.0 imblearn-0.0 scikit-learn-0.24.2

I also tried to install scikit-learn=0.23.2 via pip and I didn't get the warning in my local machine while loading the model. But scikit-learn should not installed via pip in the deployment environment. Do you have any idea?

ANSWER

Answered 2021-Aug-13 at 16:38

Explanation

Whatever is in the pip: section of a Conda environment YAML gets installed after the Conda environment is created, and is run with the pip install -U command. The -U gives Pip the permission to upgrade any existing packages if it is necessary to install the specified packages. In this particular case, the version of imblearn must be incompatible with the scikit-learn version you have selected.

Remove `imblearn`

Technically, you should be using imbalanced-learn not imblearn, as stated in the package description. That also means you don't even need to install from PyPI, since imbalanced-learn is available through Conda Forge.

If you require having scikit-learn=0.23 then you must use imbalanced-learn=0.7. This should be under the regular dependencies, not in the pip: section.

Source https://stackoverflow.com/questions/68770542

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install imbalanced-learn

You can install using 'pip install imbalanced-learn' or download it from GitHub, PyPI.
You can use imbalanced-learn like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .