Sign in with
By continuing you indicate that you have read and agree to our Terms of service and Privacy policy
pycaret | An open-source , low-code machine learning library in Python | Machine Learning library
by pycaret Jupyter Notebook Version: 3.0.0rc6 License: MIT
by pycaret Jupyter Notebook Version: 3.0.0rc6 License: MIT
kandi X-RAY | pycaret Summary
pycaret is a Jupyter Notebook library typically used in Artificial Intelligence, Machine Learning, Deep Learning, Tensorflow applications. pycaret has no bugs, it has no vulnerabilities, it has a Permissive License and it has medium support. You can download it from GitHub.
PyCaret is an open-source, low-code machine learning library in Python that automates machine learning workflows. It is an end-to-end machine learning and model management tool that speeds up the experiment cycle exponentially and makes you more productive. In comparison with the other open-source machine learning libraries, PyCaret is an alternate low-code library that can be used to replace hundreds of lines of code with few lines only. This makes experiments exponentially fast and efficient. PyCaret is essentially a Python wrapper around several machine learning libraries and frameworks such as scikit-learn, XGBoost, LightGBM, CatBoost, spaCy, Optuna, Hyperopt, Ray, and few more. The design and simplicity of PyCaret are inspired by the emerging role of citizen data scientists, a term first used by Gartner. Citizen Data Scientists are power users who can perform both simple and moderately sophisticated analytical tasks that would previously have required more technical expertise.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse
Support
- pycaret has a medium active ecosystem.
- It has 6843 star(s) with 1556 fork(s). There are 127 watchers for this library.
- There were 3 major release(s) in the last 6 months.
- There are 260 open issues and 1651 have been closed. On average issues are closed in 49 days. There are 6 open pull requests and 0 closed requests.
- It has a neutral sentiment in the developer community.
- The latest version of pycaret is 3.0.0rc6
pycaret Support
Best in #Machine Learning
Average in #Machine Learning
pycaret Support
Best in #Machine Learning
Average in #Machine Learning
Quality
- pycaret has 0 bugs and 0 code smells.
pycaret Quality
Best in #Machine Learning
Average in #Machine Learning
pycaret Quality
Best in #Machine Learning
Average in #Machine Learning
Security
- pycaret has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
- pycaret code analysis shows 0 unresolved vulnerabilities.
- There are 0 security hotspots that need review.
pycaret Security
Best in #Machine Learning
Average in #Machine Learning
pycaret Security
Best in #Machine Learning
Average in #Machine Learning
License
- pycaret is licensed under the MIT License. This license is Permissive.
- Permissive licenses have the least restrictions, and you can use them in most projects.
pycaret License
Best in #Machine Learning
Average in #Machine Learning
pycaret License
Best in #Machine Learning
Average in #Machine Learning
Reuse
- pycaret releases are available to install and integrate.
- Installation instructions, examples and code snippets are available.
- It has 23953 lines of code, 781 functions and 89 files.
- It has high code complexity. Code complexity directly impacts maintainability of the code.
pycaret Reuse
Best in #Machine Learning
Average in #Machine Learning
pycaret Reuse
Best in #Machine Learning
Average in #Machine Learning
Top functions reviewed by kandi - BETA
kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample Here
Get all kandi verified functions for this library.
Get all kandi verified functions for this library.
pycaret Key Features
An open-source, low-code machine learning library in Python
pycaret Examples and Code Snippets
See all related Code Snippets
Community Discussions
Trending Discussions on pycaret
- is test data used in Pycaret time series(beta) completely unseen by the model(s)?
- get a string without ""
- Pycaret - 'Make_Time_Features' object has no attribute 'list_of_features'
- Pycaret anomaly detection setup: ValueError: Setting a random_state has no effect since shuffle is False
- Weird Time-Series Graph Using Pycaret and plotly
- Creating 12 month MA in PyCaret time series gives a column with NA values
- What does the Anomaly_Score in Pycaret.anomaly library mean?
- How to install PyCaret in AWS Glue
- Unable to install sklearn 0.23.2 with pip 20.2.4 and python 3.9
- Pycaret.regression.compare_models: Evaluation table neither returned nor shown
Trending Discussions on pycaret
QUESTION
is test data used in Pycaret time series(beta) completely unseen by the model(s)?
Asked 2022-Apr-04 at 10:51Post checking official documentation and example, I am still confused if test data passed to the setup function is completely unseen by the model???
from pycaret.datasets import get_data
from pycaret.internal.pycaret_experiment import TimeSeriesExperiment
# get data
y = get_data('airline', verbose=False)
# no of future steps to forecast
fh = 12 # or alternately fh = np.arange(1,13)
fold = 3
# setup
exp = TimeSeriesExperiment()
exp.setup(data=y, fh=fh, fold = fold)
exp.models()
which gives description as:
Also, checked at cv-graph, we can conclude that test data set is not used while cv. But, Still as it's not mentioned anywhere about it, need a concrete evidence.
Train-Test split
Train c-v splits
ANSWER
Answered 2022-Apr-04 at 10:51If you notice the cv splits, they do not use the test data at all. So any step such as create_model, tune_model, blend_model, compare_models that use Cross-Validation, will not use the test data at all for training.
Once you are happy with the models from these steps, you can finalize the model using finalize_model. In this case, whatever model you pass to finalize_model is trained on the complete dataset (train + test) so that you can make true future predictions.
QUESTION
get a string without ""
Asked 2021-Dec-25 at 16:51I am trying to automate the setup-process in PyCaret Library. The goal is to create a list of "experiments" and the script should iretate over the list of elements.
In the setup could be following preprocessing steps chosen: feature_selection = True, remove_outliers = True, etc.
All such pre-processing steps i would like to store in a list and then run a loop in order to find the best setting (best performance) for the model.
For example:
a = "feature_selection = True"
b = "remove_outliers = True"
etc..
But when i call the variable "a" in the setup i get:
File "<ipython-input-31-0d8c8dcb30cf>", line 7
a
^
SyntaxError: positional argument follows keyword argument
because the variable "a" does not the job well.
reg = setup(data = df, silent = True, test_data = dff, target = 'Thickness',
verbose = False,
numeric_features = ['Airbubbles','PreHeat','PlugTemp','AirPres','VacPress','MoldTemp'],
a
)
If i call the variable "a" the following is being shown: 'feature_selection = True'
Thefore i need to store strings for the list in some other way, in order to be displayed as: feature_selection = True (without '') AND accepted by the setup from PyCaret.
ANSWER
Answered 2021-Dec-23 at 23:28Say you have a string like this:
>>> a = "feature_selection = True"
>>> a
"feature_selection = True"
In order to pass that as a keyword argument (kwarg) to a function (e.g., my_func(feature_selection=True)), you need to convert it to a dict:
>>> import re
>>> d = dict([re.split('\s*=\s*', a)])
{'feature_selection': 'True'}
You might want to convert the keys to actual literal values, e.g. 'True' (string) -> True (boolean):
import ast
for k, v in d.items():
d[k] = ast.literal_eval(v)
Then, you can pass it to a function like this:
my_func(**d)
e.g.
reg = setup(data = df, silent = True, test_data = dff, target = 'Thickness',
verbose = False,
numeric_features = ['Airbubbles','PreHeat','PlugTemp','AirPres','VacPress','MoldTemp'],
**d,
)
QUESTION
Pycaret - 'Make_Time_Features' object has no attribute 'list_of_features'
Asked 2021-Oct-22 at 13:02I am trying to create a model using pycaret just as:
from pycaret.classification import *
clf1 = setup(data = dt, target = 'group')
lr = create_model('lr')
Then I get:
AttributeError: 'Simple_Imputer' object has no attribute 'fill_value_categorical'
So, following here, I added:
clf1 = setup(data = dt, target = 'group', imputation_type='iterative' )
lr = create_model('lr')
Then I get:
AttributeError: 'Make_Time_Features' object has no attribute 'list_of_features'
My version of sklearn is 0.23.2 and pycaret is 2.3.2
ANSWER
Answered 2021-Sep-23 at 08:38You mentioned my previous question here.
I just faced the same issue as you on Colab. It is 100% issue with libraries.
Initially, I got the error for SMOTE:
- `AttributeError: 'SMOTE' object has no attribute '_validate_data'
After installing/reinstalling libraries I got exactly your error.
How did I resolve it?
- Started to run Colab and imported all common libraries (
pd,np,scikit, etc). - Installed PyCaret via
pip install. Thenimport pycaretandfrom pycaret.classification import * - Colab reacted: you have issues with
scipy,sklearn,lightgbm, please restart your runtime. - Restarted my runtime on Colab
- Imported all libraries again as I did in step 1
- Ran
import pycaretandfrom pycaret.classification import *only
My final code:
# Initialize the setup with SMOTE
clf_smote = setup(
data,
session_id = 123,
target = 'Target',
remove_multicollinearity = True,
multicollinearity_threshold = 0.95,
fix_imbalance = True
)
I did not use imputation_type='iterative' as in my question above.
Proof of running:
It worked, but it was my solution. Would be great to have a more detailed guide on how to deal with such issues, using this amazing library.
QUESTION
Pycaret anomaly detection setup: ValueError: Setting a random_state has no effect since shuffle is False
Asked 2021-Sep-09 at 12:22I have recently transitioned from R to python, and I am not sure how to problem solve the following.
When I run the setup for pycaret anomaly detection, following the instructions that can be found here, on my own data I get the following error.
# Inital setup
s = setup(data, session_id = 1230)
ValueError: Setting a random_state has no effect since shuffle is False. You should leave random_state to its default (None), or set shuffle=True.
The only difference from the example is that I have some additional xregs (base_price(float64), sale_price(float64), promotion_flag(int64; 0 or 1)). As I understand it, this shouldn't affect the results, after all there are a number of time features. Everything else is the same. So I don't understand why this error is occurring.
I tried the answers on this stackoverflow question, along with adding shuffle=True, but these all resulted in a unexpected keyword argument.
I appreciate this may be a bit basic, but the error doesn't make sense to me, based on what I have done. Thanks for your time.
Python 3.7.10
ANSWER
Answered 2021-Sep-09 at 12:22The answer to this question is that the environment had a library versions such as numpy that were too new for pycaret to work with, for example, pycaret need numpy (1.19.5 and will not work with newer).
My solution was to create a new environment in anaconda, which used pip install pycaret[full], and added nothing else to the environment. It worked after this.
QUESTION
Weird Time-Series Graph Using Pycaret and plotly
Asked 2021-Sep-09 at 10:35I am trying to visualize Air Quality Data as time-series charts using pycaret and plotly dash python libraries , but i am getting very weird graphs, below is my code:
import pandas as pd
import plotly.express as px
data = pd.read_csv('E:/Self Learning/Djang_Dash/2019-2020_5.csv')
data['Date'] = pd.to_datetime(data['Date'], format='%d/%m/%Y')
#data.set_index('Date', inplace=True)
# combine store and item column as time_series
data['OBJECTID'] = ['Location_' + str(i) for i in data['OBJECTID']]
#data['AQI_Bins_AI'] = ['Bin_' + str(i) for i in data['AQI_Bins_AI']]
data['time_series'] = data[['OBJECTID']].apply(lambda x: '_'.join(x), axis=1)
data.drop(['OBJECTID'], axis=1, inplace=True)
# extract features from date
data['month'] = [i.month for i in data['Date']]
data['year'] = [i.year for i in data['Date']]
data['day_of_week'] = [i.dayofweek for i in data['Date']]
data['day_of_year'] = [i.dayofyear for i in data['Date']]
data.head(4000)
data['time_series'].nunique()
for i in data['time_series'].unique():
subset = data[data['time_series'] == i]
subset['moving_average'] = subset['CO'].rolling(window = 30).mean()
fig = px.line(subset, x="Date", y=["CO","moving_average"], title = i, template = 'plotly_dark')
fig.show()
ANSWER
Answered 2021-Sep-09 at 10:35- data has not been provided in a usable way. Sought out publicly available similar data. found: https://www.kaggle.com/rohanrao/air-quality-data-in-india?select=station_hour.csv
- using this data, with a couple of cleanups of your code, no issues with plots. I suspect your data has one of these issues
- date is not
datetime64[ns]in your data frame - date is not sorted, leading to lines being drawn in way you have noted
- date is not
- by refactoring way moving average is calculated, you can use animation instead of lots of separate figures
import kaggle.cli
import sys, math
import pandas as pd
from pathlib import Path
from zipfile import ZipFile
import plotly.express as px
# download data set
# https://www.kaggle.com/rohanrao/air-quality-data-in-india?select=station_hour.csv
sys.argv = [
sys.argv[0]
] + "datasets download rohanrao/air-quality-data-in-india".split(
" "
)
kaggle.cli.main()
zfile = ZipFile("air-quality-data-in-india.zip")
print([f.filename for f in zfile.infolist()])
plot using code from question
import pandas as pd
import plotly.express as px
from pathlib import Path
from distutils.version import StrictVersion
# data = pd.read_csv('E:/Self Learning/Djang_Dash/2019-2020_5.csv')
# use kaggle data
# dfs = {f.filename:pd.read_csv(zfile.open(f)) for f in zfile.infolist() if f.filename in ['station_day.csv',"stations.csv"]}
# data = pd.merge(dfs['station_day.csv'],dfs["stations.csv"], on="StationId")
# data['Date'] = pd.to_datetime(data['Date'])
# # kaggle data is different from question, make it compatible with questions data
# data = data.assign(OBJECTID=lambda d: d["StationId"])
# sample data from google drive link
data2 = pd.read_csv(Path.home().joinpath("Downloads").joinpath("AQI.csv"))
data2["Date"] = pd.to_datetime(data2["Date"])
data = data2
# as per very first commment - it's important data is ordered !
data = data.sort_values(["Date","OBJECTID"])
data['time_series'] = "Location_" + data["OBJECTID"].astype(str)
# clean up data, remove rows where there is no CO value
data = data.dropna(subset=["CO"])
# can do moving average in one step (can also be used by animation)
if StrictVersion(pd.__version__) < StrictVersion("1.3.0"):
data["moving_average"] = data.groupby("time_series",as_index=False)["CO"].rolling(window=30).mean().to_frame()["CO"].values
else:
data["moving_average"] = data.groupby("time_series",as_index=False)["CO"].rolling(window=30).mean()["CO"]
# just first two for purpose of demonstration
for i in data['time_series'].unique()[0:3]:
subset = data.loc[data['time_series'] == i]
fig = px.line(subset, x="Date", y=["CO","moving_average"], title = i, template = 'plotly_dark')
fig.show()
can use animation
px.line(
data,
x="Date",
y=["CO", "moving_average"],
animation_frame="time_series",
template="plotly_dark",
).update_layout(yaxis={"range":[data["CO"].min(), data["CO"].quantile(.97)]})
QUESTION
Creating 12 month MA in PyCaret time series gives a column with NA values
Asked 2021-Aug-19 at 23:02I am trying to use PyCaret for time series, according to this tutorial.
My analysis did not work. When I created a new column
data['MA12'] = data['variable'].rolling(12).mean()
I got this new MA12 column with NA values only.
As a resulted I decided to replicate the code from the tutorial, using AirPassangers dataset, but got the same issue.
When I print data, I get
Month Passengers MA12
0 1949-01-01 112 NaN
1 1949-02-01 118 NaN
2 1949-03-01 132 NaN
3 1949-04-01 129 NaN
4 1949-05-01 121 NaN
I would greatly appreciate any tips on what is going on here.
My only guess, I use a default version of PyCaret, maybe I need to install a full one. Tried this too - the same result.
ANSWER
Answered 2021-Aug-19 at 23:02Since you want the previous 12 reads, the first 11 will be NaN. You need more rows than 12 before you get a moving average of 12. You can see this on the link you provided. The chart of MA doesn't start up right away.
QUESTION
What does the Anomaly_Score in Pycaret.anomaly library mean?
Asked 2021-Aug-12 at 15:14If we use the assign_model() function from pycaret.anomaly library we get a dataframe with two additional columns Anomaly and Anomaly_Score as output. What does the Anomaly_Score column mean in this dataframe and how is its value calculated?
ANSWER
Answered 2021-Aug-12 at 15:14PyCaret uses the PyOd library for anomaly detection. The anomaly score of an input sample is computed based on different detector algorithms. For consistency, outliers are assigned with larger anomaly scores. Wow it is calculated depends on the algorithm used for anomaly detection. Check out the documentation: https://pyod.readthedocs.io/en/latest/pyod.models.html and search for: "anomaly score".
QUESTION
How to install PyCaret in AWS Glue
Asked 2021-Jul-08 at 17:01How can I properly install PyCaret in AWS Glue?
Methods I tried:
--additional-python-modulesand--python-modules-installer-optionPython library patheasy_installas described in Use AWS Glue Python with NumPy and Pandas Python Packages
I am using Glue Version 2.0. I used --additional-python-modules and set to pycaret as shown in the picture.
Then I got this error log.
INFO 2021-07-05 18:12:15,107 18690 com.amazonaws.services.glue.PythonModuleInstaller [main] Collecting pycaret Downloading https://files.pythonhosted.org/packages/da/99/18f151991b0f06107af9723417c64e304ae2133587f85ea734a90136b4ae/pycaret-2.3.1-py3-none-any.whl (261kB)Collecting numpy==1.19.5 (from pycaret) Downloading https://files.pythonhosted.org/packages/b1/e1/8c4c5632adaffc18dba4e03e97458dc1cb00583811e6982fc620b9d88515/numpy-1.19.5-cp37-cp37m-manylinux1_x86_64.whl (13.4MB)Requirement already satisfied: matplotlib in /home/spark/.local/lib/python3.7/site-packages (from pycaret)Collecting pandas-profiling>=2.8.0 (from pycaret) Downloading https://files.pythonhosted.org/packages/3b/a3/34519d16e5ebe69bad30c5526deea2c3912634ced7f9b5e6e0bb9dbbd567/pandas_profiling-3.0.0-py2.py3-none-any.whl (248kB)Collecting wordcloud (from pycaret) Downloading https://files.pythonhosted.org/packages/1b/06/0516bdba2ebdc0d5bd476aa66f94666dd0ad6b9abda723fdf28e451db919/wordcloud-1.8.1-cp37-cp37m-manylinux1_x86_64.whl (366kB)Collecting lightgbm>=2.3.1 (from pycaret) Downloading https://files.pythonhosted.org/packages/18/b2/fff8370f48549ce223f929fe8cab4ee6bf285a41f86037d91312b48ed95b/lightgbm-3.2.1-py3-none-manylinux1_x86_64.whl (2.0MB)Collecting plotly>=4.4.1 (from pycaret) Downloading https://files.pythonhosted.org/packages/95/8d/ac1560f7ccc2ace85cd1e9619bbec1975b5d2d92e6c6fdbbdaa994c6ab4d/plotly-5.1.0-py2.py3-none-any.whl (20.6MB)Collecting umap-learn (from pycaret) Downloading https://files.pythonhosted.org/packages/75/69/85e7f950bb75792ad5d666d86c5f3e62eedbb942848e7e3126513af9999c/umap-learn-0.5.1.tar.gz (80kB)Collecting scikit-plot (from pycaret) Downloading https://files.pythonhosted.org/packages/7c/47/32520e259340c140a4ad27c1b97050dd3254fdc517b1d59974d47037510e/scikit_plot-0.3.7-py3-none-any.whlCollecting Boruta (from pycaret) Downloading https://files.pythonhosted.org/packages/b2/11/583f4eac99d802c79af9217e1eff56027742a69e6c866b295cce6a5a8fc2/Boruta-0.3-py3-none-any.whl (56kB)Collecting pyod (from pycaret) Downloading https://files.pythonhosted.org/packages/71/8a/faa04a753bc32aeef00b9acf8e23d0b914b03844b89dcc6062b28e7ab1c5/pyod-0.9.0.tar.gz (105kB)Collecting yellowbrick>=1.0.1 (from pycaret) Downloading https://files.pythonhosted.org/packages/3a/15/58feb940b6a2f52d3335cccf9e5d00704ec5ba62782da83f7e2abeca5e4b/yellowbrick-1.3.post1-py3-none-any.whl (271kB)Collecting cufflinks>=0.17.0 (from pycaret) Downloading https://files.pythonhosted.org/packages/1a/18/4d32edaaf31ba4af9745dac676c4a28c48d3fc539000c29e855bd8db3b86/cufflinks-0.17.3.tar.gz (81kB)Collecting spacy<2.4.0 (from pycaret) Downloading https://files.pythonhosted.org/packages/79/1c/7c5f7541eb883181b564a8c8ba15d21b2d7b8a38ae32f31763575cf8857d/spacy-2.3.7.tar.gz (5.8MB)
Complete output from command python setup.py egg_info:
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-mrzlr566/blis/
Traceback (most recent call last):
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
subprocess.check_call(cmd)
File "/usr/lib64/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError:
Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp__iwgkr5', '--quiet', 'blis<0.8.0,>=0.4.0']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-build-mafqizyu/spacy/setup.py", line 252, in <module> setup_package()
File "/tmp/pip-build-mafqizyu/spacy/setup.py", line 247, in setup_package cmdclass={"build_ext": build_ext_subclass},
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/__init__.py", line 143, in setup _install_setup_requires(attrs)
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/__init__.py", line 138, in _install_setup_requires
dist.fetch_build_eggs(dist.setup_requires)
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
replace_conflicting=True,
File "/home/spark/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 783, in resolve
replace_conflicting=replace_conflicting
File "/home/spark/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1066, in best_match
return self.obtain(req, installer)
File "/home/spark/.local/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1078, in obtain
return installer(requirement)
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
return fetch_build_egg(self, req)
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
raise DistutilsError(str(e))
distutils.errors.DistutilsError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp__iwgkr5', '--quiet', 'blis<0.8.0,>=0.4.0']'
returned non-zero exit status 1. ----------------------------------------
INFO 2021-07-05 18:12:15,108 18691 com.amazonaws.services.glue.PythonModuleInstaller [main]
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-mafqizyu/spacy/
I tried to exclude spacy from the list of dependencies by downloading PyCaret's source code, removing spacy from requirements.txt, packaging the source code into a whl file, and trying to install PyCaret through the whl file. Then I got an error message saying, Failed building wheel for numba Failed building wheel for llvmlite Failed building wheel
Log:
[truncated because of the limit of number of characters]copying numba/_hashtable.h -> build/lib.linux-x86_64-3.7/numba copying numba/_typeof.h -> build/lib.linux-x86_64-3.7/numba copying numba/_devicearray.h -> build/lib.linux-x86_64-3.7/numba copying numba/_numba_common.h -> build/lib.linux-x86_64-3.7/numba copying numba/typed/py.typed -> build/lib.linux-x86_64-3.7/numba/typed copying numba/misc/cmdlang.gdb -> build/lib.linux-x86_64-3.7/numba/misc copying numba/pycc/modulemixin.c -> build/lib.linux-x86_64-3.7/numba/pycc copying numba/cext/dictobject.c -> build/lib.linux-x86_64-3.7/numba/cext copying numba/cext/listobject.c -> build/lib.linux-x86_64-3.7/numba/cext copying numba/cext/utils.c -> build/lib.linux-x86_64-3.7/numba/cext copying numba/cext/listobject.h -> build/lib.linux-x86_64-3.7/numba/cext copying numba/cext/cext.h -> build/lib.linux-x86_64-3.7/numba/cext copying numba/cext/dictobject.h -> build/lib.linux-x86_64-3.7/numba/cext copying numba/core/runtime/_nrt_pythonmod.c -> build/lib.linux-x86_64-3.7/numba/core/runtime copying numba/core/runtime/nrt.c -> build/lib.linux-x86_64-3.7/numba/core/runtime copying numba/core/runtime/_nrt_python.c -> build/lib.linux-x86_64-3.7/numba/core/runtime copying numba/core/runtime/nrt.h -> build/lib.linux-x86_64-3.7/numba/core/runtime copying numba/core/runtime/nrt_external.h -> build/lib.linux-x86_64-3.7/numba/core/runtime copying numba/core/annotations/template.html -> build/lib.linux-x86_64-3.7/numba/core/annotations copying numba/cuda/tests/cudadrv/data/jitlink.ptx -> build/lib.linux-x86_64-3.7/numba/cuda/tests/cudadrv/data running build_ext building 'numba._dynfunc' extension Warning: Can't read registry to find the necessary compiler setting Make sure that Python modules winreg, win32api or win32con are installed. C compiler: gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC creating build/temp.linux-x86_64-3.7 creating build/temp.linux-x86_64-3.7/numba compile options: '-I/usr/include/python3.7m -c' gcc: numba/_dynfuncmo
d.c error: Command "gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python3.7m -c numba/_dynfuncmod.c -o build/temp.linux-x86_64-3.7/numba/_dynfuncmod.o" failed with exit status 127 ---------------------------------------- Running setup.py clean for numba Running setup.py bdist_wheel for future: started Running setup.py bdist_wheel for future: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e Running setup.py bdist_wheel for sklearn: started Running setup.py bdist_wheel for sklearn: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/76/03/bb/589d421d27431bcd2c6da284d5f2286c8e3b2ea3cf1594c074 Running setup.py bdist_wheel for pynndescent: started Running setup.py bdist_wheel for pynndescent: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/ba/52/4e/4c28d04d144a28f89e2575fb63628df6e6d49b56c5ddd0c74e Running setup.py bdist_wheel for htmlmin: started Running setup.py bdist_wheel for htmlmin: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/43/07/ac/7c5a9d708d65247ac1f94066cf1db075540b85716c30255459 Running setup.py bdist_wheel for phik: started Running setup.py bdist_wheel for phik: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/c0/a3/b0/f27b1cfe32ea131a3715169132ff6d85653789e80e966c3bf6 Running setup.py bdist_wheel for prometheus-flask-exporter: started Running setup.py bdist_wheel for prometheus-flask-exporter: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/c0/e2/9c/4f3ee23964802940f81a8b476d0b9be6fb6348cb12df2e2226 Running setup.py bdist_wheel for alembic: started Running setup.py bdist_wheel for alembic: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/84/07/f7/12f7370ca47a66030c2edeedcc23dec26ea0ac22dcb4c4a0f3 Running setup.py bdist_wheel for databricks-cli: started Running setup.py bdist_wheel for databricks-cli: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/5b/24/f3/34d8e3964dac4ba849d844273c49a679111b00d5799ebb934a Running setup.py bdist_wheel for llvmlite: started Running setup.py bdist_wheel for llvmlite: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ws60mqho/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpoy9cphk5pip-wheel- --python-tag cp37: running bdist_wheel /usr/bin/python3 /tmp/pip-build-ws60mqho/llvmlite/ffi/build.py LLVM version... Traceback (most recent call last): File "/tmp/pip-build-ws60mqho/llvmlite/ffi/build.py", line 220, in <module> main() File "/tmp/pip-build-ws60mqho/llvmlite/ffi/build.py", line 210, in main main_posix('linux', '.so') File "/tmp/pip-build-ws60mqho/llvmlite/ffi/build.py", line 134, in main_posix raise RuntimeError(msg) from None RuntimeError: Could not find a `llvm-config` binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help. error: command '/usr/bin/python3' failed with exit status 1 ---------------------------------------- Running setup.py clean for llvmlite Running setup.py bdist_wheel for bottleneck: started Running setup.py bdist_wheel for bottleneck: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ws60mqho/bottleneck/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpfy0tfce1pip-wheel- --python-tag cp37: running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-3.7 creating build/lib.linux-x86_64-3.7/bottleneck copying bottleneck/__init__.py -> build/lib.linux-x86_64-3.7/bottleneck copying bottleneck/_pytesttester.py -> build/lib.linux-x86_64-3.7/bottleneck copying bottleneck/_version.py -> build/lib.linux-x86_64-3.7/bottleneck creating build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/__init__.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/util.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/input_modification_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/list_input_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/move_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/nonreduce_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/reduce_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/scalar_input_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/memory_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests copying bottleneck/tests/nonreduce_axis_test.py -> build/lib.linux-x86_64-3.7/bottleneck/tests creating build/lib.linux-x86_64-3.7/bottleneck/src copying bottleneck/src/bn_config.py -> build/lib.linux-x86_64-3.7/bottleneck/src copying bottleneck/src/__init__.py -> build/lib.linux-x86_64-3.7/bottleneck/src copying bottleneck/src/bn_template.py -> build/lib.linux-x86_64-3.7/bottleneck/src creating build/lib.linux-x86_64-3.7/bottleneck/benchmark copying bottleneck/benchmark/bench.py -> build/lib.linux-x86_64-3.7/bottleneck/benchmark copying bottleneck/benchmark/autotimeit.py -> build/lib.linux-x86_64-3.7/bottleneck/benchmark copying bottleneck/benchmark/__init__.py -> build/lib.linux-x86_64-3.7/bottleneck/benchmark copying bottleneck/benchmark/bench_detailed.py -> build/lib.linux-x86_64-3.7/bottleneck/benchmark creating build/lib.linux-x86_64-3.7/bottleneck/slow copying bottleneck/slow/nonreduce.py -> build/lib.linux-x86_64-3.7/bottleneck/slow copying bottleneck/slow/move.py -> build/lib.linux-x86_64-3.7/bottleneck/slow copying bottleneck/slow/__init__.py -> build/lib.linux-x86_64-3.7/bottleneck/slow copying bottleneck/slow/reduce.py -> build/lib.linux-x86_64-3.7/bottleneck/slow copying bottleneck/slow/nonreduce_axis.py -> build/lib.linux-x86_64-3.7/bottleneck/slow UPDATING build/lib.linux-x86_64-3.7/bottleneck/_version.py set build/lib.linux-x86_64-3.7/bottleneck/_version.py to '1.3.2' running build_ext running config compiling '_configtest.c': #pragma GCC diagnostic error "-Wattributes" int __attribute__((optimize("O3"))) have_attribute_optimize_opt_3(void*); int main(void) { return 0; } gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -c _configtest.c -o _configtest.o unable to execute 'gcc': No such file or directory failure. removing: _configtest.c _configtest.o compiling '_configtest.c': #ifndef __cplusplus static inline int static_func (void) { return 0; } inline int nostatic_func (void) { return 0; } #endif int main(void) { int r1 = static_func(); int r2 = nostatic_func(); return r1 + r2; } gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -c _configtest.c -o _configtest.o unable to execute 'gcc': No such file or directory failure. removing: _configtest.c _configtest.o compiling '_configtest.c': #ifndef __cplusplus static __inline__ int static_func (void) { return 0; } __inline__ int nostatic_func (void) { return 0; } #endif int main(void) { int r1 = static_func(); int r2 = nostatic_func(); return r1 + r2; } gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -c _configtest.c -o _configtest.o unable to execute 'gcc': No such file or directory failure. removing: _configtest.c _configtest.o compiling '_configtest.c': #ifndef __cplusplus static __inline int static_func (void) { return 0; } __inline int nostatic_func (void) { return 0; } #endif int main(void) { int r1 = static_func(); int r2 = nostatic_func(); return r1 + r2; } gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -c _configtest.c -o _configtest.o unable to execute 'gcc': No such file or directory failure. removing: _configtest.c _configtest.o building 'bottleneck.reduce' extension creating build/temp.linux-x86_64-3.7 creating build/temp.linux-x86_64-3.7/bottleneck creating build/temp.linux-x86_64-3.7/bottleneck/src gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/spark/.local/lib/python3.7/site-packages/numpy/core/include -I/usr/include/python3.7m -Ibottleneck/src -c bottleneck/src/reduce.c -o build/temp.linux-x86_64-3.7/bottleneck/src/reduce.o -O2 unable to execute 'gcc': No such file or directory error: command 'gcc' failed with exit status 1 ---------------------------------------- Running setup.py clean for bottleneck Running setup.py bdist_wheel for pandocfilters: started Running setup.py bdist_wheel for pandocfilters: finished with status 'done' Stored in directory: /home/spark/.cache/pip/wheels/93/9a/79/b2c3567908fd6209e4674ca23d9fcf005aae5fe89148913727Successfully built pyod pyLDAvis cufflinks umap-learn future sklearn pynndescent htmlmin phik prometheus-flask-exporter alembic databricks-cli pandocfiltersFailed to build numba llvmlite bottleneckInstalling collected packages: tenacity, plotly, numpy, threadpoolctl, scikit-learn, mlxtend, jupyterlab-widgets, webencodings, packaging, bleach, mistune, ipython-genutils, traitlets, pygments, jupyter-core, testpath, entrypoints, pyrsistent, zipp, typing-extensions, importlib-metadata, attrs, jsonschema, nbformat, nest-asyncio, async-generator, tornado, pyzmq, jupyter-client, nbclient, pandocfilters, defusedxml, jupyterlab-pygments, MarkupSafe, jinja2, nbconvert, ptyprocess, terminado, pickleshare, backcall, pexpect, matplotlib-inline, parso, jedi, wcwidth, prompt-toolkit, decorator, IPython, debugpy, ipykernel, prometheus-client, Send2Trash, pycparser, cffi, argon2-cffi, notebook, widgetsnbextension, ipywidgets, llvmlite, numba, pyod, lightgbm, scikit-plot, smart-open, gensim, numexpr, future, funcy, sklearn, pyLDAvis, colorlover, cufflinks, yellowbrick, Boruta, pynndescent, umap-learn, textblob, pillow, wordcloud, seaborn, requests, htmlmin, phik, pydantic, networkx, bottleneck, tangled-up-in-unicode, multimethod, PyWavelets, imagehash, visions, missingno, pandas-profiling, kmodes, imbalanced-learn, querystring-parser, greenlet, sqlalchemy, cloudpickle, gunicorn, smmap, gitdb, gitpython, protobuf, Werkzeug, itsdangerous, Flask, prometheus-flask-exporter, Mako, python-editor, alembic, sqlparse, websocket-client, docker, tabulate, databricks-cli, mlflow, pycaret Found existing installation: numpy 1.18.1 Uninstalling numpy-1.18.1: Successfully uninstalled numpy-1.18.1 Found existing installation: scikit-learn 0.22.1 Uninstalling scikit-learn-0.22.1: Successfully uninstalled scikit-learn-0.22.1 Running setup.py install for llvmlite: started Running setup.py install for llvmlite: finished with status 'error' Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ws60mqho/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r7xtmu3s-record/install-record.txt --single-version-externally-managed --compile --user --prefix=: running install running build got version from file /tmp/pip-build-ws60mqho/llvmlite/llvmlite/_version.py {'version': '0.36.0', 'full': 'e6bb8d137d922bec8beeb01a237254778759becd'} running build_ext /usr/bin/python3 /tmp/pip-build-ws60mqho/llvmlite/ffi/build.py LLVM version... Traceback (most recent call last): File "/tmp/pip-build-ws60mqho/llvmlite/ffi/build.py", line 220, in <module> main() File "/tmp/pip-build-ws60mqho/llvmlite/ffi/build.py", line 210, in main main_posix('linux', '.so') File "/tmp/pip-build-ws60mqho/llvmlite/ffi/build.py", line 134, in main_posix raise RuntimeError(msg) from None RuntimeError: Could not find a `llvm-config` binary. There are a number of reasons this could occur, please see: https://llvmlite.readthedocs.io/en/latest/admin-guide/install.html#using-pip for help. error: command '/usr/bin/python3' failed with exit status 1 ----------------------------------------
INFO 2021-07-05 17:36:34,742 81650 com.amazonaws.services.glue.PythonModuleInstaller [main] Failed building wheel for numba Failed building wheel for llvmlite Failed building wheel for bottleneckCommand "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-ws60mqho/llvmlite/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-r7xtmu3s-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-ws60mqho/llvmlite/
I attempted to install PyCaret by setting Python library path as shown in the picture below. It didn't work well because installing a Python module through Python library path didn't install dependencies automatically. I tried to give the path to PyCaret whl file along with its dependency files. It kept asking me to provide whl files that were not listed in PyCaret's requirements.txt file. So I stopped trying.
Resources I already checked:
- https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html
- https://aws.amazon.com/blogs/big-data/building-python-modules-from-a-wheel-for-spark-etl-workloads-using-aws-glue-2-0/
- Python Cannot install module spaCy
- Use AWS Glue Python with NumPy and Pandas Python Packages
- and more...
I spent many hours now. I don't know how to resolve my issue. Any suggestions or help will be greatly appreciated.
ANSWER
Answered 2021-Jul-08 at 17:01I reached out to AWS support. Meghana was in charge of this case.
Here is the reply:
I [Meghana] was able to successfully replicate the error at my end as well using the below steps. Also, please find the workaround mentioned below to avoid the same.
- Create a Glue 2.0 job and configure the Job Parameters as below
===
-- additional-python-modules pycaret==2.3.2,spacy==3.0.6
===
On running the Glue job I see that it tried to a Pip3 install, Please find the same below
===
INFO 2021-07-07 05:23:15,813 0 com.amazonaws.services.glue.PythonModuleInstaller [main] pip3 install --user pycaret==2.3.2 spacy==3.0.6
===
However, pip install can be called against a pypi package, local project or wheel hosted via HTTPS we can use this functionality to install packages publicly hosted on pypi as well as those not available publicly available. This gives us the ability to install the majority of packages to use with Glue, including those which are c-based. There is however a subset that will fail (like spacy); packages that require root privileges to install or to be compiled during installation will fail. Glue does not give root access and there is no exception made for package installation. What can be done in this case is pre-compiling the binaries into a wheel compatible with Glue and installing that wheel.
Please find the below error that I have observed while installing the above “pycaret==2.3.2,spacy==3.0.6” and import them in glue job failed as the compilation failed while replicating.
==
INFO 2021-07-07 05:23:15,813 0 com.amazonaws.services.glue.PythonModuleInstaller [main] pip3 install --user pycaret==2.3.2 spacy==3.0.6
INFO 2021-07-07 05:23:26,751 10938 com.amazonaws.services.glue.PythonModuleInstaller [main] Collecting pycaret==2.3.2 Downloading https://files.pythonhosted.org/packages/bc/b6/9d620a23a038b3abdc249472ffd9be217f6b1877d2d952bfb3f653622a28/pycaret-2.3.2-py3-none-any.whl (263kB)Collecting spacy==3.0.6 Downloading https://files.pythonhosted.org/packages/6d/0d/4379e9aa35a444b6440ffe1af4c612533460e0d5ac5c7dca1f96ff6f2e23/spacy-3.0.6.tar.gz (7.1MB) Complete output from command python setup.py egg_info: Error compiling Cython file: ------------------------------------------------------------ ... from libc.stdint cimport int64_t from libcpp.vector cimport vector from libcpp.set cimport set from cymem.cymem cimport Pool ^ ------------------------------------------------------------ spacy/strings.pxd:4:0: 'cymem/cymem.pxd' not found Error compiling Cython file: ------------------------------------------------------------ ... from libc.stdint cimport int64_t from libcpp.vector cimport vector from libcpp.set cimport set from cymem.cymem cimport Pool ^ ------------------------------------------------------------ spacy/strings.pxd:4:0: 'cymem/…
…
..
INFO 2021-07-07 05:23:26,753 10940 com.amazonaws.services.glue.PythonModuleInstaller [main] Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-ahnegkpn/spacy/
==
Thus, the importing of the above failed during compilation stage. The error that you have received below is due the python version mismatch. Here the tar file spacy-2.3.7.tar.gz is compiled using python 3.7 and the Glue job is using 3.6 and hence it failed with the below error. However, even if you provide as mentioned above it still has c-dependencies and stills fails. Please find the work-around to avoid the same.
—
INFO 2021-07-05 17:01:53,986 17142 com.amazonaws.services.glue.PythonModuleInstaller [main] Collecting setuptools Downloading …
7c5f7541eb883181b564a8c8ba15d21b2d7b8a38ae32f31763575cf8857d/spacy-2.3.7.tar.gz (5.8MB)
Complete output from command python setup.py egg_info: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_e6ejo9m/blis/ Traceback (most recent call last): File "/home/spark/.local/lib/python3.7/site-packages/setuptools/installer.py", line 128, in fetch_build_egg subprocess.check_call(cmd) File "/usr/lib64/python3.7/subprocess.py", line 363, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpo5fqe7du', '--quiet', 'blis<0.8.0,>=0.4.0']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-xoyv9lar/spacy/setup.py", line 252, in <module> setup_package() File "/tmp/pip-build-xoyv9lar/spacy/setup.py", line 247, in setup_package cmdclass={"build_ext": build_ext_subclass}, File "/home/spark/.local/lib/…
File "/home/spark/.local/lib/python3.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg raise DistutilsError(str(e)) distutils.errors.DistutilsError: Command '['/usr/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpo5fqe7du', '--quiet', 'blis<0.8.0,>=0.4.0']' returned non-zero exit status 1. ----------------------------------------
—
###Workaround
- To compile a library in a C-based language, the compiler must know the target OS and processor architecture.
- If the library is compiled against a different OS or processor architecture, then the wheel will fail to install in Glue.
- Because Glue is a manages service, we do not give users cluster-access to develop these dependencies.
- Below I will walk you through using a Docker image to prepare an environment you can use to compile wheels that will be compatible with Glue. For this example we will be compiling pycaret,spacy which requires GCC to be installed on the target device as root.
Step-1: Launch an m5.xlarge EC2 instance with Amazon Linux (2) and enough volume space for your libs.
Step-2: Install Docker on the instance, set up nonsudo access, and start it
1. sudo yum install docker -y
2. sudo usermod -a -G docker ec2-user
3. sudo service docker start
Step-3: Create a Dockerfile as below
$ vi docfile
—
# Base for Glue
FROM amazonlinux
RUN yum update -y
RUN yum install shadow-utils.x86_64 -y
RUN yum install -y java-1.8.0-openjdk.x86_64
RUN yum install -y python3
RUN yum install -y gcc autoconf automake libtool zlib-devel openssl-devel maven wget protobuf-compiler cmake make gcc-c++
# Additonal Components needed for psutil
WORKDIR /root
RUN yum install python3-devel -y
RUN yum install python-devel -y
RUN pip3 install wheel
# Install psutil
RUN pip3 install pycaret
RUN pip3 install spacy
# Create a directory for the wheel
RUN mkdir wheel_dir
# create the wheel
RUN pip3 wheel pycaret -w wheel_dir
RUN pip3 wheel spacy -w wheel_dir
—
Step-4: Run docker build to build your Dockerfile
==
restart the docker daemon
[ec2-user@ip-xxx ~]$ sudo service docker restart
[ec2-user@ip-xxx ~]$ docker build -f docfile .
[ec2-user@ip-xxx ~]$ docker build -f docfile .
Sending build context to Docker daemon 16.38kB
Step 1/17 : FROM amazonlinux
---> 7443854fbdb0
Step 2/17 : RUN yum update -y
---> Using cache
…
Removing intermediate container xxx
---> xxx
Successfully built xx
==
[ec2-user@ip-xxx~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[ec2-user@ip-xxx~]$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> xxx 24 seconds ago 4.28GB
<none> <none> xxx 43 minutes ago 3.94GB
[ec2-user@ip-xxx ~]$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Xxx yyy "/bin/bash" 37 minutes ago Exited (0) 37 minutes ago brave_meninsky
Step-5 : Extract the whl from the docker container
- Get the container ID
docker ps (get container ID)
- Run the container and keep it from exiting
docker run -dite <image>
- Verify the location of the wheel file (and get it’s filename)
docker exec -t -i <container_id> ls /root/wheel_dir/
- Copy the wheel out of docker to EC2
docker cp <containerID>:/root/wheel_dir/<wheelFile> .
===
[ec2-user@ip-xxx ~]$ docker run -dite e0e1f71b8fad
9f0b1aff06dd959f3744edd3804512e73b68aaeef178962f2c0c063b290dbf78
[ec2-user@ip-xxx~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9f0b1aff06dd e0e1f71b8fad "/bin/bash" 49 seconds ago Up 48 seconds quirky_bose
[ec2-user@ip-xxx ~]$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
52ab3413c962 e0e1f71b8fad "/bin/bash" 3 seconds ago Up 2 seconds fervent_williamson
9f0b1aff06dd e0e1f71b8fad "/bin/bash" 2 minutes ago Up 2 minutes quirky_bose
[ec2-user@ip-xxx~]$ docker exec -t -i 52ab3413c962 ls /root/wheel_dir/
Boruta-0.3-py3-none-any.whl
Bottleneck-1.3.2-cp37-cp37m-linux_x86_64.whl
…
[ec2-user@ip-xxx ~]$ aws s3 cp pycaret-2.3.2-py3-none-any.whl s3://cxx/
upload: ./pycaret-2.3.2-py3-none-any.whl to s3://cxx/pycaret-2.3.2-py3-none-any.whl
===
Step-6: Upload the wheel to S3
aws s3 cp <wheelFile> s3://path/to/wheel/
Step-7: Pass the S3 URL to Glue
Edit your Job
Expand “Security configuration, script libraries, and job parameters (optional)”
In “Job parameters enter a new Key-Value pair
--
Key --additional-python-modules
Value. <s3URI>
--
QUESTION
Unable to install sklearn 0.23.2 with pip 20.2.4 and python 3.9
Asked 2021-Jun-07 at 17:47I cant use higher version of pip other than 20.2.4, since some SSL certification errors are occuring in higher versions and cannot reinstall any settings I have now (because of some office setup). Now I am using Pycaret and currently it supports only sklearn 0.23.2.
But my sklearn version is 0.24.1 and I am unable to downgrade it with 20.2.4 version of pip.
I also tried manual installation using setup.py file and it is also not success.
I am having Winpython and unable to install anaconda too.
Can someone help me to sort this problem? I am having python 3.9 . May be that is the problem?
Update:
Error message:
Collecting scikit-learn==0.23.2
Using cached scikit-learn-0.23.2.tar.gz (7.2 MB)
Installing build dependencies ... error
ERROR: Command errored out with exit status 1:
command: 'C:\Users\me\Desktop\WPy64-3902\Scripts\python.exe' 'C:\Users\me\Desktop\WPy64-3902\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\me\AppData\Local\Temp\pip-build-env-w5t8h2zr\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.28.5' 'numpy==1.13.3; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation == '"'"'CPython'"'"'' 'numpy==1.14.0; python_version=='"'"'3.6'"'"' and platform_system!='"'"'AIX'"'"' and platform_python_implementation != '"'"'CPython'"'"'' 'numpy==1.14.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.6'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.16.0; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version>='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'scipy>=0.19.1'
cwd: None
Complete output (547 lines):
Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation == "CPython"' don't match your environment
Ignoring numpy: markers 'python_version == "3.6" and platform_system != "AIX" and platform_python_implementation != "CPython"' don't match your environment
Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.6" and platform_system == "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
Ignoring numpy: markers 'python_version >= "3.8" and platform_system == "AIX"' don't match your environment
ANSWER
Answered 2021-Jun-07 at 17:47I tried various things, and the best advice I can give you is don't bother with python 3.9 for this library; It's just not supported yet. Dependencies are not sorted out yet for pycaret yet on python 3.9, and they should probably make note of that on their github, but here's the process I went through to get it installed from a completely fresh computer (windows sandbox).
C:\Users\WDAGUtilityAccount>conda create --name py38 python=3.8C:\Users\WDAGUtilityAccount>activate py38
(py38) C:\Users\WDAGUtilityAccount>pip install -U setuptools(py38) C:\Users\WDAGUtilityAccount>pip install pycaret
(py38) C:\Users\WDAGUtilityAccount>python Python 3.8.10 (default, May 19 2021, 13:12:57) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import pycaret >>> #look ma; no errors :)
QUESTION
Pycaret.regression.compare_models: Evaluation table neither returned nor shown
Asked 2021-May-30 at 04:31pycaret is a very compact tool to compare models that I wanted to use for model selection. Unfortunately, the method compare_models does not show the typical output table that you see everywhere. I am using pycaret in PyCharm and not Jupyter Notebook which seems to be the typical approach. I do get the best model as a return value, but am really aiming for the overview table. It also doesn't seem to make a difference whether the parameter silent is set to True or False appart from being asked for the confirmation of whether the derived datatypes are correct.
Thank you very much!
The system: Python 3.6 pycaret 2.3 CentOS 7 PyCharm 2020.1 Community Edition
My code:
regression.setup(data=ml_df,
target='occupation',
n_jobs=6,
categorical_features=['cluster', 'vacation', 'long_weekend', 'month', 'hour', 'weekday'],
numeric_features=['temperature', 'precipitation'],
silent=False
)
best_regression_models = regression.compare_models()
categorisation = [
[-0.1, 'empty'],
[0.01, 'partial'],
[0.99, 'full']
]
ml_df['occupation'] = modelling_utils.convert_number_to_categorical(ml_df['occupation'], categorisation)
classification.setup(data=ml_df,
target='occupation',
n_jobs=6,
categorical_features=['cluster', 'vacation', 'long_weekend', 'month', 'hour', 'weekday'],
numeric_features=['temperature', 'precipitation'],
fix_imbalance=True,
silent=False)
best_classification_models = classification.compare_models()
The full output is somewhat lengthy and saved here.
Edit: The code works as expected in a Jupyter Notebook
ANSWER
Answered 2021-May-30 at 04:31Running PyCaret from a terminal/command line has different behavior compared to running from a Jupyter notebook. In your case, if you want to display the comparison output table, add these 2 lines after your compare_models() function call:
..
best_regression_models = regression.compare_models()
regression_results = pull()
print(regression_results)
The pull() function will also return the last score grid with other training functions, such as create_model(). Currently this function only works with the regression and classification modules.
Reference: https://pycaret.org/pull/
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install pycaret
PyCaret's default installation only installs hard dependencies as listed in the requirements.txt file.
Support
With PyCaret >= 2.2, you can train models on GPU and speed up your workflow by 10x. To train models on GPU simply pass use_gpu = True in the setup function. There is no change in the use of the API, however, in some cases, additional libraries have to be installed as they are not installed with the default version or the full version. As of the latest release, the following models can be trained on GPU:.
Find more information at:
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 650 million Knowledge Items
Find more librariesReuse Solution Kits and Libraries Curated by Popular Use Cases
Explore KitsSave this library and start creating your kit
Install
PyPI
pip install pycaret
Clone
HTTPS
https://github.com/pycaret/pycaret.git
CLI
gh repo clone pycaret/pycaret
SSH
git@github.com:pycaret/pycaret.git
Share this Page
See Similar Libraries in
Reuse Pre-built Kits with pycaret
Reuse Machine Learning Kits
Consider Popular Machine Learning Libraries
- tensorflow
by tensorflow
- youtube-dl
by ytdl-org
- models
by tensorflow
- pytorch
by pytorch
- keras
by keras-team
See all Machine Learning Libraries
Try Top Libraries by pycaret
- pycaret-demo-mlflow
by pycaret Jupyter Notebook
- pycaret-demo-dphi
by pycaret Jupyter Notebook
- pycaret-git-actions
by pycaret Python
- mlopsworld2021
by pycaret Jupyter Notebook
- pycaret-deployment-aws
by pycaret CSS
See all Libraries by this author
Compare Machine Learning Libraries with Highest Support
- youtube-dl
by ytdl-org
- scikit-learn
by scikit-learn
- models
by tensorflow
- tensorflow
by tensorflow
- keras
by keras-team
See all Machine Learning Libraries
Compare Machine Learning Libraries with Highest Quality
- DataMiningClassifier
by yangliuy
- segmentations
by asanoja
- purdue-fastr
by allr
- clusterer
by mrmans0n
- learnlib
by LearnLib
See all Machine Learning Libraries
Compare Machine Learning Libraries with Highest Security
- FirePick
by firepick1
- MachineLearningToolkit
by vpejovic
- DataMiningClassifier
by yangliuy
- segmentations
by asanoja
- purdue-fastr
by allr
See all Machine Learning Libraries
Compare Machine Learning Libraries with Highest Reuse
- clusterer
by mrmans0n
- AndroidVisionPipeline
by Credntia
- lap
by gatagat
- swarmops
by Hvass-Labs
- javaAnchorExplainer
by viadee
See all Machine Learning Libraries
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 650 million Knowledge Items
Find more librariesReuse Solution Kits and Libraries Curated by Popular Use Cases
Explore KitsSave this library and start creating your kit
Open Weaver – Develop Applications Faster with Open Source