conda-pack | Package conda environments for redistribution | Machine Learning library

The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.

To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.

For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.

See the cpr repository for details on use.

https://towardsdatascience.com/python-the-system-path-and-how-conda-and-pyenv-manipulate-it-234f8e8bbc3e

This article helped me debug this issue. I just had to make sure I deactivated out of conda environment completely even the (base) environment. For some reason even after deactivating from my po environment, it went to base environment

If you want to create a custom environment you can use the below code to set the env configuration.

myenv = Environment(name="Environment")

myenv.docker.enabled = True

myenv.python.conda_dependencies = CondaDependencies.create(conda_packages = ['numpy','scikit-learn','pip','pandas'], pip_packages = ['azureml-defaults~= 1.34.0','azureml','azureml-core~= 1.34.0',"azureml-sdk",'inference-schema','azureml-telemetry~= 1.34.0','azureml- train-automl~= 1.34.0','azure-ml-api-sdk','python-dotenv','azureml-contrib-server','azureml-inference-server-http'])

Ref doc: https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.environment(class)?view=azure-ml-py#:~:text=Upload%20the%20private%20pip%20wheel,in%20the%20workspace%20storage%20blob.&text=Build%20a%20Docker%20image%20for%20this%20environment%20in%20the%20cloud.&text=Build%20the%20local%20Docker%20or%20conda%20environment.

Consider creating a separate environment, e.g.,

Build tags in you environment.yml are quite strict requirements to satisfy and most often not needed. In your case, changing the yml file to

I wasn't able to get the monkey patching to work, but I was able to figure out that ctypes is not part of numpy and rather is part of Python's standard library. So conda pack could very well treat Python standard libraries a bit differently.

So I gave up on monkey patching and found out that upgrading my Python version fixed the underlying issue.

Thanks 🙏

I resolved the issue by manually installing pyspark and making a minor change in the environment variables.

After downloading the required version of spark, you need to configure environment variables. There are a few Spark home paths you need to add to the user profile as follows,

Most of the conflicts are superfluous. The key ones are right at the end: all those packages require glibc >= 2.17 and your system (i.e., OS) only has GLIBC 2.12. So, we're talking CentOS 6 or similar RHEL, and this is a known issue that makes the newer Miniconda builds uninstallable for you. If you're deadset on Miniconda, you'll have to hit up the archive for an old version, as suggested on the install page (which, BTW, notes CentOS 7+). Unfortunately, I don't know which Miniconda version was the last to support GLIBC 2.12.

Fortunately, most of Conda Forge continues to build on COS6 images, so try out a Miniforge variant instead of Miniconda. I highly recommend Mambaforge.

And yes, testing on the centos6 Docker image, the latest Mambaforge installs and runs just fine.

The conda-unpack script is in the .tar.gz that you have created with conda-pack. It is located in the Scripts folder of the extracted environment. It should therefore be available after unpacking and activating the environment.