papermill | 📚 Parameterize , execute , and analyze notebooks | Machine Learning library

The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.

To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.

For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.

See the cpr repository for details on use.

Apart from Automation, you can use Azure Functions as mentioned in this document:

· To run a PowerShell-based Jupyter Notebook, you can use PowerShell in an Azure function to call the Invoke-ExecuteNotebook cmdlet. This is similar to the technique described above for Automation jobs. For more information, see Azure Functions PowerShell developer guide.

· To run a SQL-based Jupyter Notebook, you can use PowerShell in an Azure function to call the Invoke-SqlNotebook cmdlet. For more information, see Azure Functions PowerShell developer guide.

· To run a Python-based Jupyter Notebook, you can use Python in an Azure function to call papermill. For more information, see Azure Functions Python developer guide.

References: Run Jupyter Notebook on the Cloud in 15 mins #Azure | by Anish Mahapatra | Towards Data Science, How to run a Jupyter notebook with Python code automatically on a daily basis? - Stack Overflow and Scheduled execution of Python script on Azure - Stack Overflow

Seems to be a bug in ipykernel 6.9.0 - options that worked for me:

• upgrade to 6.9.1 (latest version as of 2022-02-22); e.g. via pip install ipykernel --upgrade
• downgrade to 6.8.0 (if upgrading messes with other dependencies you might have); e.g. via pip install ipykernel==6.8.0

It depends which of the formats you chose for your .py files, but assuming you've chosen the "percent" format, where each new cell is marked with #%%, to add tags you write:

Have you tried using the subprocess module? It seems like a better option for you instead of multiprocessing. It allows you to asynchronously spawn sub-processes that will run in parallel, this can be used to invoke commands and programs as if you were using the shell. I find it really useful to write python scripts instead of bash scripts.

So you could use your main notebook to run your other notebooks as independent sub-processes in parallel with subprocesses.run(your_function_with_papermill).

The clear problem attribution points towards ZeroMQ unable to successfully .bind()

_{In case one has never worked with ZeroMQ,
one may here enjoy to first look at "ZeroMQ: Principles in less than Five Seconds"
before diving into further details}

The error message : zmq.error.ZMQError: Address already in use is easier to explain. Whereas ZeroMQ AccessPoint-s can, for obvious reasons freely try to .connect() to many counterparts, yet one and only one can .bind() onto a particular Transport Class' Address Target.

There are three potential reasons for this happening :

1 ) an accidental calling some code (without knowing internal details)
via { multiprocessing.Process | joblib.Parallel | Docker-wrapped | ... }-spawned replicas, which each tries to acquire an ownership of some ZeroMQ Transport Class address, which will for obvious reasons fail to succeed for any attempt after a first one succeeded.

2 ) a rather fatal situation, where some "previously"-run process did not manage to release such Transport Class specific address for further use ( do not remember that ZeroMQ might be just one of more other interested candidates - a Configuration Management flaw ), or in such cases, where previous runs failed to gracefully terminate such resource usage and left remained a Context()-instance still waiting ( infinitely in some cases till an O/S reboot ) listening for something, that will never happen.

3 ) an indeed bad engineering practice in module software design, not to handle the ZeroMQ API documented EADDRINUSE error / exception less brutally than to just crash the whole circus (at all associated costs of that)

The other error message : RuntimeError: Kernel died before replying to kernel_info related to a state, that the notebook's kernel was trying so long to establish all internal connections with its own components ( pool-peers ) that it took waiting longer than a configured or hardcoded timeout and the kernel-process simply stopped waiting anymore and threw itself into an otherwise unhandled exception you observed and reported.

Check for any hanging address-owners first, reboot all nodes if in doubts on this, next verify there are no colliding attempts "hidden" inside your own code / { multiprocessing.Process() | joblib.Parallel() | ... the likes }-calls, that after distributed may try to .bind() onto the same target. If none of these steps salvage the trouble within your domain of control, ask the modules' used Support, to analyze and help you refactor & validate your still colliding use-case.

I like to start off by visualizing the desired flow, which I understand to look like:

This means that t1 and t2 need to start concurrently and then you need to join on both:

As of now, nope. You can't do that directly. What you can do though (what we are doing) is as follows :

1. Create a python environment on your EMR masternode using the hadoop user
2. Install sparkmagic in your environment and configure all kernels as described in the README.md file for sparkmagic
3. Copy your notebook to master node/use it directly from s3 location

4. Install papermill and run with papermill :

   papermill s3://path/to/notebook/input.ipynb s3://path/to/notebook/output.ipynb -p param=1

First step is to create Jupyter Lab Notebook. If you want to use additional libraries, install them and restart the kernel (Restart Kernel and Clear All Outputs option). Then, define the processing inside your Notebook.

When it's ready, remove all the runs, peeks and dry runs before you start the scheduling phase.

Now, you need to set up Cloud Composer environment (remember about installing additional packages, that you defined in first step). To schedule workflow, go to Jupyter Lab and create second notebook which generates DAG from workflow.

The final step is to upload the zipped workflow to the Cloud Composer DAGs folder. You can manage your workflow using Airflow UI.

I recommend you to take a look for this article.

Another solution that you can use is Kubeflow, which aims to make running ML workloads on Kubernetes. Kubeflow adds some resources to your cluster to assist with a variety of tasks, including training and serving models and running Jupyter Notebooks. You can find interesting tutorial on codelabs.

I hope you find the above pieces of information useful.