singularity | REST API | Machine Learning library

You need to start celery beat, because that him that will read the database and execute your task.

install : https://github.com/celery/django-celery-beat

so in CLI, you need to execute :

As mentioned in a separate discussion, Xvfb is not supposed to be start through startx or /usr/bin/X but rather with the supplied run script.

If you don't need to use yum as the bootstrap, you can easily build it from docker

It is not possible to build a singularity image based on a vagrant VM, though you can use a vagrant VM to do the actual building.

original answer:

You can just vagrant ssh to the VM and use singularity normally. The user guide docs are fairly comprehensive and searchable.

The Singularity admin guide has info specifically on using Singularity in Vagrant on a Windows host OS. It is not the first place I'd personally expect it to be, but it is there and should get you up and going anyway.

I found a bug / problem!

In the file /singularity/mconfig

is some code to check the version?

Given that some parameters and options are done during execution and not saved to the container, outputting an exact replicate as a recipe file does not seem to be possible. You can always inspect the docker image and convert the Dockerfile to singularity recipe with few tweaks though.

YOu are connecting to an interactive command shell inside a shell script, what you need is only execute the commands.

Depending on the CI software you're using, you can have certain jobs run only when specific files have changed. I use Gitlab CI, which has the only/except:changes rule. There is probably something similar for most other CI platforms, but you'll have to check their docs.

Otherwise, your solution is probably the simplest.

It's a mixture of design, convenience and technical necessity.

The biggest reason is that, unless you use certain params that say otherwise, Singularity images are read-only filesystems. You need somewhere to write output and any temporary files that get created along the way. Maybe you know to mount in your output dir, but there are all sorts of files that get created / modified / deleted in the background that we don't ever think about. Implicit automounts give reasonable defaults that work in most situations.

Simplistic example: you're doing a big sort and filter operation on some data, but you're print the results to console so you don't bother to mount in anything but the raw data. But even after some manipulation and filtering, the size of the data exceeds available memory so sort falls back to using small files in /tmp before being deleted when the process finishes. And then it crashes because you can't write to /tmp.

You can require a user to manually specify a what to mount to /tmp on run, or you can use a sane default like /tmp and also allow that to be overridden by the user (SINGULARITY_TMPDIR, -B $PWD/fake_tmp:/tmp, --contain/--containall). These are all also configurable, so the admins can set sane defaults specific the running environment.

There are also technical reasons for some of the mounts. e.g., /etc/passwd and /etc/group are needed to match permissions on the host OS. The docs on bind paths and mounts are actually pretty good and have more specifics on the whats and whys, and even the answer to your third question: --no-mount. The --contain/--containall flags will probably also be of interest. If you really want to deep dive, there are also the admin docs and the source code on github.

A simple but real singularity use case, with explanation: