singularity | Singularity : Application containers for Linux | Continuous Deployment library

You should be able to build a Singularity image from a Docker image hosted on Docker hub with:

The $(...) statement is evaluated by bash and its output is what is sent to the singularity container. What you probably want is just:

I made some changes in the dockerfile based on the method to install singularity in linux given here.

The complete dockerfile with which I was able to run successfully nextflow, java and singularity within singularity is given below,

This is a hard problem to solve in general, and even a fairly hard problem to solve in specific cases.

I think the difference arose between version 1.1.27.1 and 1.1.28, probably from this NEWS item:

construction of interacting factors (e.g. when f1:f2 or f1/f2 occur in random effects terms) is now more efficient for partially crossed designs (doesn't try to create all combinations of f1 and f2) (GH #635 and #636)

My guess is that this changes the ordering of the components in the Z matrix, which in turn means that results of various linear algebra operations are not identical (e.g. floating point arithmetic is not associative, so while binary addition is commutative (a + b == b + a), left-to-right evaluation of a sum may not be the same as right-to-left evaluation ((a+b) + c != a + (b+c)) ...)

My attempt at reproducing the problem uses the same version of R ("under development 2022-02-25 r81818") and compares only lme4 package versions 1.18.1 with 1.1.28.9000 (development); any upstream packages such as Rcpp, RcppEigen, Matrix use the same versions. (I had to backport a few changes from the development version of lme4 to 1.1.18.1 to get it to install under the most recent version of R, but I don't think any of those modifications would affect numerical results.)

I did the comparison by installing different versions of the lme4 package before running the code in a fresh R session. My results differed between versions 1.1.18.1 and 1.1.28 less than yours did (both fits were singular, and the relative differences in the theta estimates were of the order of 2e-7 — still greater than your desired 1e-8 tolerance but much smaller than 1e-4 ...)

The results from 1.1.18.1 and 1.1.27.1 were identical.

• Q1: Why are your results more different between versions than mine?
  • in general/anecdotally, numerical results on Windows are slightly more unstable/differ more from other platforms
  • there are more differences between your two test platforms than among mine: R version, upstream packages (Matrix/Rcpp/RcppEigen/minqa), possibly the compiler versions and settings used to build everything [all of which could make a difference]
• Q2: how should one deal with this kind of problem?
  • as a minor frame challenge, why (other than not understanding what's going on, which is a perfectly legitimate reason to be concerned) does this worry you? The differences in the results are way smaller than the magnitude of statistical uncertainty, and differences this large are also likely to occur across different platforms (OS/compiler version/etc.) even for otherwise identical environments (versions of R, lme4, and other packages).
  • you could revert to version 1.1.27.1 for now ...
  • I do take the differences between 1.1.27.1 as a bug, of sorts — at the very least it's an undocumented change in the package. If it were sufficiently high-priority I could investigate the code changes described above and see if there is a way to fix the problems they addressed without breaking backward compatibility (in theory this should be possible, but it could be annoyingly difficult ...)

Here's how far I've gotten:

• the exponential rates are supposed to be specified as logs of the rates (to make sure that the rates themselves stay positive, i.e. that we have exponential decay curves rather than growth curves)
• I simplified the model significantly, taking out the random effects in T1 and T2.

Nextflow processes are designed to run isolated from each other, but this can be circumvented somewhat when the command-line input and/or outputs are specified using params. Using params like this can be problematic because if, for example, a params variable specifies an absolute path but your output declaration expects files in the Nextflow working directory (e.g. ./work/fc/0249e72585c03d08e31ce154b6d873), you will get the 'Missing output file(s) expected by process' error you're seeing.

The solution is to ensure your inputs are localized in the working directory using an input declaration block and that the outputs are also written to the work dir. Note that only files specified in the output declaration block can be published using the publishDir directive.

Also, best to avoid calling Singularity manually in your script block. Instead just add singularity.enabled = true to your nextflow.config. This should also work nicely with the beforeScript process directive to initialize your environment:

As you found, instance list only lists singularity processes that were started as instances. Instance logs (both running and past) are stored in ~/.singularity/instances/logs/$HOSTNAME/$USER.

Active processes from non-instance commands aren't tracked anywhere. This is largely because Singularity does not use a centralized service model like Docker. Similarly, there's no "default" log location because stdout/stderr are handled that same as running any other command. As such, tracking down Singularity processes can be done using the standard shell tools.

Some examples:

You've to use absolute path instead of relative path

Modify your index.js with the below code:

Short answer:

bug your HPC admins to install Singularity

Longer answer:

There are two ways to install Singularity, as a privileged installation or an unprivileged / user namespace installation. The first way is the default, but requires sudo/root for certain actions (mostly singularity build). The latter removes the need for root, but has other system requirements. It's possible additional OS configuration is necessary for Singularity to function as expected.

In addition to privileged/unprivileged installations, disk storage in clusters is usually on NFS or another networked/distributed filesystem so all nodes have access to the same data. Unfortunately, as is usually the case any time it is involved, NFS is a likely cause for your problem. Singularity relies on SUID for its core functionality, but for (quite good) security reasons SUID is disabled on NFS by default. It is unlikely the cluster admins will enable that option, so your best bet is to ask them install it locally on whichever computer/interactive nodes you need it on.