snakemake-wrappers | development home of the Snakemake wrapper repository | BPM library

TLDR I'm getting the following error:

The 'conda' command is not available inside your singularity container image. Snakemake mounts your conda installation into singularity. Sometimes, this can fail because of shell restrictions. It has been tested to work with docker://ubuntu, but it e.g. fails with docker://bash

I had created a Snakemake workflow and converted the shell: commands to rule-based package management via Snakemake wrappers: .

However, I ran into issues running this on HPC and one of the HPC support staff strongly recommended against using conda on any HPC system as:

"if the builder [of wrapper] is not super careful, dynamic libraries present in the conda environment that relies on the host libs (there are always a couple present because builder are most of the time carefree) will break. I think that relying on Singularity for your pipeline would make for a more robust system." - Anon

I did some reading over the weekend and according to this document, it's possible to combine containers with conda-based package management; by defining a global conda docker container and per-rule yaml files.

Note: In contrast to the example in the link above (Figure 5.4), which uses a predefined yaml and shell: command, here I've use conda wrappers which download these yaml files into the Singularity container (if I'm thinking correctly) so I thought should function the same - see the Note: at the end though...

I'm in the process of porting Snakemake Shell: into Snakemake Wrappers and have noticed that the lambda wildcards: I've successfully used for other wrappers is failing for the BWA MEM wrapper.

I've only managed to get the wrapper to work if it's hard-coded like:

I have a rule using the bwa mem wrapper that sometimes fails due to cluster time limits. As this only happens occasionally, I do not want to generally increase the time limit for that job, but instead increase it with the number of attempts.

However, after failing due to cluster time limit, a lot of bwa mem tmp files are left in the output directory, which cause bwa mem to immediately fail in the next attempt. The generated tmp files are numbered out.tmp.1.bam .. out.tmp.n.bam, where n is some number as bwa mem sees fit, so I cannot simply mark these as temp files in Snakemake and rely on them being deleted on failure (I'm not even sure that this would happen - I don't know exactly when the deletion of files marked as temp is triggered...).

I considered the following solutions:

• Delete these files first (by not using the wrapper, but instead copy the wrapper code, and modify it to delete all out.tmp.*.bam files before running bwa mem), but this seems ugly.

• Use a shadow directory, in the hope that this is directory cleared after each attempt, but the documentation says

  Shadow directories are stored one per rule execution in .snakemake/shadow/, and are cleared on successful execution.

  Hence, for a failed execution, the temp files would still be there, which will cause subsequent attempts to fail as well. I guess that this is done in order to be able to debug failed runs. But here, it hinders restarts.

• An alternative solution would be to have onstart, onsuccess, and onerror hooks per rule, as previously suggested in #133, but that is an option for the future...

I have posted a feature request for this problem already, but maybe there already is a pure Snakemake solution already out there. Any help appreciated!

Thanks, Lucas

I am writing a snakemake file that shall perform multiple operations on multiple samples. After I validated the workflow running on my local computer, I am now working on running the workflow on a cluster.

My first two rules are independent from one another, the first uses fastqc and the other bwa mem

These two rules look like (at this point I am only calling the workflow on a single SAMPLE = 'NIPT-PearlPPlasma-03-PPx_S3downSample'):

When snakemake wrappers are used with snakemake scripts, do they get fetched everytime from remote URL or are they stored locally? I don't see them stored anywhere in .snakemake directory, which makes me think it gets fetched everytime even if same version of wrapper script is used everytime.