snakefiles | common RNA-seq data analysis workflows | Genomics library

I am trying to clean a data pipeline by using snakemake. It looks like wildcards are what I need but I don't manage to make it work in params

My function needs a parameter that depends on the wildcard value. For instance, let's say it depends on sample that can either be A or B.

I tried the following (my example is more complicated but this is basically what I am trying to do) :

I'm performing parameter exploration using Paramspace utility as described here. I've read parameters in a pandas dataframe and next I wish to pass these as values of options of a shell command but can't figure out how.

In the below minimal example, I wish to pass parameter s (read in dataframe df) as the value of option -n for head command in the shell directive.

I am working in a high performance computing grid environment, where large-scale data transfers are done via Globus. I would like to use Snakemake to pull data from a Globus path, process the data, and then push the processed data to a different Globus path. Globus has a command-line interface.

Pulling the data is no problem, for I'd just create a rule that would run globus transfer to create the requisite local file. But for pushing the data back to Globus, I think I'll need a rule that can "see" that the file is missing at the remote location, and then work backwards to determine what needs to happen to create the file.

I could create local "proxy" files that represent the remote files. For example I could make a rule for creating 'processed_data_1234.tar.gz' output files in a directory. These files would just be created using touch (thus empty), and the same rule will run globus transfer to push the files remotely. But then there's the overhead of making sure that the proxy files don't get out of sync with the real Globus-hosted files.

Is there a more elegant way to do this akin to the Remote File capability? Is it difficult to add a Globus CLI support for Snakemake? Thanks in advance for any advice!

Before this, I checked this, snakemake's documentation, this,and this. Maybe they actually answered this question but I just didn't understand it.

In short, I create in one rule a number of files from other files, that both conform to a wildcard format. I don't know how many of these I create, since I don't know how many I originally download.

In all of the examples I've read so far, the output is directory("the/path"), while I have a "the/path/{id}.txt. So this I guess modifies how I call the checkpoints in the function itself. And the use of expand.

The rules in question are:

download_mv

textgrid_to_ctm_txt

get_MV_IDs

merge_ctms

The order of the rules should be:

download_mv (creates {MV_ID}.TEX and .wav (though not necessarily the same amount)

textgrid_to_ctm_txt (creates from {MV_ID}.TEX matching .txt and .ctm)

get_MV_IDs (should make a list of the .ctm files)

merge_ctms (should concatenate the ctm files)

kaldi_align (from the .wav and .txt directories creates one ctm file)

analyse_align (compares ctm file from kaldi_align the the merge_ctms)

upload_print_results

I have tried with the outputs of download_mv being directories, and then trying to get the IDs but I had different errors then. Now with snakemake --dryrun I get

I have a snakemake file where one rule produces a file from witch i would like to extract the header and use as wildcards in my rule all. The Snakemake guide provides an example where it creates new folders named like the wildcards, but if I can avoid that it would be nice since in some cases it would need to create 100-200 folders then. Any suggestions on how to make it work?

link to snakemake guide: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

I am using Snakemake to run the defense-finder program. This program creates and overwites generic temporary files in /tmp/defense-finder, i.e. the file names do not contain unique identifiers. When running my rule across separate cores on different input files, Snakemake crashes due to clashes in /tmp/defense-finder.

It appears that Shadow rules can help when different jobs write to the same files within the working directory. Is there a way to use Shadow rules when a program writes to the /tmp directory?

Below an example of Snakemake Rmd report with a custom.css.

I am trying to run a snakemake rule with an external script that contains a wildcard as noted in the snakemake reathedocs. However I am running into KeyError when running snakemake.

For example, if we have the following rule:

I have a job that might fail with a specific configuration, what I want to do is let it run once and if it fails run a slightly different configuration. I found the attempt parameter but I don't find a way to access it outside the resources tag...

do you know how to access it or any alternative?