fastq | Fast, in memory work queue

Loop over the files. In bash, you can use parameter expansion to remove the extension from the file name:

I think the Snakefile is ok, SRR3350597_GSM2112330_RA_hip_3_Homo_sapiens_RNA-Seq_1.fastq.gz is simply missing. See the ls output of yours, that file is not in it.

I think the problem is that rule catFasta doesn't contain the wildcard barcode. If you think about it, what job name would you expect in {wildcards.barcode}.{rule}.{jobid}?

Maybe a solution could be to add to each rule a jobname parameter that could be {barcode} for guppyplex and minion and 'all_barcodes' for catFasta. Then use --jobname "{params.jobname}.{rule}.{jobid}"

The rule that fails is rule guppyplex, which looks for an input in the form of {FASTQ_PATH}/{{barcode}}.

Looks like the wildcard {barcode} is filled with barcode49/barcode49.consensus.fasta, which happened because of two reasons I think:

First (and most important): The workflow does not find a better way to produce the final output. In rule catFasta, you give an input file which is never described as an output in your workflow. The rule minion has the directory as an output, but not the file, and it is not perfectly clear for the workflow where to produce this input file.

It therefore infers that the {barcode} wildcard somehow has to contain this .consensus.fasta that it has never seen before. This wildcard is then handed over to the top, where the workflow crashes since it cannot find a matching input file.

Second: This initialisation of the wildcard with sth. you don't want is only possible since you did not constrain the wildcard properly. You can for example forbid the wildcard to contain a . (see wildcard_constraints here)

However, the main problem is that catFasta does not find the desired input. I'd suggest changing the output of minion to "nanopolish/{barcode}/{barcode}.consensus.fasta", since the you already take the OUTDIR from the params, that should not hurt your rule here.

Edit: Dummy test example:

Ok, I have a solution that has not failed for me: I create a try catch with a max attempt iterator, default 5 attempts with a wait time of 1 second, in addition to a general wait time of 0.05 seconds per accepted url request.

Let me know if anyone has a safer idea:

Here's an async variation of my answer here:

In general, I would always try to run a program on a much smaller input first (1 MB, 10 MB, 100 MB?) to see if the program works correctly, and if so, how long time it takes per MB. Then I can calculate how long time it would approximately take for the full file and how much progress would be expected at which time in the progress.

Maybe you can even make those small file tests while leaving it running on the big file, to at least see that the program actually works and will eventually finish (to not lose your current progress). Try with a very small file first (maybe the first 1 MB of the large file), then perhaps increase in size if that worked flawlessly.

Looking at the actual program, though, I would definitely not collect the whole data in memory and only write it in the end. I would write to the output file continously. That is much more efficient and won't use the same likely huge amount of virtual memory as you use with the current program.

So, something like this (didn't test, as I can't):

Incorporating @shellter comments and help, the easiest script to get rid of non-ASCII junks and get the desired output that I came up with was following;

The underlying problem was my misunderstanding of getline() getline() c reference

To store lines in different variables, an associated n is necessary for each line pointer *lineptr.

If *lineptr is set to NULL and *n is set 0 before the call, then getline() will allocate a buffer for storing the line.

Alternatively, before calling getline(), *lineptr can contain a pointer to a malloc(3)-allocated buffer *n bytes in size. If the buffer is not large enough to hold the line, getline() resizes it with realloc(3), updating *lineptr and *n as necessary.

The n (or seed in my code) will hold the size of the buffer allocated for the pointer, where getline() puts the incoming line. As I set the same buffer variable for different pointers, getline was given the wrong information of the size of the char* line_xxx.

As fastq files are usually in this shape: