bwa | Wheeler Aligner for short-read alignment | Genomics library

Hi I am attempting to use bash to iterate through a .txt file which contains the following lines. This is a smaller subset of the full list of fastq files, but all samples follow the same patterns.

I'm running into an error of 'Wildcards' object has no attribute 'output', similar to this earlier question 'Wildcards' object has no attribute 'output', when I submit Snakemake to my cluster. I'm wondering if you have any suggestions for how to make this compatible with the cluster?

While my rule annotate_snps works when I test it locally, I get the following error on the cluster:

This is a follow-up of a previous question about using a Python dictionary to generate a list of files to include as input for a single step. In this case, I'm interested in merging BAM files for a single sample that have been generated by mapping FASTQ files from multiple runs.

I am running into an error in my rule combine_bams only for a single sample:

I have a dataset with about 50 columns (all indicators I got from World Bank), Country Code and Year. These 50 columns are not all complete, and I would like to fill in the missing values based on an lm fit for the column for that specific country. For example:

Doing this for a single country and a single column is absolutely fine when following these steps here: Filling NA using linear regression in R

However, I have over 180 different countries I want to do this to. And I want this to work for each indicator per country (so 50 columns total) So in a way, each country and each column would have its own linear regression model that fills out the missing values.

Here is how it looked after I did the steps above: This is the expected output for ONE column. I would like to do this for EVERY column by individual country groups.

However, the data looks like this:

There are numerous countries and columns that I want to perform this on just like the post above.

This is for a project I am working on for my data-mining / statistics class. Any help would be appreciated and thanks so much in advance!

EDIT

I tried this:

The following Snakefile fails with AmbiguousRuleException:

I am using Synth() package (see ftp://cran.r-project.org/pub/R/web/packages/Synth/Synth.pdf) in R.

This is a part of my data frame:

I'm using shiny and I'm having trouble inserting multiple links in the same table cell. Every link should allow the user to download local files found on the computer. Here is an image of what I mean: table with links

For columns 2, 3, and 4, whose rows include at most only 1 link, it works perfectly; when I click on the hyperlinks I am able to download the corresponding file from my pc. However, for column 5, which includes multiple hyperlinks in each cell, I am unable to do so. Clicking on the links returns nothing; no file is downloaded (but I don't get an error).

This is the code I'm using for column 5:

Given that I have a file of N size. For the sake of example 30GB file.

Facts about the file content is that it has proprotional amount of lines. This is interleaved FastQ file. (not important for the question but usefull for someone)

File content is paired or interleaved DNA sequence of strings. Each pair is 8 lines long.

I want to process the interleaved FastQ with GNU parallel in order to speed up the process. Reason for using parallel instead of native bwa tool threads feature is that parallel helps to reduce amount of RAM needed because the nature of bwa memory allocation.

Given that interleaved file is 30GB of size I want to process chunks of --block 500M, command line params looks like --pipe --block 500M -L 8 -j 10 this then is sent as stdin to bwa and will run 10 bwa tasks each getting 500M chunks with a record of 8 lines.

Is my assumption correct that --block 500M and -L 8 will be managed by parallel and I can be certain that my bwa tool will always get 8 lines times N MB of data?

What I am not clear is, will parallel "repeat" last "chunk" if 8 lines are not present? And will it apropriatelly controll other chunk inputs for N processes I start with parallel?

Or this --block 500M "blindly" sends 500M chunk to single process regardless if last part of the 500M chunk does not contain 8 lines so to speak?

Update:

After whole day reading questions and answers on biostars and seqanswers I've realised that my testing/"benchmarking" was wrong.

But this helped to realise that I need to update the question and will make separate question.

I was testing inside Docker container which by default has very low /dev/shm thus I have mislead my self to go totaly different path.

I have a dataset that shows bilateral exports for several countries. Because the data fluctuates, I need to calculate the mean of year groups. All the countries do not cover exactly the years. Some start later, some have gaps in between - this means, some years are missing (but without having NA entries). I already managed to cut the data into pieces whith the help of an amazing community member: year_group.

Below I am listing two further problems along with my code, the wrong output and on the bottom some sample input data for the dataset total_trade

Problem 1

I am facing the issue, that the code does not calculate the right means. When I calculate the results manually, I get different results than my code. (see below)

This is my code