htslib | C library for high-throughput sequencing data formats | Genomics library

In bioinformatics we have the bgzip file, which is block-compressed, meaning that you can compress a file (let's say a CSV), and then if you want to access some data in the middle of that file, you can decompress only the middle chunk, rather than the entire file.

As is explained here, Arrow (and therefore Feather v2, the file format) seems to support chunked reads and writes, and also compression. However it isn't clear if the compression applies to the entire file, or if individual chunks can be decompressed. This is my questions: can we separately compress chunks of an Arrow/Feather v2 and then later decompress a single chunk without decompressing everything?

I have the following .txt file:

Suppose I have a text file such as below. My question is how to read the lines starting with SN in R. The whole file has over 10k rows but I just wanted the lines starting with SN.

I could not install htslib v1.12 with conda using either commands:

I am using htslib for extracting all the information contained in a VCF file in C++.

Currently, thanks to the VCF specification and the documentation in the file vcf.h, I have successfully extracted all the metadata information in the header (Meta-Information Lines), and most of the information contained in each row of the body of the file (Data Lines).

However, I don't know how to extract the genotype information (sample columns).

I am using example files from the 1000G project. This is an example of two rows of the file, it shows the Format field and two samples (The file has more than 1000 samples per each row, I would like to extract the data for all of them):

I have a development laptop (Mint 19.3), and a test server (Ubuntu 18.04.4 LTS).

The laptop is Docker version 19.03.5, build 633a0ea838, the server is Docker version 19.03.12, build 48a66213fe

I'm running Python 3.6 code inside the container, which uses subprocess (code below) to create an sshfs mount to a third server, after which the python code walks through the mounted directory.

Everything works fine on my development laptop. But on the server, the directory mounts (and is seen with the mount command) however cd'ing into the directory just hangs, and the Python code's subsequent walk just hangs. (NOTE: The python code never crashes or errors out. It just hangs forever.)

HOWEVER, if I manually use the same sshfs command at the container's command line, the directory works fine.

I'm at a loss as to how to troubleshoot this.

===2020-09-25 UPDATE===

OK. Since the Python code uses subprocess, the sshfs mount is obviously available to any terminal windows that wants to use it.

I have tried accessing the mount from a new terminal window inside the container, but when I cd to the mount - the window just freezes.

Well, I left everything sitting overnight - and now when I try to cd into the mount ... it works. It's like the mount has to sit for hours before it will work.

Any ideas?

Python code

I have a pipeline which uses a global singularity image and rule-based conda wrappers.

However, some of the tools don't have wrappers (i.e. htslib's bgzip and tabix).

Now I need to learn how to run jobs in containers.

In the official documentation link it says:

"Allowed image urls entail everything supported by singularity (e.g., shub:// and docker://)."

Now I've tried the following image from singularity hub but I get an error:

I want my Makefile to behave differently depending on whether I'm compiling on my own machine or inside a docker container. This is the Makefile: