darkq | A message queue for genomic surveillance

darkq is a Python library. darkq has no bugs, it has no vulnerabilities and it has low support. However darkq build file is not available. You can download it from GitHub.

Genomic surveillance is not only limited by the speed at which genomes can be shared, but by how fast they can reach the right person or algorithm to analyse them. If you are monitoring a regional outbreak of a multiresistant Klebsiella pneumoniae and I send you some viral genomes, this will not be too useful. DarkQ is a messaging queue for microbial genomes. It tries to solve two problems: First, how to know about interesting genomes having been sequenced elsewhere, without having to download these genomes first. And second, how to retrieve the original genomes easily once we observe something interesting. With DarkQ, publishers (P) send (infectious) messages and consumers (C) subscribe to a filtered subset, depending on their preferences as expressed by a set of tags (see below). Say I publish all the stuff we sequence in my lab during the month, amongst other things some Klebsiella. Now you subscribe to the Klebsiella queue, or even more general to all Enterobacterales from Germany, and you filter the messages for genomes that are similar to the isolates in your current outbreak. You might discover that some of my isolates are from the same lineage as your outbreak isolates. You quickly and automatically obtain the original genome sequences, analyse them in detail, and indeed, our houses are part of the same outbreak. Now you can contact me and we can take collective action to limit the spread of this strain. DarkQ relies on two main components, a message queue and a file sharing protocol. Messages are MinHash signatures of the underlying genomes -- think lossy compression [1]. They can be wired efficiently across the message queue and you can compare the similarity of a pair of genomes through these signatures, too. DarkQ uses the sourmash implementation of MinHash [2]. If a given message passes the filters (subscription, genome similarity), it is downloaded through the second component of DarkQ, namely the Interplanetary File System (IPFS) protocol. Basically, it allows decentralized, peer-to-peer file sharing, which we think is crucial for effective genomic surveillance. [1]: Ondov et al., 2016. “Mash: Fast Genome and Metagenome Distance Estimation Using MinHash.” Genome Biology 17 (1): 132. [2]: Pierce et al., 2019. “Large-Scale Sequence Comparisons with sourmash.” F1000Research 8 (July): 1006.