sstable | bigdata processing in golang

It isn't possible to increase the number of streaming threads. In any case, there are several factors which affect the speed of the streaming, not just network throughput. The type of disks as well as the data model have a significant impact on how quick the JVM can serialise the data to stream as well as how quick it can cleanup the heap (GC).

I see that you've already tried to increase the streaming throughput. Note that you'll need to increase it for both the sending and receiving nodes (and really, all nodes) otherwise, the stream will only be as fast as the slowest node. Cheers!

is memtable durable?

There is the memtable which is flushed to disk based on size / a few other settings, but at the point the write is accepted - it is not durable in the memtable. There is also an entry placed in the commitlog which by default will flush every 10 seconds. (so on RF 3, you would expect a flush every 3.33 seconds). The flushing of the commitlog makes it durable to that specific node. To entirely lose the write before this flush has occurred would require all replicas to have failed before any of them had performed a commit log flush. As long as 1 of them flushed, it would be durable.

if there is heavy update qps does it mean that there is going to be multiple versions of stale data in both memtable and sstable such that read latency can increase?

In terms of the memtable, no there will not be stale data. In terms of the SSTables on disk, yes, there can be multiple versions of a record as it is updated over time which would lead to an increase in read latencies. A good metric to look at is the SSTablesPerRead metric which will give you the histogram of how many SSTables are being accessed per DB Table for the queries you run. The p95 or higher is the main value to look at, these will be the scenarios that will be causing slowness.

how does cassandra get the latest data? and how is multiple version of data stored?

During the read of the data, it will use the read path (bloom filters, partition summary etc) and read all versions of the row - and discard the parts which are not needed, before returning the records to the calling application. The multiple versions of the row is a facet of it existing in more than 1 sstable.

Part of the role of compaction is to manage this scenario and to bring together the multiple copies, older and newer versions of a record, and writing out new SStables which only retain the newer version. (and the SSTables it compacted together are removed).

if there is heavy update qps does this mean there is alot of tombstone?

This depends on the type of update, for most normal updates - no, this does not generate tombstones. Updates on list collection types though can and will generate tombstones. If you are issuing deletions, then yes, it will generate tombstones.

If you are going to be running a scenario of heavy updates, then I would recommend considering LeveledCompactionStrategy instead of a default SizeTieredCompactionStrategy - it is likely to provide you better read performance, but at a higher compaction IO cost.

for exporting/importing data from Apache Cassandra®, there is an efficient tool -- DataStax Bulk Loader (aka DSBulk). You could refer to more documentation and examples here. For getting consistent reads and writes, you could leverage --datastax-java-driver.basic.request.consistency LOCAL_QUORUM in your unload & load commands.

The error is here: Too many open files <- you need to increase the limit on the number of open files. This could be done with ulimit command and make it permanent as described in this answer.

I personally would start with checking if the whole space is occupied by actual data, and not by snapshots - use nodetool listsnapshots to list them, and nodetool clearsnapshot to remove them. Because if you did snapshot for some reason, then after compaction they are occupying the space as original files were removed.

The next step would be to try to cleanup tombstones & deteled data from the small tables using nodetool garbagecollect, or nodetool compact with -s option to split table into files of different size. For big table I would try to use nodetool compact with --user-defined option on the individual files (assuming that there will be enough space for them). As soon as you free > 200Gb, you can sstablesplit (node should be down!) to split big SSTable into small files (~1-2Gb), so when node starts again, the data will be compacted

If new data is coming, they are landing in new SSTables, not modifying existing ones. Each SSTable is read separately, and then data is consolidated from all SSTables and memtable, and then put into the correct order in memory before sending. See this doc, for example, on how data is read.

Something like this, but you need to make sure that you have data on the host system, or in the Docker volume (it's good idea anyway):

• stop container
• execute docker run -it ...volume_config... --rm cassandra sstable_command
• start container

P.S. But it really depends on the command - I remember that some commands were documented as required stop, but not really required

The logs show that Cassandra failed in the validation phase of the anti-entropy repair process.

As "Cannot start multiple repair sessions over the same sstables" this means there're multiple repair sessions on the same token range at the same time.

You need to make sure that you have no repair session currently running on your cluster, and no anti-compaction.

I suggest rolling restart in order to stop all running repairs. then try to repair node by node.

One last suggestion is to try https://github.com/thelastpickle/cassandra-reaper it used to run automated repairs for Cassandra.

Nodetool tpstats revealed that there were indeed active repair jobs, but they were actually not running or compactionstats did not show any running jobs. So I restarted just the nodes on which the repair was stuck and this cleared up those stuck repair jobs and I was able to run a fresh repair after that.