keyspace | consistently replicated , fault-tolerant key | Key Value Database library

To get rid of the first issue I figured I had overlapping prefixes I was using during configuration.

Next I modified session earlier using super.cassandraSession() to instead use

Data in Cassandra is ordered only inside the Cassandra partition, but partitions themselves aren't sorted by value, and organized by hash of partition key. So when you read data, you can read nearby Cassandra partitions, but they may belong to completely different dates.

So if you have data sorted in Spark, you need to explicitly sort data using the .orderBy

It isn't possible to increase the number of streaming threads. In any case, there are several factors which affect the speed of the streaming, not just network throughput. The type of disks as well as the data model have a significant impact on how quick the JVM can serialise the data to stream as well as how quick it can cleanup the heap (GC).

I see that you've already tried to increase the streaming throughput. Note that you'll need to increase it for both the sending and receiving nodes (and really, all nodes) otherwise, the stream will only be as fast as the slowest node. Cheers!

No, the hint is not saved because the CL is not met. Hints are not included in the CL, so since you have 2 nodes out of 3 down, the quorum CL is not met and the driver will receive an UnavailableException from the coordinator.

It is a valid upgrade path but bear in mind that there could be some disadvantages with your proposed approach. For example, if a node goes down (say for a hardware failure) then you won't be able to decommission it.

Any operation that requires streaming will not work in a mixed-version cluster. Those operations include bootstrap, decommission, repairs.

To answer your questions directly:

• A. Yes, it will work but with some gotchas.
• B. No, adding a new DC is the only way you can change the number of tokens.
• C. Yes, replication is designed to work in mixed-versions.

To answer the question you didn't ask: rolling back an upgrade is actually quite rare in my experience. You would typically upgrade one node at a time. If you run into a problem on a node, you would fix that node then proceed with the rolling upgrade until all nodes in the cluster have been upgraded.

During the rolling upgrade, your application should continue to work and so there should be no reason to perform a rollback. Cheers!

It looks like you cannot. From the doc mentioned in the original post, the supported syntax is:

As I understand, you want to use a different schema/keyspace (cassandra!?) in different environments!?

Make your sql "schema/keyspace-free":

You can't use connector compiled for Scala 2.11 with Spark 3.2.0 that is compiled with Scala 2.12. You need to use appropriate version - right now it's 3.1.0 with coordinates com.datastax.spark:spark-cassandra-connector_2.12:3.1.0

P.S. Please note that although basic functionality will work, more advanced functionality won't work until the SPARKC-670 is fixed (see this PR)

Actually, after trying all day, I found a way using Pandas to insert all the data from the .csv file: treatment using pd.Dataframe and a "for" inserting item by item, as below: