partitioner | This is the text partitioner project for Python | Data Manipulation library

I have a spring boot application that sends messages into a Kafka topic, something like:

I'm using the latest version of the S3 Sink Connector (v10.0.5) and enabled both the kafka.keys and kafka.headers but only value files are being created. Here's a copy of our config:

I'm trying to read parquet files and dump it onto mongodb collection (sharded). When i do it without sharding, the write throughput is really good. But after sharding it has gone down drastically.

A single task is taking 30 mins plus, which is only processing 16 mb data

I'm using below Spark config

It's my first Kafka program.

From a kafka_2.13-3.1.0 instance, I created a Kafka topic poids_garmin_brut and filled it with this csv:

I want to implement a custom s3 partitioner class to include some avro message fields and some extra logic to generate the output s3 path prefix

The project is in kotlin, this is my class:

I am executing model for several folds. After each fold I want to clear the TPU memory so that I don't get OOM error.

Full trace of the current error.

I have a case where I want to access StepExecution from inside a Processor which is part of a CompositeItemProcessor which is part of a partitioned step.

I am able to access the StepExecution in the very first partition but am unable to access it in the subsequent partitions.

Here is my code:

Job:

As we know from Cassandra's documentation[Link to doc] that partitioner should be such that the data is distributed evenly across multiple nodes to avoid read hotspots. Cassandra offers various partitioning algorithms for that - Murmur3Partitioner, RandomPartitioner, ByteOrderedPartitioner .

Murmur3Partitioner is the default Partitioning Algorithm set by Cassandra. It hashes the partition key and converts into the hash values ranges from -2^63 to +2^63-1. My query here is, we have different data sets which has different partition key. For example, one can set partition key with uuid type data, other can set first name and last name as partitioning key, other can set timestamp as their partitioning key and one can also set city name in partitioning key.

Now assume a data set with city as partitioning key, let's say

Node 1 stores Houston data

Node 2 stores Chicago data

Node 3 stories Phoenix data and so on...

And our data gets more entries of data with Chicago city at one instant of time, then Node 2 will have maximum records of our database and there will be hotspots in that case. In this scenario how will Cassandra manage to evenly distribute data across these nodes?

We have an http endpoint that receives some data and sends it to Kafka. We would like to immediately respond with an error if the broker is down, instead of retrying asynchronously. Is this possible at all?

What we are doing is starting the application, shutting down the broker and sending messages to see what happens. We are sending the messages using the blocking option described here.