kafka-oauth | An AuthenticateCallbackHandler implementation for Kafka | OAuth library

This is quite a broad topic with questions that require some thorough answers. Anyway, most importantly:

• in general, Kafka scales with the number of partitions in a topic
• Spark scales with the number of worker nodes and available cores/slots
• each partition of the Kafka topic can only be consumed by a single Spark task (parallelsim then depends on the number of Spark wcores)
• if you have multiple Spark workers but only one Kafka topic partition, only one core can consume the data
• Likewise, if you have multiple Kafka topic partitions but only one worker node with a single core, the "parallelism" is 1
• remember that a formular usually represents a theory which for simplicity leaves out details. The formular you have cited is a good starting point but in the end it depends on your environment such as: requirements for latency or theoughput, network bandwith/traffic, available hardware, costs etc. That being said, only you can do testing for optimisations.

As a side note, when writing to Kafka from Spark Structured Streaming, if your Dataframe contains the column "partition" it will be used to send the record to the corresponding partition (starting from 0). You can also have the column "topic" in the dataframe which allows you to send the record to a certain topic.

Spark Structured Streaming will send each record individually to Kafka.