linger | Busy-indicator for the terminal | Command Line Interface library

So I discovered what the problem was. The issue really wasn't the database's fault or even the queries that were being used. It turns out that our system was using the same Transaction subsystem for both it our Data Source (Postgres Database) and our JMS messaging system. When a JMS message was sent, it created a transaction and every transactional based action that followed during the life cycle of that tread/transaction would be treated as part of that original transaction. Which includes all of our database calls.....

This explains why a query as simple as insert into a message log was touching all of our relations in the database. The debug queries only showed me the first query/statement sent to the database, not all of the others that were used during the life cycle of the JMS message. There were several ways to fix this but my team opted for the easiest which was preventing the Data Source from using the JBoss provided Transaction Manager.

Is apache running?

You must also spoof your DNS to point to your local ip, so your machine does not ask internet dns servers for an ip, which they would not be able to find.

I'm assuming you're on mac, so edit /etc/hosts and add:

This is, sadly, a known current bug:

Access doesn't close properly. A remaining background process can only be terminated in task manager

There is no official info or remedy yet.

producer.send puts data into an internal queue, which is only sent to the broker when the producer is flushed (which is the effect of calling .get().

If you need to detect a connection before calling .send, then you need to actually make the connection beforehand, for example, using an AdminClient.describeCluster method call

There are multiple ways in which a python script can communicate with a subprocess when it comes to keypresses.

• pywin32
• pynput
• pyautogui
• ctypes + user32.dll

(credits to @john-hen -> inspired from https://stackoverflow.com/a/8117562/858565)

Package: https://pypi.org/project/pywin32/

We identified some test sequence dependency between the Kafka tests.

We updated our Gradle version to 7.3-rc-3 which has a more deterministic approach to test scanning. This update "solved" our problem while we prepare to fix the tests' dependencies.

There's no straightforward non-hacky way. (And I don't even know if there is a hacky way).

What you propose violates the encapsulation principle. A class hides its nitty gritty dirty internals and only exposes a neat interface with promised behavior.

Inheritance is not a mechanism to violate this principle.

In your concrete example, the issue comes from a bad interface design of T. If T had a method compute_a() that would return self.p + self.b then in your inherited class you can of course call self.compute_a().

But only do this if a is more than a mere internal implementation detail!

The configurations spark.streaming.[...] you are referring to belong to the Direct Streaming (aka Spark Streaming) and not to Structured Streaming.

In case you are unaware of the difference, I recommend to have a look at the separate programming guides:

• Structured Streaming: processing structured data streams with relation queries (using Datasets and DataFrames, newer API than DStreams)
• Spark Streaming: processing data streams using DStreams (old API)

Structured Streaming does not provide a backpressure mechanism. As you are consuming from Kafka you can use (as you are already doing) the option maxOffsetsPerTrigger to set a limit on read messages on each trigger. This option is documented in the Structured Streaming and Kafka Integration Guide as:

"Rate limit on maximum number of offsets processed per trigger interval. The specified total number of offsets will be proportionally split across topicPartitions of different volume."

In case you are still interested in the title question

How is spark.streaming.kafka.maxRatePerPartition related to spark.streaming.backpressure.enabled in case of spark streaming with Kafka?

This relation is explained in the documentation on Spark's Configuration:

"Enables or disables Spark Streaming's internal backpressure mechanism (since 1.5). This enables the Spark Streaming to control the receiving rate based on the current batch scheduling delays and processing times so that the system receives only as fast as the system can process. Internally, this dynamically sets the maximum receiving rate of receivers. This rate is upper bounded by the values spark.streaming.receiver.maxRate and spark.streaming.kafka.maxRatePerPartition if they are set (see below)."

All details on the backpressure mechanism available in Spark Streaming (DStream, not Structured Streaming) are explained in the blog that you have already linked Enable Back Pressure To Make Your Spark Streaming Application Production Ready.

Typically, if you enable backpressure you would set spark.streaming.kafka.maxRatePerPartition to be 150% ~ 200% of the optimal estimated rate.

The exact calculation of the PID controller can be found in the code within the class PIDRateEstimator.

As you asked for an example, here is one that I have done in one of my productive applications:

• Kafka topic has 16 partitions
• Spark runs with 16 worker cores, so each partitions can be consumed in parallel
• Using Spark Streaming (not Structured Streaming)
• Batch interval is 10 seconds
• spark.streaming.backpressure.enabled set to true
• spark.streaming.kafka.maxRatePerPartition set to 10000
• spark.streaming.backpressure.pid.minRate kept at default value of 100
• The job can handle around 5000 messages per second per partition
• Kafka topic contains multiple millions of messages in each partitions before starting the streaming job

• In the very first batch the streaming job fetches 16000 (= 10 seconds * 16 partitions * 100 pid.minRate) messages.
• The job is processing these 16000 message quite fast, so the PID controller estimates an optimal rate of something larger than the masRatePerPartition of 10000.
• Therefore, in the second batch, the streaming job fetches 16000 (= 10 seconds * 16 partitions * 10000 maxRatePerPartition) messages.
• Now, it takes around 22 seconds for the second batch to finish
• Because our batch interval was set to 10 seconds, after 10 seconds the streaming job schedules already the third micro-batch with again 1600000. The reason is that the PID controller can only use performance information from finished micro-batches.
• Only in the sixth or seventh micro-batch the PID controller finds the optimal processing rate of around 5000 messages per second per partition.

This is quite a broad topic with questions that require some thorough answers. Anyway, most importantly:

• in general, Kafka scales with the number of partitions in a topic
• Spark scales with the number of worker nodes and available cores/slots
• each partition of the Kafka topic can only be consumed by a single Spark task (parallelsim then depends on the number of Spark wcores)
• if you have multiple Spark workers but only one Kafka topic partition, only one core can consume the data
• Likewise, if you have multiple Kafka topic partitions but only one worker node with a single core, the "parallelism" is 1
• remember that a formular usually represents a theory which for simplicity leaves out details. The formular you have cited is a good starting point but in the end it depends on your environment such as: requirements for latency or theoughput, network bandwith/traffic, available hardware, costs etc. That being said, only you can do testing for optimisations.

As a side note, when writing to Kafka from Spark Structured Streaming, if your Dataframe contains the column "partition" it will be used to send the record to the corresponding partition (starting from 0). You can also have the column "topic" in the dataframe which allows you to send the record to a certain topic.

Spark Structured Streaming will send each record individually to Kafka.