DLT | Direct Linear Transform for 3D to 2D mapping | 3D Animation library

This is a documented limitation of Spark Structured Streaming:

Changes in the number or type (i.e. different source) of input sources: This is not allowed.

But from your description I see that you may not need to use UnionByName - you can just have N independent streams that will write into the same table. In case when you just do append to a table, concurrent appends won't lead to a write conflicts (each stream is independent!):

It seems there are two separate problems.

One is that you seem to already have records in the topics, and if you have it configured to earliest the app will read all those records when it starts up. You can either set ConsumerConfig.AUTO_OFFSET_RESET_CONFIG to latest, or, if you're running locally on docker, you can stop the Kafka container and prune the volumes with something like docker system prune --volumes (note that this will erase data from all your stopped containers - use wisely).

Can you try one of these and test again?

The other problem is that the framework is wrongly setting the default maxDelay of 30s even though the annotation states the default is to ignore. I'll open an issue for that and add the link here.

For now You can set a maxDelay such as @Backoff(delay = 600000, multiplier = 3.0, maxDelay = 5400000), then the application should have the correct delays for 10, 30 and 90 minutes as you wanted.

Let me know if that works out for you, or if you have any other problems related to this issue.

EDIT: Issue opened, you can follow the development there https://github.com/spring-projects/spring-kafka/issues/2137

It should be fixed in the next release.

EDIT 2: Actually the phrasing in the @BackOff annotation is rather ambiguous, but seems like the behavior is correct and you should explicitly set a larger maxDelay.

The documentation should clarify this behavior in the next release.

EDIT 3: To answer your question in the comments, the way retryable topics work is the partition is paused for the duration of the delay, but the consumer keeps polling the broker, so longer delays don't trigger rebalancing.

From your logs the rebalancing is from the main topic's partitions, so it's unlikely it has anything to do with this feature.

EDIT 4: The Retryable Topics feature was released in Spring for Apache Kafka 2.7.0, which uses kafka-clients 2.7.0. However, there have been several improvements to the feature, so I recommend using the latest Spring Kafka version (currently 2.8.3) if possible to benefit from those.

The stacking of the coefficient should be of the below format.

Don't create set the dataframe during the loop. The strategy is to collect records into a list of dictionary and the end, create your dataframe.

New code (search # <- HERE)

With the RecoveringBatchErrorHandler (DefaultErrorHandler in version 2.8 and later), the listener must throw a BatchListenerFailedException; this tells the error handler where, in the batch, the error occurred.

Records prior to that one will have their offsets committed; the failed record (and subsequent records in the batch) are retried; when retries are exhausted, the failed record is sent to the DLT and the remaining records are retries.

See https://docs.spring.io/spring-kafka/docs/current/reference/html/#default-eh

and

https://docs.spring.io/spring-kafka/docs/2.7.x/reference/html/#recovering-batch-eh

for versions before 2.8.

Instead of creating a h1 element, you should find it.

So replace:

Just separate them into different sections and put the child button on top with position: absolute here is a sandbox example I made to show that

Example

Don't remove the element, just clear its contents.

I'm not sure why you're doing msgs.classList.add("okay"), since it already has that class in the HTML.

No autoCreateTopics = "false" simply means that the framework will not create the topics; it has no bearing on whether the broker is configured to create them.

@EmbeddedKafka(topics = ...) will always create the topics.

To disable topic creation in the embedded kafka broker, use

@EmbeddedKafka(brokerProperties = "auto.create.topics.enable:false")

and don't specify any topics there.

Or you can set the consumer property allow.auto.create.topics to false; it overrides the broker property (if it is true).

https://kafka.apache.org/documentation/#consumerconfigs_allow.auto.create.topics