watermarker | Automatic advanced watermark producing tool | Computer Vision library

If you were to use per-partition watermarking, which you can do by calling assignTimestampsAndWatermarks directly on the Flink Kafka consumer [1], then I believe the idle partition(s) would consistently hold back the overall watermark.

With per-partition watermarking, each Kafka source task will apply the watermarking separately to each partition it is handling, and then emit as its watermark the minimum of those per-partition watermarks. At least one of the Kafka source tasks will therefore have a watermark of 0, and assuming you have a keyBy after the watermarking and before the process function, that will hold back the watermark at the process function.

Otherwise, if you apply watermarking to the output of the Kafka source tasks, then whether the watermarking tasks have a watermark of 0 or not depends on whether their corresponding Kafka source task has any non-idle partitions. If the assignment of partitions to instances isn't deterministic, this could explain why you are seeing different results in IntelliJ.

Note that the handling of idle sources has been reworked in Flink 1.11 [2], and bug fixes related to that are still pending [3].

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/connectors/kafka.html#kafka-consumers-and-timestamp-extractionwatermark-emission.
[2] https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/event_timestamps_watermarks.html#dealing-with-idle-sources.
[3] https://issues.apache.org/jira/browse/FLINK-18934

I Solved the Problem By passing New "WaterMarkNewHeight" and "WaterMarkNewWidth" to the class "resizeImage".

For the most part, this should be workable -- one can achieve reproducible, deterministic results when processing historic or simulated data. But it's also fairly easy to do things that lead to non-determinism. Without more information, one can only speculate on the cause(s).

Non-determinism in Flink arises from one of two things: (1) writing a job that is sensitive to races between parallel pipelines, and from (2) the use of processing time rather than event time (which you might think of as races between the events and the system clock).

For any given key, there will be only one single-threaded instance of the window operator, and it will trigger its event time windows in order, as the watermarks ascend. If the results are inconsistent within a single key, then I suspect that either (1) the watermarking is incorrectly implemented, or depends in some way on the system clock rather than event time timestamps, or (2) the events for a single key come from multiple source instances, and your windowing computation is sensitive to races between these independent sources.

And even if the results are consistent within each key, if you are then combining these per-key results into some sort of global result, it would be easy enough to introduce some non-determinism in that final combining stage.

Update:

I believe your watermarking is going to produce some indeterminate number of late events, which then leads to irreproducible results. Let me explain.

The source is being read from in parallel, and if I've understood correctly, is sorted by timestamp. However, the parallelism is then reduced down to one for watermarking. At this point multiple (possibly) ordered streams are being interleaved together, resulting in a single out-of-order stream. Then the watermarking uses maxEventTime as the current watermark, without subtracting some delay from it to account for out-of-orderness, which then can cause out-of-order events to be late.

You can confirm this diagnosis by adding side outputs to the windows to surface any late events. The easiest fix would be to set the parallelism of the source to one.

I played around with numbers and managed to put my watermark position at the 'Top Left' with the following codes:

Your incremental (i++) is wrong so all processes try to output to the same file (Watermarked0.mp4) due to the fact that i stays 0 given the current code. You need to use Interlocked.Increment(ref i) like this: