flink | Apache Flink is an open source project | SQL Database library

Update: The answer is that the KafkaSource behaves differently than FlinkKafkaConsumer in the case where the number of Kafka partitions is smaller than the parallelism of Flink's kafka source operator. See https://stackoverflow.com/a/70101290/2000823 for details.

Original answer:

The problem is almost certainly something related to the timestamps and watermarks.

To verify that timestamps and watermarks are the problem, you could do a quick experiment where you replace the 3-hour-long event time sliding windows with short processing time tumbling windows.

In general it is preferred (but not required) to have the KafkaSource do the watermarking. Using forMonotonousTimestamps in a watermark generator applied after the source, as you are doing now, is a risky move. This will only work correctly if the timestamps in all of the partitions being consumed by each parallel instance of the source are processed in order. If more than one Kafka partition is assigned to any of the KafkaSource tasks, this isn't going to happen. On the other hand, if you supply the forMonotonousTimestamps watermarking strategy in the fromSource call (rather than noWatermarks), then all that will be required is that the timestamps be in order on a per-partition basis, which I imagine is the case.

As troubling as that is, it's probably not enough to explain why the windows don't produce any results. Another possible root cause is that the test data set doesn't include any events with timestamps after the first window, so that window never closes.

Do you have a sink? If not, that would explain things.

You can use the Flink dashboard to help debug this. Look to see if the watermarks are advancing in the window tasks. Turn on checkpointing, and then look to see how much state the window task has -- it should have some non-zero amount of state.

Some OSS components are offered as Dataproc Optional Components. Not of all them have a job submit API, some (e.g., Anaconda, Jupyter) don't need one, some (e.g., Flink, Druid) might add in the future.

Some other OSS components are offered as libraries, e.g., GCS connector, BigQuery connector, Apache Parquet.

Following approaches can be used to read from the database and create a datastream :

1. You can use RichParallelSourceFunction where you can do a custom query to your database and get the datastream from it. An SQL with JDBC driver can be fired in the extension of RichParallelSourceFunction class.

2. Using Table DataStream API - It is possible to query a Database by creating a JDBC catalog and then transform it into a stream

3. An alternative to this, a more expensive solution perhaps - You can use a Flink CDC connectors which provides source connectors for Apache Flink, ingesting changes from different databases using change data capture (CDC)

Then you can add Kafka as source and get a datastream.

So, briefly your your pipeline could look like follows : You have both the sources transformed as a datastreams, you can join these streams using, for example coprocess function which will also give you possibility to maintain a state and use it in your business logic. And finally Sink your final output to either a database, to Kafka or to even AWS S3 buckets using a Sink function.

You'll find some relevant details in the docs for the Table / JDBC connector: https://ci.apache.org/projects/flink/flink-docs-stable/docs/connectors/table/jdbc/#features. See especially the section describing the Lookup Cache, which says

JDBC connector can be used in temporal join as a lookup source (aka. dimension table). Currently, only sync lookup mode is supported.

By default, lookup cache is not enabled. You can enable it by setting both lookup.cache.max-rows and lookup.cache.ttl.

The lookup cache is used to improve performance of temporal join the JDBC connector. By default, lookup cache is not enabled, so all the requests are sent to external database. When lookup cache is enabled, each process (i.e. TaskManager) will hold a cache. Flink will lookup the cache first, and only send requests to external database when cache missing, and update cache with the rows returned. The oldest rows in cache will be expired when the cache hit to the max cached rows lookup.cache.max-rows or when the row exceeds the max time to live lookup.cache.ttl. The cached rows might not be the latest, users can tune lookup.cache.ttl to a smaller value to have a better fresh data, but this may increase the number of requests send to database. So this is a balance between throughput and correctness.

You can view Temporal Table Function as a previous version of some sorts of versioned tables (This is also stated in the legacy features part of Flink).

In your first example, you create an append only table which will hold multiple keys of the same currency (i.e. EURO), which makes it ineligible to be a primary key.

Can you call an append only table a versioned table? The definition on Versioned Tables in doc says:

Flink SQL can define versioned tables over any dynamic table with a PRIMARY KEY constraint and time attribute.

This implies that an append only table cannot serve as a versioned table since it can't, as we said, hold true the primary key constraint.

But, since the append only table in your example does hold the relevant information to become a versioned table, containing inserts as well as updates and deletes, we can turn it to one with a deduplication query, as you posted above.

Triggers are for windows, but windowing doesn't seem appropriate for your use case.

A better solution would be to use Timers with a KeyedProcessFunction. Depending on whether you want to wait for 10 minutes of processing time, or 10 minutes of event time, you'll choose processing time timers or event time timers.

You'll also need to use Flink state to store the events that need to be processed later.

You'll find the documentation for process functions here. There are some additional examples in the Flink training, here and here.

FWIW, Flink's Stateful Functions API might be a better fit for what you're doing, in which case you would use delayed messages.

There are 3 options that I can see:

1. Try out the JDBC 1.13 connector with your Flink version. There is a good chance it might just work.
2. If that doesn't work immediately, check if you can backport it to 1.11. There shouldn't be too many changes.
3. Write your own 2-phase-commit sink, either by extending TwoPhaseCommitSinkFunction or implement your own SinkFunction with CheckpointedFunction and CheckpointListener. Basically, you create a new transaction after a successful checkpoint and commit it with notifyCheckpointCompleted.

When the docs says NonInferrableReturnType it means that we can use the type variable , or any other letter that you prefer. So you can create a MapFunction that return a T. But then you have to use .returns(TypeInformation.of(String.class) for example, if your goal is to return a String.

Yes, sinks are required as part of Flink's execution model:

DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., filtering, updating state, defining windows, aggregating). The data streams are initially created from various sources (e.g., message queues, socket streams, files). Results are returned via sinks, which may for example write the data to files, or to standard output (for example the command line terminal)

One could argue that your that the call to your datastore is the actual sink implementation that you could use. You could define your own sink and execute the datastore call there.

I am not keen on the details of your datastore, but one could assume that you are serializing these events and sending them to the datastore in some way. In that case, you could flow all your elements to the sink operator, and store each of these elements in some ListState which you can continuously offload and send. This way, if your application needs to be upgraded, in flight records will not be lost and will be recovered and sent once the job has restored.