flink | Docker base image for creating Apache Flink clusters

Yes, sinks are required as part of Flink's execution model:

DataStream programs in Flink are regular programs that implement transformations on data streams (e.g., filtering, updating state, defining windows, aggregating). The data streams are initially created from various sources (e.g., message queues, socket streams, files). Results are returned via sinks, which may for example write the data to files, or to standard output (for example the command line terminal)

One could argue that your that the call to your datastore is the actual sink implementation that you could use. You could define your own sink and execute the datastore call there.

I am not keen on the details of your datastore, but one could assume that you are serializing these events and sending them to the datastore in some way. In that case, you could flow all your elements to the sink operator, and store each of these elements in some ListState which you can continuously offload and send. This way, if your application needs to be upgraded, in flight records will not be lost and will be recovered and sent once the job has restored.

The reason for the difference has two parts, both of them defined in GroupAggFunction, which is the process function used to process this query.

The first is this part of the code:

From the comments on the source code

In Flink 1.13 we reorganized the state backends because the old way had resulted in many misunderstandings about how things work. So these two concerns were decoupled:

1. Where your working state is stored (the state backend). (In the case of RocksDB, it should be configured to use the fastest available local disk.)
2. Where checkpoints are stored (the checkpoint storage). In most cases, this should be a distributed filesystem.

With the old API, the fact that two different filesystems are involved in the case of RocksDB was obscured by the way the checkpointing path was passed to the RocksDBStateBackend constructor. So that bit of configuration has been moved elsewhere (see below).

This table shows the relationships between the legacy state backends and the new ones (in combination with checkpoint storage):

In your case you want to use the EmbeddedRocksDBStateBackend with FileSystemCheckpointStorage. The problem you are currently having is that you are using in-memory checkpoint storage (JobManagerCheckpointStorage) with RocksDB, which severely limits how much state can be checkpointed.

You can fix this by either specifying a checkpoint directory inflink-conf.yaml

The KafkaTopicPartitionAssigner distributes the partitions of each topic uniformly across the subtasks in a round-robin fashion. The subtask to which partition 0 is assigned is determined using the topic name.

This is intended to evenly distribute the load among the parallel workers without requiring any intervention on your part. But if you do want explicit, fine-grained control, you should stick to instantiating separate consumers.

If using minikube you need to first mount the volume using

Kafka Streams only interacts between topics of the same cluster, not with external systems.

Kafka Connect HDFS2 connector maintains offsets in an internal offsets topic. Older versions of it maintained offsets in the filenames and used a write-ahead log to ensure file delivery

The record you are wondering about

With Flink SQL we speak of the duality between tables and streams -- that a stream can be thought of as a (dynamic) table, and vice versa. There are two types of streams/tables: appending and updating. An append stream corresponds to a dynamic table that only performs INSERT operations; nothing is ever deleted or updated. And an update stream corresponds to a dynamic table where rows can be updated and deleted.

Your source table is an upsert-kafka table, and as such, is an update table (not an appending table). An upsert-kafka source corresponds to a compacted topic, and when compactions occur, that leads to updates/retractions where the existing values for various keys are updated over time.

When an updating table is converted into a stream, there are two possible results: you either get an upsert stream or a retraction stream. Some sinks support one or the other of these types of update streams, and some support both.

What you are seeing is that the upsert-kafka sink can handle upserts, and the print sink cannot. So the same update table is being fed to Kafka as a stream of upsert (and possibly deletion) events, and it's being sent to stdout as a stream with an initial insert (+I) for each key, followed by update_before/update_after pairs encoded as -U +U for each update (and deletions, were any to occur).