kafka-connect | equivalent to kafka-connect wrench for nodejs

As discussed in the comment section, in order to further contribuite to the community, I am posting the summary of our discussion as an answer.

Since your data's destination is BigQuery, you can use the Kafka to BigQuery template in Dataflow, you can use this template to load json messages from Kafka to BigQuery. In addition, according to the documentation,

How to use this Dataflow template Kafka to BigQuery This template creates a streaming pipeline that ingests JSON data from Kafka, executes an optional JavaScript user defined function (UDF), and writes the resulting records to BigQuery. Any errors during the transformation of the data, execution of the UDF, or writing into BigQuery will be written into a separate errors table in BigQuery. The errors table will be created if it does not exist.

Pipeline Requirements

• The Kafka topic(s) exists and the message is encoded as a valid JSON.

• The BigQuery output table exists.

• The Kafka brokers are reachable from the Dataflow worker machines.

On the other hand, you can create your own template with your specific requirements using the KafkaIO method, you can check this tutorial to understand better how to start with.

MSK doesn't offer Kafka Connect as a service. You'll need to install this on your own computer, or on other AWS compute resources. From there, you need to install the Camel connector plugins

If you need to add a static field, Kafka comes with a built-in transform to do exactly that...

Regarding your issue, reading the code, it never tests or uses records that have schemas, and never builds a new Struct type

Therefore, I think your input is limited to primitive schema types such as string/integer/boolean

In other words, "Struct{a=111,b=222}" + "foo" would "work fine" and you'd end up "Struct{a=111,b=222}foo" but the string representation of the Avro record, "Struct{a=111,b=222}", has no Javascript property foo, and so it can't be set

Your alternative/workaround would be to make sure that you're consuming with the standard JSONConverter, then using JSON.parse to build an object that you can set JS properties into

Kafka Streams only interacts between topics of the same cluster, not with external systems.

Kafka Connect HDFS2 connector maintains offsets in an internal offsets topic. Older versions of it maintained offsets in the filenames and used a write-ahead log to ensure file delivery

You can whitelist Kafka topics based regex by replacing the topics property with topics.regex.

The Connect container starts Connect Distributed Server already. You should use HTTP and JSON properties to configure the Elastic connector rather than exec into the container shell and issue connect-standalone commands which default to using a broker running in the container itself.

Similarly, the Elastic quickstart file expects Elasticsearch running within the Connect container, by default

It worked only when I started my minikube without a driver (to be created on the storage of the machine and not as a VM) and specifying the 9.x ingress network ip (to get it I ran: ip a):

The error is not saying your driver doesn't exist, it's saying the Connector doesn't. Scan over your error for each PluginDesc{klass=class and you'll notice the connector.class you're trying to use isn't there

The latest Kafka Connect images from Confluent include no connectors, outside of those pre-bundled with Kafka (and some ones from Control Center, which aren't really useful), so you must install others on your own - described here

If you want to follow the 5.0 documentation, use the appropriate tagged docker image rather than latest (the old images do have the connectors installed)

Also, you would need to place the jdbc driver directly into the jdbc connector folder for it to properly be detected on the classpath; it is not a "plugin" in Connect terminology. The above link also shows an example of this

JsonConverter will be unable to consume Avro encoded data since the binary format contains a schema ID from the registry that's needed to be extracted before the converter can determine what the data looks like

You'll want to use the registryless-avro-converter, which will create a Structured object, and then should be able to converted to a Parquet record.