ink-table | 📐A table component for Ink | Frontend Framework library

I followed (ZIP compressed input for Apache Flink) and wrote the following code piece to process .gz log files in a dir with a simple TextInputFormat. It works on my local test directory, scans and automatically opens the .gz file contents. However, when I run it with an s3 bucket source, it does not process .gz compressed files. This Flink job still opens .log files on the s3 bucket though. Seems it just does not uncompress the .gz files. How can I get this resolved on s3 file system?

Background: I'm trying to get an event-time temporal join working with two 'large(r)' datasets/tables that are read from a CSV-file (16K+ rows in left table, somewhat less in right table). Both tables are append-only tables, i.e. their datasources are currently CSV-files, but will become CDC changelogs emitted by Debezium over Pulsar. I am using the fairly new SYSTEM_TIME AS OF syntax.

The problem: join results are only partly correct, i.e. at the start (first 20% or so) of the execution of the query, rows of the left-side are not matched with rows from the right side, while in theory, they should. After a couple of seconds, there are more matches, and by the time the query ends, rows of the left side are getting matched/joined correctly with rows of the right side. Every time that I run the query it shows other results in terms of which rows are (not) matched.

Both datasets are not ordered by their respective event-times. They are ordered by their primary key. So it's really this case, only with more data.

In essence, the right side is a lookup-table that changes over time, and we're sure that for every left record there was a matching right record, as both were created in the originating database at +/- the same instant. Ultimately our goal is a dynamic materialized view that contains the same data as when we'd join the 2 tables in the CDC-enabled source database (SQL Server).

Obviously, I want to achieve a correct join over the complete dataset as explained in the Flink docs
Unlike simple examples and Flink test-code with a small dataset of only a few rows (like here), a join of larger datasets does not yield correct results.

I suspect that, when the probing/left table starts flowing, the build/right table is not yet 'in memory' which means that left rows don't find a matching right row, while they should -- if the right table would have started flowing somewhat earlier. That's why the left join returns null-values for the columns of the right table.

I've included my code:

I am trying to connect to Kafka. When I run a simple JAR file, I get the following error:

env: HDP: 3.1.5(hadoop: 3.1.1, hive: 3.1.0), Flink: 1.12.2 Java code:

I have a simple Flink streaming app. It runs well in a cluster created by start-cluster.sh command.

Now based on the Flink tutorial, I hope to deploy it in application mode natively in a Kubernetes cluster created by k3d on macOS.

First, I created a cluster by k3d cluster create dev.

Here is my Dockerfile:

When I upgrade my Flink Java app from 1.12.2 to 1.12.3, I get a new runtime error. I can strip down my Flink app to this two liner:

I have a Flink job that runs well locally but fails when I try to flink run the job on cluster. It basically reads from Kafka, do some transformation, and writes to a sink. The error happens when trying to load data from Kafka via 'connector' = 'kafka'.

Here is my pom.xml, note flink-connector-kafka is included.

I'm trying to map the Map[String,String] object output of my Scala UDF (scala.collection.immutable.map) to some valid data type in the Table API, namely via Java type (java.util.Map) as recommended here: Flink Table API & SQL and map types (Scala). However I get below error.

Any idea about right way to proceed ? If yes, is there a way to generalize the conversion to a (nested) Scala object of type Map[String,Any] ?

Code

Scala UDF

I'm trying to register a Scala UDF in Pyflink using an external JAR as follows, but get below error.

Scala UDF: