japi | Used to generate a beautiful API Java document | REST library

Just to extend the comment that has been added. So, basically if You use ConfluentRegistryAvroDeserializationSchema.forGeneric the data produced my the consumer isn't really String but rather GenericRecord. So, the moment You will try to use it in Your map that expects String it will fail, because your DataStream is not DataStream but rather DataStream.

Now, it works if You remove the map only because You havent specified the type when defining FlinkKafkaConsumer and your FlinkKafkaProducer, so Java will just try to cast every object to required type. Your FlinkKafkaProducer is actually FlinkKafkaProducer so there will be no problem there and thus it will work as it should.

In this particular case You don't seem to be needing Avro at all, since the data is just raw CSV.

UPDATE: Seems that You are actually processing Avro, in this case You need to change the type of Your DataStream to DataStream and all the functions You gonna write are going to work using GenericRecord not String.

So, You need something like:

Credits to Dian Fu from Flink community.

Regarding outcome 2, it is because the input date (see below) has double quotes. Handling the double quotes properly will fix the issue.

Credits to Dian Fu from Apache Flink community. See thread.

For Pandas UDF, the input type for each input argument is Pandas.Series and the result type should also be a Pandas.Series. Besides, the length of the result should be the same as the inputs. Could you check if this is the case for your Pandas UDF implementation?

Then I decide to add a pytest unit test for my UDF to verify the input and output type. Here is how:

It must be because I setup my env using pip. I have pip install-ed a few things: numpy, torch, scipy, scikit_learn, etc, and finally, apache-flink. I realize this may be problematic, therefore I setup a brand new environment with apache-flink installed only, and that resolves the above problem.

The root cause is:

You are most likely referencing akka.actor.typed.javadsl.ActorContext and not akka.actor.typed.scaladsl.ActorContext as you expect. Check your imports

I think your main class have some dependencies on the jar mentioned in the pom.xml. You're simply creating a target jar which doesn't have those dependencies included. You need to create the uber/fat jar which include all the relevant dependencies. You can use this following plugin maven-assembly-plugin for creating the target jar.

Assumption: Main.java class is under package owmapi.

The problem is that you want to give your JobManager pods unique addresses when using standby JobManagers. Hence, you must not configure a service which the components use to communicate with each other. Instead you should start your JobManager pods with the pod IP as its jobmanager.rpc.address.

In order to start each JobManager pod with its IP you must not configure a ConfigMap which contains the Flink configuration, because it would be the same configuration for every JobManager pod. Instead you need to add the following snippet to your JobManager deployment:

using confluent kafka python API for sending message

Then you must use Flink's Confluent Avro deserializer

Your error is because you're trying to consume plain Avro, which requires the schema to be part of the message (it can't find it, so throws array out of bounds)