flink-training-exercises | repository contains reference solutions and utility classes

I am very new to Apache Flink. I am using v1.9.0. I want to join multiple streams example. I am getting following exception while running following example.

Exception:

I am trying to run the data artisans examples available at github. I read the tutorial and added the needed SDKs and downloaded the files for NYCFares and Rides. Whenever i am running the RideCount.java example i get a Job Execution Failed. Here is the link to the git repo for the RideCount class file. Github repo RideCount.java

here is the error

I got a example for the CEP in the following URL https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataartisans/flinktraining/exercises/datastream_java/cep/LongRides.java

And the "goal for this exercise is to emit START events for taxi rides that have not been matched by an END event during the first 2 hours of the ride." However from the code below, it seems get a pattern to find rides have been completed in 2 hours instead of have NOT been completed in 2 hours.

It looks like the pattern firstly find the Start event , then find the End Event(!ride.isStart), and within 2 hours, so doesn't it explains as a pattern to find rides have been completed in 2 hours?

I have cloned Flink Training repo and followed instructions on building and deploying from here in order to get familiar with Apache Flink. However, there are the errors in the projects after building and importing into Eclipse IDE. In the Flink Training Exercises project i find errors in the pom Plugin execution not covered by lifecycle configuration: net.alchim31.maven:scala-maven-plugin:3.1.4:testCompile. There are also errors in the project flink-quickstart-java . Some dependencies are not being resolved e.g. ExecutionEnvironment cannot be resolved in the BatchJob class.

I'm currently working through this tutorial on Stream processing in Apache Flink and am a little confused on how the TimeCharacteristics of a StreamEnvironment effect the order of the data values in the stream and in respect to which time an onTimer function of a ProcessFunction is called.

In the tutorial, they set the characteristics to EventTime, since we want to compare the start & end events based on the time they store and not the time they are received in the stream.

Now in the reference solution they set a timerService to fire 2 hours after an events timestamp for each key.

What really confuses me is when this timer actually fires during runtime. Possible explanation I came up with:

Setting the TimeCharacteristics to EventTime makes the stream to process the entries ordered by their event timestamp and this way the timer can be fired for each rideId, when an event arrives with a timestamp > rideId.timeStamp + 2 hours (2 hours coming from exercise context).

But with this explanation a startEvent of a Taxi ride would always be processed before an endEvent (I'm assuming that a ride can't end before it started), and we wouldn't have to check if a matching EndEvent has already arrived like they do in the processElement function.

In the documentation of ProcessFunction they state that the timer is called

"When a timer’s particular time is reached"

but since we have a (potentially infinite) stream of data and we don't care when the data point arrives but only when it happened, how can we be sure that there will not arrive a matching data point for a startEvent somewhere in the future that would trigger the criteria with 2 hours stated in the exercise?

If someone could link me an explanation of this or correct me where I'm wrong that would be highly appreciated.

I'm going through Flink tutorial materials from dataArtisans and for some reason when I get to the sample file PopularPlacesFromKafka.scala I don't get any output sent to stdout.

We are planning to use Flink to process a stream of data from a kafka topic (Logs in Json format).

But for that processing, we need to use input files which change every day, and the information within can change completely (not the format, but the contents).

Each time one of those input files changes we will have to reload those files into the program and keep the stream processing going on.

Re-loading of the data could be done same way as it is done now:

I am working through this Apache Flink training where you create a simple application to reads data from a file and filters it. I am using Scala as the language to write the Flink application, and the final code looks like this: