flink-training | Apache Flink Training Excercises

I am getting started with flink and having a look at one of the official tutorials.

To my understanding the goal of this exercise is to join the two streams on the time attribute.

Task:

The result of this exercise is a data stream of Tuple2 records, one for each distinct rideId. You should ignore the END events, and only join the event for the START of each ride with its corresponding fare data.

The resulting stream should be printed to standard out.

Question: How is the EnrichmentFunction able to join the two streams aka. how does it know which fair to join with which ride? I expected it to buffer multiple fairs/rides until for an incoming fair/ride there is a matching partner.

In my understanding it just saves every ride/fair it sees and combines it with the next best ride/fair. Why is this a proper join?

Provided Solution:

I have a Flink streaming application that spends roughly 20% of its CPU time in Kyro.copy. I can evade that by turning on object reuse mode, but I have a slight problem: I'd like to modify input objects to my operators.

The general contract for object reuse mode seems to state: Do not modify input objects or remember input objects after returning from your map function. You may modify objects after output and re-emit them. (e.g.: Slide 6)

Now, my question is: If I immediately dispose of all references to objects after output-ing them from my operators, is it safe to modify input objects? Or is there some other combination of rules that can make it safe to modify input objects?

I am very new to Apache Flink. I am using v1.9.0. I want to join multiple streams example. I am getting following exception while running following example.

Exception:

I am trying to run the data artisans examples available at github. I read the tutorial and added the needed SDKs and downloaded the files for NYCFares and Rides. Whenever i am running the RideCount.java example i get a Job Execution Failed. Here is the link to the git repo for the RideCount class file. Github repo RideCount.java

here is the error

I got a example for the CEP in the following URL https://github.com/dataArtisans/flink-training-exercises/blob/master/src/main/java/com/dataartisans/flinktraining/exercises/datastream_java/cep/LongRides.java

And the "goal for this exercise is to emit START events for taxi rides that have not been matched by an END event during the first 2 hours of the ride." However from the code below, it seems get a pattern to find rides have been completed in 2 hours instead of have NOT been completed in 2 hours.

It looks like the pattern firstly find the Start event , then find the End Event(!ride.isStart), and within 2 hours, so doesn't it explains as a pattern to find rides have been completed in 2 hours?

I have cloned Flink Training repo and followed instructions on building and deploying from here in order to get familiar with Apache Flink. However, there are the errors in the projects after building and importing into Eclipse IDE. In the Flink Training Exercises project i find errors in the pom Plugin execution not covered by lifecycle configuration: net.alchim31.maven:scala-maven-plugin:3.1.4:testCompile. There are also errors in the project flink-quickstart-java . Some dependencies are not being resolved e.g. ExecutionEnvironment cannot be resolved in the BatchJob class.

I'm currently working through this tutorial on Stream processing in Apache Flink and am a little confused on how the TimeCharacteristics of a StreamEnvironment effect the order of the data values in the stream and in respect to which time an onTimer function of a ProcessFunction is called.

In the tutorial, they set the characteristics to EventTime, since we want to compare the start & end events based on the time they store and not the time they are received in the stream.

Now in the reference solution they set a timerService to fire 2 hours after an events timestamp for each key.

What really confuses me is when this timer actually fires during runtime. Possible explanation I came up with:

Setting the TimeCharacteristics to EventTime makes the stream to process the entries ordered by their event timestamp and this way the timer can be fired for each rideId, when an event arrives with a timestamp > rideId.timeStamp + 2 hours (2 hours coming from exercise context).

But with this explanation a startEvent of a Taxi ride would always be processed before an endEvent (I'm assuming that a ride can't end before it started), and we wouldn't have to check if a matching EndEvent has already arrived like they do in the processElement function.

In the documentation of ProcessFunction they state that the timer is called

"When a timer’s particular time is reached"

but since we have a (potentially infinite) stream of data and we don't care when the data point arrives but only when it happened, how can we be sure that there will not arrive a matching data point for a startEvent somewhere in the future that would trigger the criteria with 2 hours stated in the exercise?

If someone could link me an explanation of this or correct me where I'm wrong that would be highly appreciated.

I'm going through Flink tutorial materials from dataArtisans and for some reason when I get to the sample file PopularPlacesFromKafka.scala I don't get any output sent to stdout.

We are planning to use Flink to process a stream of data from a kafka topic (Logs in Json format).

But for that processing, we need to use input files which change every day, and the information within can change completely (not the format, but the contents).

Each time one of those input files changes we will have to reload those files into the program and keep the stream processing going on.

Re-loading of the data could be done same way as it is done now: