flink | Apache Flink is an open source project | SQL Database library

by apache Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | flink Summary

flink is a Java library typically used in Telecommunications, Media, Media, Entertainment, Database, SQL Database, Spark, Hadoop applications. flink has build file available, it has a Permissive License and it has medium support. However flink has 2490 bugs and it has 21 vulnerabilities. You can download it from GitHub, Maven.

Apache Flink is an open source project of The Apache Software Foundation (ASF). The Apache Flink project originated from the Stratosphere research project.

Support

Quality

Security

License

Reuse

kandi-support Support

flink has a medium active ecosystem.
It has 18609 star(s) with 10596 fork(s). There are 929 watchers for this library.
It had no major release in the last 12 months.
flink has no issues reported. There are 871 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of flink is current.

flink Support

Best in #SQL Database

Average in #SQL Database

flink Support

Best in #SQL Database

Average in #SQL Database

quality kandi Quality

flink has 2490 bugs (137 blocker, 120 critical, 1174 major, 1059 minor) and 38707 code smells.

flink Quality

Best in #SQL Database

Average in #SQL Database

flink Quality

Best in #SQL Database

Average in #SQL Database

security Security

flink has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
flink code analysis shows 21 unresolved vulnerabilities (15 blocker, 5 critical, 0 major, 1 minor).
There are 1446 security hotspots that need review.

flink Security

Best in #SQL Database

Average in #SQL Database

flink Security

Best in #SQL Database

Average in #SQL Database

license License

flink is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

flink License

Best in #SQL Database

Average in #SQL Database

flink License

Best in #SQL Database

Average in #SQL Database

build Reuse

flink releases are not available. You will need to build from source code and install.
Deployable package is available in Maven.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
flink saves you 4780464 person hours of effort in developing the same functionality from scratch.
It has 1703645 lines of code, 119488 functions and 14240 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

flink Reuse

Best in #SQL Database

Average in #SQL Database

flink Reuse

Best in #SQL Database

Average in #SQL Database

Top functions reviewed by kandi - BETA

kandi has reviewed flink and discovered the below as its top functions. This is intended to give you an instant insight into flink implemented functionality, and help decide if they suit your requirements.

Initialize handlers .

Starts the application .

Generate a select logical plan .

Registers an alias from the given node .

Perform a phase 1 .

Performs the actual operation .

Returns the converter for the given data type .

Runs the loop .

Load resource .

Builds an execution graph .

Get all kandi verified functions for this library.

flink Key Features

A streaming-first runtime that supports both batch processing and data streaming programs

Elegant and fluent APIs in Java and Scala

A runtime that supports very high throughput and low event latency at the same time

Support for event time and out-of-order processing in the DataStream API, based on the Dataflow Model

Flexible windowing (time, count, sessions, custom triggers) across different time semantics (event time, processing time)

Fault-tolerance with exactly-once processing guarantees

Natural back-pressure in streaming programs

Libraries for Graph processing (batch), Machine Learning (batch), and Complex Event Processing (streaming)

Built-in support for iterative programs (BSP) in the DataSet (batch) API

Custom memory management for efficient and robust switching between in-memory and out-of-core data processing algorithms

Compatibility layers for Apache Hadoop MapReduce

Integration with YARN, HDFS, HBase, and other components of the Apache Hadoop ecosystem

flink Examples and Code Snippets

See all related Code Snippets

Streaming Example

Copy

Download

case class WordWithCount(word: String, count: Long)

val text = env.socketTextStream(host, port, '\n')

val windowCounts = text.flatMap { w =&gt; w.split("\\s") }
  .map { w =&gt; WordWithCount(w, 1) }
  .keyBy("word")
  .window(TumblingProcessingTimeWindow.of(Time.seconds(5)))
  .sum("count")

windowCounts.print()

Batch Example

Copy

Download

case class WordWithCount(word: String, count: Long)

val text = env.readTextFile(path)

val counts = text.flatMap { w =&gt; w.split("\\s") }
  .map { w =&gt; WordWithCount(w, 1) }
  .groupBy("word")
  .sum("count")

counts.writeAsCsv(outputPath)

Building Apache Flink from Source

Copy

Download

git clone https://github.com/apache/flink.git
cd flink
mvn clean package -DskipTests # this will take up to 10 minutes

Flink capped MapState

Copy

Download

map[nextIndex++] = thisEvent;
if (nextIndex - oldestIndex > n) {
  map[oldestIndex++].clear();
}

javadoc of SingleOutputStreamOperator#returns(TypeHint<T> typeHint) method

Copy

Download

public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}

public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}

import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}

public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}

public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}

import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}

public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}

public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}

import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}

toChanglelogStream prints different kinds of changes

Copy

Download

// update aggregate result and set to the newRow
if (isAccumulateMsg(input)) {
    // accumulate input
    function.accumulate(input);
} else {
    // retract input
    function.retract(input);
}

} else {
    // we retracted the last record for this key
    // sent out a delete message
    if (!firstRow) {
        // prepare delete message for previous row
        resultRow.replace(currentKey, prevAggValue).setRowKind(RowKind.DELETE);
        out.collect(resultRow);
    }
    // and clear all state
    accState.clear();
    // cleanup dataview under current key
    function.cleanup();
}

if (!recordCounter.recordCountIsZero(accumulators)) {
    // we aggregated at least one record for this key

    // update the state
    accState.update(accumulators);

    // if this was not the first row and we have to emit retractions
    if (!firstRow) {
        if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue, newAggValue)) {
            // newRow is the same as before and state cleaning is not enabled.
            // We do not emit retraction and acc message.
            // If state cleaning is enabled, we have to emit messages to prevent too early
            // state eviction of downstream operators.
            return;
        } else {
            // retract previous result
            if (generateUpdateBefore) {
                // prepare UPDATE_BEFORE message for previous row
                resultRow
                        .replace(currentKey, prevAggValue)
                        .setRowKind(RowKind.UPDATE_BEFORE);
                out.collect(resultRow);
            }
            // prepare UPDATE_AFTER message for new row
            resultRow.replace(currentKey, newAggValue).setRowKind(RowKind.UPDATE_AFTER);
        }

// update aggregate result and set to the newRow
if (isAccumulateMsg(input)) {
    // accumulate input
    function.accumulate(input);
} else {
    // retract input
    function.retract(input);
}

} else {
    // we retracted the last record for this key
    // sent out a delete message
    if (!firstRow) {
        // prepare delete message for previous row
        resultRow.replace(currentKey, prevAggValue).setRowKind(RowKind.DELETE);
        out.collect(resultRow);
    }
    // and clear all state
    accState.clear();
    // cleanup dataview under current key
    function.cleanup();
}

if (!recordCounter.recordCountIsZero(accumulators)) {
    // we aggregated at least one record for this key

    // update the state
    accState.update(accumulators);

    // if this was not the first row and we have to emit retractions
    if (!firstRow) {
        if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue, newAggValue)) {
            // newRow is the same as before and state cleaning is not enabled.
            // We do not emit retraction and acc message.
            // If state cleaning is enabled, we have to emit messages to prevent too early
            // state eviction of downstream operators.
            return;
        } else {
            // retract previous result
            if (generateUpdateBefore) {
                // prepare UPDATE_BEFORE message for previous row
                resultRow
                        .replace(currentKey, prevAggValue)
                        .setRowKind(RowKind.UPDATE_BEFORE);
                out.collect(resultRow);
            }
            // prepare UPDATE_AFTER message for new row
            resultRow.replace(currentKey, newAggValue).setRowKind(RowKind.UPDATE_AFTER);
        }

// update aggregate result and set to the newRow
if (isAccumulateMsg(input)) {
    // accumulate input
    function.accumulate(input);
} else {
    // retract input
    function.retract(input);
}

} else {
    // we retracted the last record for this key
    // sent out a delete message
    if (!firstRow) {
        // prepare delete message for previous row
        resultRow.replace(currentKey, prevAggValue).setRowKind(RowKind.DELETE);
        out.collect(resultRow);
    }
    // and clear all state
    accState.clear();
    // cleanup dataview under current key
    function.cleanup();
}

if (!recordCounter.recordCountIsZero(accumulators)) {
    // we aggregated at least one record for this key

    // update the state
    accState.update(accumulators);

    // if this was not the first row and we have to emit retractions
    if (!firstRow) {
        if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue, newAggValue)) {
            // newRow is the same as before and state cleaning is not enabled.
            // We do not emit retraction and acc message.
            // If state cleaning is enabled, we have to emit messages to prevent too early
            // state eviction of downstream operators.
            return;
        } else {
            // retract previous result
            if (generateUpdateBefore) {
                // prepare UPDATE_BEFORE message for previous row
                resultRow
                        .replace(currentKey, prevAggValue)
                        .setRowKind(RowKind.UPDATE_BEFORE);
                out.collect(resultRow);
            }
            // prepare UPDATE_AFTER message for new row
            resultRow.replace(currentKey, newAggValue).setRowKind(RowKind.UPDATE_AFTER);
        }

Got "pyflink.util.exceptions.TableException: findAndCreateTableSource failed." when running PyFlink example

Copy

Download

t_env = BatchTableEnvironment.create(
    environment_settings=EnvironmentSettings.new_instance()
    .in_batch_mode().use_blink_planner().build())
t_env._j_tenv.getPlanner().getExecEnv().setParallelism(1)

my_source_ddl = """
    create table mySource (
        word VARCHAR
    ) with (
        'connector' = 'filesystem',
        'format' = 'csv',
        'path' = '/tmp/input'
    )
"""

my_sink_ddl = """
    create table mySink (
        word VARCHAR,
        `count` BIGINT
    ) with (
        'connector' = 'filesystem',
        'format' = 'csv',
        'path' = '/tmp/output'
    )
"""

t_env.execute_sql(my_source_ddl)
t_env.execute_sql(my_sink_ddl)

tab = t_env.from_path('mySource')
tab.group_by(tab.word) \
    .select(tab.word, lit(1).count) \
    .execute_insert('mySink').wait()

How to force Apache Flink using a modified operator placement?

Copy

Download

  .map(...).startNewChain().slotSharingGroup("exceptional")
  // or
  .filter(...).startNewChain().slotSharingGroup("default")

How do I handle out-of-order events with Apache flink?

Copy

Download

yourStream.keyBy(...lookup p1 or p2 here...).process(new YourProcessFunction())

SELECT *
    FROM event
    MATCH_RECOGNIZE (
        PARTITION BY particleId
        ORDER BY ts
        MEASURES 
            b.ts, 
            b.particleId, 
            velocity(a, b)
        AFTER MATCH SKIP TO NEXT ROW
        PATTERN (a b)
        DEFINE
            a AS TRUE,
            b AS TRUE
    )

Flink job with CassandrSink fails with Error writing

Copy

Download

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-cassandra_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
</dependency>

[INFO] +- org.apache.flink:flink-connector-cassandra_2.12:jar:1.10.0:provided
[INFO] |  +- (io.dropwizard.metrics:metrics-core:jar:3.1.2:provided - omitted for conflict with 3.1.5)
[INFO] |  \- (org.apache.flink:force-shading:jar:1.10.0:provided - omitted for duplicate)

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-cassandra_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
</dependency>

[INFO] +- org.apache.flink:flink-connector-cassandra_2.12:jar:1.10.0:provided
[INFO] |  +- (io.dropwizard.metrics:metrics-core:jar:3.1.2:provided - omitted for conflict with 3.1.5)
[INFO] |  \- (org.apache.flink:force-shading:jar:1.10.0:provided - omitted for duplicate)

Flink co group outer join fails with High Backpressure

Copy

Download

public class Aggregator {

  private FlatHighFrequencyAnalog flatHighFrequencyAnalog;
  private EVWindow evWindow;

  public Aggregator(FlatHighFrequencyAnalog flatHighFrequencyAnalog) {
    this.flatHighFrequencyAnalog = flatHighFrequencyAnalog;
  }

  public Aggregator(EVWindow evWindow) {
    this.evWindow = evWindow;
  }

  public FlatHighFrequencyAnalog getFlatHighFrequencyAnalog() {
    return flatHighFrequencyAnalog;
  }

  public EVWindow getEVWindow() {
    return evWindow;
  }
}

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream1 =
    environment
        .addSource(createHFAConsumer())
        .map(hfa -> Either.Left(hfa));

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream2 = 
    environment
        .addSource(createHFDConsumer())
        .map(hfd -> Either.Right(hfd));
    
DataStream<Message> pStream =
        stream1
          .union(stream2)
          .assignTimestampsAndWatermarks(
              WatermarkStrategy
                  .<Either<EVWindow, FlatHighFrequencyAnalog>>forBoundedOutOfOrderness(
                    ofSeconds(MAX_OUT_OF_ORDERNESS))
                .withTimestampAssigner((input, timestamp) -> input.isLeft() ? input.left().getTimeStamp() : input.right().getTimeStamp()))
          .keyBy(value -> value.isLeft() ? value.left().getId() : value.right().getId())
          .window(TumblingEventTimeWindows.of(Time.minutes(MINUTES)))
          .process(new ProcessWindowFunction());

List<EVWindow> evWindows =
        Streams.stream(elements)
            .filter(Either::isLeft)
            .map(Either::left)
            .collect(Collectors.toList());

List<FlatHighFrequencyAnalog> highFrequencyAnalogs =
        Streams.stream(elements)
            .filter(Either::isRight)
            .map(Either::right)
            .collect(Collectors.toList());

public class Aggregator {

  private FlatHighFrequencyAnalog flatHighFrequencyAnalog;
  private EVWindow evWindow;

  public Aggregator(FlatHighFrequencyAnalog flatHighFrequencyAnalog) {
    this.flatHighFrequencyAnalog = flatHighFrequencyAnalog;
  }

  public Aggregator(EVWindow evWindow) {
    this.evWindow = evWindow;
  }

  public FlatHighFrequencyAnalog getFlatHighFrequencyAnalog() {
    return flatHighFrequencyAnalog;
  }

  public EVWindow getEVWindow() {
    return evWindow;
  }
}

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream1 =
    environment
        .addSource(createHFAConsumer())
        .map(hfa -> Either.Left(hfa));

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream2 = 
    environment
        .addSource(createHFDConsumer())
        .map(hfd -> Either.Right(hfd));
    
DataStream<Message> pStream =
        stream1
          .union(stream2)
          .assignTimestampsAndWatermarks(
              WatermarkStrategy
                  .<Either<EVWindow, FlatHighFrequencyAnalog>>forBoundedOutOfOrderness(
                    ofSeconds(MAX_OUT_OF_ORDERNESS))
                .withTimestampAssigner((input, timestamp) -> input.isLeft() ? input.left().getTimeStamp() : input.right().getTimeStamp()))
          .keyBy(value -> value.isLeft() ? value.left().getId() : value.right().getId())
          .window(TumblingEventTimeWindows.of(Time.minutes(MINUTES)))
          .process(new ProcessWindowFunction());

List<EVWindow> evWindows =
        Streams.stream(elements)
            .filter(Either::isLeft)
            .map(Either::left)
            .collect(Collectors.toList());

List<FlatHighFrequencyAnalog> highFrequencyAnalogs =
        Streams.stream(elements)
            .filter(Either::isRight)
            .map(Either::right)
            .collect(Collectors.toList());

public class Aggregator {

  private FlatHighFrequencyAnalog flatHighFrequencyAnalog;
  private EVWindow evWindow;

  public Aggregator(FlatHighFrequencyAnalog flatHighFrequencyAnalog) {
    this.flatHighFrequencyAnalog = flatHighFrequencyAnalog;
  }

  public Aggregator(EVWindow evWindow) {
    this.evWindow = evWindow;
  }

  public FlatHighFrequencyAnalog getFlatHighFrequencyAnalog() {
    return flatHighFrequencyAnalog;
  }

  public EVWindow getEVWindow() {
    return evWindow;
  }
}

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream1 =
    environment
        .addSource(createHFAConsumer())
        .map(hfa -> Either.Left(hfa));

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream2 = 
    environment
        .addSource(createHFDConsumer())
        .map(hfd -> Either.Right(hfd));
    
DataStream<Message> pStream =
        stream1
          .union(stream2)
          .assignTimestampsAndWatermarks(
              WatermarkStrategy
                  .<Either<EVWindow, FlatHighFrequencyAnalog>>forBoundedOutOfOrderness(
                    ofSeconds(MAX_OUT_OF_ORDERNESS))
                .withTimestampAssigner((input, timestamp) -> input.isLeft() ? input.left().getTimeStamp() : input.right().getTimeStamp()))
          .keyBy(value -> value.isLeft() ? value.left().getId() : value.right().getId())
          .window(TumblingEventTimeWindows.of(Time.minutes(MINUTES)))
          .process(new ProcessWindowFunction());

List<EVWindow> evWindows =
        Streams.stream(elements)
            .filter(Either::isLeft)
            .map(Either::left)
            .collect(Collectors.toList());

List<FlatHighFrequencyAnalog> highFrequencyAnalogs =
        Streams.stream(elements)
            .filter(Either::isRight)
            .map(Either::right)
            .collect(Collectors.toList());

Cannot change TEdit Text in Delphi

Copy

Download

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  FEdit.Name := 'SomeUniqueNameHere'; // <-- for you to decide on...
  ...
 
  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FDictionary.Add(FEdit.Name, FEvent);
end;

TFormBase = class(TForm)
private
  type TEventBeforeInsertPair = TPair<TComponent, TEventBeforeInsert>;
  FBeforeInsertEvents: TList<TEventBeforeInsertPair>;
  ...
public
  constructor Create;
  destructor Destroy; override;
  ...
end;

...

constructor TFormBase.Create;
begin
  inherited;
  FBeforeInsertEvents := TList<TEventBeforeInsertPair>.Create;
end;

destructor TFormBase.Destroy;
begin
  FBeforeInsertEvents.Free;
  inherited;
end;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  ...

  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FBeforeInsertEvents.Add(TEventBeforeInsertPair.Create(FEdit, FEvent));
end;

procedure TFormBase.rectInsertClick(Sender: TObject);
var
  Item: TEventBeforeInsertPair;
begin
  for Item in FBeforeInsertEvents do
  begin
    if not Item.Value(Item.Key, Self) then
      Exit;
  end;
end;

...

TEventBeforeInsert = function(AComponent: TComponent; AForm: TForm): Boolean of Object;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  FEdit.Name := 'SomeUniqueNameHere'; // <-- for you to decide on...
  ...
 
  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FDictionary.Add(FEdit.Name, FEvent);
end;

TFormBase = class(TForm)
private
  type TEventBeforeInsertPair = TPair<TComponent, TEventBeforeInsert>;
  FBeforeInsertEvents: TList<TEventBeforeInsertPair>;
  ...
public
  constructor Create;
  destructor Destroy; override;
  ...
end;

...

constructor TFormBase.Create;
begin
  inherited;
  FBeforeInsertEvents := TList<TEventBeforeInsertPair>.Create;
end;

destructor TFormBase.Destroy;
begin
  FBeforeInsertEvents.Free;
  inherited;
end;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  ...

  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FBeforeInsertEvents.Add(TEventBeforeInsertPair.Create(FEdit, FEvent));
end;

procedure TFormBase.rectInsertClick(Sender: TObject);
var
  Item: TEventBeforeInsertPair;
begin
  for Item in FBeforeInsertEvents do
  begin
    if not Item.Value(Item.Key, Self) then
      Exit;
  end;
end;

...

TEventBeforeInsert = function(AComponent: TComponent; AForm: TForm): Boolean of Object;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  FEdit.Name := 'SomeUniqueNameHere'; // <-- for you to decide on...
  ...
 
  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FDictionary.Add(FEdit.Name, FEvent);
end;

TFormBase = class(TForm)
private
  type TEventBeforeInsertPair = TPair<TComponent, TEventBeforeInsert>;
  FBeforeInsertEvents: TList<TEventBeforeInsertPair>;
  ...
public
  constructor Create;
  destructor Destroy; override;
  ...
end;

...

constructor TFormBase.Create;
begin
  inherited;
  FBeforeInsertEvents := TList<TEventBeforeInsertPair>.Create;
end;

destructor TFormBase.Destroy;
begin
  FBeforeInsertEvents.Free;
  inherited;
end;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  ...

  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FBeforeInsertEvents.Add(TEventBeforeInsertPair.Create(FEdit, FEvent));
end;

procedure TFormBase.rectInsertClick(Sender: TObject);
var
  Item: TEventBeforeInsertPair;
begin
  for Item in FBeforeInsertEvents do
  begin
    if not Item.Value(Item.Key, Self) then
      Exit;
  end;
end;

...

TEventBeforeInsert = function(AComponent: TComponent; AForm: TForm): Boolean of Object;

Can Flink OutputTag be reused?

Copy

Download

final OutputTag<String> errors = new OutputTag<String>("errors"){};

SingleOutputStreamOperator<Integer> task1 = ...;
SingleOutputStreamOperator<Integer> task2 = ...;
SingleOutputStreamOperator<Integer> task3 = ...;

DataStream<String> exceptions1 = task1.getSideOutput(errors);
DataStream<String> exceptions2 = task2.getSideOutput(errors);
DataStream<String> exceptions3 = task3.getSideOutput(errors);

DataStream<String> exceptions = exceptions1.union(exceptions2, exceptions3);

exceptions.addSink(new FlinkKafkaProducer(...));

See all related Code Snippets

Community Discussions

Trending Discussions on flink

Flink capped MapState
Migrating from FlinkKafkaConsumer to KafkaSource, no windows executed
OSS supported by Google Cloud Dataproc
Questions for reading data from JDBC source in DataStream Flink
how flink interacts with MySQL for the temporal join with mysql
What's the difference between temporal table function and versioned table
Apache Flink delay processing of certain events
Flink, Kafka and JDBC sink
javadoc of SingleOutputStreamOperator#returns(TypeHint<T> typeHint) method
Flink pipeline without a data sink with checkpointing on

Trending Discussions on flink

QUESTION

Flink capped MapState

Asked 2022-Apr-08 at 09:03

Background

We want to keep in a Flink operator's state the last n unique id's. When the n+1 unique id arrives, we want to keep it and drop the oldest unique id in the state. This is in order to avoid an ever-growing state.

We already have a TTL (expiration time) mechanism in place. The size limit is another restriction we're looking to put in place.

Not every element holds a unique id.

Question

Does Flink provide an API that limits the number of elements in the state?

Things tried

Using MapState with a StateTtlConfig generated TTL/expiration mechanism.
Window limited the number of processed elements, but not the number of elements in state.

ANSWER

Answered 2022-Apr-07 at 14:30

I don't think Flink has a state type that would support this out of the box. The closest thing I can think of is to use ListState. With ListState you can append elements like you would a regular list.

For your use case, you would read the state, call .get() which would give you an iterable that you can iterate on, removing the item you'd like to drop and then pushing the state back.

From a performance perspective, the iteration may not be ideal but on the other hand, it would not be significant in comparison to accessing state from disk (in case you're using RocksDB as a state backend) which incurs a heavy cost due to serialization and deserialization.

Source https://stackoverflow.com/questions/71782690

QUESTION

Migrating from FlinkKafkaConsumer to KafkaSource, no windows executed

Asked 2021-Nov-30 at 05:29

I am a kafka and flink beginner. I have implemented FlinkKafkaConsumer to consume messages from a kafka-topic. The only custom setting other than "group" and "topic" is (ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest") to enable re-reading the same messages several times. It works out of the box for consuming and logic. Now FlinkKafkaConsumer is deprecated, and i wanted to change to the successor KafkaSource.

Initializing KafkaSource with the same parameters as i do FlinkKafkaConsumer produces a read of the topic as expected, i can verify this by printing the stream. De-serialization and timestamps seem to work fine. However execution of windows are not done, and as such no results are produced.

I assume some default setting(s) in KafkaSource are different from that of FlinkKafkaConsumer, but i have no idea what they might be.

KafkaSource - Not working

 KafkaSource<TestData> source = KafkaSource.<TestData>builder()
                .setBootstrapServers(propertiesForKafka.getProperty("bootstrap.servers"))
                .setTopics(TOPIC)
                .setDeserializer(new CustomDeserializer())
                .setGroupId(GROUP_ID)
                .setStartingOffsets(OffsetsInitializer.earliest())
                .build();

        DataStream<TestData> testDataStreamSource = env.fromSource(
                source,
                WatermarkStrategy.
                        <TestData>noWatermarks(),
                "Kafka Source"
        );

Kafka consumer - Working (Properties contains group.id,bootstrap.servers and zookeeper.connect)

    propertiesForKafka.setProperty(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
    FlinkKafkaConsumer<TestData> flinkKafkaConsumer = new  FlinkKafkaConsumer(TOPIC, new CustomDeserializer(), propertiesForKafka);
    DataStreamSource<TestData> testDataStreamSource = env.addSource(flinkKafkaConsumer)

Both streams use the same pipeline that looks like this

testDataStreamSource
           .assignTimestampsAndWatermarks(WatermarkStrategy.
                        <TestData>forMonotonousTimestamps().
                        withTimestampAssigner((event, timestamp) -> event.getTimestamp()))
            .keyBy(TestData::getKey)
            .window(SlidingEventTimeWindows.of(Time.hours(3), Time.hours(1)))
            .process(new ProcessWindowFunction<TestData, TestDataOutput, String, TimeWindow>() {
                    @Override
                    public void process(
            ....
            });

Things tried

I've tried to experiment with setting committing of offsets, but it has not improved the situation.
Setting timestamps already in the source.

ANSWER

Answered 2021-Nov-24 at 18:39

Update: The answer is that the KafkaSource behaves differently than FlinkKafkaConsumer in the case where the number of Kafka partitions is smaller than the parallelism of Flink's kafka source operator. See https://stackoverflow.com/a/70101290/2000823 for details.

Original answer:

The problem is almost certainly something related to the timestamps and watermarks.

To verify that timestamps and watermarks are the problem, you could do a quick experiment where you replace the 3-hour-long event time sliding windows with short processing time tumbling windows.

In general it is preferred (but not required) to have the KafkaSource do the watermarking. Using forMonotonousTimestamps in a watermark generator applied after the source, as you are doing now, is a risky move. This will only work correctly if the timestamps in all of the partitions being consumed by each parallel instance of the source are processed in order. If more than one Kafka partition is assigned to any of the KafkaSource tasks, this isn't going to happen. On the other hand, if you supply the forMonotonousTimestamps watermarking strategy in the fromSource call (rather than noWatermarks), then all that will be required is that the timestamps be in order on a per-partition basis, which I imagine is the case.

As troubling as that is, it's probably not enough to explain why the windows don't produce any results. Another possible root cause is that the test data set doesn't include any events with timestamps after the first window, so that window never closes.

Do you have a sink? If not, that would explain things.

You can use the Flink dashboard to help debug this. Look to see if the watermarks are advancing in the window tasks. Turn on checkpointing, and then look to see how much state the window task has -- it should have some non-zero amount of state.

Source https://stackoverflow.com/questions/69765972

QUESTION

how flink interacts with MySQL for the temporal join with mysql

Asked 2021-Aug-07 at 14:24

I am reading at

https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/table/sql/queries/joins/#lookup-join,

It is using the MySQL as a lookup table in the temporal table join as

-- Customers is backed by the JDBC connector and can be used for lookup joins
CREATE TEMPORARY TABLE Customers (
  id INT,
  name STRING,
  country STRING,
  zip STRING
) WITH (
  'connector' = 'jdbc',
  'url' = 'jdbc:mysql://mysqlhost:3306/customerdb',
  'table-name' = 'customers'
);

-- enrich each order with customer information
SELECT o.order_id, o.total, c.country, c.zip
FROM Orders AS o
  JOIN Customers FOR SYSTEM_TIME AS OF o.proc_time AS c
    ON o.customer_id = c.id;

I am wondering about how flink interacts with MySQL and whether there are performance issue at the mysql side for temporal join mysql.

The basic question is how flink works for temporal join with mysql.

ANSWER

Answered 2021-Aug-07 at 14:24

You'll find some relevant details in the docs for the Table / JDBC connector: https://ci.apache.org/projects/flink/flink-docs-stable/docs/connectors/table/jdbc/#features. See especially the section describing the Lookup Cache, which says

JDBC connector can be used in temporal join as a lookup source (aka. dimension table). Currently, only sync lookup mode is supported.

By default, lookup cache is not enabled. You can enable it by setting both lookup.cache.max-rows and lookup.cache.ttl.

The lookup cache is used to improve performance of temporal join the JDBC connector. By default, lookup cache is not enabled, so all the requests are sent to external database. When lookup cache is enabled, each process (i.e. TaskManager) will hold a cache. Flink will lookup the cache first, and only send requests to external database when cache missing, and update cache with the rows returned. The oldest rows in cache will be expired when the cache hit to the max cached rows lookup.cache.max-rows or when the row exceeds the max time to live lookup.cache.ttl. The cached rows might not be the latest, users can tune lookup.cache.ttl to a smaller value to have a better fresh data, but this may increase the number of requests send to database. So this is a balance between throughput and correctness.

Source https://stackoverflow.com/questions/68689647

QUESTION

What's the difference between temporal table function and versioned table

Asked 2021-Aug-02 at 04:34

In Flink 1.12, Flink introduced a new concept which is called versioned table, It is very similar with temporal table function,but I am kind of confused between these two concepts.

temporal table function only supports append-only table(Please correct me if I am wrong),such as:

(changelog kind) update_time   currency   rate
================ ============= =========  ====
+(INSERT)        09:00:00      Yen        102
+(INSERT)        09:00:00      Euro       114
+(INSERT)        09:00:00      USD        1
+(INSERT)        11:15:00      Euro       119
+(INSERT)        11:49:00      Pounds     108

Can we call the above append-only table also a versioned table, From the official documentation it looks the answer is NO? . But I think it can be called a versioned table since it has kept the change history, and I can get the version(snapshot) of the table for the given time point (this is the confusing part to me)

In the following link, it is converting the append-only table to a versioned table by defining a versioned-table-views, with this additional conversion, it looks that append-only table is not versioned table?

Could you please explain about what is versioned table and the difference between versioned table and temporal table function. The official documentation doesn't explain very clearly.

https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/streaming/versioned_tables.html#versioned-table-views,


-- Define a versioned view
CREATE VIEW versioned_rates AS              
SELECT currency, rate, update_time              -- (1) `update_time` keeps the event time
  FROM (
      SELECT *,
      ROW_NUMBER() OVER (PARTITION BY currency  -- (2) the inferred unique key `currency` can be a primary key
         ORDER BY update_time DESC) AS rownum 
      FROM currency_rates)
WHERE rownum = 1; 

-- the view `versioned_rates` will produce a changelog as the following.
(changelog kind) update_time currency   rate
================ ============= =========  ====
+(INSERT)        09:00:00      Yen        102
+(INSERT)        09:00:00      Euro       114
+(INSERT)        09:00:00      USD        1
+(UPDATE_AFTER)  10:45:00      Euro       116
+(UPDATE_AFTER)  11:15:00      Euro       119
+(INSERT)        11:49:00      Pounds     108

ANSWER

Answered 2021-Aug-02 at 04:34

You can view Temporal Table Function as a previous version of some sorts of versioned tables (This is also stated in the legacy features part of Flink).

In your first example, you create an append only table which will hold multiple keys of the same currency (i.e. EURO), which makes it ineligible to be a primary key.

Can you call an append only table a versioned table? The definition on Versioned Tables in doc says:

Flink SQL can define versioned tables over any dynamic table with a PRIMARY KEY constraint and time attribute.

This implies that an append only table cannot serve as a versioned table since it can't, as we said, hold true the primary key constraint.

But, since the append only table in your example does hold the relevant information to become a versioned table, containing inserts as well as updates and deletes, we can turn it to one with a deduplication query, as you posted above.

Source https://stackoverflow.com/questions/68607727

QUESTION

Flink, Kafka and JDBC sink

Asked 2021-Jul-01 at 13:09

I have a Flink 1.11 job that consumes messages from a Kafka topic, keys them, filters them (keyBy followed by a custom ProcessFunction), and saves them into the db via JDBC sink (as described here: https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/connectors/jdbc.html)

The Kafka consumer is initialized with these options:

properties.setProperty("auto.offset.reset", "earliest")
kafkaConsumer = new FlinkKafkaConsumer(topic, deserializer, properties)
kafkaConsumer.setStartFromGroupOffsets()
kafkaConsumer.setCommitOffsetsOnCheckpoints(true)

Checkpoints are enabled on the cluster.

What I want to achieve is a guarantee for saving all filtered data into the db, even if the db is down for, let's say, 6 hours, or there are programming errors while saving to the db and the job needs to be updated, redeployed and restarted.

For this to happen, any checkpointing of the Kafka offsets should mean that either

Data that was read from Kafka is in Flink operator state, waiting to be filtered / passed into the sink, and will be checkpointed as part of Flink operator checkpointing, OR
Data that was read from Kafka has already been committed into the db.

While looking at the implementation of the JdbcSink, I see that it does not really keep any internal state that will be checkpointed/restored - rather, its checkpointing is a write out to the database. Now, if this write fails during checkpointing, and Kafka offsets do get saved, I'll be in a situation where I've "lost" data - subsequent reads from Kafka will resume from committed offsets and whatever data was in flight when the db write failed is now not being read from Kafka anymore nor is in the db.

So is there a way to stop advancing the Kafka offsets whenever a full pipeline (Kafka -> Flink -> DB) fails to execute - or potentially the solution here (in pre-1.13 world) is to create my own implementation of GenericJdbcSinkFunction that will maintain some ValueState until the db write succeeds?

ANSWER

Answered 2021-Jul-01 at 13:09

There are 3 options that I can see:

Try out the JDBC 1.13 connector with your Flink version. There is a good chance it might just work.
If that doesn't work immediately, check if you can backport it to 1.11. There shouldn't be too many changes.
Write your own 2-phase-commit sink, either by extending TwoPhaseCommitSinkFunction or implement your own SinkFunction with CheckpointedFunction and CheckpointListener. Basically, you create a new transaction after a successful checkpoint and commit it with notifyCheckpointCompleted.

Source https://stackoverflow.com/questions/68146383

QUESTION

javadoc of SingleOutputStreamOperator#returns(TypeHint<T> typeHint) method

Asked 2021-Jun-30 at 09:16

I am reading the source code of SingleOutputStreamOperator#returns, its javadoc is:

/**
 * Adds a type information hint about the return type of this operator. This method
 * can be used in cases where Flink cannot determine automatically what the produced
 * type of a function is. That can be the case if the function uses generic type variables
 * in the return type that cannot be inferred from the input type.
 *
 * <p>Use this method the following way:
 * <pre>{@code
 *     DataStream<Tuple2<String, Double>> result =
 *         stream.flatMap(new FunctionWithNonInferrableReturnType())
 *               .returns(new TypeHint<Tuple2<String, Double>>(){});
 * }</pre>
 *
 * @param typeHint The type hint for the returned data type.
 * @return This operator with the type information corresponding to the given type hint.
 */

It mentions FunctionWithNonInferrableReturnType to show case the necessity of returns method, but I am unable to write such a class that is NonInferrableReturnType. Could you please help write a simple one? Thanks!

ANSWER

Answered 2021-Jun-30 at 08:55

When the docs says NonInferrableReturnType it means that we can use the type variable <T>, or any other letter that you prefer. So you can create a MapFunction that return a T. But then you have to use .returns(TypeInformation.of(String.class) for example, if your goal is to return a String.

public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}

Here I am using the classes of your last question Compiling fails when creating MapFunction with super type . The same code without .returns(TypeInformation.of(String.class)) compiles but throw the runtime exception:

could not be determined automatically, due to type erasure. You can give type information hints by using the returns(...) method on the result of the transformation call, or by letting your function implement the 'ResultTypeQueryable' interface.

public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}

In case you wish, here is the integration test for this example:

import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}

Source https://stackoverflow.com/questions/68187973

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

CVE-2020-17519 HIGH

A change introduced in Apache Flink 1.11.0 (and released in 1.11.1 and 1.11.2 as well) allows attackers to read any file on the local filesystem of the JobManager through the REST interface of the JobManager process. Access is restricted to files accessible by the JobManager process. All users should upgrade to Flink 1.11.3 or 1.12.0 if their Flink instance(s) are exposed. The issue was fixed in commit b561010b0ee741543c3953306037f00d7a9f0801 from apache/flink:master.

CVE-2020-17518 HIGH

Apache Flink 1.5.1 introduced a REST handler that allows you to write an uploaded file to an arbitrary location on the local file system, through a maliciously modified HTTP HEADER. The files can be written to any location accessible by Flink 1.5.1. All users should upgrade to Flink 1.11.3 or 1.12.0 if their Flink instance(s) are exposed. The issue was fixed in commit a5264a6f41524afe8ceadf1d8ddc8c80f323ebc4 from apache/flink:master.

CVE-2020-1960 MEDIUM

A vulnerability in Apache Flink (1.1.0 to 1.1.5, 1.2.0 to 1.2.1, 1.3.0 to 1.3.3, 1.4.0 to 1.4.2, 1.5.0 to 1.5.6, 1.6.0 to 1.6.4, 1.7.0 to 1.7.2, 1.8.0 to 1.8.3, 1.9.0 to 1.9.2, 1.10.0) where, when running a process with an enabled JMXReporter, with a port configured via metrics.reporter.reporter_name>.port, an attacker with local access to the machine and JMX port can execute a man-in-the-middle attack using a specially crafted request to rebind the JMXRMI registry to one under the attacker's control. This compromises any connection established to the process via JMX, allowing extraction of credentials and any other transferred data.

Install flink

You can download it from GitHub, Maven.
You can use flink like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the flink component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .