kandi background
Explore Kits

flink | Apache Flink is an open source project | SQL Database library

 by   apache Java Version: Current License: Apache-2.0

 by   apache Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | flink Summary

flink is a Java library typically used in Telecommunications, Media, Media, Entertainment, Database, SQL Database, Spark, Hadoop applications. flink has build file available, it has a Permissive License and it has medium support. However flink has 2490 bugs and it has 21 vulnerabilities. You can download it from GitHub, Maven.
Apache Flink is an open source project of The Apache Software Foundation (ASF). The Apache Flink project originated from the Stratosphere research project.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • flink has a medium active ecosystem.
  • It has 18609 star(s) with 10596 fork(s). There are 929 watchers for this library.
  • It had no major release in the last 12 months.
  • flink has no issues reported. There are 871 open pull requests and 0 closed requests.
  • It has a neutral sentiment in the developer community.
  • The latest version of flink is current.
flink Support
Best in #SQL Database
Average in #SQL Database
flink Support
Best in #SQL Database
Average in #SQL Database

quality kandi Quality

  • flink has 2490 bugs (137 blocker, 120 critical, 1174 major, 1059 minor) and 38707 code smells.
flink Quality
Best in #SQL Database
Average in #SQL Database
flink Quality
Best in #SQL Database
Average in #SQL Database

securitySecurity

  • flink has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • flink code analysis shows 21 unresolved vulnerabilities (15 blocker, 5 critical, 0 major, 1 minor).
  • There are 1446 security hotspots that need review.
flink Security
Best in #SQL Database
Average in #SQL Database
flink Security
Best in #SQL Database
Average in #SQL Database

license License

  • flink is licensed under the Apache-2.0 License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
flink License
Best in #SQL Database
Average in #SQL Database
flink License
Best in #SQL Database
Average in #SQL Database

buildReuse

  • flink releases are not available. You will need to build from source code and install.
  • Deployable package is available in Maven.
  • Build file is available. You can build the component from source.
  • Installation instructions are not available. Examples and code snippets are available.
  • flink saves you 4780464 person hours of effort in developing the same functionality from scratch.
  • It has 1703645 lines of code, 119488 functions and 14240 files.
  • It has medium code complexity. Code complexity directly impacts maintainability of the code.
flink Reuse
Best in #SQL Database
Average in #SQL Database
flink Reuse
Best in #SQL Database
Average in #SQL Database
Top functions reviewed by kandi - BETA

kandi has reviewed flink and discovered the below as its top functions. This is intended to give you an instant insight into flink implemented functionality, and help decide if they suit your requirements.

  • Initialize handlers .
  • Starts the application .
  • Generate a select logical plan .
  • Registers an alias from the given node .
  • Perform a phase 1 .
  • Performs the actual operation .
  • Returns the converter for the given data type .
  • Runs the loop .
  • Load resource .
  • Builds an execution graph .

flink Key Features

A streaming-first runtime that supports both batch processing and data streaming programs

Elegant and fluent APIs in Java and Scala

A runtime that supports very high throughput and low event latency at the same time

Support for event time and out-of-order processing in the DataStream API, based on the Dataflow Model

Flexible windowing (time, count, sessions, custom triggers) across different time semantics (event time, processing time)

Fault-tolerance with exactly-once processing guarantees

Natural back-pressure in streaming programs

Libraries for Graph processing (batch), Machine Learning (batch), and Complex Event Processing (streaming)

Built-in support for iterative programs (BSP) in the DataSet (batch) API

Custom memory management for efficient and robust switching between in-memory and out-of-core data processing algorithms

Compatibility layers for Apache Hadoop MapReduce

Integration with YARN, HDFS, HBase, and other components of the Apache Hadoop ecosystem

Streaming Example

copy iconCopydownload iconDownload
case class WordWithCount(word: String, count: Long)

val text = env.socketTextStream(host, port, '\n')

val windowCounts = text.flatMap { w => w.split("\\s") }
  .map { w => WordWithCount(w, 1) }
  .keyBy("word")
  .window(TumblingProcessingTimeWindow.of(Time.seconds(5)))
  .sum("count")

windowCounts.print()

Batch Example

copy iconCopydownload iconDownload
case class WordWithCount(word: String, count: Long)

val text = env.readTextFile(path)

val counts = text.flatMap { w => w.split("\\s") }
  .map { w => WordWithCount(w, 1) }
  .groupBy("word")
  .sum("count")

counts.writeAsCsv(outputPath)

Building Apache Flink from Source

copy iconCopydownload iconDownload
git clone https://github.com/apache/flink.git
cd flink
mvn clean package -DskipTests # this will take up to 10 minutes

Flink capped MapState

copy iconCopydownload iconDownload
map[nextIndex++] = thisEvent;
if (nextIndex - oldestIndex > n) {
  map[oldestIndex++].clear();
}

javadoc of SingleOutputStreamOperator#returns(TypeHint<T> typeHint) method

copy iconCopydownload iconDownload
public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}
public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}
import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}
-----------------------
public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}
public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}
import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}
-----------------------
public class MyMapFunctionNonInferrableReturnType<T> implements MapFunction<AbstractDataModel, T> {
    @Override
    public T map(AbstractDataModel value) throws Exception {
        return (T) value.getValue();
    }
}
public class NonInferrableReturnTypeStreamJob {

    private final List<AbstractDataModel> abstractDataModelList;
    private final ValenciaSinkFunction sink;

    public NonInferrableReturnTypeStreamJob() {
        this.abstractDataModelList = new ArrayList<AbstractDataModel>();
        this.abstractDataModelList.add(new ConcreteModel("a", "1"));
        this.abstractDataModelList.add(new ConcreteModel("a", "2"));
        this.sink = new ValenciaSinkFunction();
    }

    public NonInferrableReturnTypeStreamJob(List<AbstractDataModel> abstractDataModelList, ValenciaSinkFunction sink) {
        this.abstractDataModelList = abstractDataModelList;
        this.sink = sink;
    }

    public static void main(String[] args) throws Exception {
        NonInferrableReturnTypeStreamJob concreteModelTest = new NonInferrableReturnTypeStreamJob();
        concreteModelTest.execute();
    }

    public void execute() throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        env.fromCollection(this.abstractDataModelList)
                .map(new MyMapFunctionNonInferrableReturnType())
                .returns(TypeInformation.of(String.class))
                .addSink(sink);

        env.execute();
    }
}
import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
import org.apache.flink.test.util.MiniClusterWithClientResource;
import org.junit.ClassRule;
import org.junit.Test;
import org.sense.flink.examples.stream.valencia.ValenciaSinkFunction;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import static junit.framework.TestCase.assertEquals;
import static org.junit.Assert.assertTrue;

public class NonInferrableReturnTypeStreamJobTest {

    @ClassRule
    public static MiniClusterWithClientResource flinkCluster;
    private final int minAvailableProcessors = 4;
    private final boolean runInParallel;

    public NonInferrableReturnTypeStreamJobTest() {
        int availableProcessors = Runtime.getRuntime().availableProcessors();
        this.runInParallel = availableProcessors >= minAvailableProcessors;
        if (this.runInParallel) {
            flinkCluster = new MiniClusterWithClientResource(
                    new MiniClusterResourceConfiguration.Builder()
                            .setNumberSlotsPerTaskManager(minAvailableProcessors)
                            .setNumberTaskManagers(1)
                            .build());
        }
    }

    @Test
    public void execute() throws Exception {
        List<AbstractDataModel> abstractDataModelList = new ArrayList<AbstractDataModel>();
        abstractDataModelList.add(new ConcreteModel("a", "1"));
        abstractDataModelList.add(new ConcreteModel("a", "2"));
        ValenciaSinkFunction.values.clear();

        NonInferrableReturnTypeStreamJob streamJob = new NonInferrableReturnTypeStreamJob(abstractDataModelList, new ValenciaSinkFunction());
        streamJob.execute();

        List<String> results = ValenciaSinkFunction.values;
        assertEquals(2, results.size());
        assertTrue(results.containsAll(Arrays.asList("1", "2")));
    }
}

toChanglelogStream prints different kinds of changes

copy iconCopydownload iconDownload
// update aggregate result and set to the newRow
if (isAccumulateMsg(input)) {
    // accumulate input
    function.accumulate(input);
} else {
    // retract input
    function.retract(input);
}
} else {
    // we retracted the last record for this key
    // sent out a delete message
    if (!firstRow) {
        // prepare delete message for previous row
        resultRow.replace(currentKey, prevAggValue).setRowKind(RowKind.DELETE);
        out.collect(resultRow);
    }
    // and clear all state
    accState.clear();
    // cleanup dataview under current key
    function.cleanup();
}
if (!recordCounter.recordCountIsZero(accumulators)) {
    // we aggregated at least one record for this key

    // update the state
    accState.update(accumulators);

    // if this was not the first row and we have to emit retractions
    if (!firstRow) {
        if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue, newAggValue)) {
            // newRow is the same as before and state cleaning is not enabled.
            // We do not emit retraction and acc message.
            // If state cleaning is enabled, we have to emit messages to prevent too early
            // state eviction of downstream operators.
            return;
        } else {
            // retract previous result
            if (generateUpdateBefore) {
                // prepare UPDATE_BEFORE message for previous row
                resultRow
                        .replace(currentKey, prevAggValue)
                        .setRowKind(RowKind.UPDATE_BEFORE);
                out.collect(resultRow);
            }
            // prepare UPDATE_AFTER message for new row
            resultRow.replace(currentKey, newAggValue).setRowKind(RowKind.UPDATE_AFTER);
        }
-----------------------
// update aggregate result and set to the newRow
if (isAccumulateMsg(input)) {
    // accumulate input
    function.accumulate(input);
} else {
    // retract input
    function.retract(input);
}
} else {
    // we retracted the last record for this key
    // sent out a delete message
    if (!firstRow) {
        // prepare delete message for previous row
        resultRow.replace(currentKey, prevAggValue).setRowKind(RowKind.DELETE);
        out.collect(resultRow);
    }
    // and clear all state
    accState.clear();
    // cleanup dataview under current key
    function.cleanup();
}
if (!recordCounter.recordCountIsZero(accumulators)) {
    // we aggregated at least one record for this key

    // update the state
    accState.update(accumulators);

    // if this was not the first row and we have to emit retractions
    if (!firstRow) {
        if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue, newAggValue)) {
            // newRow is the same as before and state cleaning is not enabled.
            // We do not emit retraction and acc message.
            // If state cleaning is enabled, we have to emit messages to prevent too early
            // state eviction of downstream operators.
            return;
        } else {
            // retract previous result
            if (generateUpdateBefore) {
                // prepare UPDATE_BEFORE message for previous row
                resultRow
                        .replace(currentKey, prevAggValue)
                        .setRowKind(RowKind.UPDATE_BEFORE);
                out.collect(resultRow);
            }
            // prepare UPDATE_AFTER message for new row
            resultRow.replace(currentKey, newAggValue).setRowKind(RowKind.UPDATE_AFTER);
        }
-----------------------
// update aggregate result and set to the newRow
if (isAccumulateMsg(input)) {
    // accumulate input
    function.accumulate(input);
} else {
    // retract input
    function.retract(input);
}
} else {
    // we retracted the last record for this key
    // sent out a delete message
    if (!firstRow) {
        // prepare delete message for previous row
        resultRow.replace(currentKey, prevAggValue).setRowKind(RowKind.DELETE);
        out.collect(resultRow);
    }
    // and clear all state
    accState.clear();
    // cleanup dataview under current key
    function.cleanup();
}
if (!recordCounter.recordCountIsZero(accumulators)) {
    // we aggregated at least one record for this key

    // update the state
    accState.update(accumulators);

    // if this was not the first row and we have to emit retractions
    if (!firstRow) {
        if (stateRetentionTime <= 0 && equaliser.equals(prevAggValue, newAggValue)) {
            // newRow is the same as before and state cleaning is not enabled.
            // We do not emit retraction and acc message.
            // If state cleaning is enabled, we have to emit messages to prevent too early
            // state eviction of downstream operators.
            return;
        } else {
            // retract previous result
            if (generateUpdateBefore) {
                // prepare UPDATE_BEFORE message for previous row
                resultRow
                        .replace(currentKey, prevAggValue)
                        .setRowKind(RowKind.UPDATE_BEFORE);
                out.collect(resultRow);
            }
            // prepare UPDATE_AFTER message for new row
            resultRow.replace(currentKey, newAggValue).setRowKind(RowKind.UPDATE_AFTER);
        }

Got &quot;pyflink.util.exceptions.TableException: findAndCreateTableSource failed.&quot; when running PyFlink example

copy iconCopydownload iconDownload
t_env = BatchTableEnvironment.create(
    environment_settings=EnvironmentSettings.new_instance()
    .in_batch_mode().use_blink_planner().build())
t_env._j_tenv.getPlanner().getExecEnv().setParallelism(1)

my_source_ddl = """
    create table mySource (
        word VARCHAR
    ) with (
        'connector' = 'filesystem',
        'format' = 'csv',
        'path' = '/tmp/input'
    )
"""

my_sink_ddl = """
    create table mySink (
        word VARCHAR,
        `count` BIGINT
    ) with (
        'connector' = 'filesystem',
        'format' = 'csv',
        'path' = '/tmp/output'
    )
"""

t_env.execute_sql(my_source_ddl)
t_env.execute_sql(my_sink_ddl)

tab = t_env.from_path('mySource')
tab.group_by(tab.word) \
    .select(tab.word, lit(1).count) \
    .execute_insert('mySink').wait()

How to force Apache Flink using a modified operator placement?

copy iconCopydownload iconDownload
  .map(...).startNewChain().slotSharingGroup("exceptional")
  // or
  .filter(...).startNewChain().slotSharingGroup("default")

How do I handle out-of-order events with Apache flink?

copy iconCopydownload iconDownload
yourStream.keyBy(...lookup p1 or p2 here...).process(new YourProcessFunction())
-----------------------
SELECT *
    FROM event
    MATCH_RECOGNIZE (
        PARTITION BY particleId
        ORDER BY ts
        MEASURES 
            b.ts, 
            b.particleId, 
            velocity(a, b)
        AFTER MATCH SKIP TO NEXT ROW
        PATTERN (a b)
        DEFINE
            a AS TRUE,
            b AS TRUE
    )

Flink job with CassandrSink fails with Error writing

copy iconCopydownload iconDownload
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-cassandra_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
</dependency>
[INFO] +- org.apache.flink:flink-connector-cassandra_2.12:jar:1.10.0:provided
[INFO] |  +- (io.dropwizard.metrics:metrics-core:jar:3.1.2:provided - omitted for conflict with 3.1.5)
[INFO] |  \- (org.apache.flink:force-shading:jar:1.10.0:provided - omitted for duplicate)
-----------------------
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-cassandra_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
</dependency>
[INFO] +- org.apache.flink:flink-connector-cassandra_2.12:jar:1.10.0:provided
[INFO] |  +- (io.dropwizard.metrics:metrics-core:jar:3.1.2:provided - omitted for conflict with 3.1.5)
[INFO] |  \- (org.apache.flink:force-shading:jar:1.10.0:provided - omitted for duplicate)

Flink co group outer join fails with High Backpressure

copy iconCopydownload iconDownload
public class Aggregator {

  private FlatHighFrequencyAnalog flatHighFrequencyAnalog;
  private EVWindow evWindow;

  public Aggregator(FlatHighFrequencyAnalog flatHighFrequencyAnalog) {
    this.flatHighFrequencyAnalog = flatHighFrequencyAnalog;
  }

  public Aggregator(EVWindow evWindow) {
    this.evWindow = evWindow;
  }

  public FlatHighFrequencyAnalog getFlatHighFrequencyAnalog() {
    return flatHighFrequencyAnalog;
  }

  public EVWindow getEVWindow() {
    return evWindow;
  }
}
SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream1 =
    environment
        .addSource(createHFAConsumer())
        .map(hfa -> Either.Left(hfa));

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream2 = 
    environment
        .addSource(createHFDConsumer())
        .map(hfd -> Either.Right(hfd));
    
DataStream<Message> pStream =
        stream1
          .union(stream2)
          .assignTimestampsAndWatermarks(
              WatermarkStrategy
                  .<Either<EVWindow, FlatHighFrequencyAnalog>>forBoundedOutOfOrderness(
                    ofSeconds(MAX_OUT_OF_ORDERNESS))
                .withTimestampAssigner((input, timestamp) -> input.isLeft() ? input.left().getTimeStamp() : input.right().getTimeStamp()))
          .keyBy(value -> value.isLeft() ? value.left().getId() : value.right().getId())
          .window(TumblingEventTimeWindows.of(Time.minutes(MINUTES)))
          .process(new ProcessWindowFunction());
List<EVWindow> evWindows =
        Streams.stream(elements)
            .filter(Either::isLeft)
            .map(Either::left)
            .collect(Collectors.toList());

List<FlatHighFrequencyAnalog> highFrequencyAnalogs =
        Streams.stream(elements)
            .filter(Either::isRight)
            .map(Either::right)
            .collect(Collectors.toList());
-----------------------
public class Aggregator {

  private FlatHighFrequencyAnalog flatHighFrequencyAnalog;
  private EVWindow evWindow;

  public Aggregator(FlatHighFrequencyAnalog flatHighFrequencyAnalog) {
    this.flatHighFrequencyAnalog = flatHighFrequencyAnalog;
  }

  public Aggregator(EVWindow evWindow) {
    this.evWindow = evWindow;
  }

  public FlatHighFrequencyAnalog getFlatHighFrequencyAnalog() {
    return flatHighFrequencyAnalog;
  }

  public EVWindow getEVWindow() {
    return evWindow;
  }
}
SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream1 =
    environment
        .addSource(createHFAConsumer())
        .map(hfa -> Either.Left(hfa));

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream2 = 
    environment
        .addSource(createHFDConsumer())
        .map(hfd -> Either.Right(hfd));
    
DataStream<Message> pStream =
        stream1
          .union(stream2)
          .assignTimestampsAndWatermarks(
              WatermarkStrategy
                  .<Either<EVWindow, FlatHighFrequencyAnalog>>forBoundedOutOfOrderness(
                    ofSeconds(MAX_OUT_OF_ORDERNESS))
                .withTimestampAssigner((input, timestamp) -> input.isLeft() ? input.left().getTimeStamp() : input.right().getTimeStamp()))
          .keyBy(value -> value.isLeft() ? value.left().getId() : value.right().getId())
          .window(TumblingEventTimeWindows.of(Time.minutes(MINUTES)))
          .process(new ProcessWindowFunction());
List<EVWindow> evWindows =
        Streams.stream(elements)
            .filter(Either::isLeft)
            .map(Either::left)
            .collect(Collectors.toList());

List<FlatHighFrequencyAnalog> highFrequencyAnalogs =
        Streams.stream(elements)
            .filter(Either::isRight)
            .map(Either::right)
            .collect(Collectors.toList());
-----------------------
public class Aggregator {

  private FlatHighFrequencyAnalog flatHighFrequencyAnalog;
  private EVWindow evWindow;

  public Aggregator(FlatHighFrequencyAnalog flatHighFrequencyAnalog) {
    this.flatHighFrequencyAnalog = flatHighFrequencyAnalog;
  }

  public Aggregator(EVWindow evWindow) {
    this.evWindow = evWindow;
  }

  public FlatHighFrequencyAnalog getFlatHighFrequencyAnalog() {
    return flatHighFrequencyAnalog;
  }

  public EVWindow getEVWindow() {
    return evWindow;
  }
}
SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream1 =
    environment
        .addSource(createHFAConsumer())
        .map(hfa -> Either.Left(hfa));

SingleOutputStreamOperator<Either<EVWindow, FlatHighFrequencyAnalog>> stream2 = 
    environment
        .addSource(createHFDConsumer())
        .map(hfd -> Either.Right(hfd));
    
DataStream<Message> pStream =
        stream1
          .union(stream2)
          .assignTimestampsAndWatermarks(
              WatermarkStrategy
                  .<Either<EVWindow, FlatHighFrequencyAnalog>>forBoundedOutOfOrderness(
                    ofSeconds(MAX_OUT_OF_ORDERNESS))
                .withTimestampAssigner((input, timestamp) -> input.isLeft() ? input.left().getTimeStamp() : input.right().getTimeStamp()))
          .keyBy(value -> value.isLeft() ? value.left().getId() : value.right().getId())
          .window(TumblingEventTimeWindows.of(Time.minutes(MINUTES)))
          .process(new ProcessWindowFunction());
List<EVWindow> evWindows =
        Streams.stream(elements)
            .filter(Either::isLeft)
            .map(Either::left)
            .collect(Collectors.toList());

List<FlatHighFrequencyAnalog> highFrequencyAnalogs =
        Streams.stream(elements)
            .filter(Either::isRight)
            .map(Either::right)
            .collect(Collectors.toList());

Cannot change TEdit Text in Delphi

copy iconCopydownload iconDownload
function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  FEdit.Name := 'SomeUniqueNameHere'; // <-- for you to decide on...
  ...
 
  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FDictionary.Add(FEdit.Name, FEvent);
end;
TFormBase = class(TForm)
private
  type TEventBeforeInsertPair = TPair<TComponent, TEventBeforeInsert>;
  FBeforeInsertEvents: TList<TEventBeforeInsertPair>;
  ...
public
  constructor Create;
  destructor Destroy; override;
  ...
end;

...

constructor TFormBase.Create;
begin
  inherited;
  FBeforeInsertEvents := TList<TEventBeforeInsertPair>.Create;
end;

destructor TFormBase.Destroy;
begin
  FBeforeInsertEvents.Free;
  inherited;
end;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  ...

  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FBeforeInsertEvents.Add(TEventBeforeInsertPair.Create(FEdit, FEvent));
end;

procedure TFormBase.rectInsertClick(Sender: TObject);
var
  Item: TEventBeforeInsertPair;
begin
  for Item in FBeforeInsertEvents do
  begin
    if not Item.Value(Item.Key, Self) then
      Exit;
  end;
end;

...
TEventBeforeInsert = function(AComponent: TComponent; AForm: TForm): Boolean of Object;
-----------------------
function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  FEdit.Name := 'SomeUniqueNameHere'; // <-- for you to decide on...
  ...
 
  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FDictionary.Add(FEdit.Name, FEvent);
end;
TFormBase = class(TForm)
private
  type TEventBeforeInsertPair = TPair<TComponent, TEventBeforeInsert>;
  FBeforeInsertEvents: TList<TEventBeforeInsertPair>;
  ...
public
  constructor Create;
  destructor Destroy; override;
  ...
end;

...

constructor TFormBase.Create;
begin
  inherited;
  FBeforeInsertEvents := TList<TEventBeforeInsertPair>.Create;
end;

destructor TFormBase.Destroy;
begin
  FBeforeInsertEvents.Free;
  inherited;
end;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  ...

  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FBeforeInsertEvents.Add(TEventBeforeInsertPair.Create(FEdit, FEvent));
end;

procedure TFormBase.rectInsertClick(Sender: TObject);
var
  Item: TEventBeforeInsertPair;
begin
  for Item in FBeforeInsertEvents do
  begin
    if not Item.Value(Item.Key, Self) then
      Exit;
  end;
end;

...
TEventBeforeInsert = function(AComponent: TComponent; AForm: TForm): Boolean of Object;
-----------------------
function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  FEdit.Name := 'SomeUniqueNameHere'; // <-- for you to decide on...
  ...
 
  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FDictionary.Add(FEdit.Name, FEvent);
end;
TFormBase = class(TForm)
private
  type TEventBeforeInsertPair = TPair<TComponent, TEventBeforeInsert>;
  FBeforeInsertEvents: TList<TEventBeforeInsertPair>;
  ...
public
  constructor Create;
  destructor Destroy; override;
  ...
end;

...

constructor TFormBase.Create;
begin
  inherited;
  FBeforeInsertEvents := TList<TEventBeforeInsertPair>.Create;
end;

destructor TFormBase.Destroy;
begin
  FBeforeInsertEvents.Free;
  inherited;
end;

function TFormBase.addStringField(AField: TFieldBase; ATab: TTabsNames): TFormBase;
var
  ...
  FEdit: TEdit;
  FEvent: TEventBeforeInsert;
begin
  ...
  FEdit := TEdit.Create(Self);
  ...

  FEvent := AField.EventBeforeInsert;
  if Assigned(FEvent) then
    FBeforeInsertEvents.Add(TEventBeforeInsertPair.Create(FEdit, FEvent));
end;

procedure TFormBase.rectInsertClick(Sender: TObject);
var
  Item: TEventBeforeInsertPair;
begin
  for Item in FBeforeInsertEvents do
  begin
    if not Item.Value(Item.Key, Self) then
      Exit;
  end;
end;

...
TEventBeforeInsert = function(AComponent: TComponent; AForm: TForm): Boolean of Object;

Can Flink OutputTag be reused?

copy iconCopydownload iconDownload
final OutputTag<String> errors = new OutputTag<String>("errors"){};

SingleOutputStreamOperator<Integer> task1 = ...;
SingleOutputStreamOperator<Integer> task2 = ...;
SingleOutputStreamOperator<Integer> task3 = ...;

DataStream<String> exceptions1 = task1.getSideOutput(errors);
DataStream<String> exceptions2 = task2.getSideOutput(errors);
DataStream<String> exceptions3 = task3.getSideOutput(errors);

DataStream<String> exceptions = exceptions1.union(exceptions2, exceptions3);

exceptions.addSink(new FlinkKafkaProducer(...));

Community Discussions

Trending Discussions on flink
  • Flink capped MapState
  • Migrating from FlinkKafkaConsumer to KafkaSource, no windows executed
  • OSS supported by Google Cloud Dataproc
  • Questions for reading data from JDBC source in DataStream Flink
  • how flink interacts with MySQL for the temporal join with mysql
  • What's the difference between temporal table function and versioned table
  • Apache Flink delay processing of certain events
  • Flink, Kafka and JDBC sink
  • javadoc of SingleOutputStreamOperator#returns(TypeHint&lt;T&gt; typeHint) method
  • Flink pipeline without a data sink with checkpointing on
Trending Discussions on flink

QUESTION

Flink capped MapState

Asked 2022-Apr-08 at 09:03

Background

We want to keep in a Flink operator's state the last n unique id's. When the n+1 unique id arrives, we want to keep it and drop the oldest unique id in the state. This is in order to avoid an ever-growing state.

We already have a TTL (expiration time) mechanism in place. The size limit is another restriction we're looking to put in place.

Not every element holds a unique id.

Question

Does Flink provide an API that limits the number of elements in the state?

Things tried

  1. Using MapState with a StateTtlConfig generated TTL/expiration mechanism.
  2. Window limited the number of processed elements, but not the number of elements in state.

ANSWER

Answered 2022-Apr-07 at 14:30

I don't think Flink has a state type that would support this out of the box. The closest thing I can think of is to use ListState. With ListState you can append elements like you would a regular list.

For your use case, you would read the state, call .get() which would give you an iterable that you can iterate on, removing the item you'd like to drop and then pushing the state back.

From a performance perspective, the iteration may not be ideal but on the other hand, it would not be significant in comparison to accessing state from disk (in case you're using RocksDB as a state backend) which incurs a heavy cost due to serialization and deserialization.

Source https://stackoverflow.com/questions/71782690

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

A change introduced in Apache Flink 1.11.0 (and released in 1.11.1 and 1.11.2 as well) allows attackers to read any file on the local filesystem of the JobManager through the REST interface of the JobManager process. Access is restricted to files accessible by the JobManager process. All users should upgrade to Flink 1.11.3 or 1.12.0 if their Flink instance(s) are exposed. The issue was fixed in commit b561010b0ee741543c3953306037f00d7a9f0801 from apache/flink:master.
Apache Flink 1.5.1 introduced a REST handler that allows you to write an uploaded file to an arbitrary location on the local file system, through a maliciously modified HTTP HEADER. The files can be written to any location accessible by Flink 1.5.1. All users should upgrade to Flink 1.11.3 or 1.12.0 if their Flink instance(s) are exposed. The issue was fixed in commit a5264a6f41524afe8ceadf1d8ddc8c80f323ebc4 from apache/flink:master.
A vulnerability in Apache Flink (1.1.0 to 1.1.5, 1.2.0 to 1.2.1, 1.3.0 to 1.3.3, 1.4.0 to 1.4.2, 1.5.0 to 1.5.6, 1.6.0 to 1.6.4, 1.7.0 to 1.7.2, 1.8.0 to 1.8.3, 1.9.0 to 1.9.2, 1.10.0) where, when running a process with an enabled JMXReporter, with a port configured via metrics.reporter.reporter_name>.port, an attacker with local access to the machine and JMX port can execute a man-in-the-middle attack using a specially crafted request to rebind the JMXRMI registry to one under the attacker's control. This compromises any connection established to the process via JMX, allowing extraction of credentials and any other transferred data.

Install flink

You can download it from GitHub, Maven.
You can use flink like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the flink component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

Don’t hesitate to ask!. Contact the developers and community on the mailing lists if you need any help. Open an issue if you found a bug in Flink.

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Share this Page

share link
Reuse Pre-built Kits with flink
Consider Popular SQL Database Libraries
Compare SQL Database Libraries with Highest Support
Compare SQL Database Libraries with Highest Quality
Compare SQL Database Libraries with Highest Security
Compare SQL Database Libraries with Permissive License
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.