pulsar | Apache Pulsar distributed pubsub messaging system | Pub Sub library

by apache Java Version: v2.10.0 License: Apache-2.0

Download this library from

kandi X-RAY | pulsar Summary

pulsar is a Java library typically used in Messaging, Pub Sub, Kafka applications. pulsar has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub, Maven.

Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. Learn more about Pulsar at https://pulsar.apache.org.

Support

Quality

Security

License

Reuse

kandi-support Support

pulsar has a highly active ecosystem.
It has 10675 star(s) with 2795 fork(s). There are 422 watchers for this library.
There were 3 major release(s) in the last 6 months.
There are 1695 open issues and 3122 have been closed. On average issues are closed in 83 days. There are 361 open pull requests and 0 closed requests.
It has a positive sentiment in the developer community.
The latest version of pulsar is v2.10.0

pulsar Support

Best in #Pub Sub

Average in #Pub Sub

pulsar Support

Best in #Pub Sub

Average in #Pub Sub

quality kandi Quality

pulsar has 0 bugs and 0 code smells.

pulsar Quality

Best in #Pub Sub

Average in #Pub Sub

pulsar Quality

Best in #Pub Sub

Average in #Pub Sub

security Security

pulsar has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
pulsar code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

pulsar Security

Best in #Pub Sub

Average in #Pub Sub

pulsar Security

Best in #Pub Sub

Average in #Pub Sub

license License

pulsar is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

pulsar License

Best in #Pub Sub

Average in #Pub Sub

pulsar License

Best in #Pub Sub

Average in #Pub Sub

build Reuse

pulsar releases are available to install and integrate.
Deployable package is available in Maven.
Build file is available. You can build the component from source.
Installation instructions, examples and code snippets are available.
It has 444636 lines of code, 28552 functions and 3482 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

pulsar Reuse

Best in #Pub Sub

Average in #Pub Sub

pulsar Reuse

Best in #Pub Sub

Average in #Pub Sub

Top functions reviewed by kandi - BETA

kandi has reviewed pulsar and discovered the below as its top functions. This is intended to give you an instant insight into pulsar implemented functionality, and help decide if they suit your requirements.

Creates a BitSetRecyclable from the specified ByteBuffer .
Prints statistics for a topic .
Handles a command producer .
Update a function .
Update a source .
Updates an existing sink .
Calculate cursor back cursors counters .
Returns command to execute .
Truncate ledgers .
Returns the internal statistics for this topic .

pulsar Key Features

Horizontally scalable (Millions of independent topics and millions of messages published per second)

Strong ordering and consistency guarantees

Low latency durable storage

Topic and queue semantics

Load balancer

Designed for being deployed as a hosted service: Multi-tenant Authentication Authorization Quotas Support mixing very different workloads Optional hardware isolation

Keeps track of consumer cursor position

REST API for provisioning, admin and stats

Geo replication

Transparent handling of partitioned topics

Transparent batching of messages

pulsar Examples and Code Snippets

See all related Code Snippets

Build Pulsar

Copy

Download

$ mvn install -DskipTests

Minimal build (This skips most of external connectors and tiered storage handlers)

Copy

Download

mvn install -Pcore-modules,-main -DskipTests

Build custom docker images

Copy

Download

mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

error 404 when try to make oauth in pulsar stream native cluster

Copy

Download

pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651

client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))

-----------------------

pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651

client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))

Apache Pulsar: Access state storage in LocalRunner not working

Copy

Download

2022-02-07T11:09:04,916-0800 [main] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
>>> PRODUCER SENT
2022-02-07T11:09:05,267-0800 [public/default/TestFunction-0] INFO  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Opening state table for function public/default/TestFunction
2022-02-07T11:09:05,279-0800 [client-scheduler-OrderedScheduler-7-0] INFO  org.apache.bookkeeper.clients.SimpleStorageClientImpl - Retrieved table properties for table public_default/TestFunction : stream_id: 1024

2022-02-07T11:09:05,527-0800 [pulsar-client-io-1-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [test-topic-input][Test Function Sub] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
>>> GOT input Hello World!

Pass private key as header in curl PUT returning error for illegal character

Copy

Download

Authorization: Bearer $AUTHORIZATION

-----------------------

bin/pulsar tokens create --private-key file:///path/to/my-private.key \
            --subject test-user

How to pass authorization key in shell script curl command without header

Copy

Download

AUTHORIZATION=$(cat $REPO_ROOT/pulsar/tls/broker/broker.key.pem)

Event-time Temporal Join in Apache Flink only works with small datasets

Copy

Download

WATERMARK FOR MUT_TS AS MUT_TS - INTERVAL '2' MINUTE

Why does the data streamed using the Java client appear as encoded on Apache Pulsar?

Copy

Download

Schema.AVRO(DavisMessage.class)

Pulsar client thread balance

Copy

Download

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .ioThreads(16)
    .connectionsPerBroker(16)
    .create();

Pulsar function fails to deserialize message because of wrong schema type (JSON instead of AVRO)

Copy

Download

Map<String, ConsumerConfig> inputSpecs = new HashMap<String, ConsumerConfig> ();
inputSpecs.put("persistent://orders/inbound/food-orders", 
    ConsumerConfig.builder().schemaType("avro").build());
FunctionConfig functionConfig = 
    FunctionConfig.builder()
        ...
        .inputSpecs(inputSpecs)
        ...
        .build();

inputSpecs:
 $topicName:
  schemaType: AVRO

-----------------------

Map<String, ConsumerConfig> inputSpecs = new HashMap<String, ConsumerConfig> ();
inputSpecs.put("persistent://orders/inbound/food-orders", 
    ConsumerConfig.builder().schemaType("avro").build());
FunctionConfig functionConfig = 
    FunctionConfig.builder()
        ...
        .inputSpecs(inputSpecs)
        ...
        .build();

inputSpecs:
 $topicName:
  schemaType: AVRO

Pulsar: If a message gets nack'd (negativeAcknowledge()) when will it be redelivered?

Copy

Download

Consumer<byte[]> consumer = client.newConsumer()
    .topic("my-topic")
    .subscriptionName("my-sub")
    .negativeAckRedelivery(10, TimeUnit.SECONDS)
    .subscribe()

How to organize multiple Dockerfiles, docker-compose.yaml and .dockerignore

Copy

Download

COPY requirements.txt ./

version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

COPY python_producer/requirements.txt ./

version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

-----------------------

COPY requirements.txt ./

version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

COPY python_producer/requirements.txt ./

version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

-----------------------

COPY requirements.txt ./

version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

COPY python_producer/requirements.txt ./

version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

-----------------------

COPY requirements.txt ./

version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

COPY python_producer/requirements.txt ./

version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

See all related Code Snippets

Community Discussions

Trending Discussions on pulsar

error 404 when try to make oauth in pulsar stream native cluster
Kubernetes Statefulsets: Restart all pods concurrently (instead of in sequence)
How to work with protobuf schema messages in Python Apache pulsar pulsar-client package?
Apache Pulsar: Access state storage in LocalRunner not working
non-persistent message is lost when throughput is high
ProducerBlockedQuotaExceededError: Cannot create producer on topic with backlog quota exceeded
Pass private key as header in curl PUT returning error for illegal character
How to pass authorization key in shell script curl command without header
Event-time Temporal Join in Apache Flink only works with small datasets
does pulsar support multiple bookkeeper replicas in different clusters

Trending Discussions on pulsar

QUESTION

error 404 when try to make oauth in pulsar stream native cluster

Asked 2022-Mar-24 at 21:35

Hello I am triying to connect to apache pulsar cluster using stream native, I don't have problems with token oauth, but when I try to make Oauth I always get malformed responde or 404 I am using curl and python client, and following their instructions,like this.

params = '''
{
    "issuer_url": "https://auth.streamnative.cloud/",
    "private_key": test.json",
    "audience": "urn:sn:pulsar:test:test"
}
'''

pulsar_client = pulsar.Client(
    PULSAR_URL,
    authentication=AuthenticationOauth2(params)
)

and the error is the following:

AuthOauth2:223 | Response failed for getting the well-known configuration https://auth.streamnative.cloud/. response Code 404

But the params and url I get from https://console.streamnative.cloud/test/test/clients, but nothing.

Any Idea about how can I connect to the cluster with the oauth?

EDIT

With client the credentials works, looks like a bug in python package of pulsar 2.9.1.

Thanks

ANSWER

Answered 2022-Mar-24 at 20:25

You may need the full path to the private key. make sure it has permissions.

also make sure your audience is correct

what is pulsar URL format?

pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651

I have an example python app here

https://github.com/tspannhw/FLiP-Pi-BreakoutGarden/blob/c056b0eccc7ad18f8b18740bb30c224467991310/consume.sh

client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))

https://docs.streamnative.io/cloud/stable/connect/client/connect-python

https://github.com/streamnative/examples/blob/master/cloud/README.md

Also make sure you gave the userid associated with your key permissions.

If that doesn't work, chat with me on the Pulsar slack.

Source https://stackoverflow.com/questions/71577989

QUESTION

Kubernetes Statefulsets: Restart all pods concurrently (instead of in sequence)

Asked 2022-Feb-21 at 00:34

I have a use-case for concurrent restart of all pods in a statefulset.

Does kubernetes statefulset support concurrent restart of all pods?

According to the statefulset documentation, this can be accomplished by setting the pod update policy to parallel as in this example:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-db
spec:
  podManagementPolicy: Parallel
  replicas: 3

However this does not seem to work in practice, as demonstrated on this statefulset running on EKS:

Apply this:

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: producer
  namespace: ragnarok
spec:
  selector:
    matchLabels:
      app: producer
  replicas: 10
  podManagementPolicy: "Parallel"
  serviceName: producer-service
  template:
    metadata:
      labels:
        app: producer
    spec:
      containers:
      - name: producer
        image: archbungle/load-tester:pulsar-0.0.49
        imagePullPolicy: IfNotPresent

Rollout restart happens in sequence as if disregarding the rollout policy setting:


(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0                               1/1     Running             0          3m58s
producer-1                               1/1     Running             0          3m56s
producer-2                               1/1     Running             0          3m53s
producer-3                               1/1     Running             0          3m47s
producer-4                               1/1     Running             0          3m45s
producer-5                               1/1     Running             0          3m43s
producer-6                               1/1     Running             1          3m34s
producer-7                               0/1     ContainerCreating   0          1s
producer-8                               1/1     Running             0          16s
producer-9                               1/1     Running             0          19s

(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0                               1/1     Running       0          4m2s
producer-1                               1/1     Running       0          4m
producer-2                               1/1     Running       0          3m57s
producer-3                               1/1     Running       0          3m51s
producer-4                               1/1     Running       0          3m49s
producer-5                               1/1     Running       0          3m47s
producer-6                               0/1     Terminating   1          3m38s
producer-7                               1/1     Running       0          5s
producer-8                               1/1     Running       0          20s
producer-9                               1/1     Running       0          23s

(base) welcome@Traianos-MacBook-Pro eks-deploy % kubectl get pods -n ragnarok | egrep producer
producer-0                               1/1     Running       0          4m8s
producer-1                               1/1     Running       0          4m6s
producer-2                               1/1     Running       0          4m3s
producer-3                               1/1     Running       0          3m57s
producer-4                               1/1     Running       0          3m55s
producer-5                               0/1     Terminating   0          3m53s
producer-6                               1/1     Running       0          4s
producer-7                               1/1     Running       0          11s
producer-8                               1/1     Running       0          26s
producer-9                               1/1     Running       0          29s

ANSWER

Answered 2022-Feb-21 at 00:34

As the document pointed, Parallel pod management will effective only in the scaling operations. This option only affects the behavior for scaling operations. Updates are not affected.

Maybe you can try something like kubectl scale statefulset producer --replicas=0 -n ragnarok and kubectl scale statefulset producer --replicas=10 -n ragnarok

According to documentation, all pods should be deleted and created together by scaling them with the Parallel policy.

Reference : https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#parallel-pod-management

Source https://stackoverflow.com/questions/71196230

QUESTION

Apache Pulsar: Access state storage in LocalRunner not working

Asked 2022-Feb-07 at 19:16

I'm trying to implement a simple Apache Pulsar Function and access the State API in LocalRunner mode, but it's not working.

pom.xml snippet

<dependencies>
    <dependency>
    <groupId>org.apache.pulsar</groupId>
    <artifactId>pulsar-client-original</artifactId>
    <version>2.9.1</version>
  </dependency>
    <dependency>
      <groupId>org.apache.pulsar</groupId>
      <artifactId>pulsar-functions-local-runner-original</artifactId>
      <version>2.9.1</version>
    </dependency>
    <dependency>
      <groupId>com.fasterxml.jackson.core</groupId>
      <artifactId>jackson-core</artifactId>
      <version>2.13.1</version>
    </dependency>
  </dependencies>

Function class

public class TestFunction implements Function<String, String> {

  public String process(String input, Context context) throws Exception {
    System.out.println(">>> GOT input "+input);
    context.incrCounter("counter",1); //--> try to access state
    return input;
  }
}

Main

public class Main {
  public static final String BROKER_URL = "pulsar://localhost:6650";
  public static final String TOPIC_IN = "test-topic-input";
  public static final String TOPIC_OUT = "test-topic-output";

  public static void main(String[] args) throws Exception {

    FunctionConfig functionConfig = FunctionConfig
        .builder()
        .className(TestFunction.class.getName())
        .inputs(Collections.singleton(TOPIC_IN))
        .name("Test Function")
        .runtime(Runtime.JAVA)
        .subName("Test Function Sub")
        .build();

    LocalRunner localRunner = LocalRunner.builder()
        .brokerServiceUrl(BROKER_URL)
        .stateStorageServiceUrl("bk://127.0.0.1:4181")
        .functionConfig(functionConfig)
        .build();
    localRunner.start(false);

    PulsarClient client = PulsarClient.builder().serviceUrl(BROKER_URL).build();
    Producer<String> producer = client.newProducer(Schema.STRING).topic(TOPIC_IN).create();

    producer.send("Hello World!");
    System.out.println(">>> PRODUCER SENT");
  }

I'm running Pulsar in a local docker container (Docker Desktop on Win10), started like this:

Docker container

docker run -it 
-p 6650:6650  
-p 8080:8080 
-p 4181:4181 #I added that port to expose the bookkeeper, otherwise the function will hang and don't do anything
--mount source=pulsardata,target=/pulsar/data --mount source=pulsarconf,target=/pulsar/conf 
apachepulsar/pulsar:2.9.1 bin/pulsar standalone

When I start the application the Console shows these logs:

2022-01-07T12:39:48,757+0100 [public/default/Test Function-0] WARN  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Encountered issue Invalid stream name : public_default on fetching state stable metadata, re-attempting in 100 milliseconds
2022-01-07T12:39:48,863+0100 [public/default/Test Function-0] WARN  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Encountered issue Invalid stream name : public_default on fetching state stable metadata, re-attempting in 100 milliseconds
2022-01-07T12:39:48,970+0100 [public/default/Test Function-0] WARN  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Encountered issue Invalid stream name : public_default on fetching state stable metadata, re-attempting in 100 milliseconds
2022-01-07T12:39:49,076+0100 [public/default/Test Function-0] WARN  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Encountered issue Invalid stream name : public_default on fetching state stable metadata, re-attempting in 100 milliseconds

... and it goes on an on ...

Pulsar logging shows:

2022-01-07T12:13:48,882+0000 [grpc-default-executor-0] ERROR org.apache.bookkeeper.stream.storage.impl.metadata.RootRangeStoreImpl - Invalid stream name Test Function

2022-01-07T12:13:48,989+0000 [grpc-default-executor-0] ERROR org.apache.bookkeeper.stream.storage.impl.metadata.RootRangeStoreImpl - Invalid stream name Test Function

2022-01-07T12:13:49,097+0000 [grpc-default-executor-0] ERROR org.apache.bookkeeper.stream.storage.impl.metadata.RootRangeStoreImpl - Invalid stream name Test Function

2022-01-07T12:13:49,207+0000 [grpc-default-executor-0] ERROR org.apache.bookkeeper.stream.storage.impl.metadata.RootRangeStoreImpl - Invalid stream name Test Function

... goes on and on ...

What am I doing wrong?

ANSWER

Answered 2022-Feb-07 at 19:16

The issue is with the name you chose for your function, "Test Function". Since it has a space in it, that causes issues later on inside Pulsar's state store when it uses that name for the internal storage stream.

If you remove the space and use "TestFunction" instead, it will work just fine. I have confirmed this myself just now.

2022-02-07T11:09:04,916-0800 [main] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
>>> PRODUCER SENT
2022-02-07T11:09:05,267-0800 [public/default/TestFunction-0] INFO  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Opening state table for function public/default/TestFunction
2022-02-07T11:09:05,279-0800 [client-scheduler-OrderedScheduler-7-0] INFO  org.apache.bookkeeper.clients.SimpleStorageClientImpl - Retrieved table properties for table public_default/TestFunction : stream_id: 1024

2022-02-07T11:09:05,527-0800 [pulsar-client-io-1-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [test-topic-input][Test Function Sub] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
>>> GOT input Hello World!

Source https://stackoverflow.com/questions/70621132

QUESTION

Event-time Temporal Join in Apache Flink only works with small datasets

Asked 2021-Dec-10 at 09:31

Background: I'm trying to get an event-time temporal join working with two 'large(r)' datasets/tables that are read from a CSV-file (16K+ rows in left table, somewhat less in right table). Both tables are append-only tables, i.e. their datasources are currently CSV-files, but will become CDC changelogs emitted by Debezium over Pulsar. I am using the fairly new SYSTEM_TIME AS OF syntax.

The problem: join results are only partly correct, i.e. at the start (first 20% or so) of the execution of the query, rows of the left-side are not matched with rows from the right side, while in theory, they should. After a couple of seconds, there are more matches, and by the time the query ends, rows of the left side are getting matched/joined correctly with rows of the right side. Every time that I run the query it shows other results in terms of which rows are (not) matched.

Both datasets are not ordered by their respective event-times. They are ordered by their primary key. So it's really this case, only with more data.

In essence, the right side is a lookup-table that changes over time, and we're sure that for every left record there was a matching right record, as both were created in the originating database at +/- the same instant. Ultimately our goal is a dynamic materialized view that contains the same data as when we'd join the 2 tables in the CDC-enabled source database (SQL Server).

Obviously, I want to achieve a correct join over the complete dataset as explained in the Flink docs
Unlike simple examples and Flink test-code with a small dataset of only a few rows (like here), a join of larger datasets does not yield correct results.

I suspect that, when the probing/left table starts flowing, the build/right table is not yet 'in memory' which means that left rows don't find a matching right row, while they should -- if the right table would have started flowing somewhat earlier. That's why the left join returns null-values for the columns of the right table.

I've included my code:

@Slf4j(topic = "TO_FILE")
public class CsvTemporalJoinTest {

    private final String emr01Ddl =
            "CREATE TABLE EMR01\n" +
                    "(\n" +
                    "    SRC_NO         STRING,\n" +
                    "    JRD_ETT_NO     STRING,\n" +
                    "    STT_DT         DATE,\n" +
                    "    MGT_SLT_DT     DATE,\n" +
                    "    ATM_CRT_DT     DATE,\n" +
                    "    LTD_MDT_IC     STRING,\n" +
                    "    CPN_ORG_NO     STRING,\n" +
                    "    PTY_NO         STRING,\n" +
                    "    REG_USER_CD    STRING,\n" +
                    "    REG_TS         TIMESTAMP,\n" +
                    "    MUT_USER_CD    STRING,\n" +
                    "    MUT_TS         TIMESTAMP(3),\n" +
                    "    WATERMARK FOR MUT_TS AS MUT_TS,\n" +
                    "    PRIMARY KEY (CPN_ORG_NO) NOT ENFORCED\n" +
                    ") WITH (\n" +
                    "   'connector' = 'filesystem',\n" +
                    "   'path' = '" + getCsv1() + "',\n" +
                    "   'format' = 'csv'\n" +
                    ")";

    private final String emr02Ddl =
            "CREATE TABLE EMR02\n" +
                    "(\n" +
                    "    CPN_ORG_NO  STRING,\n" +
                    "    DSB_TX      STRING,\n" +
                    "    REG_USER_CD STRING,\n" +
                    "    REG_TS      TIMESTAMP,\n" +
                    "    MUT_USER_CD STRING,\n" +
                    "    MUT_TS      TIMESTAMP(3),\n" +
                    "    WATERMARK FOR MUT_TS AS MUT_TS,\n" +
                    "    PRIMARY KEY (CPN_ORG_NO) NOT ENFORCED\n" +
                    ") WITH (\n" +
                    "   'connector' = 'filesystem',\n" +
                    "   'path' = '" + getCsv2() + "',\n" +
                    "   'format' = 'csv'\n" +
                    ")";

    @Test
    public void testEventTimeTemporalJoin() throws Exception {
        var env = StreamExecutionEnvironment.getExecutionEnvironment();
        var tableEnv = StreamTableEnvironment.create(env);

        tableEnv.executeSql(emr01Ddl);
        tableEnv.executeSql(emr02Ddl);

        Table result = tableEnv.sqlQuery("" +
                "SELECT *" +
                "   FROM EMR01" +
                "   LEFT JOIN EMR02 FOR SYSTEM_TIME AS OF EMR01.MUT_TS" +
                "       ON EMR01.CPN_ORG_NO = EMR02.CPN_ORG_NO");

        tableEnv.toChangelogStream(result).addSink(new TestSink());
        env.execute();

        System.out.println("[Count]" + TestSink.values.size());
        //System.out.println("[Row 1]" + TestSink.values.get(0));
        //System.out.println("[Row 2]" + TestSink.values.get(1));
        AtomicInteger i = new AtomicInteger();
        TestSink.values.listIterator().forEachRemaining(value -> log.info("[Row " + i.incrementAndGet() + " ]=" + value));
    }

    private static class TestSink implements SinkFunction<Row> {

        // must be static
        public static final List<Row> values = Collections.synchronizedList(new ArrayList<>());

        @Override
        public void invoke(Row value, SinkFunction.Context context) {
            values.add(value);
        }
    }

    String getCsv1() {
        try {
            return new ClassPathResource("/GBTEMR01.csv").getFile().getAbsolutePath();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

    String getCsv2() {
        try {
            return new ClassPathResource("/GBTEMR02.csv").getFile().getAbsolutePath();
        } catch (Exception e) {
            throw new RuntimeException(e);
        }
    }

}

Is there a way to solve this? E.g. is there a way to FIRST load the right side into Flink state, and THEN start loading/streaming the left side? Would this be a good approach because this question begs: how much later? what is the time that the left side can start flowing?

We're using Flink 1.13.3.

ANSWER

Answered 2021-Dec-10 at 09:31

This sort of temporal/versioned join depends on having accurate watermarks. Flink relies on the watermarks to know which rows can safely be dropped from the state being maintained (because they can no longer affect the results).

The watermarking you've used indicates that the rows are ordered by MUT_TS. Since this isn't true, the join isn't able to produce complete results.

To fix this, the watermarks should be defined with something like this

WATERMARK FOR MUT_TS AS MUT_TS - INTERVAL '2' MINUTE

where the interval indicates how much out-of-orderness needs to be accommodated.

Source https://stackoverflow.com/questions/70295647

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install pulsar

Compile and install individual module.
Java JDK 11 or JDK 8
Maven 3.6.1+
zip
Run Individual Unit Test:.
Docker images must be built with Java 8 for branch-2.7 or previous branches because of issue 8445. Java 11 is the recommended JDK version in master/branch-2.8. This builds the docker images apachepulsar/pulsar-all:latest and apachepulsar/pulsar:latest. After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to localhost:32000/apachepulsar.
Apache Pulsar is using lombok so you have to ensure your IDE setup with required plugins.
Refer to the docs README.