kandi background
Explore Kits

pulsar | Apache Pulsar distributed pubsub messaging system | Pub Sub library

 by   apache Java Version: v2.10.0 License: Apache-2.0

 by   apache Java Version: v2.10.0 License: Apache-2.0

Download this library from

kandi X-RAY | pulsar Summary

pulsar is a Java library typically used in Messaging, Pub Sub, Kafka applications. pulsar has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub, Maven.
Pulsar is a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API. Learn more about Pulsar at https://pulsar.apache.org.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • pulsar has a highly active ecosystem.
  • It has 10675 star(s) with 2795 fork(s). There are 422 watchers for this library.
  • There were 3 major release(s) in the last 6 months.
  • There are 1695 open issues and 3122 have been closed. On average issues are closed in 83 days. There are 361 open pull requests and 0 closed requests.
  • It has a positive sentiment in the developer community.
  • The latest version of pulsar is v2.10.0
pulsar Support
Best in #Pub Sub
Average in #Pub Sub
pulsar Support
Best in #Pub Sub
Average in #Pub Sub

quality kandi Quality

  • pulsar has 0 bugs and 0 code smells.
pulsar Quality
Best in #Pub Sub
Average in #Pub Sub
pulsar Quality
Best in #Pub Sub
Average in #Pub Sub

securitySecurity

  • pulsar has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • pulsar code analysis shows 0 unresolved vulnerabilities.
  • There are 0 security hotspots that need review.
pulsar Security
Best in #Pub Sub
Average in #Pub Sub
pulsar Security
Best in #Pub Sub
Average in #Pub Sub

license License

  • pulsar is licensed under the Apache-2.0 License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
pulsar License
Best in #Pub Sub
Average in #Pub Sub
pulsar License
Best in #Pub Sub
Average in #Pub Sub

buildReuse

  • pulsar releases are available to install and integrate.
  • Deployable package is available in Maven.
  • Build file is available. You can build the component from source.
  • Installation instructions, examples and code snippets are available.
  • It has 444636 lines of code, 28552 functions and 3482 files.
  • It has medium code complexity. Code complexity directly impacts maintainability of the code.
pulsar Reuse
Best in #Pub Sub
Average in #Pub Sub
pulsar Reuse
Best in #Pub Sub
Average in #Pub Sub
Top functions reviewed by kandi - BETA

kandi has reviewed pulsar and discovered the below as its top functions. This is intended to give you an instant insight into pulsar implemented functionality, and help decide if they suit your requirements.

  • Creates a BitSetRecyclable from the specified ByteBuffer .
  • Prints statistics for a topic .
  • Handles a command producer .
  • Update a function .
  • Update a source .
  • Updates an existing sink .
  • Calculate cursor back cursors counters .
  • Returns command to execute .
  • Truncate ledgers .
  • Returns the internal statistics for this topic .

pulsar Key Features

Horizontally scalable (Millions of independent topics and millions of messages published per second)

Strong ordering and consistency guarantees

Low latency durable storage

Topic and queue semantics

Load balancer

Designed for being deployed as a hosted service: Multi-tenant Authentication Authorization Quotas Support mixing very different workloads Optional hardware isolation

Keeps track of consumer cursor position

REST API for provisioning, admin and stats

Geo replication

Transparent handling of partitioned topics

Transparent batching of messages

Build Pulsar

copy iconCopydownload iconDownload
$ mvn install -DskipTests

Minimal build (This skips most of external connectors and tiered storage handlers)

copy iconCopydownload iconDownload
mvn install -Pcore-modules,-main -DskipTests

Build custom docker images

copy iconCopydownload iconDownload
mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

error 404 when try to make oauth in pulsar stream native cluster

copy iconCopydownload iconDownload
pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651 
client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))
-----------------------
pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651 
client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))

Apache Pulsar: Access state storage in LocalRunner not working

copy iconCopydownload iconDownload
2022-02-07T11:09:04,916-0800 [main] WARN  com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
>>> PRODUCER SENT
2022-02-07T11:09:05,267-0800 [public/default/TestFunction-0] INFO  org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Opening state table for function public/default/TestFunction
2022-02-07T11:09:05,279-0800 [client-scheduler-OrderedScheduler-7-0] INFO  org.apache.bookkeeper.clients.SimpleStorageClientImpl - Retrieved table properties for table public_default/TestFunction : stream_id: 1024

2022-02-07T11:09:05,527-0800 [pulsar-client-io-1-2] INFO  org.apache.pulsar.client.impl.ConsumerImpl - [test-topic-input][Test Function Sub] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
>>> GOT input Hello World!

Pass private key as header in curl PUT returning error for illegal character

copy iconCopydownload iconDownload
Authorization: Bearer $AUTHORIZATION
-----------------------
bin/pulsar tokens create --private-key file:///path/to/my-private.key \
            --subject test-user

How to pass authorization key in shell script curl command without header

copy iconCopydownload iconDownload
AUTHORIZATION=$(cat $REPO_ROOT/pulsar/tls/broker/broker.key.pem)

Event-time Temporal Join in Apache Flink only works with small datasets

copy iconCopydownload iconDownload
WATERMARK FOR MUT_TS AS MUT_TS - INTERVAL '2' MINUTE

Why does the data streamed using the Java client appear as encoded on Apache Pulsar?

copy iconCopydownload iconDownload
Schema.AVRO(DavisMessage.class)

Pulsar client thread balance

copy iconCopydownload iconDownload
PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://localhost:6650")
    .ioThreads(16)
    .connectionsPerBroker(16)
    .create();

Pulsar function fails to deserialize message because of wrong schema type (JSON instead of AVRO)

copy iconCopydownload iconDownload
Map<String, ConsumerConfig> inputSpecs = new HashMap<String, ConsumerConfig> ();
inputSpecs.put("persistent://orders/inbound/food-orders", 
    ConsumerConfig.builder().schemaType("avro").build());
FunctionConfig functionConfig = 
    FunctionConfig.builder()
        ...
        .inputSpecs(inputSpecs)
        ...
        .build();

inputSpecs:
 $topicName:
  schemaType: AVRO
-----------------------
Map<String, ConsumerConfig> inputSpecs = new HashMap<String, ConsumerConfig> ();
inputSpecs.put("persistent://orders/inbound/food-orders", 
    ConsumerConfig.builder().schemaType("avro").build());
FunctionConfig functionConfig = 
    FunctionConfig.builder()
        ...
        .inputSpecs(inputSpecs)
        ...
        .build();

inputSpecs:
 $topicName:
  schemaType: AVRO

Pulsar: If a message gets nack'd (negativeAcknowledge()) when will it be redelivered?

copy iconCopydownload iconDownload
Consumer<byte[]> consumer = client.newConsumer()
    .topic("my-topic")
    .subscriptionName("my-sub")
    .negativeAckRedelivery(10, TimeUnit.SECONDS)
    .subscribe()

How to organize multiple Dockerfiles, docker-compose.yaml and .dockerignore

copy iconCopydownload iconDownload
COPY requirements.txt ./
version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
-----------------------
COPY requirements.txt ./
version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
-----------------------
COPY requirements.txt ./
version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
-----------------------
COPY requirements.txt ./
version: '3.8'
services:
   producer:
     build: ./python_producer
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build: ./python_consumer
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
   producer:
     build:
       context: .
       dockerfile: python_producer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   consumer:
     build:
       context: .
       dockerfile: python_consumer/Dockerfile
     environment:
       - RABBITMQ_HOST=rabbitmq
   rabbitmq:
     image: rabbitmq:3
     hostname: rabbitmq # RabbitMQ is very unusual in needing to set this

Community Discussions

Trending Discussions on pulsar
  • error 404 when try to make oauth in pulsar stream native cluster
  • Kubernetes Statefulsets: Restart all pods concurrently (instead of in sequence)
  • How to work with protobuf schema messages in Python Apache pulsar pulsar-client package?
  • Apache Pulsar: Access state storage in LocalRunner not working
  • non-persistent message is lost when throughput is high
  • ProducerBlockedQuotaExceededError: Cannot create producer on topic with backlog quota exceeded
  • Pass private key as header in curl PUT returning error for illegal character
  • How to pass authorization key in shell script curl command without header
  • Event-time Temporal Join in Apache Flink only works with small datasets
  • does pulsar support multiple bookkeeper replicas in different clusters
Trending Discussions on pulsar

QUESTION

error 404 when try to make oauth in pulsar stream native cluster

Asked 2022-Mar-24 at 21:35

Hello I am triying to connect to apache pulsar cluster using stream native, I don't have problems with token oauth, but when I try to make Oauth I always get malformed responde or 404 I am using curl and python client, and following their instructions,like this.

params = '''
{
    "issuer_url": "https://auth.streamnative.cloud/",
    "private_key": test.json",
    "audience": "urn:sn:pulsar:test:test"
}
'''

pulsar_client = pulsar.Client(
    PULSAR_URL,
    authentication=AuthenticationOauth2(params)
)

and the error is the following:

AuthOauth2:223 | Response failed for getting the well-known configuration https://auth.streamnative.cloud/. response Code 404

But the params and url I get from https://console.streamnative.cloud/test/test/clients, but nothing.

Any Idea about how can I connect to the cluster with the oauth?

EDIT

With client the credentials works, looks like a bug in python package of pulsar 2.9.1.

Thanks

ANSWER

Answered 2022-Mar-24 at 20:25

You may need the full path to the private key. make sure it has permissions.

also make sure your audience is correct

what is pulsar URL format?

pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651 
client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))

https://docs.streamnative.io/cloud/stable/connect/client/connect-python

https://github.com/streamnative/examples/blob/master/cloud/README.md

Also make sure you gave the userid associated with your key permissions.

If that doesn't work, chat with me on the Pulsar slack.

Source https://stackoverflow.com/questions/71577989

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install pulsar

Compile and install individual module.
Java JDK 11 or JDK 8
Maven 3.6.1+
zip
Run Individual Unit Test:.
Docker images must be built with Java 8 for branch-2.7 or previous branches because of issue 8445. Java 11 is the recommended JDK version in master/branch-2.8. This builds the docker images apachepulsar/pulsar-all:latest and apachepulsar/pulsar:latest. After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to localhost:32000/apachepulsar.
Apache Pulsar is using lombok so you have to ensure your IDE setup with required plugins.
Refer to the docs README.

Support

Pulsar Translation

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Explore Related Topics

Share this Page

share link
Reuse Pre-built Kits with pulsar
Compare Pub Sub Libraries with Highest Support
Compare Pub Sub Libraries with Highest Reuse
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.