Support
Quality
Security
License
Reuse
kandi has reviewed pulsar and discovered the below as its top functions. This is intended to give you an instant insight into pulsar implemented functionality, and help decide if they suit your requirements.
Horizontally scalable (Millions of independent topics and millions of messages published per second)
Strong ordering and consistency guarantees
Low latency durable storage
Topic and queue semantics
Load balancer
Designed for being deployed as a hosted service: Multi-tenant Authentication Authorization Quotas Support mixing very different workloads Optional hardware isolation
Keeps track of consumer cursor position
REST API for provisioning, admin and stats
Geo replication
Transparent handling of partitioned topics
Transparent batching of messages
Build Pulsar
$ mvn install -DskipTests
Minimal build (This skips most of external connectors and tiered storage handlers)
mvn install -Pcore-modules,-main -DskipTests
Build custom docker images
mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
error 404 when try to make oauth in pulsar stream native cluster
pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651
client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))
-----------------------
pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651
client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))
Apache Pulsar: Access state storage in LocalRunner not working
2022-02-07T11:09:04,916-0800 [main] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider
>>> PRODUCER SENT
2022-02-07T11:09:05,267-0800 [public/default/TestFunction-0] INFO org.apache.pulsar.functions.instance.state.BKStateStoreProviderImpl - Opening state table for function public/default/TestFunction
2022-02-07T11:09:05,279-0800 [client-scheduler-OrderedScheduler-7-0] INFO org.apache.bookkeeper.clients.SimpleStorageClientImpl - Retrieved table properties for table public_default/TestFunction : stream_id: 1024
2022-02-07T11:09:05,527-0800 [pulsar-client-io-1-2] INFO org.apache.pulsar.client.impl.ConsumerImpl - [test-topic-input][Test Function Sub] Subscribed to topic on localhost/127.0.0.1:6650 -- consumer: 0
>>> GOT input Hello World!
Pass private key as header in curl PUT returning error for illegal character
Authorization: Bearer $AUTHORIZATION
-----------------------
bin/pulsar tokens create --private-key file:///path/to/my-private.key \
--subject test-user
How to pass authorization key in shell script curl command without header
AUTHORIZATION=$(cat $REPO_ROOT/pulsar/tls/broker/broker.key.pem)
Event-time Temporal Join in Apache Flink only works with small datasets
WATERMARK FOR MUT_TS AS MUT_TS - INTERVAL '2' MINUTE
Why does the data streamed using the Java client appear as encoded on Apache Pulsar?
Schema.AVRO(DavisMessage.class)
Pulsar client thread balance
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://localhost:6650")
.ioThreads(16)
.connectionsPerBroker(16)
.create();
Pulsar function fails to deserialize message because of wrong schema type (JSON instead of AVRO)
Map<String, ConsumerConfig> inputSpecs = new HashMap<String, ConsumerConfig> ();
inputSpecs.put("persistent://orders/inbound/food-orders",
ConsumerConfig.builder().schemaType("avro").build());
FunctionConfig functionConfig =
FunctionConfig.builder()
...
.inputSpecs(inputSpecs)
...
.build();
inputSpecs:
$topicName:
schemaType: AVRO
-----------------------
Map<String, ConsumerConfig> inputSpecs = new HashMap<String, ConsumerConfig> ();
inputSpecs.put("persistent://orders/inbound/food-orders",
ConsumerConfig.builder().schemaType("avro").build());
FunctionConfig functionConfig =
FunctionConfig.builder()
...
.inputSpecs(inputSpecs)
...
.build();
inputSpecs:
$topicName:
schemaType: AVRO
Pulsar: If a message gets nack'd (negativeAcknowledge()) when will it be redelivered?
Consumer<byte[]> consumer = client.newConsumer()
.topic("my-topic")
.subscriptionName("my-sub")
.negativeAckRedelivery(10, TimeUnit.SECONDS)
.subscribe()
How to organize multiple Dockerfiles, docker-compose.yaml and .dockerignore
COPY requirements.txt ./
version: '3.8'
services:
producer:
build: ./python_producer
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build: ./python_consumer
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
producer:
build:
context: .
dockerfile: python_producer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build:
context: .
dockerfile: python_consumer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
-----------------------
COPY requirements.txt ./
version: '3.8'
services:
producer:
build: ./python_producer
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build: ./python_consumer
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
producer:
build:
context: .
dockerfile: python_producer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build:
context: .
dockerfile: python_consumer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
-----------------------
COPY requirements.txt ./
version: '3.8'
services:
producer:
build: ./python_producer
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build: ./python_consumer
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
producer:
build:
context: .
dockerfile: python_producer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build:
context: .
dockerfile: python_consumer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
-----------------------
COPY requirements.txt ./
version: '3.8'
services:
producer:
build: ./python_producer
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build: ./python_consumer
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
COPY python_producer/requirements.txt ./
version: '3.8'
services:
producer:
build:
context: .
dockerfile: python_producer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
consumer:
build:
context: .
dockerfile: python_consumer/Dockerfile
environment:
- RABBITMQ_HOST=rabbitmq
rabbitmq:
image: rabbitmq:3
hostname: rabbitmq # RabbitMQ is very unusual in needing to set this
QUESTION
error 404 when try to make oauth in pulsar stream native cluster
Asked 2022-Mar-24 at 21:35Hello I am triying to connect to apache pulsar cluster using stream native, I don't have problems with token oauth, but when I try to make Oauth I always get malformed responde or 404 I am using curl and python client, and following their instructions,like this.
params = '''
{
"issuer_url": "https://auth.streamnative.cloud/",
"private_key": test.json",
"audience": "urn:sn:pulsar:test:test"
}
'''
pulsar_client = pulsar.Client(
PULSAR_URL,
authentication=AuthenticationOauth2(params)
)
and the error is the following:
AuthOauth2:223 | Response failed for getting the well-known configuration https://auth.streamnative.cloud/. response Code 404
But the params and url I get from https://console.streamnative.cloud/test/test/clients, but nothing.
Any Idea about how can I connect to the cluster with the oauth?
EDIT
With client the credentials works, looks like a bug in python package of pulsar 2.9.1.
Thanks
ANSWER
Answered 2022-Mar-24 at 20:25You may need the full path to the private key. make sure it has permissions.
also make sure your audience is correct
what is pulsar URL format?
pulsar+ssl://sn-myinstance.mycluster.snio.cloud:6651
I have an example python app here
client = pulsar.Client(args.service_url, authentication=AuthenticationOauth2(args.auth_params))
https://docs.streamnative.io/cloud/stable/connect/client/connect-python
https://github.com/streamnative/examples/blob/master/cloud/README.md
Also make sure you gave the userid associated with your key permissions.
If that doesn't work, chat with me on the Pulsar slack.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
No vulnerabilities reported
Save this library and start creating your kit
Save this library and start creating your kit