pulsar | Scrape web data at scale | Crawler library

I have a use-case for concurrent restart of all pods in a statefulset.

Does kubernetes statefulset support concurrent restart of all pods?

According to the statefulset documentation, this can be accomplished by setting the pod update policy to parallel as in this example:

Is there a way to publish message to an Apache Pulsar topic using Protobuf schema using pulsar-client package using python?

As per the documentation, it supports only Avro, String, Json and bytes. Any work around for this? https://pulsar.apache.org/docs/ko/2.8.1/client-libraries-python/

I'm trying to implement a simple Apache Pulsar Function and access the State API in LocalRunner mode, but it's not working.

pom.xml snippet

I found that non-persistent messages are lost sometimes even though the my pulsar client is up and running. Those non-persistent messages are lost when the throughput is high (more than 1000 messages within a very short period of time. I personally think that this is not high). If I increase the parameter receiverQueueSize or change the message type to persistent message, the problem is gone.

I check the Pulsar source code (I am not sure this is the latest one)

https://github.com/apache/pulsar/blob/35f0e13fc3385b54e88ddd8e62e44146cf3b060d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/nonpersistent/NonPersistentDispatcherMultipleConsumers.java#L185

and I think that Pulsar simply ignore those non-persistent messages if no consumer is available to handle the newly arrived non-persistent messages. "No consumer" here means

• no consumer subscribe the topic
• OR all consumers are busy on processing messages received before

Is my understanding correct?

I have a Lucidworks Fusion 5 kubernetes installation setup on AWS EKS and currently one of the services, Connector Classic REST service, is experiencing an outage. After digging into the logs I found:

I have a .pem file containing my private key that I need to pass as an authorization header.

I've tried just using the command $(cat $REPO_ROOT/pulsar/tls/broker/broker.key.pem) but I'm getting the response:

In a shell script, I need to pull a private key from a .pem file. When I set my AUTORIZATION variable to the path, the variable is only the filepath string, not the actual filepath.

If I change my AUTHORIZATION variable to cat it imports the header and footer i.e. -----BEGIN RSA PRIVATE KEY... END RSA PRIVATE KEY-----

How do I pull out the RSA key without the header and footer?

Background: I'm trying to get an event-time temporal join working with two 'large(r)' datasets/tables that are read from a CSV-file (16K+ rows in left table, somewhat less in right table). Both tables are append-only tables, i.e. their datasources are currently CSV-files, but will become CDC changelogs emitted by Debezium over Pulsar. I am using the fairly new SYSTEM_TIME AS OF syntax.

The problem: join results are only partly correct, i.e. at the start (first 20% or so) of the execution of the query, rows of the left-side are not matched with rows from the right side, while in theory, they should. After a couple of seconds, there are more matches, and by the time the query ends, rows of the left side are getting matched/joined correctly with rows of the right side. Every time that I run the query it shows other results in terms of which rows are (not) matched.

Both datasets are not ordered by their respective event-times. They are ordered by their primary key. So it's really this case, only with more data.

In essence, the right side is a lookup-table that changes over time, and we're sure that for every left record there was a matching right record, as both were created in the originating database at +/- the same instant. Ultimately our goal is a dynamic materialized view that contains the same data as when we'd join the 2 tables in the CDC-enabled source database (SQL Server).

Obviously, I want to achieve a correct join over the complete dataset as explained in the Flink docs
Unlike simple examples and Flink test-code with a small dataset of only a few rows (like here), a join of larger datasets does not yield correct results.

I suspect that, when the probing/left table starts flowing, the build/right table is not yet 'in memory' which means that left rows don't find a matching right row, while they should -- if the right table would have started flowing somewhat earlier. That's why the left join returns null-values for the columns of the right table.

I've included my code:

I have a use case where requires data backup across multiple data centers and needs strong consistency. The ideal view is each segment is replicated to three clusters located at three different data centers. pulsar supports using multiple clusters as a large bookie pool but I didn't find how to configure the replicas in different clusters. Anyone has similar use case before? i think it should be not hard to do considering pulsar separate broker and storage + replicas in different clusters