LogiKM | 一站式Apache Kafka集群指标监控与运维管控平台 | Pub Sub library
kandi X-RAY | LogiKM Summary
Support
Quality
Security
License
Reuse
- Returns true if the given object is a primitive array .
- Converts a NotificationStrategy map to a Strategy
- Produces a sorted list of the analysis broker DTOs .
- Returns the code for a Kafka topic .
- Records a consumer .
- set the running state
- Get topic metrics from database .
- Converts the cluster status to a map .
- Converts a list of MetricsVo to a broker MetricsV object .
- Get authorized apps
LogiKM Key Features
LogiKM Examples and Code Snippets
Trending Discussions on Pub Sub
Trending Discussions on Pub Sub
QUESTION
In R, I want to build json content according this Google Cloud Pub Sub message format: https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage
It have to respect :
{
"data": string,
"attributes": {
string: string,
...
},
"messageId": string,
"publishTime": string,
"orderingKey": string
}
The message built will be readed from this Python code:
def pubsub_read(data, context):
'''This function is executed from a Cloud Pub/Sub'''
message = base64.b64decode(data['data']).decode('utf-8')
file_name = data['attributes']['file_name']
This following R code builds a R dataframe and converts it to json content:
library(jsonlite)
data="Hello World!"
df <- data.frame(data)
attributes <- data.frame(file_name=c('gfs_data_temp_FULL.csv'))
df$attributes <- attributes
msg <- df %>%
toJSON(auto_unbox = TRUE, dataframe = 'columns', pretty = T) %>%
# Pub/Sub expects a base64 encoded string
googlePubsubR::msg_encode() %>%
googlePubsubR::PubsubMessage()
ANSWER
Answered 2022-Apr-16 at 09:59Not sure why, but replacing the dataframe by a list seems to work:
library(jsonlite)
df = list(data = "Hello World")
attributes <- list(file_name=c('toto.csv'))
df$attributes <- attributes
df %>%
toJSON(auto_unbox = TRUE, simplifyVector=TRUE, dataframe = 'columns', pretty = T)
Output:
{
"data": "Hello World",
"attributes": {
"file_name": "toto.csv"
}
}
QUESTION
My basic requirement was to create a pipeline to read from BigQuery Table and then convert it into JSON and pass it onto a PubSub topic.
At first I read from Big Query and tried to write it into Pub Sub Topic but got an exception error saying "Pub Sub" is not supported for batch pipelines. So I tried some workarounds and
I was able to work around this in python by
- Reading from BigQuery-> ConvertTo JSON string-> Save as text file in cloud storage (Beam pipeline)
p = beam.Pipeline(options=options)
json_string_output = (
p
| 'Read from BQ' >> beam.io.ReadFromBigQuery(
query='SELECT * FROM '\
'`project.dataset.table_name`',
use_standard_sql=True)
| 'convert to json' >> beam.Map(lambda record: json.dumps(record))
| 'Write results' >> beam.io.WriteToText(outputs_prefix)
)
p.run()
- And then from there run a normal python script to read it line from file and pass it onto PubSub Topic
# create publisher
publisher = pubsub_v1.PublisherClient()
with open(input_file, 'rb') as ifp:
header = ifp.readline()
# loop over each record
for line in ifp:
event_data = line # entire line of input file is the message
print('Publishing {0} to {1}'.format(event_data, pubsub_topic))
publisher.publish(pubsub_topic, event_data)
I'm not able to find a way to integrate both scripts within a single ApacheBeam Pipeline.
ANSWER
Answered 2021-Oct-14 at 20:27Because your pipeline does not have any unbounded PCollections, it will be automatically run in batch mode. You can force a pipeline to run in streaming mode with the --streaming command line flag.
QUESTION
We are using Pub Sub lite instances along with reservations, we want to deploy it via Terraform, on UI while creating a Pub Sub Lite we get an option to specify Peak Publish Throughput (MiB/s) and Peak Subscribe Throughput (MiB/s) which is not available in the resource "google_pubsub_lite_topic" as per this doc https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_lite_topic.
resource "google_pubsub_lite_reservation" "pubsub_lite_reservation" {
name = var.lite_reservation_name
project = var.project
region = var.region
throughput_capacity = var.throughput_capacity
}
resource "google_pubsub_lite_topic" "pubsub_lite_topic" {
name = var.topic_name
project = var.project
region = var.region
zone = var.zone
partition_config {
count = var.partitions_count
capacity {
publish_mib_per_sec = var.publish_mib_per_sec
subscribe_mib_per_sec = var.subscribe_mib_per_sec
}
}
retention_config {
per_partition_bytes = var.per_partition_bytes
period = var.period
}
reservation_config {
throughput_reservation = google_pubsub_lite_reservation.pubsub_lite_reservation.name
}
}
Currently use the above TF script to create pub sub lite instance, the problem here is we are mentioning the throughput capacity instead of setting the peak throughput capacity, and capacity block is a required field. Please help if there is any workaround to it ? we want topic to set throughput dynamically but with peak limit to the throughput, as we are setting a fix value to the lite reservation.
ANSWER
Answered 2022-Feb-20 at 21:46If you check the bottom of your Google Cloud console screenshot, you can see it suggests to have 4 partitions with 4MiB/s publish and subscribe throughput.
Therefore your Terraform partition_config should match this. Count should be 4 for the 4 partitions, with capacity of 4MiB/s publish and 4MiB/s subscribe for each partition.
The "peak throughput" in web UI is just for convenience to help you choose some numbers here. The actual underlying PubSub Lite API doesn't actually have this field, which is why there is no Terraform setting either. You will notice the sample docs require a per-partiton setting just like Terraform.
eg. https://cloud.google.com/pubsub/lite/docs/samples/pubsublite-create-topic
I think the only other alternative would be to create a reservation attached to your topic with enough throughput units for desired capacity. And then completely omit capacity block in Terraform and let the reservation decide.
QUESTION
I have a User that needs to be able to query and create Jetstream keyvalue stores. I attempted to add pub/sub access to $JS.API.STREAM.INFO.* in order to give the User the ability to query and create keyvalue stores:
96f4d12cdd02:~# nsc edit user RequestCacheService --allow-pubsub "$JS.API.STREAM.INFO.*"
[ OK ] added pub pub ".API.STREAM.INFO.*"
[ OK ] added sub ".API.STREAM.INFO.*"
[ OK ] generated user creds file `/nsc/nkeys/creds/Client/Client/RequestCacheService.creds`
[ OK ] edited user "RequestCacheService"
As you can see above, pub sub was added for ".API.STREAM.INFO.", not "$JS.API.STREAM.INFO.".
How do I allow a User permissions to query & create Jetstream keyvalue stores?
ANSWER
Answered 2022-Jan-31 at 16:16Should be:
nsc edit user RequestCacheService --allow-pubsub '$JS.API.STREAM.INFO.*'
With single-quotes around the subject. I was under the impression that double & single quotes would escape the $ but apparently only single-quote will escape special characters in the subject.
QUESTION
I am deciding if I should use MSK (managed kafka from AWS) or a combination of SQS + SNS to achieve a pub sub model?
Background
Currently, we have a micro service architecture but we don't use any messaging service and only use REST apis (dont ask why - related to some 3rd party vendors who designed the architecture). Now, I want to revamp it and start using messaging for communication between micro-services.
Initially, the plan is to start publishing entity events for any other micro service to consume - these events will also be stored in data lake in S3 which will also serve as a base for starting data team.
Later, I want to move certain features from REST to async communication.
Anyway, the main question I have is - should I decide to go with MSK or should I use SQS + SNS for the same? ( I already understand the basic concepts but wanted to understand from fellow community if there are some other pros and cons)?
Thanks in advance
ANSWER
Answered 2022-Feb-09 at 17:58MSK VS SQS+SNS is not really 1:1 comparison. The choice depends on various use cases. Please find out some of specific difference between two
- Scalability -> MSK has better scalability option because of inherent design of partitions that allow parallelism and ordering of message. SNS has limitation of 300 publish/Second, to achieve same performance as MSK, there need to have higher number of SNS topic for same purpose.
Example : Topic: Order Service in MSK -> one topic+ 10 Partitions SNS -> 10 topics
if client/message producer use 10 SNS topic for same purpose, then client needs to have information of all 10 SNS topic and distribution of message. In MSK, it's pretty straightforward, key needs to send in message and kafka will allocate the partition based on Key value.
Administration/Operation -> SNS+SQS setup is much simpler compare to MSK. Operational challenge is much more with MSK( even this is managed service). MSK needs more in depth skills to use optimally.
SNS +SQS VS SQS -> I believe you have multiple subscription(fanout) for same message thats why you have refer SNS +SQS. If you have One Subscription for one message, then only SQS is also sufficient.
Replay of message -> MSK can be use for replaying the already processed message. It will be tricky for SQS, though can be achieve by having duplicate queue so that can be use for replay.
QUESTION
After following the dataflow tutorial, I used the pub/sub topic to big query template to parse a JSON record into a table. The Job has been streaming for 21 days. During that time I have ingested about 5000 JSON records, containing 4 fields (around 250 bytes).
After the bill came this month I started to look into resource usage. I have used 2,017.52 vCPU hr, memory 7,565.825 GB hr, Total HDD 620,407.918 GB hr.
This seems absurdly high for the tiny amount of data I have been ingesting. Is there a minimum amount of data I should have before using dataflow? It seems over powered for small cases. Is there another preferred method for ingesting data from a pub sub topic? Is there a different configuration when setting up a Dataflow Job that uses less resources?
ANSWER
Answered 2022-Feb-03 at 21:43It seems that the numbers you mentioned, correspond to not customizing the job resources. By default streaming jobs use a n1-standar-4 machine:
3 Streaming worker defaults: 4 vCPU, 15 GB memory, 400 GB Persistent Disk.
4 vCPU x 24 hrs x 21 days = 2,016
15 GB x 24 hrs x 21 days = 7,560
If you really need streaming in Dataflow, you will need to pay for resources allocated even if there is nothing to process.
Options:
Optimizing Dataflow
- Considering that the number and size of the JSON string you need to process are really small, you can reduce the cost to aprox 1/4 of current charge. You just need to set the job to use a n1-standard-1 machine, which has 1vCPU and 3.75GB memory. Just be careful with max nodes, unless you are planning increase the load, one node may be enough.
Your own way
- If you don't really need streaming (not likely), you can just create a function that pulls using Synchronous Pull, and add the part that writes to BigQuery. You can schedule according to your needs.
Cloud functions (my recommendation)
- You can create a serverless Event-Driven Cloud Function with a Cloud Pub/Sub trigger. This way, considering your low volume, you can take advantage of the Free Tier and keep the real time processing:
"Cloud Functions provides a perpetual free tier for compute-time resources, which includes an allocation of both GB-seconds and GHz-seconds. In addition to the 2 million invocations, the free tier provides 400,000 GB-seconds, 200,000 GHz-seconds of compute time and 5GB of Internet egress traffic per month."[1]
QUESTION
I need to have a TCP client that listens to messages constantly (and publish pub sub events for each message)
Since there is no Kafka in GCP, I'm trying to do it using my flask service (which runs using AppEngine in GCP).
I'm planning on setting the app.yaml as:
manual_scaling:
instances: 1
But I can't figure out how to trigger code on the flask app startup.
I've tried running code in the main.py and also tried to manipulate the main function in it - but it obviously doesn't work as AppEngine doesn't run it.
Do you have any idea how can I init the listener on the Flask' app startup?
(Or any other offer regarding how should I implement a tcp client that sends pubsub events, or inserting to Big Query?)
ANSWER
Answered 2021-Dec-27 at 16:07I eventually went for implementing a Kafka connector myself and using Kafka.
QUESTION
Trigger a function which updates Cloud Firestore when a student completes assignments or assignments are added for any course.
ProblemThe official docs state that a feed for CourseWorkChangesInfo requires a courseId, and I would like to avoid having a registration and subscription for each course, each running on its own thread.
I have managed to get a registration to one course working:
def registration_body():
return { # An instruction to Classroom to send notifications from the `feed` to the
# provided destination.
"feed": { # Information about a `Feed` with a `feed_type` of `COURSE_WORK_CHANGES`.
"feedType": "COURSE_WORK_CHANGES", # Information about a `Feed` with a `feed_type` of `COURSE_WORK_CHANGES`.
"courseWorkChangesInfo": {
# This field must be specified if `feed_type` is `COURSE_WORK_CHANGES`.
"courseId": "xxxxxxxxxxxx", # The `course_id` of the course to subscribe to work changes for.
},
},
"cloudPubsubTopic": {
"topicName": "projects/xxxxx/topics/gcr-course", # The `name` field of a Cloud Pub/Sub
},
}
def create_registration(service):
"""
Creates a registration to the Google Classroom Service which will listen
for updates on Google Classroom according to the requested body.
Pub Sub will emit a payload to subscribers when a classroom upate occurs.
Args:
service (Google Classroom Service): Google Classroom Service as retrieved
from the Google Aclassroom Service builder
Returns:
Registration: Google Classroom Registration
"""
body = registration_body()
try:
registration = service.registrations().create(body=body).execute()
print(f"Registration to Google CLassroom Created\n{registration}")
return registration
except Exception as e:
print(e)
raise
And am successfully able to subscribe to those updates alongside my FastAPI server:
def init_subscription():
# [INIT PUBSUB SUBSCRIBER AND CALLBACKS]
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
"x-student-portal", "gcr-course-sub"
)
future = subscriber.subscribe(subscription_path, callback)
with subscriber:
try:
future.result()
except TimeoutError:
future.cancel()
future.result()
def callback(message):
print("message recieved")
# do_stuff(message)
print(message)
message.ack()
The registration and subscription initialization:
@app.on_event("startup")
async def startup_event():
global db
global gc_api
global gc_service
global gc_registration
global future
global pub_sub_subscription_thread
db = firestore.FirestoreDatabase()
gc_service = get_service()
gc_api = ClassroomApi(service=gc_service)
gc_registration = publisher_client.create_registration(gc_service)
pub_sub_subscription_thread = multiprocessing.Process(
target=publisher_client.init_subscription
)
pub_sub_subscription_thread.start()
I would very much like to avoid running multiple threads while still being able to subscribe to changes in all my courses.
Any advice would be appreciated.
ANSWER
Answered 2021-Dec-22 at 08:48This is not possible.
You cannot have a single registration to track course work changes for multiple courses, as you can see here:
File a feature request:Types of feeds
The Classroom API currently offers three types of feed:
- Each domain has a roster changes for domain feed, which exposes notifications when students and teachers join and leave courses in that domain.
- Each course has a roster changes for course feed, which exposes notifications when students and teachers join and leave courses in that course.
- Each course has a course work changes for course feed, which exposes notifications when any course work or student submission objects are created or modified in that course.
If you think this feature could be useful, I'd suggest you to file a feature request in Issue Tracker using this template.
Reference:QUESTION
I've a method X that's getting data from the server via pub sub. This method returns a flow. I've another method that subscribes to the flow by method X but only wants to take the first 3 values max from the flow if the data is distinct compared to previous data. I've written the following code
fun subscribeToData() : Flow> {
....
//incoming data
emit(list)
}
fun getUptoFirst3Items() {
subscribeToData()
.take(ITEM_COUNT) // ITEM_COUNT is 3
.distinctUntilChange() //only proceed if the data is different from the previous top 3 items
.mapIndex {
//do transformation
}
.collect { transformedListOf3Elements ->
}
}
Problem:
In collect{} I'm not getting 3 elements but rather I'm getting all the data that's coming in the flow.
I'm not sure what's wrong here? Can someone help me?
ANSWER
Answered 2021-Dec-17 at 19:13You have a Flow> here, which means every element of this flow is itself a list.
The take operator is applied on the flow, so you will take the 3 first lists of the flow. Each individual list is not limited, unless you use take on the list itself.
So the name transformedListOf3Elements is incorrect, because the list is of an unknown number of elements, unless you filter it somehow in the map.
QUESTION
I am working with a Java API from a data vendor providing real time streams. I would like to process this stream using Akka streams.
The Java API has a pub sub design and roughly works like this:
Subscription sub = createSubscription();
sub.addListener(new Listener() {
public void eventsReceived(List events) {
for (Event e : events)
buffer.enqueue(e)
}
});
I have tried to embed the creation of this subscription and accompanying buffer in a custom graph stage without much success. Can anyone guide me on the best way to interface with this API using Akka? Is Akka Streams the best tool here?
ANSWER
Answered 2021-Oct-26 at 13:31To feed a Source, you don't necessarily need to use a custom graph stage. Source.queue will materialize as a buffered queue to which you can add elements which will then propagate through the stream.
There are a couple of tricky things to be aware of. The first is that there's some subtlety around materializing the Source.queue so you can set up the subscription. Something like this:
def bufferSize: Int = ???
Source.fromMaterializer { (mat, att) =>
val (queue, source) = Source.queue[Event](bufferSize).preMaterialize()(mat)
val subscription = createSubscription()
subscription.addListener(
new Listener() {
def eventsReceived(events: java.util.List[Event]): Unit = {
import scala.collection.JavaConverters.iterableAsScalaIterable
import akka.stream.QueueOfferResult._
iterableAsScalaIterable(events).foreach { event =>
queue.offer(event) match {
case Enqueued => () // do nothing
case Dropped => ??? // handle a dropped pubsub element, might well do nothing
case QueueClosed => ??? // presumably cancel the subscription...
}
}
}
}
)
source.withAttributes(att)
}
Source.fromMaterializer is used to get access at each materialization to the materializer (which is what compiles the stream definition into actors). When we materialize, we use the materializer to preMaterialize the queue source so we have access to the queue. Our subscription adds incoming elements to the queue.
The API for this pubsub doesn't seem to support backpressure if the consumer can't keep up. The queue will drop elements it's been handed if the buffer is full: you'll probably want to do nothing in that case, but I've called it out in the match that you should make an explicit decision here.
Dropping the newest element is the synchronous behavior for this queue (there are other queue implementations available, but those will communicate dropping asynchronously which can be really bad for memory consumption in a burst). If you'd prefer something else, it may make sense to have a very small buffer in the queue and attach the "overall" Source (the one returned by Source.fromMaterializer) to a stage which signals perpetual demand. For example, a buffer(downstreamBufferSize, OverflowStrategy.dropHead) will drop the oldest event not yet processed. Alternatively, it may be possible to combine your Events in some meaningful way, in which case a conflate stage will automatically combine incoming Events if the downstream can't process them quickly.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install LogiKM
You can use LogiKM like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the LogiKM component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .
Support
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesExplore Kits - Develop, implement, customize Projects, Custom Functions and Applications with kandi kits
Save this library and start creating your kit
Share this Page
