flume-ng-rabbitmq | Flume plugin for RabbitMQ | Pub Sub library

by jcustenborder Java Version: Current License: Apache-2.0

Download this library from

kandi X-RAY | flume-ng-rabbitmq Summary

flume-ng-rabbitmq is a Java library typically used in Messaging, Pub Sub, Docker, Kafka, RabbitMQ applications. flume-ng-rabbitmq has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can download it from GitHub.

Flume plugin for RabbitMQ

Support

Quality

Security

License

Reuse

kandi-support Support

flume-ng-rabbitmq has a low active ecosystem.
It has 59 star(s) with 47 fork(s). There are 10 watchers for this library.
It had no major release in the last 12 months.
There are 3 open issues and 3 have been closed. On average issues are closed in 1047 days. There are 3 open pull requests and 0 closed requests.
It has a neutral sentiment in the developer community.
The latest version of flume-ng-rabbitmq is current.

flume-ng-rabbitmq Support

Best in #Pub Sub

Average in #Pub Sub

flume-ng-rabbitmq Support

Best in #Pub Sub

Average in #Pub Sub

quality kandi Quality

flume-ng-rabbitmq has 0 bugs and 0 code smells.

flume-ng-rabbitmq Quality

Best in #Pub Sub

Average in #Pub Sub

flume-ng-rabbitmq Quality

Best in #Pub Sub

Average in #Pub Sub

security Security

flume-ng-rabbitmq has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
flume-ng-rabbitmq code analysis shows 0 unresolved vulnerabilities.
There are 0 security hotspots that need review.

flume-ng-rabbitmq Security

Best in #Pub Sub

Average in #Pub Sub

flume-ng-rabbitmq Security

Best in #Pub Sub

Average in #Pub Sub

license License

flume-ng-rabbitmq is licensed under the Apache-2.0 License. This license is Permissive.
Permissive licenses have the least restrictions, and you can use them in most projects.

flume-ng-rabbitmq License

Best in #Pub Sub

Average in #Pub Sub

flume-ng-rabbitmq License

Best in #Pub Sub

Average in #Pub Sub

build Reuse

flume-ng-rabbitmq releases are not available. You will need to build from source code and install.
Build file is available. You can build the component from source.
Installation instructions are not available. Examples and code snippets are available.
flume-ng-rabbitmq saves you 207 person hours of effort in developing the same functionality from scratch.
It has 508 lines of code, 25 functions and 7 files.
It has medium code complexity. Code complexity directly impacts maintainability of the code.

flume-ng-rabbitmq Reuse

Best in #Pub Sub

Average in #Pub Sub

flume-ng-rabbitmq Reuse

Best in #Pub Sub

Average in #Pub Sub

Top functions reviewed by kandi - BETA

kandi has reviewed flume-ng-rabbitmq and discovered the below as its top functions. This is intended to give you an instant insight into flume-ng-rabbitmq implemented functionality, and help decide if they suit your requirements.

Creates a new channel .
Create a new connection .
Creates a connection factory from a context .
Extract the HTTP headers from the given properties .
close channel
This method initialises the queue .
Retrieves the set of topics
Ensure that the config or queue name or exchange name is valid
Resets the connection and channel .
Override this method to configure the connection factory .

flume-ng-rabbitmq Key Features

Flume plugin for RabbitMQ

flume-ng-rabbitmq Examples and Code Snippets

See all related Code Snippets

default

Copy

Download

The configuration of RabbitMQ sources requires that you either declare an exchange name or a queue name.

The exchange name option is helpful if you have declared an exchange in RabbitMQ, but want to use a
default named queue.  If you have a predeclared queue that you want to receive events from, then you can simply declare
the queue name and leave the exchange name out.  Another optional configuration option is the declaration of
topic routing keys that you want to listen to.  This is a comma-delimited list.

**Minimal Config Example**

	agent1.sources.rabbitmq-source1.channels = ch1
	agent1.sources.rabbitmq-source1.type = org.apache.flume.source.rabbitmq.RabbitMQSource
	agent1.sources.rabbitmq-source1.hostname = 10.10.10.173

	agent1.sources.rabbitmq-source1.queuename = log_jammin
	OR
	agent1.sources.rabbitmq-source1.exchangename = log_jammin_exchange

**Full Config Example**

	agent1.sources.rabbitmq-source1.channels = ch1
	agent1.sources.rabbitmq-source1.type = org.apache.flume.source.rabbitmq.RabbitMQSource
	agent1.sources.rabbitmq-source1.hostname = 10.10.10.173

	agent1.sources.rabbitmq-source1.queuename = log_jammin
	OR
	agent1.sources.rabbitmq-source1.exchangename = log_jammin_exchange

	agent1.sources.rabbitmq-source1.topics = topic1,topic2
	agent1.sources.rabbitmq-source1.username = rabbitmquser
	agent1.sources.rabbitmq-source1.password = p@$$w0rd!
	agent1.sources.rabbitmq-source1.port = 12345
	agent1.sources.rabbitmq-source1.virtualhost = virtualhost1

RabbitMQ Sink

See all related Code Snippets

Community Discussions

Trending Discussions on Pub Sub

Build JSON content in R according Google Cloud Pub Sub message format
BigQuery Table a Pub Sub Topic not working in Apache Beam Python SDK? Static source to Streaming Sink
Pub Sub Lite topics with Peak Capacity Throughput option
How do I add permissions to a NATS User to allow the User to query & create Jestream keyvalue stores?
MSK vs SQS + SNS
Dataflow resource usage
Run code on Python Flask AppEngine startup in GCP
Is there a way to listen for updates on multiple Google Classroom Courses using Pub Sub?
Flow.take(ITEM_COUNT) returning all the elements rather then specified amount of elements
Wrapping Pub-Sub Java API in Akka Streams Custom Graph Stage

Trending Discussions on Pub Sub

QUESTION

Build JSON content in R according Google Cloud Pub Sub message format

Asked 2022-Apr-16 at 09:59

In R, I want to build json content according this Google Cloud Pub Sub message format: https://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage

It have to respect :

{
  "data": string,
  "attributes": {
    string: string,
    ...
  },
  "messageId": string,
  "publishTime": string,
  "orderingKey": string
}

The message built will be readed from this Python code:

def pubsub_read(data, context):
    '''This function is executed from a Cloud Pub/Sub'''
    message = base64.b64decode(data['data']).decode('utf-8')
    file_name = data['attributes']['file_name']

This following R code builds a R dataframe and converts it to json content:

library(jsonlite)
data="Hello World!"
df <- data.frame(data)
attributes <- data.frame(file_name=c('gfs_data_temp_FULL.csv'))
df$attributes <- attributes

msg <- df %>%
    toJSON(auto_unbox = TRUE, dataframe = 'columns', pretty = T) %>%
    # Pub/Sub expects a base64 encoded string
    googlePubsubR::msg_encode() %>%
    googlePubsubR::PubsubMessage()

It seems good but when I visualise it with a json editor :

indexes are added.

Additionally there is the message content:

I dont'sure it respects Google Cloud Pub Sub message format...

ANSWER

Answered 2022-Apr-16 at 09:59

Not sure why, but replacing the dataframe by a list seems to work:

library(jsonlite)

df = list(data = "Hello World")
attributes <- list(file_name=c('toto.csv'))
df$attributes <- attributes

df %>%
  toJSON(auto_unbox = TRUE, simplifyVector=TRUE, dataframe = 'columns', pretty = T)

Output:

{
  "data": "Hello World",
  "attributes": {
    "file_name": "toto.csv"
  }
}

Source https://stackoverflow.com/questions/71892778

QUESTION

BigQuery Table a Pub Sub Topic not working in Apache Beam Python SDK? Static source to Streaming Sink

Asked 2022-Apr-02 at 19:41

My basic requirement was to create a pipeline to read from BigQuery Table and then convert it into JSON and pass it onto a PubSub topic.

At first I read from Big Query and tried to write it into Pub Sub Topic but got an exception error saying "Pub Sub" is not supported for batch pipelines. So I tried some workarounds and

I was able to work around this in python by

Reading from BigQuery-> ConvertTo JSON string-> Save as text file in cloud storage (Beam pipeline)

p = beam.Pipeline(options=options)

json_string_output =   (
                          p
                          | 'Read from BQ' >> beam.io.ReadFromBigQuery(
                                query='SELECT * FROM '\
                                 '`project.dataset.table_name`',
                                 use_standard_sql=True)
                          | 'convert to json' >> beam.Map(lambda record: json.dumps(record))
                          | 'Write results' >> beam.io.WriteToText(outputs_prefix)
                      )

p.run()

And then from there run a normal python script to read it line from file and pass it onto PubSub Topic

 # create publisher
    publisher = pubsub_v1.PublisherClient()

    with open(input_file, 'rb') as ifp:
        header = ifp.readline()  
        # loop over each record
        for line in ifp:
            event_data = line   # entire line of input file is the message
            print('Publishing {0} to {1}'.format(event_data, pubsub_topic))
            publisher.publish(pubsub_topic, event_data)

Python working codes repo

I'm not able to find a way to integrate both scripts within a single ApacheBeam Pipeline.

ANSWER

Answered 2021-Oct-14 at 20:27

Because your pipeline does not have any unbounded PCollections, it will be automatically run in batch mode. You can force a pipeline to run in streaming mode with the --streaming command line flag.

Source https://stackoverflow.com/questions/69549649

QUESTION

Pub Sub Lite topics with Peak Capacity Throughput option

Asked 2022-Feb-20 at 21:46

We are using Pub Sub lite instances along with reservations, we want to deploy it via Terraform, on UI while creating a Pub Sub Lite we get an option to specify Peak Publish Throughput (MiB/s) and Peak Subscribe Throughput (MiB/s) which is not available in the resource "google_pubsub_lite_topic" as per this doc https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/pubsub_lite_topic.

resource "google_pubsub_lite_reservation" "pubsub_lite_reservation" {
  name = var.lite_reservation_name
  project = var.project
  region  = var.region
  throughput_capacity = var.throughput_capacity
}

resource "google_pubsub_lite_topic" "pubsub_lite_topic" {
  name    = var.topic_name
  project = var.project
  region  = var.region
  zone    = var.zone
  partition_config {
    count = var.partitions_count
    capacity {
      publish_mib_per_sec   = var.publish_mib_per_sec
      subscribe_mib_per_sec = var.subscribe_mib_per_sec
    }
  }

  retention_config {
    per_partition_bytes = var.per_partition_bytes
    period              = var.period
  }

  reservation_config {
    throughput_reservation = google_pubsub_lite_reservation.pubsub_lite_reservation.name
  }

}

Currently use the above TF script to create pub sub lite instance, the problem here is we are mentioning the throughput capacity instead of setting the peak throughput capacity, and capacity block is a required field. Please help if there is any workaround to it ? we want topic to set throughput dynamically but with peak limit to the throughput, as we are setting a fix value to the lite reservation.

ANSWER

Answered 2022-Feb-20 at 21:46

If you check the bottom of your Google Cloud console screenshot, you can see it suggests to have 4 partitions with 4MiB/s publish and subscribe throughput.

Therefore your Terraform partition_config should match this. Count should be 4 for the 4 partitions, with capacity of 4MiB/s publish and 4MiB/s subscribe for each partition.

The "peak throughput" in web UI is just for convenience to help you choose some numbers here. The actual underlying PubSub Lite API doesn't actually have this field, which is why there is no Terraform setting either. You will notice the sample docs require a per-partiton setting just like Terraform.

eg. https://cloud.google.com/pubsub/lite/docs/samples/pubsublite-create-topic

I think the only other alternative would be to create a reservation attached to your topic with enough throughput units for desired capacity. And then completely omit capacity block in Terraform and let the reservation decide.

Source https://stackoverflow.com/questions/70210745

QUESTION

Is there a way to listen for updates on multiple Google Classroom Courses using Pub Sub?

Asked 2021-Dec-22 at 08:48

Goal

Trigger a function which updates Cloud Firestore when a student completes assignments or assignments are added for any course.

Problem

The official docs state that a feed for CourseWorkChangesInfo requires a courseId, and I would like to avoid having a registration and subscription for each course, each running on its own thread.

What I Have

I have managed to get a registration to one course working:

def registration_body():
    return {  # An instruction to Classroom to send notifications from the `feed` to the
        # provided destination.
        "feed": {  # Information about a `Feed` with a `feed_type` of `COURSE_WORK_CHANGES`.
            "feedType": "COURSE_WORK_CHANGES",  # Information about a `Feed` with a `feed_type` of `COURSE_WORK_CHANGES`.
            "courseWorkChangesInfo": { 
                # This field must be specified if `feed_type` is `COURSE_WORK_CHANGES`.
                "courseId": "xxxxxxxxxxxx",  # The `course_id` of the course to subscribe to work changes for.
            },
        },
        "cloudPubsubTopic": {
            "topicName": "projects/xxxxx/topics/gcr-course",  # The `name` field of a Cloud Pub/Sub
        },
    }


def create_registration(service):
    """
    Creates a registration to the Google Classroom Service which will listen
    for updates on Google Classroom according to the requested body.
    Pub Sub will emit a payload to subscribers when a classroom upate occurs.

    Args:
        service (Google Classroom Service): Google Classroom Service as retrieved
        from the Google Aclassroom Service builder

    Returns:
        Registration: Google Classroom Registration
    """

    body = registration_body()
    try:
        registration = service.registrations().create(body=body).execute()
        print(f"Registration to Google CLassroom Created\n{registration}")
        return registration
    except Exception as e:
        print(e)
        raise

And am successfully able to subscribe to those updates alongside my FastAPI server:

def init_subscription():
    # [INIT PUBSUB SUBSCRIBER AND CALLBACKS]
    subscriber = pubsub_v1.SubscriberClient()
    subscription_path = subscriber.subscription_path(
        "x-student-portal", "gcr-course-sub"
    )
    future = subscriber.subscribe(subscription_path, callback)
    with subscriber:
        try:
            future.result()
        except TimeoutError:
            future.cancel()
            future.result()


def callback(message):
    print("message recieved")
    # do_stuff(message)
    print(message)
    message.ack()

The registration and subscription initialization:

@app.on_event("startup")
async def startup_event():
    global db
    global gc_api
    global gc_service
    global gc_registration
    global future
    global pub_sub_subscription_thread

    db = firestore.FirestoreDatabase()
    gc_service = get_service()
    gc_api = ClassroomApi(service=gc_service)
    gc_registration = publisher_client.create_registration(gc_service)

    pub_sub_subscription_thread = multiprocessing.Process(
        target=publisher_client.init_subscription
    )
    pub_sub_subscription_thread.start()

Conclusion

I would very much like to avoid running multiple threads while still being able to subscribe to changes in all my courses.

Any advice would be appreciated.

ANSWER

Answered 2021-Dec-22 at 08:48

Answer:

This is not possible.

You cannot have a single registration to track course work changes for multiple courses, as you can see here:

Types of feeds

The Classroom API currently offers three types of feed:

Each domain has a roster changes for domain feed, which exposes notifications when students and teachers join and leave courses in that domain.

Each course has a roster changes for course feed, which exposes notifications when students and teachers join and leave courses in that course.

Each course has a course work changes for course feed, which exposes notifications when any course work or student submission objects are created or modified in that course.

File a feature request:

If you think this feature could be useful, I'd suggest you to file a feature request in Issue Tracker using this template.

Reference:

Source https://stackoverflow.com/questions/70428658

QUESTION

Wrapping Pub-Sub Java API in Akka Streams Custom Graph Stage

Asked 2021-Oct-26 at 13:31

I am working with a Java API from a data vendor providing real time streams. I would like to process this stream using Akka streams.

The Java API has a pub sub design and roughly works like this:

Subscription sub = createSubscription();
sub.addListener(new Listener() {
        public void eventsReceived(List events) {
                for (Event e : events)
                        buffer.enqueue(e)
        }
});

I have tried to embed the creation of this subscription and accompanying buffer in a custom graph stage without much success. Can anyone guide me on the best way to interface with this API using Akka? Is Akka Streams the best tool here?

ANSWER

Answered 2021-Oct-26 at 13:31

To feed a Source, you don't necessarily need to use a custom graph stage. Source.queue will materialize as a buffered queue to which you can add elements which will then propagate through the stream.

There are a couple of tricky things to be aware of. The first is that there's some subtlety around materializing the Source.queue so you can set up the subscription. Something like this:

def bufferSize: Int = ???

Source.fromMaterializer { (mat, att) =>
  val (queue, source) = Source.queue[Event](bufferSize).preMaterialize()(mat)
  val subscription = createSubscription()
  subscription.addListener(
    new Listener() {
      def eventsReceived(events: java.util.List[Event]): Unit = {
        import scala.collection.JavaConverters.iterableAsScalaIterable
        import akka.stream.QueueOfferResult._

        iterableAsScalaIterable(events).foreach { event =>
          queue.offer(event) match {
            case Enqueued => ()  // do nothing
            case Dropped => ??? // handle a dropped pubsub element, might well do nothing
            case QueueClosed => ??? // presumably cancel the subscription...
          }
        }
      }
    }
  )

  source.withAttributes(att)
}

Source.fromMaterializer is used to get access at each materialization to the materializer (which is what compiles the stream definition into actors). When we materialize, we use the materializer to preMaterialize the queue source so we have access to the queue. Our subscription adds incoming elements to the queue.

The API for this pubsub doesn't seem to support backpressure if the consumer can't keep up. The queue will drop elements it's been handed if the buffer is full: you'll probably want to do nothing in that case, but I've called it out in the match that you should make an explicit decision here.

Dropping the newest element is the synchronous behavior for this queue (there are other queue implementations available, but those will communicate dropping asynchronously which can be really bad for memory consumption in a burst). If you'd prefer something else, it may make sense to have a very small buffer in the queue and attach the "overall" Source (the one returned by Source.fromMaterializer) to a stage which signals perpetual demand. For example, a buffer(downstreamBufferSize, OverflowStrategy.dropHead) will drop the oldest event not yet processed. Alternatively, it may be possible to combine your Events in some meaningful way, in which case a conflate stage will automatically combine incoming Events if the downstream can't process them quickly.

Source https://stackoverflow.com/questions/69368519

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install flume-ng-rabbitmq

You can download it from GitHub.
You can use flume-ng-rabbitmq like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the flume-ng-rabbitmq component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .