kinesis-kafka-connector | connector based on Kafka Connect

by awslabs Java Version: Current License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(1)Vulnerabilities Install Support

kandi X-RAY | kinesis-kafka-connector Summary

kinesis-kafka-connector is a Java library typically used in Big Data, Kafka, Spark applications. kinesis-kafka-connector has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. However kinesis-kafka-connector has 1 bugs. You can download it from GitHub.

The Kafka-Kinesis-Connector is a connector to be used with Kafka Connect to publish messages from Kafka to Amazon Kinesis Streams or Amazon Kinesis Firehose. Kafka-Kinesis-Connector for Firehose is used to publish messages from Kafka to one of the following destinations: Amazon S3, Amazon Redshift, or Amazon Elasticsearch Service and in turn enabling near real time analytics with existing business intelligence tools and dashboards. Amazon Kinesis Firehose has ability to transform, batch, archive message onto S3 and retry if destination is unavailable. Kafka-Kinesis-Connector for Kinesis is used to publish messages from Kafka to Amazon Kinesis Streams. Kafka-Kinesis-Connector can be executed on on-premise nodes or EC2 machines. It can be executed in standalone mode as well as distributed mode.

Support

Quality

Security

License

Reuse

Support

kinesis-kafka-connector has a low active ecosystem.

It has 135 star(s) with 83 fork(s). There are 50 watchers for this library.

It had no major release in the last 6 months.

There are 31 open issues and 21 have been closed. On average issues are closed in 54 days. There are 8 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of kinesis-kafka-connector is current.

Quality

kinesis-kafka-connector has 1 bugs (0 blocker, 0 critical, 1 major, 0 minor) and 49 code smells.

Security

kinesis-kafka-connector has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

kinesis-kafka-connector code analysis shows 0 unresolved vulnerabilities.

There are 2 security hotspots that need review.

License

kinesis-kafka-connector is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

kinesis-kafka-connector releases are not available. You will need to build from source code and install.

Build file is available. You can build the component from source.

kinesis-kafka-connector saves you 386 person hours of effort in developing the same functionality from scratch.

It has 951 lines of code, 46 functions and 8 files.

It has high code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed kinesis-kafka-connector and discovered the below as its top functions. This is intended to give you an instant insight into kinesis-kafka-connector implemented functionality, and help decide if they suit your requirements.

Put sink record in batch
Method to perform PutRecord operation with specified record list
Method to put records
Puts records in batch
Put sink records into kinesis
Check for an earlier put exception
Validates that outstanding records in the producer has remaining data
Parse the given value into a byte buffer
Initialization method
Creates an IAM credentials provider
Returns the kinesis producer configuration
Initialize the handler
Validates the delivery stream
Returns the Kafka version
Opens a Kafka producer
Get task configurations
Destroys all Kafka partitions
Stops the kinesis producers
Sends flush to Kafka
Initialize the BO
Returns a list of task configurations
Initialize metrics
Returns the version

Get all kandi verified functions for this library.

kinesis-kafka-connector Key Features

No Key Features are available at this moment for kinesis-kafka-connector.

kinesis-kafka-connector Examples and Code Snippets

No Code Snippets are available at this moment for kinesis-kafka-connector.

Community Discussions

Trending Discussions on kinesis-kafka-connector

How to write a Spark dataframe into Kinesis Stream?

QUESTION

How to write a Spark dataframe into Kinesis Stream?

Asked 2019-Jul-11 at 16:54

I am creating a Dataframe from a kafka topic using spark streaming. I want to write the Dataframe into a Kinesis Producer. I understand that there is no official API for this as of now. But there are multiple APIs available over the internet , but sadly, none of them worked for me. Spark version : 2.2 Scala : 2.11

I tried using https://github.com/awslabs/kinesis-kafka-connector and build the jar. But getting errors due to conflicting package names between this jar and spark API. Please help.

########### Here is the code for others: ...

ANSWER

Answered 2019-Jul-09 at 16:05

Kafka Connect is a service to which you can POST your connector specifications (kinesis in this case), which then takes care of running the connector. It supports quite a few transformations as well while processing the records. Kafka Connect plugins are not intended to be used with Spark applications.

If your use case requires you to do some business logic while processing the records, then you could go with either Spark Streaming or Structured Streaming approach.

If you want to take Spark based approach, below are the 2 options I can think of.

Use Structured Streaming. You could use a Strucuted streaming connector for Kinesis. You can find one here. There may be others too. This is the only stable and open source connector I am aware of. You can find an example for using Kinesis as a sink here.
Use Kinesis Producer Library or aws-java-sdk-kinesis library to publish records from your Spark Streaming application. Using KPL is a preferred approach here. You could do mapPartitions and create a Kinesis client per partition and publish the records using these libraries. There are plenty of examples in AWS docs for these 2 libraries.

Source https://stackoverflow.com/questions/56942802

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install kinesis-kafka-connector

You can download it from GitHub.
You can use kinesis-kafka-connector like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the kinesis-kafka-connector component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: