spark-kafka | Low level integration of Spark and Kafka

by tresata Scala Version: Current License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(10)Vulnerabilities Install Support

kandi X-RAY | spark-kafka Summary

spark-kafka is a Scala library typically used in Big Data, Kafka, Spark applications. spark-kafka has no bugs, it has no vulnerabilities, it has a Permissive License and it has low support. You can download it from GitHub.

Low level integration of Spark and Kafka

Support

Quality

Security

License

Reuse

Support

spark-kafka has a low active ecosystem.

It has 134 star(s) with 39 fork(s). There are 12 watchers for this library.

It had no major release in the last 6 months.

There are 1 open issues and 2 have been closed. On average issues are closed in 366 days. There are 2 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of spark-kafka is current.

Quality

spark-kafka has 0 bugs and 1 code smells.

Security

spark-kafka has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

spark-kafka code analysis shows 0 unresolved vulnerabilities.

There are 0 security hotspots that need review.

License

spark-kafka is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

spark-kafka releases are not available. You will need to build from source code and install.

It has 468 lines of code, 45 functions and 3 files.

It has medium code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi's functional review helps you automatically verify the functionalities of the libraries and avoid rework.
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of spark-kafka

Get all kandi verified functions for this library.

spark-kafka Key Features

No Key Features are available at this moment for spark-kafka.

spark-kafka Examples and Code Snippets

No Code Snippets are available at this moment for spark-kafka.

Community Discussions

Trending Discussions on spark-kafka

Spark structured stream with tumbling window delayed and duplicate data

Spark Structured Streaming Batch Read Checkpointing

How can I set a maximum allowed execution time per task on Spark-YARN?

Infinite loop of Resetting offset and seeking for LATEST offset

spark structured streaming accessing the Kafka with SSL raised error

kafka-consumer-groups command doesnt show LAG and CURRENT-OFFSET for spark structured streaming applications(consumers)

NoClassDefFoundError: org/apache/spark/sql/internal/connector/SimpleTableProvider when running in Dataproc

Reading from the beginning of a Kafka Topic using Structured Streaming when query started

How do I create an MQTT sink for Spark Streaming?

unable to read kafka topic data using spark

QUESTION

Spark structured stream with tumbling window delayed and duplicate data

Asked 2022-Feb-09 at 14:49

I am attempting to read from a kafka topic, aggregate some data over a tumbling window and write that to a sink (I've been trying both with Kafka and console).

The problems I'm seeing are

a long delay between sending data and receiving aggregate records for a window on the sink (minutes after the expected triggers should fire)
duplicate records from previous window aggregations appearing in subsequent windows

Why is the delay so long, and what can I do to reduce it?

Why are duplicates records from previous windows showing up and how can I remove them?

The delays seem to be especially bad as the window gets shorter - it was 3+ minutes when I had the window duration set to 10 seconds, around 2 minutes when the window duration was set to 60 seconds.

With the shortest window times I'm also seeing the records getting "bunched up" so that when records are received by the sink I receive those for several windows at a time.

On the duplicate aggregate records I do have the output mode set to complete but my understanding is records should only be repeated withing the current window assuming the trigger fires multiple times within it, which mine shouldn't be.

I have a processing trigger set up matching the window time and a watermark threshold of 10% (1 or 6 seconds) and I know the stream itself works fine if I remove the tumbling window.

I get why spark might not be able to hit a certain frequency of triggers but I'd think 10 and certainly 60 seconds would be more than enough time to process the very limited amount of data I am testing with.

An example of sending data with a 60 second tumbling window and processing time trigger

send 6 payloads
wait a minute
send 1 payload
wait a while
send 3 payloads

(CreateTime is coming from kafka-console-consumer with --property print.timestamp=true). These arrive a couple of minutes after I would expect the trigger to fire based on the CreateTime timestamp and window.

...

ANSWER

Answered 2022-Feb-09 at 14:49

For long delays, it's likely caused by not enough resources to process messages according to the warning message. You could check spark UI to understand why. It could be data skew between partitions or more memory or cores needed.

For duplicate records, you may want to try update or append mode. Complete mode means the whole Result Table will be outputted to the sink after every trigger. That's the reason why you have deplicates. You can refer https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#output-modes

Source https://stackoverflow.com/questions/71036074

QUESTION

Spark Structured Streaming Batch Read Checkpointing

Asked 2021-Oct-15 at 09:04

I am fairly new to Spark and am still learning. One of the more difficult concepts I have come across is checkpointing and how Spark uses it to recover from failures. I am doing batch reads from Kafka using Structured Streaming and writing them to S3 as Parquet file as:

...

ANSWER

Answered 2021-Oct-15 at 09:04

I am not sure why batch Spark Structured Streaming with Kafka still exists now. If you wish to use it, then you must code your own Offset management. See the guide, but it is badly explained.

I would say Trigger.Once is a better use case for you; Offset management is provided by Spark as it is thus not batch mode.

Source https://stackoverflow.com/questions/69576248

QUESTION

How can I set a maximum allowed execution time per task on Spark-YARN?

Asked 2021-Aug-24 at 08:26

I have a long-running PySpark Structured Streaming job, which reads a Kafka topic, does some processing and writes the result back to another Kafka topic. Our Kafka server runs on another cluster.

It's running fine but every few hours it freezes, even though in the web UI the YARN application still has status "running". After inspecting the logs, it seems due to some transient connectivity problem with the Kafka source. Indeed, all tasks of the problematic micro-batch have completed correctly, except one which shows:

...

ANSWER

Answered 2021-Aug-24 at 08:26

I haven't found a solution to do it with YARN, but a workaround using a monitoring loop in the Pyspark driver. The loop will check status regularly and fail the streaming app if status hasn't been updated for 10 minutes

Source https://stackoverflow.com/questions/68762078

QUESTION

Infinite loop of Resetting offset and seeking for LATEST offset

Asked 2021-Feb-25 at 08:15

I am trying to execute a simple spark structured streaming application which for now does not do much expect for pulling from a local Kafka cluster and writing to local file system. The code looks as follows:

...

ANSWER

Answered 2021-Feb-25 at 08:15

As it turns out, this behaviour of seeking and resetting is perfectly desirable in case that one does not read to the topic from beginning, but from latest offset. The pipeline then only reads new data that gets sent to the Kafka topic whilst it is running and since no new data was sent, the infinite loop of seeking (new data) and resetting (to latest offset).

Bottom line, just read from beginning or send new data and the problem is solved.

Source https://stackoverflow.com/questions/65813055

QUESTION

spark structured streaming accessing the Kafka with SSL raised error

Asked 2021-Feb-06 at 01:14

I plan to extract the data from Kafka(self-signed certificate).

My consumer is the following

...

ANSWER

Answered 2021-Feb-06 at 01:14

I append another option to tell the Kafka broker communication by SSL.

Source https://stackoverflow.com/questions/66043025

QUESTION

kafka-consumer-groups command doesnt show LAG and CURRENT-OFFSET for spark structured streaming applications(consumers)

Asked 2021-Jan-22 at 19:18

I have a spark structured streaming application consuming from kafka, for this application I would like to monitor the consumer lag. I 'm using below command to check consumer lag. However I don't get the CURRENT-OFFSET and hence LAG is blank too. Is this expected ? It works for other python based consumers.

Command

...

ANSWER

Answered 2021-Jan-22 at 19:18

"However I don't get the CURRENT-OFFSET and hence LAG is blank too. Is this expected?"

Yes, this is the expected behavior as Spark Structured Streaming applications are not committing any offsets back to Kafka. Therefore, the current offset and the lag of this consumer group will not be stored in Kafka and you will see exactly the result of the consumer-groups tool what you have shown.

I have written a more comprehensive answer on Consumer Group and how Spark Structured Streaming applications manage Kafka offsets here.

Source https://stackoverflow.com/questions/65847816

QUESTION

NoClassDefFoundError: org/apache/spark/sql/internal/connector/SimpleTableProvider when running in Dataproc

Asked 2020-Jul-18 at 20:32

I am able to run my program in standalone mode. But when I am trying to run in Dataproc in cluster mode, getting following error. PLs help. My build.sbt

...

ANSWER

Answered 2020-Jul-17 at 20:21

Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.internal.connector.SimpleTableProvider

org.apache.spark.sql.internal.connector.SimpleTableProvider was added in v3.0.0-rc1 so you're using spark-submit from Spark 3.0.0 (I guess).

I only now noticed that you use --master yarn and the exception is thrown at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686).

I know nothing about Dataproc, but you should review the configuration of YARN / Dataproc and make sure they don't use Spark 3 perhaps.

Source https://stackoverflow.com/questions/62921366

QUESTION

Reading from the beginning of a Kafka Topic using Structured Streaming when query started

Asked 2020-Jul-11 at 21:34

im using structured streaming to read from the kafka topic, using spark 2.4 and scala 2.12

im using a checkpoint to make my query fault-tolerant.

however everytime i start the query it jumps to the current offset without reading the exisitng data before it connected onto the topic.

is there a config for the kafka stream im missing?

READ:

...

ANSWER

Answered 2020-Jul-11 at 15:22

So annonying... i mis-spelled the option startingOffset

the correct way to spell it is:

Source https://stackoverflow.com/questions/62850747

QUESTION

How do I create an MQTT sink for Spark Streaming?

Asked 2020-Jun-24 at 06:14

There are some examples of how to create MQTT sources [1] [2] for Spark Streaming. However, I want to create an MQTT sink where I can publish the results instead of using the print() method. I tried to create one MqttSink but I am getting object not serializable error. Then I am basing the code on this blog but I cannot find the method send that I created on the MqttSink object.

...

ANSWER

Answered 2020-Jun-18 at 16:39

this is a working example based on the blog entry Spark and Kafka integration patterns.

Source https://stackoverflow.com/questions/62429316

QUESTION

unable to read kafka topic data using spark

Asked 2020-Jun-01 at 15:15

I have data like below in one of the topics which I created named "sampleTopic"

...

ANSWER

Answered 2020-May-30 at 17:03

spark-sql-kafka jar is missing, which is having the implementation of 'kafka' datasource.

you can add the jar using config option or build fat jar which includes spark-sql-kafka jar. Please use relevant version of jar

Source https://stackoverflow.com/questions/62105605

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install spark-kafka

You can download it from GitHub.

Support

For any new features, suggestions and bugs create an issue on GitHub. If you have any questions check and ask questions on community page Stack Overflow .

Find more information at: