kafka-client | Go client library for Apache Kafka | Pub Sub library

In my application config i have defined the following properties:

It's my first Kafka program.

From a kafka_2.13-3.1.0 instance, I created a Kafka topic poids_garmin_brut and filled it with this csv:

I am trying to deploy a docker container with Kafka and Spark and would like to read to Kafka Topic from a pyspark application. Kafka is working and I can write to a topic and also spark is working. But when I try to read the Kafka stream I get the error message:

I'm having problem understanding how to connect Kafka and PySpark.

I have kafka installation on Windows 10 with topic nicely streaming data. I've installed pyspark which runs properly-I'm able to create test DataFrame without problem.

But when I try to connect to Kafka stream it gives me error:

AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of "Structured Streaming- Kafka Integration Guide".

Spark documentation is not really helpful - it says: ... groupId = org.apache.spark artifactId = spark-sql-kafka-0-10_2.12 version = 3.2.0 ...

For Python applications, you need to add this above library and its dependencies when deploying your application. See the Deploying subsection below.

And then when you go to Deploying section it says:

As with any Spark applications, spark-submit is used to launch your application. spark-sql-kafka-0-10_2.12 and its dependencies can be directly added to spark-submit using --packages, such as, ./bin/spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.0 ...

I'm developing app, I don't want to deploy it. Where and how to add these dependencies if I'm developing pyspark app?

Tried several tutorials ended up being more confused.

Saw answer saying that

"You need to add kafka-clients JAR to your --packages".so-answer

Few more steps could be useful because for someone who is new this is unclear.

versions:

• kafka 2.13-2.8.1
• spark 3.1.2
• java 11.0.12

All environmental variables and paths are correctly set.

EDIT

I've load :

I have used this document for creating kafka https://kow3ns.github.io/kubernetes-kafka/manifests/

able to create zookeeper, facing issue with the creation of kafka.getting error to connect with the zookeeper.

this is the manifest i have used for creating for kafka:

https://kow3ns.github.io/kubernetes-kafka/manifests/kafka.yaml for Zookeeper

https://github.com/kow3ns/kubernetes-zookeeper/blob/master/manifests/zookeeper.yaml

The logs of the kafka

I have a Reactor Kafka application that consumes messages from a topic indefinitely. I need to expose a health check REST endpoint that can indicate the health of this process - Essentially interested in knowing if the Kafka receiver flux sequence has terminated so that some action can be taken to start it. Is there a way to know the current status of a flux (completed/terminated etc)? The application is Spring Webflux + Reactor Kafka.

Edit 1 - doOnTerminate/doFinally do not execute

I have a test case with following configured kafka properties:

EDIT: OBE - figured it out. Provided in answer for anyone else who has this issue.

I am working in an offline environment and am unable to connect to a kafka broker, on machine 1, from a separate machine, machine 2, on a LAN connection through a single switch.

Machine 1 (where Kafka and ZK are running):

I'm trying to start a Snowflake Connector instance for Kafka, following this tutorial: https://docs.snowflake.com/en/user-guide/kafka-connector-install.html