spark-project | A solution about web , orm and authorization based on spark

I am using the requests-scala library to make an HTTP call out to an external API. My spark program workflow is like this:

(JSON_FILE:INPUT) --> (SPARK) --> (HTTP-API) --> (KAFKA:OUTPUT)

When I run it, I get the following error:

I have gone through the following questions and pages seeking an answer for my problem, but they did not solve my problem:

log from spark udf to driver

Logger is not working inside spark UDF on cluster

https://www.javacodegeeks.com/2016/03/log-apache-spark.html

We are using Spark in standalone mode, not on Yarn. I have configured the log4j.properties file in both the driver and executors to define a custom logger "myLogger". The log4j.properties file, which I have replicated in both the driver and the executors, is as follows:

I'm working on a rather big project. I need to use azure-security-keyvault-secrets, so I added following to my pom.xml file:

I have a Spring web application(built in maven) with which I connect to my spark cluster(4 workers and 1 master) and to my cassandra cluster(4 nodes). The application starts, the workers communicate with the master and the cassandra cluster is also running. However when I do a PCA(spark mllib) or any other calculation(clustering, pearson, spearman) through the interface of my web-app I get the following error:

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

which appears on this command:

I'm using Kafka docker version which is working fine (I guess ?) from that repo https://github.com/wurstmeister/kafka-docker

The docker-compose.yml file looks like that :

I have a project where I am using spark with scala. The Code does not give the compilation issue but when I run the code I get the below exception:-

I have set up Kafka and Spark on Ubuntu. I am trying to read kafka topics through Spark Streaming using pyspark(Jupyter notebook). Spark is neither reading the data nor throwing any error.

Kafka producer and consumer are communicating with each other on terminal. Kafka is configured with 3 partitions on port 9092,9093,9094. Messages are getting stored in kafka topics. Now, I want to read it through Spark Streaming. I am not sure what I am missing. Even I have explored it on internet, but couldnt find any working solution. Please help me to understand the missing part.

• Topic Name: new_topic
• Spark - 2.3.2
• Kafka - 2.11-2.1.0
• Python 3
• Java- 1.8.0_201
• Zookeeper port : 2181

Kafka Producer : bin/kafka-console-producer.sh --broker-list localhost:9092 --topic new_topic

Kafka Consumer: bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic new_topic --from-beginning

Pyspark Code (Jupyter Notebook):

I am having difficulties creating a basic spark streaming application.

Right now, am trying it on my local machine.

I have done following setup.

-Setup Zookeeper

-Setup Kafka ( Version : kafka_2.10-0.9.0.1)

-Created a topic using below command

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

-Started producer and consumer on two different cmd terminals using below commands

Producer :

kafka-console-producer.bat --broker-list localhost:9092 --topic test

Consumer :

kafka-console-consumer.bat --zookeeper localhost:2181 --topic test

Now I can receive the data which I enter in the producer terminal in consumer console.

Now am trying to integrate Kafka into Apache Spark streaming.

Below is a sample code which I referenced from official documents. Kafka & Spark Setup and Kafka & Spark Integration

I am new in spark framework. I have tried to create a sample application using spark and java. I have the following code

Pom.xml