learn_new | 每天掌握一个新知识点
kandi X-RAY | learn_new Summary
kandi X-RAY | learn_new Summary
每天掌握一个新知识点
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
Currently covering the most popular Java, JavaScript and Python libraries. See a Sample of learn_new
learn_new Key Features
learn_new Examples and Code Snippets
Community Discussions
Trending Discussions on learn_new
QUESTION
I am learning how to integrate spark
with kafka
. Currently i created a virtualenv
and installed pyspark
, py4j
packages.
I also configured these environment:
- PYSPARK_PYTHON :
C:\learn_new\learn_utils\venv\Scripts\python.exe
- SPARK_HOME :
C:\spark-2.4.3-bin-hadoop2.7
Then i want to run the example python source code under C:\spark-2.4.3-bin-hadoop2.7\examples\src\main\python\streaming\direct_kafka_wordcount.py
The script code is this:
...ANSWER
Answered 2019-Jul-24 at 02:33I mainly want to debug code locally, so i do not want to use spark-submit and add --jars or --packages parameters to run the code.
But it really need the spark-streaming-kafka-0-8-assembly_2.11-2.4.3.jar package.(here change the package version according to your spark version)
So i tried to download the package and save it to the C:\spark-2.4.3-bin-hadoop2.7\jars(change it to your spark installation path, and find the jars folder).
Then the issue is solved. Hope it will help other people.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install learn_new
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page