spark-notebook | Interactive and Reactive Data Science using Scala and Spark

I think encoding the file from here should solve the problem. So you add encoding="utf8" to your tuple of the variable listings_df.

Like shown below;

listings_df = spark.read.csv("listings.csv", encoding="utf8", header=True, mode='DROPMALFORMED')

Change url to jdbc:postgresql://postgres-dest:5432/destdb.

And make sure that PostgreSQL driver jar is present in classpath. You can download the jar from here.

Don't try to install everything in a single environment. Create a Python environment and create an R environment, and make sure they each have their respective kernel packages (ipykernel, r-irkernel) so that they can be used in Jupyter.

Seems you confused the source dir in container and target dir in local machine, try docker cp 44758917bf50:/home/jovyan/pysparkex.ipynb ./pyex.ipypnb

The exit code 8 is likely from wget meaning an error response from the server. As an example, this path that the Dockerfile tries to wget from isn't valid anymore: https://www.apache.org/dyn/closer.lua/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz

From the issues on the repo, it appears that Apache version 3.0.1 is no longer valid so you should override the APACHE_SPARK version to 3.0.2 with a --build-arg:

When you created a network you need to connect both containers

There are several problems in your setup:

• You don't add the package for Kafka support as described in docs. It's either needs to be added when starting pyspark, or when initializing session, something like this (change 3.0.1 to version that is used in your jupyter container):

I don't have much experience with PySpark but I once setup Java Spark to run on a Kubernetes cluster in client mode, like you are trying now.. and I believe the configuration should mostly be the same.

First of all, you should check if the headless service is working as expected or not. First with a:

You can run it with: