findspark | PySpark isn't on sys

You need to create an RDD of type RDD[Tuple[str]] but in your code, the line:

You can install Pyspark using PyPI as an alternative:

For Python users, PySpark also provides pip installation from PyPI. This is usually for local usage or as a client to connect to a cluster instead of setting up a cluster itself.

Spark requires Java even if you're running pyspark, so you need to install java in your image. In addition, if you're still using findspark you can specify the SPARK_HOME directory as well:

Modify your lines as per below and reorder the a,b and b,a always as a,b or vice-versa:

1. You should use spark-sql-kafka-0-10

2. You need to move findspark.init() after os.environ line. Also, you don't actually need this line, as you can provide the packages via findspark.

You are receiving the error

TypeError: unsupported operand type(s) for +: 'int' and 'str'

because your tuple values are string i.e. ("1,0") instead of (1,0), python currently will not apply this operator + or add the int and str(string) data types.

Moreover, there seems to be a logic error in your comparison in your map function where you have "word1" and "word2" in x as this will only check if "word2" is in x. I would recommend the following rewrite:

Change url to jdbc:postgresql://postgres-dest:5432/destdb.

And make sure that PostgreSQL driver jar is present in classpath. You can download the jar from here.

You can't load csv directly into pyspark from url. Try this:

Issue got fixed once I converted the .p12 file to .jks file using keytool

cmd to convert the .p12 file to .jks file =>

keytool -importkeystore -srckeystore /my-storage/ssl_certificates/elastic-certificates.p12 -destkeystore /my-storage/ssl_certificates/elastic-certificates.jks -srcstoretype PKCS12 -deststoretype JKS -deststorepass

You may get the below error if you try to execute the above command on a computer with openjdk-1.8.0. To avoid the error execute the keytool command from a computer with openjdk version "16"