spark2-submit

The options that the SparkSubmitOperator in Airflow requires can be sent in a the dictionary. Keep in mind that the keys in the dictionary should be the same as teh parameter names to the function.

Create the following two dictionaries:

On the spark docs (https://spark.apache.org/docs/latest/submitting-applications.html), it has links for sbt and maven plugins on how to package a compatible jar for submitting on spark (however there is none for gradle).

The jar structure output of the spring-boot-gradle-plugin is as follows:

The problem turned out to be that python was running from the wrong place. I had to submit from correct place this way -

PYTHONPATH=./${virtualenv}/venv/lib/python3.6/site-packages/ spark2-submit

Problem is resolved by changing the deploy-mode from client to cluster. I am triggering the spark job from oozie application. So in client mode, driver will start on oozie JVM. To avoid this, I have set the mode to cluster.

I don't know/think if this can be at configuration level (e.g lo4j.properties, etc), but there are ways we can achieve this. Here is one approach:

You will need to have a logger class/trait where you deal with all you logger management, something like :

It seems you have both Spark 2.x and 3.x jars in classpath. According to the sbt file, Spark 2.x should be used, however, JsonFileFormat was added in Spark 3.x with this issue

String "\u0001" is a unicode character, but what is passed to spark from the command line is a literal string "\\u0001". You need to explicitly unescape Unicode:

So the cause of issue was quite trivial, it is spark local binaries vs remote spark driver version mismatch. Locally I had installed spark 2.4.5 and on Cloudera it was 2.3.0, after aligning the versions to 2.3.0, the issue resolved and the spark job completed successfully.

I suspect the values for SSL is not getting picked up. As you can notice in your log the values are shown as null.