spark-yarn | Launch Spark clusters on YARN

As per the spark documentation

Spark Driver :

The Driver(aka driver program) is responsible for converting a user application to smaller execution units called tasks and then schedules them to run with a cluster manager on executors. The driver is also responsible for executing the Spark application and returning the status/results to the user.

Spark Driver contains various components – DAGScheduler, TaskScheduler, BackendScheduler and BlockManager. They are responsible for the translation of user code into actual Spark jobs executed on the cluster.

Where in Application Master is

The Application Master is responsible for the execution of a single application. It asks for containers from the Resource Scheduler (Resource Manager) and executes specific programs on the obtained containers. Application Master is just a broker that negotiates resources with the Resource Manager and then after getting some container it make sure to launch tasks(which are picked from scheduler queue) on containers.

In a nutshell Driver program will translate your custom logic into stages, job and task.. and your application master will make sure to get enough resources from RM And also make sure to check the status of your tasks running in a container.

as it is already said in your provided references the only different between client and cluster mode is

In client, mode driver will run on the machine where we have executed/run spark application/job and AM runs in one of the cluster nodes.

(AND)

In cluster mode driver run inside application master, it means the application has much more responsibility.

References :

https://luminousmen.com/post/spark-anatomy-of-spark-application#:~:text=The%20Driver(aka%20driver%20program,status%2Fresults%20to%20the%20user.

https://www.edureka.co/community/1043/difference-between-application-master-application-manager#:~:text=The%20Application%20Master%20is%20responsible,class)%20on%20the%20obtained%20containers.

Two of these things ended up solving this issue:

First, I added the following lines to all nodes in the yarn-site.xml file:

The default --deploy-mode is client. So both the below spark-submit will run in client mode.

I finally resolved the issue. I commented out all the dependencies and uncommented them one at a time. First I uncommented spark_core dependency and the issue got resolved. I uncommented another dependency in my project which again brought back the issue. Then on investigation I found that the second dependency was in turn having dependency of a different version(2.10) of spark_core which was causing the issue. I added exclusion to the dependency as below:

Can I get more details about how you are running the spark job? If you are using custom distribution such as Cloudera or Horton works, you may have to use their libraries to compile and spark-submit will use the distribution installed classpath to submit the job to cluster.

To get started, please add % provided to the library in sbt file so that it will use the particular library from the classpath of spark installation.

Scala classes are packed to spark application distributive which causes wrong version of the class in the classpath.

The fix would be to change the scope of all conflicting dependencies to provided.

I think I had two main issues.

1. Spark is not compatible with scala 2.12 yet. So moving to 2.11.12 solved one issue
2. The second issue is that for intelliJ SBT console to reload the build.sbt you either need to kill and restart the console or use the reload command which I didnt know so I was not actually using the latest build.sbt file.

The way to get rid of the problem is to provide "path" option prior to "save" operation as shown below: