spark-examples | Apache Spark jobs such as Principal Coordinate Analysis

According to dataproc batches docs, the subnetwork URI needs to be specified using argument --subnet.

Try:

Since you've checked resources and You have enough. It might be network issue. executor maybe cannot connect back to driver program. Allow traffic on both master and workers.

Submitting via EMR Step gives some additional monitoring and tooling on the AWS platform.

• EMR has CloudWatch metrics for running/completed/failed steps.
• EMR steps dispatch Eventbridge events on complete/failure which can be used as triggers.
• If your EMR cluster is running on a private subnet, you'll have to tunnel to the subnet to monitor your jobs, EMR step status does not have this limitation.
• Similar to above, if your cluster is on a private subnet, you'll have to tunnel in via ssh to call spark-submit, EMR API is publicly addressable.
• EMR RunJobFlowStep has AWS Step Functions integration if you want to run an EMR job as part of a workflow via state machine.

I'm sure there are others but these are the benefits I've seen.

Edit: One caveat - with EMR steps you'll need to submit the job via command-runner.jar and these end up as running processes on your master node for the life of the EMR step. If you're running hundreds of steps, you may end up needing a larger master node to support all of these processes.

Learned a couple things here. The first is that 143 KILLED does not seem to actually be indicative of failure but rather of executors receiving a signal to shutdown once the job is finished. So, seems draconian when found in logs but is not.

What was confusing me was that I wasn't seeing any "Pi is roughly 3.1475357376786883" text on stdout/stderr. This led me to believe the computation never got that far, which was incorrect.

The issue here is what I was using --deploy-mode cluster when --deploy-mode client actually made a lot more sense in this situation. That is because I was running an ad-hoc container through kubectl run which was not part of the existing deployment. This fits the definition of client mode better, since the submission does not come from an existing Spark worker. When running in --deploy-mode=cluster, you'll never actually see stdout since input/output of the application are not attached to the console.

Once I changed --deploy-mode to client, I also needed to add --conf spark.driver.host as documented here and here, for the pods to be able to resolve back to the invoking host.

Between all those technologies, I still wonder why spark needs to be run under spark-submit circumstances, you wont see this with Mongodb or Kafka, just spark!

to achieve this I advise you to use REST API providers like Apache Livy(Although I didn't like it as I tried to use it a year ago) or

try to make your server "GUI capable" with Xorg or something like that, log on to it, install intelij and submit your jobs in a local fashion, you can use your PC to test the scenarios as intelij can support local spark job runs and when you made sure you doing fine with your syntaxes and your algorithm, ship it to the repository of yours or copy and paste it into your server system and there work with it.

Good luck.

For more detailed information you can use kubectl describe pod . It will print a detailed description of the selected resources, including related resources such as events or controllers.

You can also use kubectl get event | grep pod/ - it will show events only for selected pod.

Solution :

HTTPS k8s API use cert and token for authentication.

First download k8s HTTPS API :

On master spark ->

Seems like you are missing the ServiceAccount/AWS role credentials so that your job can connect to the EKS cluster.

I recommend you set up fine-grained IAM roles for service accounts.

Basically, you would have something like this (after you set up the roles in AWS):

Based on the log file: