docker-spark | repository contains a Docker file | Continuous Deployment library

What you are seeing are not layers, but images that were pushed to the same registry. Basically, those are the different versions of one image.

In a repository, each image is accessible through an unique ID, its SHA value. Furthermore, one can tag images with convenient names, e.g. V1.0 or latest. These tags are not fixed, however. When an image is pushed with a tag that is already assigned to another image, the old image loses the tag and the new image gains it. Thus, a tag can move from one image to another. The tag latest is no exception. It has, however, one special property: the tag is always assigned to the most recently pushed version of an image.

The person/owner of the registry has pushed new versions of the image and not tagged the old versions. Thus, all old versions show up as "untagged".

If we pull a specific image, we will receive this image and this image only, not the complete registry.

Yes, same layers are "shared". Docker using hashes (including filesystem and commands) to identify these layers.

So docker shows you the size of the images (including the base-images) but that doesn't mean that they needs the same disk space.

The Dockerfile tells the container what port to expose.
The compose-file tells the host which ports to expose and to which ports should be the traffic forwarded inside the container.
If the source port is not specified, a random port should be generated. This statement helps in this scenario because you have multiple workers and you cannot specify a unique source port for all of them - this would result in a conflict.

Quoting docker cp Documentation,

docker cp SRC_PATH CONTAINER:DEST_PATH

If SRC_PATH specifies a file and DEST_PATH does not exist then the file is saved to a file created at DEST_PATH

From the command you tried,

The destination path /jars does not exist in the container since the actual destination should have been /usr/spark-2.4.1/jars/. Thus the jar was copied to the container with the name jars under the root (/) directory.

Try this command instead to add the jar to spark jars,

While the node has mysql installed, the container does not. What the logs are telling you is that impressions-agg_1 contains a script at /app/app.py which is trying to load mysql but cannot find it.

Did you create impressions-agg_1? Add a RUN pip install mysql step to its Dockerfile.

So the working configuration is:

• set spark.driver.host to the IP address of the host machine
• set spark.driver.bindAddress to the IP address of the container

The working Docker image is here: docker-spark-submit.

Dockerfile:

This is the command that solves my problem:

The solution is to avoid user error...

This was a total face-palm situation but I hope my misunderstanding of the Spark system can help some poor fool, like myself, who has spent too much time stuck on the same type of problem.

The answer for the last iteration (gettyimages/docker-spark Docker image) was that I was trying to run the spark-submit command without having a master or worker(s) started. In the gettyimages/docker-spark repo, you can find a docker-compose file that shows you that it creates the master and the worker nodes before any spark work is done. The way that image creates a master or a worker is by using the spark-class script and passing in the org.apache.spark.deploy.. class, respectively.

So, putting it all together, I can use the configuration I was using but I have to create the master and worker(s) first, then execute the spark-submit command the same as I was already doing.

This is a quick and dirty of one implementation, although I guarantee there's better, done by folks who actually know what they're doing:

The first 3 steps happen in a cluster boot script. I do this in an AWS Lambda, triggered by an APIGateway

1. create a cluster and a queue or some sort of message brokerage system, like zookeeper/kafka. (I'm using API-Gateway -> lambda for this)
2. pick a master node (logic in the lambda)
3. create a message with some basic information, like the master's IP or domain and put it in the queue from step 1 (happens in the lambda)

Everything below this happens in the startup script on the Spark nodes

1. create a step in the startup script that has the nodes check the queue for the message from step 3
2. add SPARK_MASTER_HOST and SPARK_LOCAL_IP to the $SPARK_HOME/conf/spark-env.sh file, using the information from the message you picked up in step 4
3. add spark.driver.bindAddress to the $SPARK_HOME/conf/spark-defaults.conf file, using the information from the message you picked up in step 4
4. use some logic in your startup script to decide "this" node is a master or a worker
5. start the master or worker. in the gettyimages/docker-spark image, you can start a master with $SPARK_HOME/bin/spark-class org.apache.spark.deploy.master.Master -h and you can start a worker with $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker -h spark://:7077
6. Now you can run the spark-submit command, which will deploy the work to the cluster.

Edit: (some code for reference) This is the addition to the lambda