docker-spark | Docker images with Apache Spark and advanced config

You're getting command not found because PATH isn't correctly established

Use the absolute path /opt/spark/bin/spark-shell

Also, I'd suggest packaging your Scala project as an uber jar to submit unless you have no external dependencies or like to add --packages/--jars manually

Okay so i found that i can use the IP of the container, as long as all my containers are on the same network. So i check the IP by running

What you are seeing are not layers, but images that were pushed to the same registry. Basically, those are the different versions of one image.

In a repository, each image is accessible through an unique ID, its SHA value. Furthermore, one can tag images with convenient names, e.g. V1.0 or latest. These tags are not fixed, however. When an image is pushed with a tag that is already assigned to another image, the old image loses the tag and the new image gains it. Thus, a tag can move from one image to another. The tag latest is no exception. It has, however, one special property: the tag is always assigned to the most recently pushed version of an image.

The person/owner of the registry has pushed new versions of the image and not tagged the old versions. Thus, all old versions show up as "untagged".

If we pull a specific image, we will receive this image and this image only, not the complete registry.

Yes, same layers are "shared". Docker using hashes (including filesystem and commands) to identify these layers.

So docker shows you the size of the images (including the base-images) but that doesn't mean that they needs the same disk space.

The Dockerfile tells the container what port to expose.
The compose-file tells the host which ports to expose and to which ports should be the traffic forwarded inside the container.
If the source port is not specified, a random port should be generated. This statement helps in this scenario because you have multiple workers and you cannot specify a unique source port for all of them - this would result in a conflict.

Quoting docker cp Documentation,

docker cp SRC_PATH CONTAINER:DEST_PATH

If SRC_PATH specifies a file and DEST_PATH does not exist then the file is saved to a file created at DEST_PATH

From the command you tried,

The destination path /jars does not exist in the container since the actual destination should have been /usr/spark-2.4.1/jars/. Thus the jar was copied to the container with the name jars under the root (/) directory.

Try this command instead to add the jar to spark jars,