hadoop-cluster-docker | Run Hadoop Custer within Docker Containers | Continuous Deployment library

At the very least, you need to expose the port.

docker run -it -p 50070:50070 sequenceiq/hadoop-docker:2.7.1

Then, if you want to continue using the old Docker Toolbox (that linked post was created in 2016, before Docker for Windows existed), you need to not use ifconfig, but rather docker-machine ip from Windows, not the container

Personally, I use Docker Compose

What you need to realize is that the Hadoop instance is running in an environment that is entirely different from the host environment. So the second you run the sudo ./start-container.sh command which is mentioned in the GitHub repository that you're following, you're ideally creating a new subsystem which is independent of your host Operating System ( that contains files under /home/ke/code ). Unfortunately in this case the Hadoop Disk File System ( HDFS ) is running inside the newly created subsystem ( known as Docker Container ) and the files you wish to transfer are present elsewhere ( in the host OS).

There is however a fix that you might be able to do to make it work out.

• Edit the start-container.sh in this way : Edit the lines 10-16 responsible for starting the hadoop master container to this :-

You should inspect the dfs.datanode.data.dir in the hdfs-site.xml file to know where data is stored to the container filesystem

The URI hdfs:///user/root/input/test is missing an authority (hostname) section and port. To write to hdfs in another container you would need to fully specify the URI and make sure the two containers were on the same network and that the HDFS container has the ports for the namenode and data node exposed.

For example, you might have set the host name for the HDFS container to be hdfs.container. Then you can write to that HDFS instance using the URI hdfs://hdfs.container:8020/user/root/input/test (assuming the Namenode is running on 8020). Of course you will also need to make sure that the path you're seeking to write has the correct permissions as well.

So to do what you want:

• Make sure your HDFS container has the namenode and datanode ports exposed. You can do this using an EXPOSE directive in the dockerfile (the container you linked does not have these) or using the --expose argument when invoking docker run. The default ports are 8020 and 50010 (for NN and DN respectively).
• Start the containers on the same network. If you just do docker run with no --network they will start on the default network and you'll be fine. Start the HDFS container with a specific name using the --name argument.
• Now modify your URI to include the proper authority (this will be the value of the docker --name argument you passed) and port as described above and it should work

The equivalent command in kubernetes is