docker-hadoop | Docker container with a full Hadoop cluster setup

I see a lot of echo statements in one the entrypoint.sh.

1. Where these logs will be stored ?

2. I believe, these will be automatically logged. Useful in debugging to see which environment variables ingested etc .. ?

A Sample entrypoint.sh file https://github.com/big-data-europe/docker-hadoop/blob/master/base/entrypoint.sh

Hello Guys, I'm trying to connect Kafka and HDFS with Kafka Connect, but I still face an issue that I can't rid of it.

I'm using this example: https://clubhouse.io/developer-how-to/how-to-set-up-a-hadoop-cluster-in-docker/

I start the HDFS first with: docker-compose up -d

Then I launch the zookeeper kafka and mysql with images from debezium website. https://debezium.io/documentation/reference/1.0/tutorial.html

docker run -it --rm --name zookeeper --network docker-hadoop-master_default -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.0

docker run -it --rm --name kafka --network docker-hadoop-master_default -e ZOOKEEPER_CONNECT=zookeeper -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.0

docker run -it --rm --name mysql --network docker-hadoop-master_default -p 3306:3306 -e MYSQL_ROOT_PASSWORD=debezium -e MYSQL_USER=mysqluser -e MYSQL_PASSWORD=mysqlpw debezium /example-mysql:1.0

I use the network on these runs because when I tried to change the network from HDFS on docker-compose.yml the resource manager shutdown and no matter how I couldn't find how I could raise up again and make him stable. So added directly on these containers zookeeper kafka and mysql.

Then, this is the most tricky part, the Kafka Connect, I used the same network to on this case which makes sense.

docker run -it --rm --name connect --network docker-hadoop-master_default -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses -e BOOTSTRAP_SERVERS="172.18.0.10:9092" -e CORE_CONF_fs_defaultFS=hdfs://172.18.0.2:9000 --link namenode:namenode --link zookeeper:zookeeper --link mysql:mysql debezium/connect:1.0

To link the source (Mysql) to Kafka I uses the connector from the debezium tutorial, the one below.

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{ "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.whitelist": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.inventory" } }'

I tested if Kafka receives any event from the source and works fine.

After setting this, I moved to the installation of the plugin, which I downloaded from the confluent web site and pasted on my local machine Linux, then I installed the Confluent-Hub, and after that the plugin on my local machine. Then I created the user kafka and change all the content from the plugin directory into kafka:kafka.

After all this I used docker cp :/kafka/connect to copy to Kafka Connect.

Then check if it is there and then restart the Kafka Connect to install it.

We can use this to check if is installed: curl -i -X GET -H "Accept:application/json" localhost:8083/connector-plugins

You need to see somewhere this: [{"class":"io.confluent.connect.hdfs.HdfsSinkConnector","type":"sink","version":"5.4.0"},…

After this step I believe is where my problem resides: curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{"name":"hdfs-sink","config":{"connector.class":"io.confluent.connect.hdfs.HdfsSinkConnector","tasks.max":1,"topics":"dbserver1,dbserver1.inventory.products,dbserver1.inventory.products_on_hand,dbserver1.inventory.customers,dbserver1.inventory.orders, dbserver1.inventory.geom,dbserver1.inventory.addresses","hdfs.url":"hdfs://172.18.0.2:9000","flush.size":3,"logs.dir":"logs","topics.dir":"kafka","format.class":"io.confluent.connect.hdfs.parquet.ParquetFormat","partitioner.class":"io.confluent.connect.hdfs.partitioner.DefaultPartitioner","partition.field.name":"day"}}'

I have no idea how to convince Kafka Connect that I want a specific IP address from the namenode, he just keeps my trowing messages that found a different IP when the expected is hdfs://namenode:9000

Also adding this -e CORE_CONF_fs_defaultFS=hdfs://172.18.0.2:9000 to the docker run our setting it inside the Kafka Connect, when I POST the Curl of hdfs-sink he trowing me the message below.

Log from Kafka Connect:

I am trying to access the webUIs from the containers on the docker-compose from a hadoop-cluster. Link: https://github.com/big-data-europe/docker-hadoop

The Docker-Compose File had the following content:

I've below docker swarm cluster.

I am trying to compose a lightweight minimal hadoop stack with the images provided by bde2020 (learning purpose). Right now, the stack includes (among others)

• a namenode
• a datanote
• hue

Basically, I started from Big Data Europe official docker compose, and added a hue image based on their documentation

Hue's file browser can't access HDFS:

Based on https://github.com/gotthardsen/docker-hadoop-spark-workbench/tree/master/swarm I have a docker swarm setup with hadoop, spark, hue and a jupyter notebook setup.

Using Hue I uploaded a file to hdfs, and I have not problem downloading or viewing the file from hue or in hdfs on the namenode. There is no missing blocks and file check says everything is fine.

But when I try to access it using pyspark in jupyter I get a:

org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-296583215-10.0.0.6-1542473394293:blk_1073741833_1009 file=/20170930.csv

I know this is not about missing block but more likely something else. But I can not figure out why. The code python code from the workbook using python2 kernel is:

I creasted a Spark Cluster using this repository and the relative README.md file.

Now I'm trying to execute through spark-submit a job to the docker container of the Spark Master so the command that I use is something similar:

I have installed https://github.com/big-data-europe/docker-hadoop-spark-workbench

Then started it up with docker-compose up . I navigated to the various urls mentioned in the git readme and all appears to be up.

I then started a local apache zeppelin with:

I have installed apache zeppelin by downloading and extracting the binary with all interpreters

I then started it up with: