docker-cloudera-quickstart | Docker Cloudera Quick Start Image | Continuous Deployment library

I have met some issues while trying to consume messages from Kafka with a Spark Streaming application in a Kerberized Hadoop cluster. I tried both of the two approaches listed here :

• receiver-based approach : KafkaUtils.createStream
• direct approach (no receivers) : KafkaUtils.createDirectStream

The receiver-based approach (KafkaUtils.createStream) throws 2 types of exceptions (different exceptions whether I am in local mode (--master local[*]) or in YARN mode (--master yarn --deploy-mode client) :

• a weird kafka.common.BrokerEndPointNotAvailableException in a Spark local application
• a Zookeeper timeout in a Spark on YARN application. I once managed to make this work (connecting to Zookeeper successfully), but no messages were received

In both modes (local or YARN), the direct approach (KafkaUtils.createDirectStream) returns an unexplained EOFException (see details below).

My final goal is to launch a Spark Streaming job on YARN, so I will leave the Spark local job aside.

Here is my test environment :

• Cloudera CDH 5.7.0
• Spark 1.6.0
• Kafka 0.10.1.0

I'm working on a single-node cluster (hostname = quickstart.cloudera) for testing purposes. For those interested to reproduce the tests, I'm working on a custom Docker container based on cloudera/quickstart (Git repo).

Below is my sample code I used in a spark-shell. Of course this code works when Kerberos is not enabled : messages produced by kafka-console-producer are received by the Spark application.