leveldbjni | A Java Native Interface to LevelDB | SQL Database library

• I cleaned my ~/.ivy2/cache directory.
• My project/plugins.sbt file :

I'm working on a rather big project. I need to use azure-security-keyvault-secrets, so I added following to my pom.xml file:

I'm able to run application from my eclipse, but when i create jar try to run from command prompt it giving error. i'm using java 1.8 and eclipse kepler

I have a Spring web application(built in maven) with which I connect to my spark cluster(4 workers and 1 master) and to my cassandra cluster(4 nodes). The application starts, the workers communicate with the master and the cassandra cluster is also running. However when I do a PCA(spark mllib) or any other calculation(clustering, pearson, spearman) through the interface of my web-app I get the following error:

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

which appears on this command:

I am getting the following error when launching my Java Application.

I need to export some hadoop related directories to the classpath before launching the application to make it work (I can't skip this step), and I am getting the following error.

Caused by: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/usr/hdp/3.0.0.0-1634/hadoop/lib/jsr311-api-1.1.1.jar!/javax/ws/rs/ext/RuntimeDelegate.class to jar:file:/tmp/blobStore-634df1c1-ffc8-4610-86af-8f39b33e4250/job_ac11246bea2bb31008c1a78212357514/blob_p-79f2d3193313ea987c15b4b28411db0fc2aa436c-f858cb54126b6d546c01e5ed453bf106!/javax/ws/rs/ext/RuntimeDelegate.class at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:146) at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:120) at javax.ws.rs.core.UriBuilder.newInstance(UriBuilder.java:95) at javax.ws.rs.core.UriBuilder.fromUri(UriBuilder.java:119) at org.glassfish.jersey.client.JerseyWebTarget.(JerseyWebTarget.java:71) at org.glassfish.jersey.client.JerseyClient.target(JerseyClient.java:290) at org.glassfish.jersey.client.JerseyClient.target(JerseyClient.java:76) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.lambda$currentSchemaRegistryTargets$0(SchemaRegistryClient.java:293) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.currentSchemaRegistryTargets(SchemaRegistryClient.java:293) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getSupportedSchemaProviders(SchemaRegistryClient.java:384) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getDefaultDeserializer(SchemaRegistryClient.java:969) at SchemaService.InitDeserializer(SchemaService.java:47) at SchemaService.deserialize(SchemaService.java:38) at org.apache.flink.streaming.connectors.kafka.internals.KafkaDeserializationSchemaWrapper.deserialize(KafkaDeserializationSchemaWrapper.java:45) at org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:140) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:712) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:93) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:57) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:97) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:302) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at java.lang.Thread.run(Thread.java:745)

After making some research, I figured out that the class the program is trying to use is present in two diferent JAR files:

• The first one is one transitive dependence on the libraries on my maven application (javax.ws.rs.jar)

• The second one is a jar located in the directory with all the hadoop depdencies from hortonworks that I need to export into my classpath to make the application work (jsr311-api-1.1.1.jar)

So I need to tell the program in some way that the maven dependency packaged in the jar of my application should be used instead of the jar located on the classpath.(javax.ws.rs.jar) without removing this jar because it is part of my Big Data cluster installation and I can't play with those jars in the classpath.

Any thoughts?

pom.xml dependency causing the issue:

I have a project where I am using spark with scala. The Code does not give the compilation issue but when I run the code I get the below exception:-

I created a Hadoop cluster with 1 master and 2 slaves. All of the services are running in nodes. Datanode and Nodemanager are active on slave1 and slave2. Namenode, Datanode, Nodemanager, ResourceManager, and SecondaryNameNode are active on the master. But the Web UI of NameNode (localhost:50070) in part of Live nodes shows 1 node (master) and the web UI of yarn shows 1 active node.

The following works are done:

• Disable firewall.
• Password-less ssh connection between all of the nodes.
• Hostname configuration.
• Transfer Hadoop config files from master to slaves.

how to solve this problem?

hadoop-hadoop-datanode-hadoopslave1.log:

I am having difficulties creating a basic spark streaming application.

Right now, am trying it on my local machine.

I have done following setup.

-Setup Zookeeper

-Setup Kafka ( Version : kafka_2.10-0.9.0.1)

-Created a topic using below command

kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

-Started producer and consumer on two different cmd terminals using below commands

Producer :

kafka-console-producer.bat --broker-list localhost:9092 --topic test

Consumer :

kafka-console-consumer.bat --zookeeper localhost:2181 --topic test

Now I can receive the data which I enter in the producer terminal in consumer console.

Now am trying to integrate Kafka into Apache Spark streaming.

Below is a sample code which I referenced from official documents. Kafka & Spark Setup and Kafka & Spark Integration

I am new in spark framework. I have tried to create a sample application using spark and java. I have the following code

Pom.xml