spark-testing-base | Base classes to use when writing tests with Spark

I think it is due to mismatch between Scala version with which the code is compiled and Scala version of the runtime.

Spark 2.4.2 was prebuilt using Scala 2.12 but Scala 2.4.5 is prebuilt with Scala 2.11 as mentioned at - https://spark.apache.org/downloads.html.

This issue should go away if you use spark libraries compiled in 2.11

My problem come from a spark error about union 2 dataframe that i can't, but the message is not explict.

If you have the same problem, you can try your test with a local spark session.

remove DataFrameSuiteBase from your test class and instead make a local spark session:

Before :

I haven't managed to figure it out myself when and why it happens, but I did experience similar resolution-related errors earlier.

Whenever I run into issues like yours I usually delete the affected directory (e.g. /Users/ashika.umagiliya/.m2/repository/org/antlr) and start over. It usually helps.

I always make sure to use the latest and greatest sbt. You seem to be on macOS so use brew update early and often.

I'd also recommend using the latest and greatest for the libraries, and more specifically, for Spark it'd be 2.4.6 (in the 2.4.x line) while Delta Lake should be 0.7.0.

By default hive use two Metastores first one meta store service, and second the database called by default metastore_db and it uses derby. so i think you have to install and configure derby with hive. But i have not seen the use of hive in your code. I hope my answer help you

To control the version of jackson-databind, add an entry to the section in which you specify the version you want. This will override all transitive definitions and is much easier to handle than various exclusions.

So in the first step, you can try to set it to 2.8.11 and try if your tests work. If not, then you need to figure out a "middle version" that works both for the applications in core-module and your tests.

Okay, after many trials and tribulations, I figured it out. The issue is not that SBT was failing to exclude libraries, it was excluding them perfectly. The issue was that even though I was excluding any version of Netty that wasn't 4.1.11.Final, Spark was using its own jars, external to SBT and my built jar.

When spark-submit is run, it includes jars from the $SPARK_HOME/lib directory. One of those is an older version of Netty 4. This problem is shown with this call:

bootstrap.getClass().getProtectionDomain().getCodeSource()

The result of that is a jar location of /usr/local/Cellar/apache-spark/2.2.0/libexec/jars/netty-all-4.0.43.Final.jar

So, Spark was including its own Netty dependency. When I created my jar in SBT, it had the right jars. Spark has a configuration for this called spark.driver.userClassPathFirst documented in the Spark config documentation however when I set this to true, I end up with issues to do with using a later version of Netty.

I decided to ditch using the Transport client, and use trusty old HTTP requests instead.

To exclude certain transitive dependencies of a dependency, use the excludeAll or exclude methods.

The exclude method should be used when a pom will be published for the project. It requires the organization and module name to exclude.

For example:

Not exactly sure how to do this in python. In scala, using sbt, it is quite straight forward. But anyways, the System.setProperty("spark.jars.packages", "org.postgresql:postgresql:42.1.1") method found here: https://github.com/holdenk/spark-testing-base/issues/187 worked for me.

So I would rec looking up how to do that with python + spark.