scalacheck-shapeless | arbitrary case classes / ADTs instances | Serialization library

GeneratorDrivenPropertyChecks seems to have been removed in ScalaTest 3.1.0

We made it private so that it would not hold up the 3.1.0 release any longer. I wanted to investigate a better way to integrate shrinking, as has been done by tools such as Hedgehog. The 3.2.0 release we wanted to be exactly the same as 3.1.0 except for modularization. After that we plan to complete and release ScalaTest's Generator. Meanwhile we figured everyone would continue to use ScalaCheckDrivenPropertyChecks and Gen, which is available here:

https://github.com/scalatest/scalatestplus-scalacheck

Instead try using ScalaCheckDrivenPropertyChecks like so

Put libraryDependencies ++= dependencies into settings.

global, core and serversupervisor are three different subprojects. They can have different library dependencies. Currently you add them to global but not to core and serversupervisor.

Alternatively you can move libraryDependencies ++= dependencies to Global or ThisBuild scope rather than specific subproject scope. You can add at top

You use this in the Compile scope (i.e. sources in src/main), so you need to remove % "test" from the ScalaCheck dependency. Or move that source to the Test scope (i.e. in src/test)

Okay, after many trials and tribulations, I figured it out. The issue is not that SBT was failing to exclude libraries, it was excluding them perfectly. The issue was that even though I was excluding any version of Netty that wasn't 4.1.11.Final, Spark was using its own jars, external to SBT and my built jar.

When spark-submit is run, it includes jars from the $SPARK_HOME/lib directory. One of those is an older version of Netty 4. This problem is shown with this call:

bootstrap.getClass().getProtectionDomain().getCodeSource()

The result of that is a jar location of /usr/local/Cellar/apache-spark/2.2.0/libexec/jars/netty-all-4.0.43.Final.jar

So, Spark was including its own Netty dependency. When I created my jar in SBT, it had the right jars. Spark has a configuration for this called spark.driver.userClassPathFirst documented in the Spark config documentation however when I set this to true, I end up with issues to do with using a later version of Netty.

I decided to ditch using the Transport client, and use trusty old HTTP requests instead.