hadoop-cli | interactive command line shell | Command Line Interface library

the EMR docs says "use our own s3: connector"...if you are running on EMR do exactly that.

you should use the s3a one on other installations, including local ones

And there

• mvnrepository a good way to get a view of what dependencies are
  * here is its summary for hadoop-aws though its 3.2.1 declaration misses out all the dependencies. it is 1.11.375
• the stack traces you are seeing are from trying to get the aws s3 sdk, core sdk, jackson and httpclient in sync.
• it's easiest to give up and just go with the full aws-java-sdk-bundle, which has a consistent set of aws artifacts and private versions of the dependencies. It is huge -but takes away all issues related to transitive dependencies

To read and write to S3 from Spark you only need these 2 dependencies:

The answer is the same as before. Make all Spark and Scala versions the exact same. What's happening is kafka_2.13 depends on Scala 2.13, and the rest of your dependencies are 2.11... Spark 2.4 doesn't support Scala 2.13

You can more easily do this with Maven properties

Spark and Scala version mismatch is what causing this. If you use below set of dependencies this problem should be resolved.

One observation I have (which might not be 100% true as well) is if we have spark-core_2.11 (or any spark-xxxx_2.11) but scala-library version is 2.12.X I always ran into issues. Easy thing to memorize might be like if we have spark-xxxx_2.11 then use scala-library 2.11.X but not 2.12.X.

Please fix scala-reflect and scala-compile versions also to 2.11.X

There is a question in my mind over whether you should do this if you have any viable alternative.

Sounds like a situation where you are trying to work around something that is just wrong. Conceptually, depending on something that has incorporated specific versions of dependent classes is clearly a potential nightmare especially as you have discovered if there are CVEs identified against one of those shaded dependencies. Depending on an uber-jar essentially breaks the dependency management model.

I'm guessing it is internally created in your organisation, rather than coming from a central repository, so can you put pressure on that team to do the right thing?

Alternatively the dependency plugin's unpack may be an option - unpack that dependency with exclusions based on package into your build - https://maven.apache.org/plugins/maven-dependency-plugin/usage.html#dependency:unpack

The following works for me as an example - unpacks the dependency without the auth package into the classes directory of target before the default-jar is built by maven-jar plugin, and then I have to exclude the original jar - this is a spring-boot project so I use the spring-boot plugin configuration, which is used during the repackage goal, if you are using the war plugin I suspect there is a similar exclusion capability.

End result is the filtered down classes from http client in my jar classes directory alongside my application classes.

Missing org.apache.parquet.hadoop.ColumnChunkPageWriteStore(org.apache.parquet.hadoop.CodecFactory$BytesCompressor,org.apache.parquet.schema.MessageType,org.apache.parquet.bytes.ByteBufferAllocator,int) [java.lang.NoSuchMethodException: org.apache.parquet.hadoop.ColumnChunkPageWriteStore.(org.apache.parquet.hadoop.CodecFactory$BytesCompressor, org.apache.parquet.schema.MessageType, org.apache.parquet.bytes.ByteBufferAllocator, int)]

Means you are using the Constructor of ColumnChunkPageWriteStore, which takes in 4 parameters, of types (org.apache.parquet.hadoop.CodecFactory$BytesCompressor, org.apache.parquet.schema.MessageType, org.apache.parquet.bytes.ByteBufferAllocator, int)

It cant find the constructor you are using. That why NoSuchMethodError

According to https://jar-download.com/artifacts/org.apache.parquet/parquet-hadoop/1.8.1/source-code/org/apache/parquet/hadoop/ColumnChunkPageWriteStore.java , you need 1.8.1 of parquet-hadoop

Change your mvn import to an older version. I looked at 1.8.1 source code and it has the proper constructor you need.

This is because sparks pushes the filter/predicate, i.e. spark optimizes the query in such a way that the filter is applied before the "projection". The same occures with select instead of drop.

This can be beneficial because the filter can be pushed to the data:

I think it is due to mismatch between Scala version with which the code is compiled and Scala version of the runtime.

Spark 2.4.2 was prebuilt using Scala 2.12 but Scala 2.4.5 is prebuilt with Scala 2.11 as mentioned at - https://spark.apache.org/downloads.html.

This issue should go away if you use spark libraries compiled in 2.11

You need to provide XGBoost libraries when submitting the job - the easiest way to do it is to specify Maven coordinates via --packages flag to spark-submit, like this: