Invoker | Generic and extensible callable invoker | Dependency Injection library

ABFS is a "real" file system, so the S3A zero rename committers are not needed. Indeed, they won't work. And the client is entirely open source - look into the hadoop-azure module.

the ADLS gen2 store does have scale problems, but unless you are trying to commit 10,000 files, or clean up massively deep directory trees -you won't hit these. If you do get error messages about Elliott to rename individual files and you are doing Jobs of that scale (a) talk to Microsoft about increasing your allocated capacity and (b) pick this up https://github.com/apache/hadoop/pull/2971

This isn't it. I would guess that actually you have multiple jobs writing to the same output path, and one is cleaning up while the other is setting up. In particular -they both seem to have a job ID of "0". Because of the same job ID is being used, what only as task set up and task cleanup getting mixed up, it is possible that when an job one commits it includes the output from job 2 from all task attempts which have successfully been committed.

I believe that this has been a known problem with spark standalone deployments, though I can't find a relevant JIRA. SPARK-24552 is close, but should have been fixed in your version. SPARK-33402 Jobs launched in same second have duplicate MapReduce JobIDs. That is about job IDs just coming from the system current time, not 0. But: you can try upgrading your spark version to see if it goes away.

My suggestions

1. make sure your jobs are not writing to the same table simultaneously. Things will get in a mess.
2. grab the most recent version spark you are happy with

The version of @PactTestFor method still needs to use version 3.

Try

findAll() method is not mocked, thus a null value is returned. You should mock userRepository.findAll() instead of userRepository.findAll().collectList().block().

Try this:

There are two separate problems from my point of view:

1. AssertJ's SoftAssertionsExtension is not yet supported by Quarkus.
2. QuarkusTestExtension does not play nicely with composed annotations.

About the latter, the issue seems to be unrelated to AssertJ. I was able to reproduce it by taking out the SoftAssertionsExtension from the custom QuarkusAssertJTest annotation and executing an empty test.

Having:

Based on your input data, I create my dataframe :

Disclaimer: it's not a answer to the question. But I wanted to share with OP my findings, which would not fit so well in a comment.

I encountered on my system a different exception:

In place of doing an if you can catch an HttpClientErrorException, so with that you can avoid boiler plate

As mentioned in the comments, this stems from a Guava version incompatibility between the GCS connector's dependency vs what you have bundled in your Spark distro. Specifically, the GCS connector hadoop3-2.2.2 depends on Guava 30.1-jre whereas Spark 3.1.2 brings Guava 14.0.1 as a "provided" dependency.

In the two different commands, it was more-or-less luck of the draw that classpath loading happened in the right order for your first approach to work, and it could end up failing unexpectedly again when other jars are added.

Ideally you'll want to host your own jarfile anyways to minimize runtime dependencies on external repositories (Maven repository), so pre-installing the jarfile is the right approach. When you do that, you should consider using the full shaded jarfile (also available on Maven central) instead of the minimal GCS connector jarfile to avoid classloading version issues in the future.