HDFS | PowerShell module provides a wrapper for the Hadoop File

The "used" in the monitoring graph is actually "DFS used", and it didn't show "non-DFS used". If you open the HDFS UI in the component gateway web interfaces you should be able to see something like:

This is possible by renaming the single parquet file in the dataset so that it always has the same file name, that way the export task will overwrite the previous file in the external system.

This can be done using raw file system access. The write_single_named_parquet_file function below validates its inputs, creates a file with a given name in the output dataset, then copies the file in the input dataset to it. The result is a schemaless output dataset that contains a single named parquet file.

Notes

• The build will fail if the input contains more than one parquet file, as pointed out in the question, calling .coalesce(1) (or .repartition(1)) is necessary in the upstream transform
• If you require transaction history in your external store, or your dataset is much larger than 512 MB this method is not appropriate, as only the latest version is kept, and you likely want multiple parquet files for use in your downstream system. The createTransactionFolders (put each new export in a different folder) and flagFile (create a flag file once all files have been written) options can be useful in this case.
• The transform does not require any spark executors, so it is possible to use @configure() to give it a driver only profile. Giving the driver additional memory should fix out of memory errors when working with larger datasets.
• shutil.copyfileobj is used because the 'files' that are opened are actually just file objects.

Full code snippet

example_transform.py

It happened the same to me last friday. I think it has something to do with Cuda instalation in Google Colab but I don't know exactly the reason

It may happen because some partition is bigger than others.

Try to trigger reducer task by adding distribute by

One of the issues is that partition is an overloaded term in Spark world and you're looking at 2 different kind of partitions:

• your dataset is organized as a Hive-partitioned table, where each partition is a separate directory named with = that may contain many data files inside. This is only useful for dynamically pruning the set of input files to read and has no effect on the actual RDD processing

• when Spark loads your data and creates a DataFrame/RDD, this RDD is organized in splits that can be processed in parallel and that are also called partitions.

df.rdd.getNumPartitions() returns the number of splits in your data and that is completely unrelated to your input table partitioning. It's determined by a number of config options but is mostly driven by 3 factors:

• computing parallelism: spark.default.parallelism in particular is the reason why you have 2 partitions in your RDD even though you don't have enough data to fill the first
• input size: spark will try to not create partitions bigger than spark.sql.files.maxPartitionBytes and thus may split a single multi-gigabyte parquet file into many partitions)
• shuffling: any operation that need to reorganize data for correct behavior (for example join or groupBy) will repartition your RDD with a new strategy and you will end up with many more partitions (governed by spark.sql.shuffle.partitions and AQE settings)

On the whole, you want to preserve this behavior since it's necessary for Spark to process your data in parallel and achieve good performance. When you use df.coalesce(1) you will coalesce your data into a single RDD partition but you will do your processing on a single core in which case simply doing your work in Pandas and/or Pyarrow would be much faster.

If what you want is to preserve the property on your output to have a single parquet file per Hive-partition attribute, you can use the following construct:

Once you can copy io.delta:delta-core_2.12:1.0.0 JAR file to $SPARK_HOME/lib and restart, this error goes away.

ABFS is a "real" file system, so the S3A zero rename committers are not needed. Indeed, they won't work. And the client is entirely open source - look into the hadoop-azure module.

the ADLS gen2 store does have scale problems, but unless you are trying to commit 10,000 files, or clean up massively deep directory trees -you won't hit these. If you do get error messages about Elliott to rename individual files and you are doing Jobs of that scale (a) talk to Microsoft about increasing your allocated capacity and (b) pick this up https://github.com/apache/hadoop/pull/2971

This isn't it. I would guess that actually you have multiple jobs writing to the same output path, and one is cleaning up while the other is setting up. In particular -they both seem to have a job ID of "0". Because of the same job ID is being used, what only as task set up and task cleanup getting mixed up, it is possible that when an job one commits it includes the output from job 2 from all task attempts which have successfully been committed.

I believe that this has been a known problem with spark standalone deployments, though I can't find a relevant JIRA. SPARK-24552 is close, but should have been fixed in your version. SPARK-33402 Jobs launched in same second have duplicate MapReduce JobIDs. That is about job IDs just coming from the system current time, not 0. But: you can try upgrading your spark version to see if it goes away.

My suggestions

1. make sure your jobs are not writing to the same table simultaneously. Things will get in a mess.
2. grab the most recent version spark you are happy with

It's ... complicated. Very complicated. I wanted to write a short answer but I'd risk being misleading on several points. Instead I'll try to give a very short summary of the very long answer.

• Hive uses staging directories for a good reason: atomicity. You don't want users reading a table while it is being re-written, so instead you write in a staging directory and rename the directory when it's done, like this.
• Problem is: Cloud storages are "object storages", not "distributed files systems" like HDFS, and some operations like folder renaming can be much slower because of that.
• Each cloud has it's own storage implementation, with it's own specificities and downsides, and with time they even propose new variants to overcome some of these downsides (e.g. Azure has 3 different storage variants: Blob Storage, Datalake Storage Gen 1 and Gen 2).
• Therefore, the best solution on one cloud isn't necessarily the best on another cloud.
• The FileSystem API implementation for various cloud storage is part of the Hadoop distribution, which Spark uses. So the solutions available to you will also depend on which version of Hadoop your Spark installation is using.
• Azure/GCS only: You could try setting [this option]:(https://spark.apache.org/docs/3.1.1/cloud-integration.html#configuring): spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2. It is faster than v1 but it also not recommended as it is not atomic and therefore less safe in case of partial failures.
• v2 is currently the default in Hadoop, but Spark 3 set it back to v1 by default and there are some discussion in the Hadoop community to deprecate it and make v1 the default again.
• There are also some ongoing development to write better output committers for Azure and GCS, based on a similar output committer done for S3.

• Alternatively, you can try switching to cloud-first formats like Apache Iceberg, Apache Hudi or Delta Lake.
• I am not very familiar with these yet, but a quick look at Delta Lake's documentation convinced me that they had to deal with same kind of issues (cloud storages not being real file systems), and depending on which cloud you're on, it may require extra configuration, especially on GCP where the feature is flagged as experimental.
• EDIT: Apache Iceberg does not have this issue as it uses metadata files to point to the real data files location. Thanks to this, changes to a table are committed via an atomic change on a single metadata file.
• I am not very familiar with Apache Hudi, and I couldn't find any mention of them dealing with these kind of issues. I'd have to dig further into their design architecture to know for sure.

Now, for the long answer, maybe I should write a blog article... I'll post it here whenever it's done.

Here, you have permission denied error message;