mapreduce | MapReduce course projects developed at Northeastern

the issue is with the airflow Docker image you are using.

The ENTRYPOINT I see is a custom .sh file you have written and that decides whether to run a webserver or scheduler.

Airflow scheduler submits a pod for the tasks with args as follows

Acording to this answer: https://stackoverflow.com/a/51236918/16651073 tomcat falls back to default logging if it can resolve the location

Can you try to save the properties without the spaces.

Like this: logging.file.name=application.logs

In researching how this might be done, I found hadoopDependencyCoordinates property here: https://druid.apache.org/docs/0.22.1/ingestion/hadoop.html#task-syntax

which seems relevant.

There are two ways how you can fix OOM in mapper: 1 - increase mapper parallelism, 2 - increase the mapper size.

Try to increase parallelism first.

Check current values of these parameters and reduce mapreduce.input.fileinputformat.split.maxsize to get more smaller mappers:

this does sound like a binding problem but I cannot see immediately where it is. At a glance you have all the right settings.

The easiest way to check that an S3 a committee is being used is to look at the _SUCCESS file . If it is a piece of JSON then a new committer was used… The text inside will then tell you more about the committer.

a 0 byte file means that the classic file output committer was still used

Just to give an update, apparently there were problems with the algorithm linked in the OP. It looks like there is something wrong with the sychronization protocol, which is hinted at by the presence of the call to the sleep() function (ie. it's not supposed to be there).

For a good working implementation of the map/reduce framework, please refer to Joe Armstrong's version in the Programming Erlang book (2nd ed).

Armstrong's version only uses one reducer, but it can be easily modified for more reducers in order to eliminate the bottleck. I have also added a function to split the input list into chunks. Each mapper will get a chunk of data.

It is possible to do it in place like this:

ABFS is a "real" file system, so the S3A zero rename committers are not needed. Indeed, they won't work. And the client is entirely open source - look into the hadoop-azure module.

the ADLS gen2 store does have scale problems, but unless you are trying to commit 10,000 files, or clean up massively deep directory trees -you won't hit these. If you do get error messages about Elliott to rename individual files and you are doing Jobs of that scale (a) talk to Microsoft about increasing your allocated capacity and (b) pick this up https://github.com/apache/hadoop/pull/2971

This isn't it. I would guess that actually you have multiple jobs writing to the same output path, and one is cleaning up while the other is setting up. In particular -they both seem to have a job ID of "0". Because of the same job ID is being used, what only as task set up and task cleanup getting mixed up, it is possible that when an job one commits it includes the output from job 2 from all task attempts which have successfully been committed.

I believe that this has been a known problem with spark standalone deployments, though I can't find a relevant JIRA. SPARK-24552 is close, but should have been fixed in your version. SPARK-33402 Jobs launched in same second have duplicate MapReduce JobIDs. That is about job IDs just coming from the system current time, not 0. But: you can try upgrading your spark version to see if it goes away.

My suggestions

1. make sure your jobs are not writing to the same table simultaneously. Things will get in a mess.
2. grab the most recent version spark you are happy with

The reason is that map and filter have a special implementation for AbstractString. They iterate a string and return a string. Therefore, in map it is required that the function you pass returns AbstractChar. Here is an example: