HDP

So, after many hours and trials, here's a working solution. First of all, the most important thing is to use drivers that correlate with your version of Hadoop. Needed are jar files like 'hive-jdbc-standalone' and 'hadoop-common' in their respective versions and to avoid adding all of them in the 'Artifact' field of the %jdbc interpreter in Zeppelin it is best to use one complete file containing all required dependencies. Thanks to Tim Veil it is available in his Github repository below:

https://github.com/timveil/hive-jdbc-uber-jar/

This is my complete Zeppelin %jdbc interpreter settings:

As the output says, group doesn't exist with --zookeeper

You need to keep your arguments consistent; use --bootstrap-server to list, delete, and describe, assuming your cluster supports this

However, groups delete themselves with no active consumers, so you shouldn't need to run this

No, auto-compaction (and auto-optimize) is only the future on Databricks. For non-Databricks installations you can consult documentation on delta.io.

1. you need the spark-hadoop-cloud module for the release of spark you are using
2. the committer is happy using the local fs (it's now the public integration test suites work https://github.com/hortonworks-spark/cloud-integration. all that's needed is a "real" filesystem shared across all workers and the spark driver, so the driver gets the manifests of each pending commit.
3. print the _SUCCESS file after a job to see what the committer did: 0 byte file == old committer, JSON with diagnostics == new one

It should be outputCol, not outputCols.

For spark 2.3.1, you can refer to: https://spark.apache.org/docs/2.3.1/api/python/pyspark.ml.html#pyspark.ml.feature.StringIndexer

Hi I have made a few changes in your HTML

like changing img tag to div with the background image. For now, I have added inline CSS, you can put it in your CSS as per your usage

Read about CSS layout and position for further knowledge css positions and layouts

preview:

The rebalance in Kafka is a protocol and is used by various components (Kafka connect, Kafka streams, Schema registry etc.) for various purposes.

In the most simplest form, a rebalance is triggered whenever there is any change in the metadata.

Now, the word metadata can have many meanings - for example:

• In the case of a topic, it's metadata could be the topic partitions and/or replicas and where (which broker) they are stored
• In the case of a consumer group, it could be the number of consumers that are a part of the group and the partitions they are consuming the messages from etc.

The above examples are by no means exhaustive i.e. there is more metadata for topics and consumer groups but I wouldn't go into more details here.

So, if there is any change in:

• The number of partitions or replicas of a topic such as addition, removal or unavailability
• The number of consumers in a consumer group such as addition or removal
• Other similar changes...

A rebalance will be triggered. In the case of consumer group rebalancing, consumer applications need to be robust enough to cater for such scenarios.

So rebalances are a feature. However, in your case it appears that it is happening very frequently so you may need to investigate the logs on your client application and the cluster.

Following are a couple of references that might help:

1. Rebalance protocol - A very good article on medium on this subject
2. Consumer rebalancing - Another post on SO focusing on consumer rebalancing

ABFS is a "real" file system, so the S3A zero rename committers are not needed. Indeed, they won't work. And the client is entirely open source - look into the hadoop-azure module.

the ADLS gen2 store does have scale problems, but unless you are trying to commit 10,000 files, or clean up massively deep directory trees -you won't hit these. If you do get error messages about Elliott to rename individual files and you are doing Jobs of that scale (a) talk to Microsoft about increasing your allocated capacity and (b) pick this up https://github.com/apache/hadoop/pull/2971

This isn't it. I would guess that actually you have multiple jobs writing to the same output path, and one is cleaning up while the other is setting up. In particular -they both seem to have a job ID of "0". Because of the same job ID is being used, what only as task set up and task cleanup getting mixed up, it is possible that when an job one commits it includes the output from job 2 from all task attempts which have successfully been committed.

I believe that this has been a known problem with spark standalone deployments, though I can't find a relevant JIRA. SPARK-24552 is close, but should have been fixed in your version. SPARK-33402 Jobs launched in same second have duplicate MapReduce JobIDs. That is about job IDs just coming from the system current time, not 0. But: you can try upgrading your spark version to see if it goes away.

My suggestions

1. make sure your jobs are not writing to the same table simultaneously. Things will get in a mess.
2. grab the most recent version spark you are happy with

got the directory structure I wanted, but now I can't read the files

This is due to the binary structure of Parquet files. They have header/footer metadata that stores the schemas and the number of records in the file... getmerge therefore is really only useful for row-delimited, non-binary data formats.

What you can do instead is have spark.read.path("/data_folder"), then repartition or coalesce that dataframe, then output to a new "merged" output location

Another alternative is Gobbilin - https://gobblin.apache.org/docs/user-guide/Compaction/