HDP | Accurate estimation of conditional categorical probability | Analytics library

The correct signature for the handler method for the SelectedDateChanged event is:

Running df -h and du -h -d1 /some/paths/of/interest on the machine doing the Spark calls just taking a guess from the "writing to local FS" and "No space on disk" messages in the errors (running clush -ab df -h / across all the hadoop nodes, I could see that the client node initiating the Spark jobs was the only one with high disk utilization), I found that there was only 1GB of disk space remaining on the machine that was calling the Spark jobs (due to other issues) that eventually threw this error for some of them and have since fixed that issue, but not sure if that is related or not (as my understanding is that Spark does the actual processing on other nodes in the cluster).

I suspect that this was the problem, but if anyone with more experience could explain more what is going wrong under the surface here, that would be very helpful for future debugging and a better actual answer to this post. Eg.

1. Why would the lack of free disk space on one of the cluster nodes (in this case, a client node) cause the RM heap to remain at such a high utilization percentage even when no jobs were reported running in the RM UI?
2. Why would the lack of disk space on the local machine affect the Spark jobs (as my understanding is that Spark does the actual processing on other nodes in the cluster)?

If the disk space on the local machine calling the spark jobs was indeed the problem, this question could potentially be marked as a duplicate to the question answered here: https://stackoverflow.com/a/18365738/8236733.

There are two sections for set hive.tez.container.size in Ambari Hive Config page. One of them appears in the SETTINGS tab and the other that has related to LLAP goes under the Advanced hive-interactive-site in the ADVANCED tab. I was trying with hive.tez.container.size value the SETTINGS tab instead of Advanced hive-interactive-site section. Finally, I set the following configs and the error solved:

To clear out the space consumed by jornal edit, you are on right track. However the values are too less and if something goes wrong, you might loose data.

The default value for dfs.namenode.num.extra.edits.retained and dfs.namenode.max.extra.edits.segments.retained is set to 1000000 and 10000 respectively.

I would suggest following values:-

hive.metastore.warehouse.dir - is a warehouse root directory.

When you create the database, specify MANAGEDLOCATION - a location root for managed tables and LOCATION - root for external tables.

MANAGEDLOCATION is within hive.metastore.warehouse.dir

Setting the metastore.warehouse.tenant.colocation property to true allows a common location for managed tables (MANAGEDLOCATION) outside the warehouse root directory, providing a tenant-based common root for setting quotas and other policies.

See more details in this manual: Hive managed location.

By using those two commends make a configuration.then install python:

yum-config-manager --save --setopt=HDP-SOLR-2.3-100.skip_if_unavailable=true

First, I missed zk connect string in state-management.xml (thanks to @BenYaakobi for noticing).

Second, Hive processors work with Hive3ConnectionPool from nifi-hive3-nar library. Library contains Hive3* processors, but Hive1* (e.g. SelectHiveQL, GenerateTableFetch) processors work with Hive3 connector as well.

Restrar the ResourceManager and NodeManager manually on hosts: in Apache Ambari Go to Hosts on each Host in the Components section click on the menu for ResourceManager and NodeManager and Select Restart.

Run this like below, It should work: