metastore | david hardeman 's util for managing file

I’m trying to integrate spark(3.1.1) and hive local metastore (3.1.2) to use spark-sql.

i configured the spark-defaults.conf according to https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html and hive jar files exists in correct path.

but an exception occurred when execute 'spark.sql("show tables").show' like below.

any mistakes, hints, or corrections would be appreciated.

I have installed the frontend of a webpage project and when I try to start it, I get the following error message:

I have started spark-thrift server and connected to the thrift server using beeline. when trying to query create a table in hive metastore and i am getting the following error.

creating table

I have an AWS CLI cluster creation command that I am trying to modify so that it enables my driver and executor to work with a customized log4j.properties file. With Spark stand-alone clusters I have successfully used the approach of using the --files switch together with setting -Dlog4j.configuration= specified via spark.driver.extraJavaOptions, and spark.executor.extraJavaOptions.

I tried many different permutations and variations, but have yet to get this working with the Spark job that I am running on an AWS EMR clusters.

I use the AWS CLI's 'create cluster' command with an intermediate step that downloads my spark jar, unzips it to get at the log4j.properties packaged with that .jar. I then copy the log4j.properties to my hdfs /tmp folder and attempt to distribute that log4j.properties file via '--files'.

Note, I have also tried this without hdfs (specifying --files log4j.properties instead of --files hdfs:///tmp/log4j.properties) and that didn't work either.

My latest non-working version of this command (using hdfs) is given below. I'm wondering if anyone can share a recipe that actually does work. The output of the command from the driver when I run this version is:

Our setup is configured that we have a default Data Lake on AWS using S3 as storage and Glue Catalog as our metastore.

We are starting to use Apache Hudi and we could get it working following de AWS documentation. The issue is that, when using the configuration and JARs indicated in the doc, we are unable to run spark.sql on our Glue metastore.

Here follows some information.

We are creating the cluster with boto3:

I am able to successfully import data from SQL Server to HDFS using sqoop. However, when it tries to link to HIVE I get an error. I am not sure I understand the error correctly

I am working on some benchmarks and need to compare ORC, Parquet and CSV formats. I have exported TPC/H (SF1000) to ORC based tables. When I want to export it to Parquet I can run:

I am trying to sqoop hive view to SQL server database however i'm getting "object not found error". Does sqoop export works for hive views?

I am using spark v2.4.4 via the python API

According to the spark documentation I can force spark to download all the hive jars for interacting with my hive_metastore by setting the following config

• spark.sql.hive.metastore.version=${my_version}
• spark.sql.hive.metastore.jars=maven

However, when I run the following python code, no jar files are downloaded from maven.