metastore | Store and restore metadata from a filesystem | Cloud Storage library

Attempting to read a view which was created on AWS Athena (based on a Glue table that points to an S3's parquet file) using pyspark over a Databricks cluster throws the following error for an unknown reason:

I'm trying to create local spark environment in Windows 11 with python.
I am using python 3.9 and spark version 3.2.1. I have set my environmental variables to:

I'm using a dockerized version of the Confluent Platform v 7.0.1:

HIVE has a metastore and HIVESERVER2 listens for SQL requests; with the help of metastore, the query is executed and the result is passed back. The Thrift framework is actually customised as HIVESERVER2. In this way, HIVE is acting as a service. Via programming language, we can use HIVE as a database.

The relationship between Spark-SQL and HIVE is that:

Spark-SQL just utilises the HIVE setup (HDFS file system, HIVE Metastore, Hiveserver2). When we invoke /sbin/start-thriftserver2.sh (present in spark installation), we are supposed to give hiveserver2 port number, and the hostname. Then via spark's beeline, we can actually create, drop and manipulate tables in HIVE. The API can be either Spark-SQL or HIVE QL. If we create a table / drop a table, it will be clearly visible if we login into HIVE and check(say via HIVE beeline or HIVE CLI). To put in other words, changes made via Spark can be seen in HIVE tables.

My understanding is that Spark does not have its own meta store setup like HIVE. Spark just utilises the HIVE setup and simply the SQL execution happens via Spark SQL API.

Is my understanding correct here?

Then I am little confused about the usage of bin/spark-sql.sh (which is also present in Spark installation). Documentation says that via this SQL shell, we can create tables like we do above (via Thrift Server/Beeline). Now my question is: How the metadata information is maintained by spark then?

Or like the first approach, can we make spark-sql CLI to communicate to HIVE (to be specific: hiveserver2 of HIVE) ? If yes, how can we do that ?

Thanks in advance!

I use Spark to write data from Hive Table to Kinetica using this jar: kinetica-spark-7.0.6.1-jar-with-dependencies.jar. However, when I run spark-submit, the logger from the jar is printing the JDBC connection string with its credentials as follows:

I'm unable to connect to snowflake via a dockerized pyspark container. I do not find the snowflake documentation to be helpful nor the pyspark documentation at this point in time.

I'm using the following configuration installed & can be seen below in the Dockerfile

• python 3.7.12
• pyspark 3.1.1
• Hadoop 3.2
• jre-1.8.0-openjdk
• snowflake-jdbc-3.13.15.jar
• spark-snowflake_2.12-2.10.0-spark_3.1.jar
• snowflake-connector-python 2.7.4

I'm running spark 3.1.1 and an AWS emr-6.3.0 cluster with the following hive/metastore configurations:

I'd like connect to Delta using JDBC and would like to run the Spark Thrift Server (STS) in local mode to kick the tyres.

I start STS using the following command:

I'm learning Spark SQL, when I'm using spark-sql to uncache a table which has previously cached, but after submitted the uncache command, I can still query the cache table. Why this happened?

Spark version 3.2.0(Pre-built for Apache Hadoop 2.7)

Hadoop version 2.7.7

Hive metastore 2.3.9

Linux Info