parquet-format | Apache Parquet

I just discovered Parquet and it met my "big" data processing / (local) storage needs:

• faster than relational databases, which are designed to run over the network (creating overhead) and just aren't as fast as a solution designed for local storage
• compared to JSON or CSV: good for storing data efficiently into types (instead of everything being a string) and can read specific chunks from the file more dynamically than JSON or CSV

But to my dismay while Node.js has a fully functioning library for it, the only Parquet lib for Python seems to be quite literally a half-measure:

parquet-python is a pure-python implementation (currently with only read-support) of the parquet format ... Not all parts of the parquet-format have been implemented yet or tested e.g. nested data

So what gives? Is there something better than Parquet already supported by Python that lowers interest in developing a library to support it? Is there some close alternative?

Let say I have a pyarrow table with a column Timestamp containing float64. These floats are actually timestamps experessed in s. For instance:

I am in the process of learning Apache Avro and I would like to know how is it represented internally. If I were to describe Apache Parquet for the same question, I can say each Parquet file is composed of row_groups, each row_groups contains column chunks and column chunks has multiple pages with different encodings. Finally the metadata about all of these is stored on the file footer. This file representation is clearly documented in the Github page as well in its official Apache page.

To find the same internal representation for Apache Avro I looked into multiple pages like Github page, Apache Avro's home and the book Hadoop definitive guide and many more tutorials online but I am not able to find what I am looking for. I understand Apache Avro is row oriented file format and each of the file has the schema also along with the data in the file. All of them is fine but I wanted to know how the data is further broken down for interal organization perhaps like pages for RDBMS tables.

Any pointers related to this will be highly appreciated.

Somewhat recently, the parquet-format project added a UUID logical type. Specifically, this was added in revision 2.4 of the parquet format. I'm interested in using the parquet-mr library in Java to create some parquet files, but I can't seem to figure out how to use the UUID logical type in a parquet schema. A simple schema like this does not seem to work as I would hope:

I'm working on a rather big project. I need to use azure-security-keyvault-secrets, so I added following to my pom.xml file:

I am trying to write a Dataframe like this to Parquet:

I have a Spring web application(built in maven) with which I connect to my spark cluster(4 workers and 1 master) and to my cassandra cluster(4 nodes). The application starts, the workers communicate with the master and the cassandra cluster is also running. However when I do a PCA(spark mllib) or any other calculation(clustering, pearson, spearman) through the interface of my web-app I get the following error:

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

which appears on this command:

Until recently parquet did not support null values - a questionable premise. In fact a recent version did finally add that support:

https://github.com/apache/parquet-format/blob/master/LogicalTypes.md

However it will be a long time before spark supports that new parquet feature - if ever. Here is the associated (closed - will not fix) JIRA:

https://issues.apache.org/jira/browse/SPARK-10943

So what are folks doing with regards to null column values today when writing out dataframe's to parquet ? I can only think of very ugly horrible hacks like writing empty strings and .. well .. I have no idea what to do with numerical values to indicate null - short of putting some sentinel value in and having my code check for it (which is inconvenient and bug prone).

I have a project where I am using spark with scala. The Code does not give the compilation issue but when I run the code I get the below exception:-