parquet-tools | easy install parquet-tools | Cloud Storage library
kandi X-RAY | parquet-tools Summary
kandi X-RAY | parquet-tools Summary
easy install parquet-tools
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Implements the command line interface
- Execute select
- Return a pandas dataframe
- Get the dataframe from a list of objects
- Read the data from an IProt
parquet-tools Key Features
parquet-tools Examples and Code Snippets
Lines of Code : 37License : Strong Copyleft (CC BY-SA 4.0)
Copyspark-shell --conf spark.sql.parquet.outputTimestampType=TIMESTAMP_MICROS
2020-03-16 11:37:50 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting defaulnt = sqlCtx.sql("SELECT COUNT(*) AS pageCount FROM table1 WHERE pp_count>=500") \
.collect()
$> parquet-tools head data.parquet/
a = 1
pp_count = 500
a = 2
pp_count = 750
a = 3
pp_count = 400
a val df=Seq((1,"a"),(2,"b")).toDF("id","name")
df.coalesce(1).write.partitionBy("id").csv("/user/shu/temporary2") //write csv file.
hadoop fs -cat /user/shu/temporary2/id=1/part-00000-dc55f08e-9143-4b60-a94e-e28b1d7Community Discussions
Trending Discussions on parquet-tools
QUESTION
i'm using parquetjs to create parquet files and push to google cloud storage.
Problem is that bigquery cannot read the data from file but when i use parquet-tools everything looks healthy.
ANSWER
Answered 2021-Nov-29 at 15:07just pass useDataPageV2: false as option to parquet.ParquetWriter.openFile(...)
Like this:
QUESTION
I am new to Snowflake, but my company has been using it successfully.
Parquet files are currently being written with an existing Avro Schema, using Java parquet-avro v1.10.1.
I have been updating the dependencies in order to use latest Avro, and part of that bumped Parquet to 1.11.0.
The Avro Schema is unchanged. However when using the COPY INTO Snowflake command, I receive a LOAD FAILED with error: Error parsing the parquet file: Logical type Null can not be applied to group node but no other error details :(
The problem is that there are no null columns in the files.
I've cut the Avro schema down, and found that the presence of a MAP type in the Avro schema is causing the issue.
The field is
...ANSWER
Answered 2020-Jun-22 at 09:19Logical type Null can not be applied to group node
Looking up the error above, it appears that a version of Apache Arrow's parquet libraries is being used to read the file.
However, looking closer, the real problem lies in the use of legacy types within the Avro based Parquet Writer implementation (the following assumes Java was used to write the files).
The new logicalTypes schema metadata introduced in Parquet defines many types including a singular MAP type. Historically, the former convertedTypes schema field supported use of MAP AND MAP_KEY_VALUE for legacy readers. The new writers that use logicalTypes (1.11.0+) should not be using the legacy map type anymore, but work hasn't been done yet to update the Avro to Parquet schema conversions to drop the MAP_KEY_VALUE types entirely.
As a result, the schema field for MAP_KEY_VALUE gets written out with an UNKNOWN value of logicalType, which trips up Arrow's implementation that only understands logicalType values of MAP and LIST (understandably).
Consider logging this as a bug against the Apache Parquet project to update their Avro writers to stop nesting the legacy MAP_KEY_VALUE type when transforming an Avro schema to a Parquet one. It should've ideally been done as part of PARQUET-1410.
Unfortunately this is hard-coded behaviour and there are no configuration options that influence map-types that can aid in producing a correct file for Apache Arrow (and for Snowflake by extension). You'll need to use an older version of the writer until a proper fix is released by the Apache Parquet developers.
QUESTION
I have an S3 bucket full of .gz.parquet files. I want to make them accessible in Athena. In order to do this I am creating a table in Athena that points at the s3 bucket:
...ANSWER
Answered 2020-Jun-17 at 20:16You can use an AWS Glue Crawler to automatically derive the schema from your Parquet files.
Defining AWS Glue Crawlers: https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html
QUESTION
I am using BigQuery to query an external data source (also known as a federated table), where the source data is a hive-partitioned parquet table stored in google cloud storage. I used this guide to define the table.
My first query to test this table looks like the following
...ANSWER
Answered 2020-Apr-13 at 15:53Note that, the schema of the external table is inferred from the last file sorted by the file names lexicographically among the list of all files that match the source URI of the table. So any chance that particular Parquet file in your case has a different schema than the one you described, e.g., a INT32 column with DATE logical type for the "visitor_partition" field -- which BigQuery would infer as DATE type.
QUESTION
I am running a spark job to write to parquet. I want to enable dictionary encoding for the files written. When I check the files, I see they are 'plain dictionary'. However, I do not see any stats for these columns
Let me know if I am missing anything
...ANSWER
Answered 2020-Mar-27 at 23:18Got the answer. The parquet tools version I was using was 1.6. Upgrading to 1.10 solved the issue
QUESTION
I'd like to convert an int96 value such as ACIE4NxJAAAKhSUA into a readable timestamp format like 2020-03-02 14:34:22 or whatever that could be normally interpreted...I mostly use python so I'm looking to build a function that does this conversion. If there's another function that can do the reverse -- even better.
BackgroundI'm using parquet-tools to convert a raw parquet file (with snappy compression) to raw JSON via this commmand:
...ANSWER
Answered 2020-Mar-16 at 19:01parquet-tools will not be able to change format type from INT96 to INT64. What you are observing in json output is a String representation of the timestamp stored in INT96 TimestampType. You will need spark to re-write this parquet with timestamp in INT64 TimestampType and then the json output will produce a timestamp (in the format you desire).
You will need to set a specific config in Spark -
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install parquet-tools
You can use parquet-tools like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
