parquet-cpp | Apache Parquet

You may want to try destroying your environment and starting fresh. Also, it looks like you have almost cloned the base environment, are you sure you need all of those packages?

To remove the environment:

The offset of the first data page is always larger than the offset of the dictionary. In other words, the dictionary comes first and only then the data pages. There are two metadata fields meant to store these offsets: dictionary_page_offset (aka DO) and data_page_offset (aka FPO). Unfortunately, these metadata fields are not filled in correctly by parquet-mr.

For example, if the dictionary starts at offset 1000 and the the first data page starts at offset 2000, then the correct values would be:

• dictionary_page_offset = 1000
• data_page_offset = 2000

Instead, parquet-mr stores

• dictionary_page_offset = 0
• data_page_offset = 1000

Applied to your example, this means that in spite of parquet-tools showing DO: 0, columns x and y are dictionary encoded nonetheless (column z is not).

It is worth mentioning that Impala follows the specification correctly, so you can not rely on every file having this deficiency.

This is how parquet-mr handles this situation during reading:

Parquet uses different column encodings to store low entropy data very efficiently. For example:

• It can use delta encoding to only store differences between values. For example 9192631770, 9192631773, 9192631795, 9192631797 would be stored effectively as 9192631770, +3, +12, +2.
• It can use dictionary encoding to shortly refer to common values. For example, Los Angeles, Los Angeles, Los Angeles, San Francisco, San Francisco would be stored as a dictionary of 0 = Los Angeles, 1 = San Francisco and the references 0, 0, 0, 1, 1
• It can use run-length encoding to only store the number of repeating values. For example, Los Angeles, Los Angeles, Los Angeles would be effectively stored as Los Angeles×3. (Actually as far as I know pure RLE is only used for boolean types at this moment, but the idea is the same.)
• A combination of the above, specifically RLE and dictionary encoding. For example, Los Angeles, Los Angeles, Los Angeles, San Francisco, San Francisco would be stored as a dictionary of 0 = Los Angeles, 1 = San Francisco and the references 0×3, 1×2

With the 3 to 5 values of the examples above, the savings are not that significant, but the more values you have the bigger the gain. Since you have 150k identical records, the gains will be huge, since with RLE dictionary encoding, each column value will only have to be stored once, and then marked as repeating 150k times.

However, it seems that pyarrow does not use these space-saving encodings. You can confirm this by taking a look at the metadata of the two files using parquet-tools meta. Here is a sample output:

Perhaps you're using an outdated miniconda on one of the build machine, try doing docker build --pull --no-cache.

Docker doesn't necessarily pull the latest image from the repository, so unless you do a --pull, it is possible that some of your machines may be starting the build with outdated base image.

I arranged a script to download dependencies sources, set the environment variables and run your cmake line at the end. Just change the DEPDIR variable value, setting it to a directory of choice.

So cmake does use its own version of curl.

I had to download cmake sources here https://cmake.org/download/ and use ./bootstrap --system-curl, make and make install to have a cmake version which uses the system curl. I also needed to install the package libcurl-devel.

You forgot to include the path for the header files in the compilation instruction. You need to find directory containing parquet/api/reader.h and include it in the compilation command