dictionaries | Misc dictionaries for directory/file enumeration | Security Testing library

Since Python 3.7, dictionaries are ordered. So why I can't get keys by index?

I have a dataframe like this

I am working on a large Pandas DataFrame which needs to be converted into dictionaries before being processed by another API.

The required dictionaries can be generated by calling the .to_dict(orient='records') method. As stated in the docs, the returned value depends on the orient option:

Returns: dict, list or collections.abc.Mapping

Return a collections.abc.Mapping object representing the DataFrame. The resulting transformation depends on the orient parameter.

For my case, passing orient='records', a list of dictionaries is returned. When dealing with lists, the complete memory required to store the list items, is reserved/allocated. As my dataframe can get rather large, this might lead to memory issues especially as the code might be executed on lower spec target systems.

I could certainly circumvent this issue by processing the dataframe chunk-wise and generate the list of dictionaries for each chunk which is then passed to the API. Furthermore, calling iter(df.to_dict(orient='records')) would return the desired generator, but would not reduce the required memory footprint as the list is created intermediately.

Is there a way to directly return a generator expression from df.to_dict(orient='records') instead of a list in order to reduce the memory footprint?

I'm experimenting with Hunspell and how to interact with it using Java Project Panama (Build 19-panama+1-13 (2022/1/18)). I was able to get some initial testing done, as in creating a handle to Hunspell and subsequently using that to perform a spell check. I'm now trying something more elaborate, letting Hunspell give me suggestions for a word not present in the dictionary. This is the code that I have for that now:

I have a some python code below that walk down a tree but I want it to work down a tree checking taking some paths conditioally based on values. I want to get the LandedPrice for branches of tree based on condition and fulfillmentChannel

I would like to convert a pandas dataframe to a multi key dictionary, using 2 ore more columns as the dictionary key, and I would like these keys to be order irrelevant.

Here's an example of converting a pandas dictionary to a regular multi-key dictionary, where order is relevant.

I'm surprised to find out that this compiles:

I have a list of 'Id's' that I wish to associate with a property from another list, their 'rows'. I have found a way to do it by making smaller dictionaries and concatenating them together which works, but I wondered if there was a more pythonic way to do it?

Code

I'm seeking advice from people deeply familiar with the binary layout of Apache Parquet:

Having a data transformation F(a) = b where F is fully deterministic, and same exact versions of the entire software stack (framework, arrow & parquet libraries) are used - how likely am I to get an identical binary representation of dataframe b on different hosts every time b is saved into Parquet?

In other words how reproducible Parquet is on binary level? When data is logically the same what can cause binary differences?

• Can there be some uninit memory in between values due to alignment?
• Assuming all serialization settings (compression, chunking, use of dictionaries etc.) are the same, can result still drift?

I'm working on a system for fully reproducible and deterministic data processing and computing dataset hashes to assert these guarantees.

My key goal has been to ensure that dataset b contains an idendital set of records as dataset b' - this is of course very different from hashing a binary representation of Arrow/Parquet. Not wanting to deal with the reproducibility of storage formats I've been computing logical data hashes in memory. This is slow but flexible, e.g. my hash stays the same even if records are re-ordered (which I consider an equivalent dataset).

But when thinking about integrating with IPFS and other content-addressable storages that rely on hashes of files - it would simplify the design a lot to have just one hash (physical) instead of two (logical + physical), but this means I have to guarantee that Parquet files are reproducible.

I decided to continue using logical hashing for now.

I've created a new Rust crate arrow-digest that implements the stable hashing for Arrow arrays and record batches and tries hard to hide the encoding-related differences. The crate's README describes the hashing algorithm if someone finds it useful and wants to implement it in another language.

I'll continue to expand the set of supported types as I'm integrating it into the decentralized data processing tool I'm working on.

In the long term, I'm not sure logical hashing is the best way forward - a subset of Parquet that makes some efficiency sacrifices just to make file layout deterministic might be a better choice for content-addressability.