DeepLy | PHP client for the DeepL.com translation API | Translation library

I'm seeking advice from people deeply familiar with the binary layout of Apache Parquet:

Having a data transformation F(a) = b where F is fully deterministic, and same exact versions of the entire software stack (framework, arrow & parquet libraries) are used - how likely am I to get an identical binary representation of dataframe b on different hosts every time b is saved into Parquet?

In other words how reproducible Parquet is on binary level? When data is logically the same what can cause binary differences?

• Can there be some uninit memory in between values due to alignment?
• Assuming all serialization settings (compression, chunking, use of dictionaries etc.) are the same, can result still drift?

I'm working on a system for fully reproducible and deterministic data processing and computing dataset hashes to assert these guarantees.

My key goal has been to ensure that dataset b contains an idendital set of records as dataset b' - this is of course very different from hashing a binary representation of Arrow/Parquet. Not wanting to deal with the reproducibility of storage formats I've been computing logical data hashes in memory. This is slow but flexible, e.g. my hash stays the same even if records are re-ordered (which I consider an equivalent dataset).

But when thinking about integrating with IPFS and other content-addressable storages that rely on hashes of files - it would simplify the design a lot to have just one hash (physical) instead of two (logical + physical), but this means I have to guarantee that Parquet files are reproducible.

I decided to continue using logical hashing for now.

I've created a new Rust crate arrow-digest that implements the stable hashing for Arrow arrays and record batches and tries hard to hide the encoding-related differences. The crate's README describes the hashing algorithm if someone finds it useful and wants to implement it in another language.

I'll continue to expand the set of supported types as I'm integrating it into the decentralized data processing tool I'm working on.

In the long term, I'm not sure logical hashing is the best way forward - a subset of Parquet that makes some efficiency sacrifices just to make file layout deterministic might be a better choice for content-addressability.

I have a recursive type that extracts the indices from a nested object and puts them in a flat, strongly-typed tuple, like so:

I want to define deeply nested compositions of applicative functors. For example something like this:

I have a rather large Vue 3 application (~550 components). It takes two minutes just to run vue-cli-service serve and around 20 seconds to re-build it after each change. Hot reload stopped working a long time ago so it always needs to be refreshed in the browser even after a small style change. Moreover, the app is still not finished and it will probably get 2-3 times this big in the next year so it will probably be even worse.

Because of these problems, I've decided to migrate it from Vue CLI to Vite. I have already resolved a lot of problems and the app seems to work with Vite now with so much better loading times.

However, it sometimes gets stuck when I start a dev server (vite command) and open it in a browser. The page keeps loading and I can see a lot of pending requests in the Network tab of Chrome DevTools. There's nothing special in the output of vite --debug and running vite --force doesn't help either.

When this problem occurs, the browser always loads a lot of modules (~900) and then it gets stuck on 10-20 modules. The status of all these HTTP requests is simply Pending and they never finish. There are no errors in the browser or on the command line.

I don't think any particular file causes this. Maybe the problem is in my deeply nested folder structure with a lot of re-exports using index.ts file on each level. It mostly gets stuck on my own modules but I've also seen cases when it was waiting for a module of some external library.

Has anybody experienced a similar problem? How did you solve it?

EDIT: I have discovered that this issue only occurs in Chromium-based browsers (Google Chrome, Brave, etc.) on Linux. It works without any problems in Chrome on MacOS and Windows as well as in other browsers (Firefox, GNOME Web, etc.) on Linux.

I am writing a react application, and for some reason I can not wrap my head around this issue. Below is a snippet, the only thing that should be needed:

I want to select paths of a deeply nested map to keep.

For example:

How can I get the names of the leafs of a nested list (containing a dataframe)

While I was going through the haskell-excercises questions. I saw the following code which creates an aggregate Constraint by applying each type to a Constraint constructor. In GHC it seems deeply nested tuples of Constraints are still of kind Constraint (perhaps flattened ?).

I'm working on a streaming rules engine, and some of my customers have a few hundred rules they'd like to evaluate on every event that arrives at the system. The rules are pure (i.e. non-side-effecting) Boolean expressions, and they can be nested arbitrarily deeply.

Customers are creating, updating and deleting rules at runtime, and I need to detect and adapt to the population of rules dynamically. At the moment, the expression evaluation uses an interpreter over the internal AST, and I haven't started thinking about codegen yet.

As always, some of the predicates in the tree are MUCH cheaper to evaluate than others, and I've been looking for an algorithm or data structure that makes it easier to find the predicates that are cheap, and that are validly interpretable as controlling the entire expression. My mental headline for this pattern is "ANDs all the way to the root", i.e. any predicate for which all ancestors are ANDs can be interpreted as controlling.

Despite several days of literature search, reading about ROBDDs, CNF, DNF, etc., I haven't been able to close the loop from what might be common practice in the industry to my particular use case. One thing I've found that seems related is Analysis and optimization for boolean expression indexing but it's not clear how I could apply it without implementing the BE-Tree data structure myself, as there doesn't seem to be an open source implementation.

I keep half-jokingly mentioning to my team that we're going to need a SAT solver one of these days. 😅 I guess it would probably suffice to write a recursive algorithm that traverses the tree and keeps track of whether every ancestor is an AND or an OR, but I keep getting the "surely this is a solved problem" feeling. :)

Edit: After talking to a couple of friends, I think I may have a sketch of a solution!

1. Transform the expressions into Conjunctive Normal Form, in which, by definition, every node is in a valid short-circuit position.
2. Use the Tseitin algorithm to try to avoid exponential blowups in expression size as a result of the CNF transform
3. For each AND in the tree, sort it in ascending order of cost (i.e. cheapest to the left)
4. ???
5. Profit!^Weval as usual :)