duperemove | Tools for deduping file systems | Hashing library

I'm implementing my own class, with custom __eq__. And I'd like to return True for things that are not "equal" in a mathematical sense, but "match" in a fuzzy way.

An issue with this is, however, that this leads to loss of transitivity in a mathematical sense, i.e. a == b && b ==c, while a may not be equal to c.

Question: is Python dependent on __eq__ being transitive? Will what I'm trying to do break things, or is it possible to do this as long as I'm careful myself not to assume transitivity?

I want to match telephone numbers with one another, while those may be either formatted internationally, or just for domestic use (without a country code specified). If there's no country code specified, I'd like a number to be equal to a number with one, but if it is specified, it should only be equal to numbers with the same country-code, or without one.

So:

• Of course, +31 6 12345678 should equal +31 6 12345678, and 06 12345678 should equal 06 12345678
• +31 6 12345678 should equal 06 12345678 (and v.v.)
• +49 6 12345678 should equal 06 12345678 (and v.v.)
• But +31 6 12345678 should not be equal to +49 6 12345678

Edit: I don't have a need for hashing (and so won't implement it), so that at least makes life easier.

I know that in hashing you, by definition, lose information. However, as email addresses can be restricted - such as with the information available I would know a potential domain of the email, and that it must have an @. Do these constraints change anything about the problem? Or is the best way to simply make a guess and see if the hash is the same? Also MD5 is no longer as secure as it once was.

Thanks

I'm trying to learn Go while writing a small program. The program should parse a PATH recursivelys as efficient and fast as possible and output the full filename (with the path included) and the sha256 file hash of the file.

If the file hashing generates fails, I wanna keep the error and add it to the string (at the hash position).

The result should return a string on the console like: fileXYZ||hash

Unfortunately, the programs hangs at some point. I guess some of my channels are not closing properly and waiting indefinitely for input. I've been trying for quite some time to fix the problem, but without success.

Does anyone have an idea why the output hangs? Many many thx in advance, any input/advice for a Go newcomer is welcome too ;-).

(I wrote separate functions as I wanna add additional features after having fixed this issue.)

Thanks a lot! Didier

Here is the code:

Assume we have two tables (think as in SQL tables), where the primary key in one of them is the foreign key in the other. I'm supposed to write a simple algorithm that would imitate the joining of these two tables. I thought about iterating over each element in the primary key column in the first table, having a second loop where it checks if the foreign key matches, then store it in an external array or list. However, this would take O(N*M) and I need to find something better. There is a hint in the textbook that it involves hashing, however, I'm not sure how hashing could be implemented here or how it would make it better?

Editing to add an example:

I'm seeking advice from people deeply familiar with the binary layout of Apache Parquet:

Having a data transformation F(a) = b where F is fully deterministic, and same exact versions of the entire software stack (framework, arrow & parquet libraries) are used - how likely am I to get an identical binary representation of dataframe b on different hosts every time b is saved into Parquet?

In other words how reproducible Parquet is on binary level? When data is logically the same what can cause binary differences?

• Can there be some uninit memory in between values due to alignment?
• Assuming all serialization settings (compression, chunking, use of dictionaries etc.) are the same, can result still drift?

I'm working on a system for fully reproducible and deterministic data processing and computing dataset hashes to assert these guarantees.

My key goal has been to ensure that dataset b contains an idendital set of records as dataset b' - this is of course very different from hashing a binary representation of Arrow/Parquet. Not wanting to deal with the reproducibility of storage formats I've been computing logical data hashes in memory. This is slow but flexible, e.g. my hash stays the same even if records are re-ordered (which I consider an equivalent dataset).

But when thinking about integrating with IPFS and other content-addressable storages that rely on hashes of files - it would simplify the design a lot to have just one hash (physical) instead of two (logical + physical), but this means I have to guarantee that Parquet files are reproducible.

I decided to continue using logical hashing for now.

I've created a new Rust crate arrow-digest that implements the stable hashing for Arrow arrays and record batches and tries hard to hide the encoding-related differences. The crate's README describes the hashing algorithm if someone finds it useful and wants to implement it in another language.

I'll continue to expand the set of supported types as I'm integrating it into the decentralized data processing tool I'm working on.

In the long term, I'm not sure logical hashing is the best way forward - a subset of Parquet that makes some efficiency sacrifices just to make file layout deterministic might be a better choice for content-addressability.

I have an Angular 12 application that has different build environments (dev/staging/prod) and I have configured these with output hashing on in angular.json:

Trying to understand how to get and then save user in client (using JWT token in Http only cookie), so that I can do conditional rendering. What I'm having difficulty with is how to continously know if the user is logged in or not, without having to send a request to the server each time the user changes/refresh page. (Note: the problem is not how do I get the token in the Http only cookie, I know that this is done through withCredentials: true)

So my problem is how do you get/store the access token so that the client will not have to make a request to the server each time the user does something on the website. For example the Navbar should do conditional renderingen depending on if the user is logged in or not, then I don't want to do "ask the server if the user has a access token, then if not check if user has refresh token, then return a new access token if true else redirect to login page" every single time the user switches page.

Client:

UserContext.js

I need web app with base url as

I have an Ionic project I'm working with that is having trouble building to Android. I inherited this project, so that's why I'm not 100% familiar with Fastlane and how it's building the java files. Additionally, I'm on WSL2 and using sdkmanager with the following installed packages: