semblance | Supports | Reverse Engineering library

Here is a tidyverse approach. First, reshape to a long df and aggregate the groups. Then, reshape the variables back to a wide df.

You can use eval. Just be careful with this command though. It can be devastating. Unfortunately unlike bash, sh does not support declare.

The set type in python is not explicitly ordered. It can appear ordered based on the implementation, but is not guaranteed to be so. If you need a ordered representation, you should use something like sorted(set(input_sequence)) which will return a sorted list after removing the duplicates. Note that sorting lists with types that are not comparable is not supported without some sort of custom comparator (so you can't sort ['a', 1] out of the box).

If you want to share a list of custom objects (such as particle in the question) among processes, you can consider a simplified example here:

Is it safe to assume that every document created in a given collection will have a unique timestamp and there would be no collision?

No, it's not safe to assume that. But it's also extremely unlikely that there will be a collision, depending on how the writes actually occur. If you need a guaranteed order, add another random piece of data to the document in another field, and use its sort order to break any ties in a deterministic fashion. You will have to decide for yourself if this is worthwhile for your use case.

Does Firestore queue up the writes for a collection or are the writes executed in parallel?

You should consider all writes to be in parallel. No guarantees are made about the order of writes, as that does not scale well at all.

Does Amazon AWS S3 Glacier support some semblance of file hierarchy inside a Vault for Archives?

No, there's no hierarchy other than "archives exist inside a vault".

For example, in AWS S3, objects are given hierarchy via /. For example: all_logs/some_sub_category/log.txt

This is actually incorrect.

S3 doesn't have any inherent hierarchy. The character / is absolutely no different than any other character valid for the key of an S3 Object.

The S3 Console — and most S3 client tools, including AWS's CLI — treat the / character in a special way. But notice that it is a client-side thing. The client will make sure that listing happens in such a way that a / behaves as most people would expect, that is, as a "hierarchy separator".

If file hierarchy inside S3 Glacier is possible, can you provide brief instructions for how to do so?

You need to keep track of your hierarchy separately. For example, when you store an archive in Glacier, you could write metadata about that archive in a database (RDS, DynamoDB, etc).

As a side note, be careful about .tar.gz in Glacier, especially if you're talking about (1) a very large archive (2) that is composed of a large number of small individual files (3) which you may want to access individually.

If those conditions are met (and in my experience they often are in real-world scenarios), then using .tar.gz will often lead to excessive costs when retrieving data.

The reason is because you pay per number of requests as well as per size of request. So while having one huge .tar.gz file may reduce your costs in terms of number of requests, the fact that gzip uses DEFLATE, which is a non-splittable compression algorithm, means that you'll have to retrieve the entire .tar.gz archive, decompress it, and finally get the one file that you actually want.

An alternative approach that solves the problem I described above — and that, at the same time, relates back to your question and my answer — is to actually first gzip the individual files, and then tar them together. The reason this solves the problem is that when you tar the files together, the individual files actually have clear bounds inside the tarball. And then, when you request a retrieval from glacier, you can request only a range of the archive. E.g., you could say, "Glacier, give me bytes between 105MB and 115MB of archive X". That way you can (1) reduce the total number of requests (since you have a single tar file), and (2) reduce the total size of the requests and storage (since you have compressed data).

Now, to know which range you need to retrieve, you'll need to store metadata somewhere — usually the same place where you will keep your hierarchy! (like I mentioned above, RDS, DynamoDB, Elasticsearch, etc).

Anyways, just an optimization that could save a tremendous amount of money in the future (and I've worked with a ton of customers who wasted a lot of money because they didn't know about this).

I have executed your code and I managed to reproduce your scenario, encountering the same problem that only some rows seem to be inserted.

The problem is that you are using a REPLACE statement. As stated in MySQL's documentation a REPLACE statement works exactly like an INSERT operation but if the PRIMARY KEY already exists said row is replaced instead of failing the query.

As per your shared code it seems like you are using the field code as PRIMARY KEY but you are checking the field id as an indicator of an inserted web page. What happens is that multiple rows have the same code, thus every time one is inserted the previous one is removed.

I have solved your issue simply making the field id the PRIMARY KEY instead of code. Once you change that (remember to drop the original table), you can run your code again and you will see no missing ids. You can verify it using:

Your current implementation treats each individual unit of the string as a std::string rather than as a single char, which introduces some unnecessary overhead. Here's a rewrite that uses chars: