Priori | A fast C dynamic_cast alternative

From a precision standpoint - not much difference.

To avoid avoid underflow (product becomes 0.0), use (big * scale) * scale; as scale * scale may become 0.

I now see "... and the scale is not that "small" to cause underflow when multiply themselves. " - oh well.

From the excellent C# 9.0 in a Nutshell book:

Waiting or signaling an AutoResetEvent or ManualResetEvent takes about one microsecond (assuming no blocking).

ManualResetEventSlim and CountdownEvent can be up to 50 times faster in short-wait scenarios because of their nonreliance on the OS and judicious use of spinning constructs. In most scenarios, however, the overhead of the signaling classes themselves doesn't create a bottleneck; thus, it is rarely a consideration.

Hopefully that's enough to give you a rough order of magnitude.

You just need to create a row identifier, which you can do with dplyr and then use tidyr::pivot_wider() to generate all your resX variables.

You can use itertools.product():

You should use the github API rather than scraping the website. This way, you can get the file names and the download links into a nice two-column data frame by doing:

If the highest dimension contains always a small number of element like 6, then your code is not the best one.

First of all, myarray == 0, np.all and ~ creates temporary arrays that introduces some additional overhead as they needs to be written and read back. The overhead is dependent of the this of the temporary array and the biggest one is myarray == 0.

Moreover, Numpy calls perform some unwanted checks that Cython is not able to remove. These checks introduce a constant time overhead. Thus, is can be quite big for small input arrays but not big input arrays.

Additionally, the code of np.all can be faster if it would know the exact size of the last dimension which is not the case here. Indeed, the loop of np.all could theoretically be unrolled since the last dimension is small. Unfortunately, Cython does not optimize Numpy calls and Numpy is compiled for a variable input size, so not known at compile-time.

Finally, the computation can be parallelized if lenpropen is huge (otherwise this will not be faster and could actually be slower). However, note that a parallel implementation requires the computation to be done in two steps: np.all(myarray == 0, axis=1) needs to be computed in parallel and then you can create the resulting array and write it by computing myarray[~result] in parallel. In sequential, you can directly overwrite myarray by filtering lines in-place and then produce a view of the filtered lines. This pattern is known as the erase-remove idiom. Note that this assume the array is contiguous.

To conclude, a faster implementation consists writing 2 nested loops iterating on myarray with a constant number of iterations for the innermost one. Regarding the size of lenpropen, you can either use a sequential in-place implementation base on the erase-remove idiom, or a parallel out-of-place implementation with two steps (and a temporary array).

A function can be created to extract the desired matrix for a given array and vector.

the abstract state machine

The abstract machine is a term used by the formal C standard to describe the core about how a C program is supposed to behave, particularly in terms of code generation, order of execution and optimizations. It's a somewhat advanced topic so if you are a beginner, I'd advise to just ignore it for now. Otherwise, I wrote a brief explanation here.

Things that are not completely specified as such by the standard are, for example, how the sign of a signed type is represented the (sign representation), and the precision to which a double floating-point operation is performed (floating-point representation).

This refers to integers and floats having different sizes, different signedness formats, different endianess and so on depending on system. Meaning that those types are usually not portable.

C only imposes properties on representations such that the results of operations can be deduced a priori from two different sources:

• The values of the operands
• Some characteristic values that describe the particular platform

This is very broad, it doesn't mean much, basically just that in some cases the outcome of using an operator is well-defined by the language and in some cases it is not.

For example, the operations on the type size_t can be entirely determined when inspecting the value of SIZE_MAX in addition to the operands. We call the model to represent values of a given type on a given platform the binary representation of the type.

Generally, all information we need to determine that model is within reach of any C program: the C library headers provide the necessary information through named values (such as SIZE_MAX), operators, and function calls.

This probably means that for example we can check if an operator applied to operands of size_t will give the expected result or not:

Yes, Databricks will take implicit advantage of this partitioning through data skipping because there will be min/max statistics associated with specific data files. The min/max information will be loaded into memory from the transaction log, but it will need to make decision which files it need to hit on every query. But because everything is in memory, it shouldn't be very big performance overhead, until you have hundreds of thousands files.

One thing that you may consider - use generated column instead of explicit date column. Declare it as date GENERATED ALWAYS AS (CAST(timestampColumn AS DATE)), and partition by it. The advantage is that when you're doing a query on timestampColumn, then it should do partition filtering on the date column automatically.