bottleneck | Job scheduler and rate limiter, supports Clustering | Runtime Evironment library

I have a numerical routine that I need to run to solve a certain equation, which contains a few nested four loops. I initially wrote this routine into Python, using numba.jit to achieve an acceptable performance. For large system sizes however, this method becomes quite slow, so I have been rewriting the routine into Fortran hoping to achieve a speed-up. However I have found that my Fortran version is much slower than the first version in Python, by a factor of 2-3.

I believe the bottleneck is a linear interpolation function that is called at each innermost loop. In the Python implementation I use numpy.interp, which seems to be pretty fast when combined with numba.jit. In Fortran I wrote my own interpolation function, which reads,

I have a local python project called jive that I would like to use in an another project. My current method of using jive in other projects is to activate the conda env for the project, then move to my jive directory and use python setup.py install. This works fine, and when I use conda list, I see everything installed in the env including jive, with a note that jive was installed using pip.

But what I really want is to do this with full conda. When I want to use jive in another project, I want to just put jive in that projects environment.yml.

So I did the following:

1. write a simple meta.yaml so I could use conda-build to build jive locally
2. build jive with conda build .
3. I looked at the tarball that was produced and it does indeed contain the jive source as expected
4. In my other project, add jive to the dependencies in environment.yml, and add 'local' to the list of channels.
5. create a conda env using that environment.yml.

When I activate the environment and use conda list, it lists all the dependencies including jive, as desired. But when I open python interpreter, I cannot import jive, it says there is no such package. (If use python setup.py install, I can import it.) How can I fix the build/install so that this works?

Here is the meta.yaml, which lives in the jive project top level directory:

I've implemented a custom validation message on my input for the pattern validation rule while leaving the default message for required as is. However, when I do so, once the input becomes invalid, it never becomes valid again, even though I am meeting the pattern criteria.

Consider the following code, running on an ARM Cortex-A72 processor (optimization guide here). I have included what I expect are resource pressures for each execution port:

Although uzp2 could run on either the F0 or F1 ports, I chose to attribute it entirely to F1 due to high pressure on F0 and zero pressure on F1 other than this instruction.

There are no dependencies between loop iterations, other than the loop counter and array pointers; and these should be resolved very quickly, compared to the time taken for the rest of the loop body.

Thus, my intuition is that this code should be throughput limited, and considering the worst pressure is on F0, run in 8 cycles per iteration (unless it hits a decoding bottleneck or cache misses). The latter is unlikely given the streaming access pattern, and the fact that arrays comfortably fit in L1 cache. As for the former, considering the constraints listed on section 4.1 of the optimization manual, I project that the loop body is decodable in only 8 cycles.

Yet microbenchmarking indicates that each iteration of the loop body takes 12.5 cycles on average. If no other plausible explanation exists, I may edit the question including further details about how I benchmarked this code, but I'm fairly certain the difference can't be attributed to benchmarking artifacts alone. Also, I have tried to increase the number of iterations to see if performance improved towards an asymptotic limit due to startup/cool-down effects, but it appears to have done so already for the selected value of 128 iterations displayed above.

Manually unrolling the loop to include two calculations per iteration decreased performance to 13 cycles; however, note that this would also duplicate the number of load and store instructions. Interestingly, if the doubled loads and stores are instead replaced by single LD1/ST1 instructions (two-register format) (e.g. ld1 { v3.4s, v4.4s }, [x1], #32) then performance improves to 11.75 cycles per iteration. Further unrolling the loop to four calculations per iteration, while using the four-register format of LD1/ST1, improves performance to 11.25 cycles per iteration.

In spite of the improvements, the performance is still far away from the 8 cycles per iteration that I expected from looking at resource pressures alone. Even if the CPU made a bad scheduling call and issued uzp2 to F0, revising the resource pressure table would indicate 9 cycles per iteration, still far from actual measurements. So, what's causing this code to run so much slower than expected? What kind of effects am I missing in my analysis?

EDIT: As promised, some more benchmarking details. I run the loop 3 times for warmup, 10 times for say n = 512, and then 10 times for n = 256. I take the minimum cycle count for the n = 512 runs and subtract from the minimum for n = 256. The difference should give me how many cycles it takes to run for n = 256, while canceling out the fixed setup cost (code not shown). In addition, this should ensure all data is in the L1 I and D cache. Measurements are taken by reading the cycle counter (pmccntr_el0) directly. Any overhead should be canceled out by the measurement strategy above.

I've got a function func that may cost ~50s when running on a single core. Now I want to run it on a server which has got 192-core CPUs for many times. But when I add worker processes to say, 180, the performance of each core slows down. The worst CPU takes ~100s to calculate func.

Can someone help me, please?

Here is the pseudo code

In a cross platform (Linux and windows) real-time application, I need the fastest way to share data between a C++ process and a python application that I both manage. I currently use sockets but it's too slow when using high-bandwith data (4K images at 30 fps).

I would ultimately want to use the multiprocessing shared memory but my first tries suggest it does not work. I create the shared memory in C++ using Boost.Interprocess and try to read it in python like this:

Is there a faster way to do the following, where in the real application, df has many rows (and therefore list_of_colnames has the same number of elements):

I have the following class:

Let X be a set of distinct 64-bit unsigned integers std::uint64_t, each one being interpreted as a bitset representing a subset of {1,2,...,64}.

I want a function to do the following: given a std::uint64_t A, not necessarily in X, list all B in X, such that B is a subset of A, when A, B are interpreted as subsets of {1,2,...,64}.

(Of course, in C++ this condition is just (A & B) == B).

Since A itself need not be in X, I believe that this is not a duplicate of other questions.

X will grow over time (but nothing will be deleted), although there will be far more queries than additions to X.

I am free to choose the data structure representing the elements of X.

Obviously, we could represent X as a std::set or sorted std::vector of std::uint64_t, and I give one algorithm below. But can we do better?

What are good data structures for X and algorithms to do this efficiently? This should be a standard problem but I couldn't find anything.

EDIT: sorry if this is too vague. Obviously, if X were a std::set we could search through all subsets of A, taking time O(2^m log |X|) with m <= N, or all elements of X in time O(|X| log |X|).

Assume that in most cases, the number of B is quite a bit smaller than both 2^m (the number of subsets of A) and |X|. So, we want some kind of algorithm to run in time much less than |X| or 2^m in such cases, ideally in time O(number of B) but that's surely too optimistic. Obviously, O(|X|) cannot be beaten in the worst case.

Obviously some memory overhead for X is expected, and memory is less of a bottleneck than time for me. Using memory roughly 10 * (the memory of X stored as a std::set) is fine. Much more than this is too much. (Asymptotically, anything more than O(|X|) or O(|X| log |X|) memory is probably too much).

Obviously, the use of C++ is not essential: the algorithms/data structures are the important things here.

In the case that X is fixed, maybe Hasse diagrams could work.

It looks like Hasse diagrams would be quite time-consuming to construct each time X grows. (But still maybe worth a try if nothing else comes up). EDIT: maybe not so slow to update, so better than I thought.

The below is just my idea so far; maybe something better can be found?

FINAL edit: since it's closed, probably fairly - the "duplicate" question is pretty close - I won't bother with any further edits. I will probably do the below, but using a probabilistic skip list structure instead of a std::set, and augmented with skip distances (so you can quickly calculate how many X elements remain in an interval, and thus reduce the number of search intervals, by switching to linear search when the intersection gets small). This is similar to Order Statistic Trees given in this question, but skip lists are a lot easier to reimplement than std::set (especially as I don't need deletions).

Represent X as a std::set or sorted std::vector of 64-bit unsigned integers std::uint64_t, using the ordinary numerical order, and do recursive searches within smaller and smaller intervals.

E.g., my query element is A = 10011010. Subsets of A containing the first bit lie in the inclusive interval [10000000, 10011010].

Subsets of A containing the second bit but not the first lie in the interval [00010000, 00011010].

Those with the third but not the second bit are in [00001000, 00001010].

Those with the fourth but not the third bit are in [00000010, 00000010].

Now, within the first interval [10000000, 10011010] you could make two subintervals to search, based on the second bit: [10000000, 10001010] and [10010000, 10011010].

Thus you can break it down recursively in this manner. The total length of search intervals is getting smaller all the time, so this is surely going to be better asymptotically than a trivial linear search through all of X.

E.g., if X = {00000010, 00001000, 00110111, 10011100} then only the first, third, fourth depth-1 intervals would have nonempty intersection with X. The final returned result would be [00000010, 00001000].

Of course this is unbalanced if the X elements are distributed fairly uniformly. We might want the search intervals to have roughly equal width at each depth, and they don't; above, the sizes of the four depth-1 search intervals are, I think, 27, 11, 3, 1, and for larger N the discrepancies could be much bigger.

If there are k bits in the query set A, then you'll have to construct k initial search intervals at depth 1 (searching on ONE bit), then up to 2k search intervals at depth 2, 4k at depth 3, etc.

If I've got it right, since log |X| = O(N) the number of search intervals is O(k + 2k + 4k + ... + 2^n . k) = O(k^2) = O(N^2), where 2^n = O(k), and each one takes O(N) time to construct (actually a bit less since it's the log of a smaller number, but the log doesn't increase much), so it seems like this is an O(N^3) algorithm to construct the search intervals.

Of course the full algorithm is not O(N^3), because each interval may contain many elements, so listing them all cannot be better than O(2^N) in general, but let's ignore this and assume that there are not enough elements of X to overwhelm the O(N^3) estimate.

Another issue is that std::map cannot tell you how many elements lie within an interval (unlike a sorted std::vector) so you don't know when to break off the partitioning and search through all remaining X elements in the interval. Of course, you have an upper bound on the number of X elements (the size of the full interval) but it may be quite poor.

EDIT: the answer to another question shows how to have a std::set-like structure which also quickly gives you the number of elements in a range, which obviously could be adapted to std::map-like structures. This would work well here for pruning (although annoying that, for C++, you'd have to reimplement most of std::map!)