itertools | Library - Python-inspired library | Widget library

Firstly, you can use a list comprehension to give you all of the possible combinations (regardless of the duplicates):

I would use following approach, where I find the index and use slicing to create under and over:

I think this gets you pretty close; might just need to adjust the various name columns and drop the extra data (I kept the grouping column).

The main idea is to recursively use pd.json_normalize with pd.concat for all availalable children levels.

EDIT: Put everything into a single function and added section to collapse the name columns like the expected output.

It is going to be quite hard to get numpy to go as fast as the filtered python iterator because numpy processes whole structures that will inevitably be larger than the result of filtering sets.

Here is the best I could come up with to process the product of arrays in such a way that the result is filtered on unique combinations of distinct values:

Numba currently uses LLVM-Lite to compile the code efficiently to a binary (after the Python code has been translated to an LLVM intermediate representation). The code is optimized like en C++ code would be using Clang with the flags -O3 and -march=native. This last parameter is very important as is enable LLVM to use wider SIMD instructions on relatively-recent x86-64 processors: AVX and AVX2 (possible AVX512 for very recent Intel processors). Otherwise, by default Clang and GCC use only the SSE/SSE2 instructions (because of backward compatibility).

Another difference come from the comparison between GCC and the LLVM code from Numba. Clang/LLVM tends to aggressively unroll the loops while GCC often don't. This has a significant performance impact on the resulting program. In fact, you can see that the generated assembly code from Clang:

With Clang (128 items per loops):

This isn't fully fleshed-out, and it's also a bit dorky:

This is an initial answer (which is incorrect, as I incorrectly understood the question, see edit below for a corrected answer).

A natural way to do it is:

I believe this will do it:

Once you know the items shared between the two lists, you can iterate over each list separately to sum up the items in between the shared items, thus constructing a list of partial sums. These lists will have the same length for both input lists, because the number of shared items is the same.

The maximum path sum can then be found by taking the maximum between the two lists for each stretch between shared values: