array_split | Python package for decomposing multi | Data Manipulation library

What I can wrap-up after invesigating this myself:

1. joblib.Parallel is not obliged to terminate processes after successfull single invocation
2. Loky backend doesn't terminate workers physically and it is intentinal design explained by authors: Loky Code Line
3. If you want explicitly release workers you can use my snippet:

Here you just need to select the given elements. first row, first column and first row, second column

Assume vector length is always a multiple of 5, you can use reshape instead of array_split to convert the vector into a (5, 4) array and then use normal array indexing to remove the last two columns:

I think a good approach has been found. Credits to a colleague.

The idea is to sort the group sizes (in descending order) and put groups into bins in a "backward S"-pattern. Let me illustrate with an example. Assume n = 3 (number of bins) and the following data:

The problem is that cell_text is not reset to an empty list after each loop, so each successive table will also include the previous one(s). Anyway, cell_text is not actually needed as the cell values can be accessed with table.values.

In the following example, the figure dimensions are switched around to have a portrait orientation of the A4 pages to fit the tables on single pages. Also, the column to improve the format of the table. The pyplot interface is used exclusively so as to simplify the code a bit.

if the schema is unknown in advance, you could try something like this (using mv-apply, summarize make_bag() and bag_unpack())

vstack create a new matrix for every iteration. This is the main source of slowdown since the complexity of the algorithm is O(U^3). You can just append the new lines in a Python list and then vstack the list of lines. Alternatively, a better approach is just to use the following Numpy expression :

original_sparse_matrix[random_indices, :]

An interesting question, with a few possible solutions. As you indicated, it is possible to use np.array_split, but since we are only interested in the indices, we can also use np.unravel_index, which would mean that we only have to loop over all the indices (the size) of the array to get the index.

Now there are two great ideas for multiprocessing:

1. Create a (thread safe) shared memory of the array and splitting the indices across the different processes.
2. Only update the array in a main thread, but provide a copy of the required data to the processes and let them return the value that has to be updated.

Both solutions will work for any np.ndarray, but have different advantages. Creating a shared memory doesn't create copies, but can have a large insertion penalty if it has to wait on other processes (the computational time, is small compared to the write time.)

There are probably many more solutions, but I will work out the first solution, where a Shared Memory object is created and a range of indices is provided to every process.

Required imports:

Just ran your code, and it works if you use dist = np.concatenate(dist, axis=0) instead of dist = np.asarray(dist).ravel().reshape(num_row(data),-1) Same thing for memb