NDArray | A Multidimensional Array library for Swift | Machine Learning library

That's working as designed. from_tensor_slices is really only useful for small amounts of data. Dataset is designed for large datasets that need to be streamed from disk.

The hard way but ideal way to do this would be to write your numpy array data to TFRecords then read them in as a dataset via TFRecordDataset. Here's the guide.

https://www.tensorflow.org/tutorials/load_data/tfrecord

The easier way but less performant way to do this would be Dataset.from_generator. Here is a minimal example:

This is a performance issue of the current Numpy implementation. I can also reproduce this problem on Windows (using an Intel Skylake Xeon processor with Numpy 1.20.3). np.logical_and(a, b) executes a very-inefficient scalar assembly code based on slow conditional jumps while np.logical_and(a.view(bool), b.view(bool)) executes relatively-fast SIMD instructions.

Currently, Numpy uses a specific implementation for bool-types. Regarding the compiler used, the general-purpose implementation can be significantly slower if the compiler used to build Numpy failed to automatically vectorize the code which is apparently the case on Windows (and explain why this is not the case on other platforms since the compiler is likely not exactly the same). The Numpy code can be improved for non-bool types. Note that the vectorization of Numpy is an ongoing work and we plan optimize this soon.

Here is the assembly code executed by np.logical_and(a, b):

As already answered by Michael Szczesny, more Pythonic way would be:

If the highest dimension contains always a small number of element like 6, then your code is not the best one.

First of all, myarray == 0, np.all and ~ creates temporary arrays that introduces some additional overhead as they needs to be written and read back. The overhead is dependent of the this of the temporary array and the biggest one is myarray == 0.

Moreover, Numpy calls perform some unwanted checks that Cython is not able to remove. These checks introduce a constant time overhead. Thus, is can be quite big for small input arrays but not big input arrays.

Additionally, the code of np.all can be faster if it would know the exact size of the last dimension which is not the case here. Indeed, the loop of np.all could theoretically be unrolled since the last dimension is small. Unfortunately, Cython does not optimize Numpy calls and Numpy is compiled for a variable input size, so not known at compile-time.

Finally, the computation can be parallelized if lenpropen is huge (otherwise this will not be faster and could actually be slower). However, note that a parallel implementation requires the computation to be done in two steps: np.all(myarray == 0, axis=1) needs to be computed in parallel and then you can create the resulting array and write it by computing myarray[~result] in parallel. In sequential, you can directly overwrite myarray by filtering lines in-place and then produce a view of the filtered lines. This pattern is known as the erase-remove idiom. Note that this assume the array is contiguous.

To conclude, a faster implementation consists writing 2 nested loops iterating on myarray with a constant number of iterations for the innermost one. Regarding the size of lenpropen, you can either use a sequential in-place implementation base on the erase-remove idiom, or a parallel out-of-place implementation with two steps (and a temporary array).

Yes, there's no reason why constexpr std::initializer_list would be unusable in compile-time initialization.

From your code snippet, it's unclear whether you've used an in-class initialization for StaticArray members, so one of the issues you could've run into is that a constexpr constructor can't use a trivial constructor for members which would initialize them to unspecified run-time values.

So the fix for your example is to default-initialize StaticArray members and specify constexpr for the constructor, checkDims, addList and data. To initialize a runtime StaticArray with constexpr std::initializer_list validated at compile-time, you can make the initializer expression manifestly constant-evaluated using an immediate function.

As you probably realize, it is impossible to initialize a run-time variable at compile-time so that's the best one can do.

If what you wanted is to validate at compile-time the dimensions of an std::initializer_list that depends on runtime variables, it can't be done -- the std::initializer_list is not constexpr, so its size isn't either. Instead, you can define a wrapper type around Scalar, mark its default constructor as deleted, and accept an aggregate type of these wrappers in the StaticArray constructor, for example a nested std::array of the desired dimensions, or, to avoid double braces, a C-style multidimensional array. Then, if the dimensions don't match, compilation will fail for either of the two reasons: too many initializers or the use of the deleted default constructor.

The code below compiles on godbolt with every GCC, Clang, MSVC version that supports C++20.

You can't change this behavior in the API, either when loading the parquet file into an arrow table or converting the arrow table to pandas.

But you can write your own function that would look at the schema of the arrow table and convert every list field to a python list

Here, copying of data doesn't play a big role: the bottle neck is fast how the tanh-function is evaluated. There are many algorithms: some of them are faster some of them are slower, some are more precise some less.

Different numpy-distributions use different implementations of tanh-function, e.g. it could be one from mkl/vml or the one from the gnu-math-library.

Depending on numba version, also either the mkl/svml impelementation is used or gnu-math-library.

The easiest way to look inside is to use a profiler, for example perf.

For the numpy-version on my machine I get:

constinit has exactly and only one semantic: the expression used to initialize the variable must be a constant expression.

That's it. That's all it does: it causes a compile error if the initializing expression isn't a valid constant expression. In every other way, such a variable is identical to not having constinit there. Indeed, the early versions of the proposal had constinit as an attribute rather than a keyword, since it didn't really do anything. It only gave a compile error for invalid initialization of the variable.

For this case, there is no reason not to make the variable constexpr. You clearly don't intend to change the variable, so there's no point in making the variable modifiable.

For this problem, I don't recommend using Kmeans color quantization since this technique is usually reserved for a situation where there are various colors and you want to segment them into dominant color blocks. Take a look at this previous answer for a typical use case. Since your CT scan images are grayscale, Kmeans would not perform very well. Here's a potential solution using simple image processing with OpenCV:

1. Obtain binary image. Load input image, convert to grayscale, Otsu's threshold, and find contours.

2. Create a blank mask to extract desired objects. We can use np.zeros() to create a empty mask with the same size as the input image.

3. Filter contours using contour area and aspect ratio. We search for the lung objects by ensuring that contours are within a specified area threshold as well as aspect ratio. We use cv2.contourArea(), cv2.arcLength(), and cv2.approxPolyDP() for contour perimeter and contour shape approximation. If we have have found our lung object, we utilize cv2.drawContours() to fill in our mask with white to represent the objects that we want to extract.

4. Bitwise-and mask with original image. Finally we convert the mask to grayscale and bitwise-and with cv2.bitwise_and() to obtain our result.

Here is our image processing pipeline visualized step-by-step:

Grayscale -> Otsu's threshold

Detected objects to extract highlighted in green -> Filled mask

Bitwise-and to get our result -> Optional result with white background instead

Code