npy | Save and load NumPy npy and npz files in Ruby | Data Manipulation library

I have this 4D array (numpy.ndarray) that I need to save in a way that its format does not change as I save it (since it should remain unchanged), and then reuse it in my Google Colab file. I have tried saving it in different formats and when I upload it and preview it within my code, the previous format is no longer preserved even when I save it in the .npy format. I have also tried importing the date using the raw link from my GitHub repository or uploading it from my local device, but still no chance. I would appreciate your comments regarding the issue!

Further elaboration:

Here is the code that I use to generate my 4D array:

I tried reading into R a boolean vector stored as a numpy array (.npy) with RcppCNPy package like this:

I am getting some results in the form of np.arrays of size (no_of_rows, 2), where no_of_rows varies from array to array.

I want to save these results to disk for later usage (plotting). In any way, it doesn't matter how, but if possible in the most efficient way in terms of REOPENING/USING time-consumed-wise.

I can do anything with these 2D arrays. My thought was to create a .npy file containing a 3D tensor of shape (no_of_matrices, ? , 2), but that ? is my problem. Each 2D array can potentially have a different no_of_rows. So I abandoned this idea.

I now made a list of these 2D np.arrays. I now want to save that list to disk. I have read about np.savez, but this doesn't preserve the order of saving. That is, when loading back in memory, I won't know which array is which, apart from many if's statements to check the no_of_lines of each array and match it to what it means for when plotting it later.

Do I have any chance to store these arrays in 1 SINGLE file, or do I just have to resort to creating multiple files, named in a distinctive way, one for each 2D array, and then access these files individually when plotting?

[PS: I can only have a maximum of 100-ish of such 2D arrays, usually having less than 10.]

Thanks

I have the following code:

I need to do Semantic image segmentation based on Unet.

I have to work with Pascal VOC 2012 dataset, however I don't know how to do it, do I manually select images for the train & val and convert them into numpy and then load them into the model? Or is there another way?

If this is the first one I would like to know how to convert all the images present in a folder into .npy.

I have a folder in which I have 100+ .npy files. The path to this folder is '/content/drive/MyDrive/lung_cancer/subset0/trainImages'.

This folder has the .npy files as shown in the image the .npy files

The shape of each of these .npy files is (3,512,512)

I want to combine all of these files into one single file with the name trainImages.npy so that I can train my unet model with it.

My unet model takes input of the shape (1,512,512). I will load the above trainImages.npy file into imgs_train as below to pass it as input into unet model

imgs_train = np.load(working_path+"trainImages.npy").astype(np.float32)

Can someone please tell me how do i concatenate all those .npy files into one single .npy file?? Thanks.

I'm currently training my first neural network on a larger dataset. I have splitted my training data to several .npy binary files, that each contain batches of 20k training samples. I'm loading the data from the npy files, apply some simple pre-processing operations, and then start to train my network by applying the partial_fit method several times in a loop:

I thought it's easy to deploy a python api project to somewhere. BUT I was wrong, I cannot deploy to any platforms.

So far I have tried:

1. Azure, Webapp and Function App
2. PythonAnywhere
3. Heroku

They all have issues when I'm trying to install dependency packages for this one: scikit-fmm

here is the error message:

Python Version is 3.7.10 Linux

I have this code: