pyastro | Python astronomical library | Dataset library

I was taking a look at Hub—the dataset format for AI—and noticed that hub integrates with GCP and AWS. I was wondering if it also supported integrations with MinIO.

I know that Hub allows you to directly stream datasets from cloud storage to ML workflows but I’m not sure which ML workflows it integrates with.

I would like to use MinIO over S3 since my team has a self-hosted MinIO instance (aka it's free).

I have a map-stype dataset, which is used for instance segmentation tasks. The dataset is very imbalanced, in the sense that some images have only 10 objects while others have up to 1200.

How can I limit the number of objects per batch?

A minimal reproducible example is:

I'm tackling a exercise which is supposed to exactly benchmark the time complexity of such code.

The data I'm handling is made up of pairs of strings like this hbFvMF,PZLmRb, each string is present two times in the dataset, once on position 1 and once on position 2 . so the first string would point to zvEcqe,hbFvMF for example and the list goes on....

example dataset of 50k pairs

I've been able to produce code which doesn't have much problem sorting these datasets up to 50k pairs, where it takes about 4-5 minutes. 10k gets sorted in a matter of seconds.

The problem is that my code is supposed to handle datasets of up to 5 million pairs. So I'm trying to see what more I can do. I will post my two best attempts, initial one with vectors, which I thought I could upgrade by replacing vector with unsorted_map because of the better time complexity when searching, but to my surprise, there was almost no difference between the two containers when I tested it. I'm not sure if my approach to the problem or the containers I'm choosing are causing the steep sorting times...

Attempt with vectors:

I'm trying to load the DomainNet dataset into a tensorflow dataset. Each of the domains contain two .txt files for the training and test data respectively, which is structured as follows:

I have a csv dataset with the values 0-1 for the features of the elements. I want to iterate each cell and replace the values 1 with the name of its column. There are more than 500 thousand rows and 200 columns and, because the table is exported from another annotation tool which I update often, I want to find a way in Python to do it automatically. This is not the table, but a sample test which I was using while trying to write a code I tried some, but without success. I would really appreciate it if you can share your knowledge with me. It will be a huge help. The final result I want to have is of the type: (abonojnë, token_pos_verb). If you know any method that I can do this in Excel without the help of Python, it would be even better. Thank you, Brikena

I want to download only person class and binary segmentation from COCO dataset. How can I do it?

I have a vector of words, like the below. This is an oversimplification, my real vector is over 600 words:

I have an image dataset that looks like this: Dataset

The timestep of each image is 15 minutes (as you can see, the timestamp is in the filename).

Now I would like to group those images in 3hrs long sequences and save those sequences inside subfolders that would contain respectively 12 images(=3hrs). The result would ideally look like this: Sequences

I have tried using os.walk and loop inside the folder where the image dataset is saved, then I created a dataframe using pandas because I thought I could handle the files more easily but I think I am totally off target here.

I've got a huge CSV file, which looks like this: