DataGenerator | generating various test data for machine learning | Machine Learning library

I have a dataset that includes video frames partially 1000 real videos and 1000 deep fake videos. each video after preprocessing phase converted to the 300 frames in other worlds I have a dataset with 300000 images with Real(0) label and 300000 images with Fake(1) label. I want to train MesoNet with this data. I used costum DataGenerator class to handle train, validation, test data with 0.8,0.1,0.1 ratios but when I run the project show this message:

I am using Databricks Labs Data Generator to send synthetic data to Event Hub.

Everything appears to be working fine for a about two minutes but then the streaming stops and provides the following error:

I set aside a validation split like below:

This is my first question on stack overflow. I apologise in advance for the poor formatting and indentation due to my troubles with the interface.

Environment specifications:

Tensorflow version - 2.7.0 GPU (tested and working properly)

Python version - 3.9.6

CPU - Intel Core i7 7700HQ

GPU - NVIDIA GTX 1060 3GB

RAM - 16GB DDR4 2400MHz

HDD - 1TB 5400 RPM

Problem Statement:

I wish to train a TensorFlow 2.7.0 model to perform multilabel classification with six classes on CT scans stored as DICOM images. The dataset is from Kaggle, link here. The training labels are stored in a CSV file, and the DICOM image names are of the format ID_"random characters".dcm. The images have a combined size of 368 GB.

Approach used:

1. The CSV file containing the labels is imported into a pandas DataFrame and the image filenames are set as the index.

2. A simple data generator is created to read the DICOM image and the labels by iterating on the rows of the DataFrame. This generator is used to create a training dataset using tf.data.Dataset.from_generator. The images are pre-processed using bsb_window().

3. The training dataset is shuffled and split into a training(90%) and validation set(10%)

4. The model is created using Keras Sequential, compiled, and fit using the training and validation datasets created earlier.

code:

I have an issue training my tensorflow model which is seemingly related to batch size. If I set the batch size to 1 it executes fine.

If I set the batch size to 6 and provide 13 records I receive this error:

I have a function used to create an object graph in my app for testing purposes. The data structure is very simple at present with a one-to-many relationship between Patient and ParameterMeasurement entities.

As setup of the test state involves around 800 entries it makes sense to do this as a batch insert which works...until you try and establish the relationship between ParameterMeasurement and Patient (which, in the reciprocal, is a one-to-one) at which point the app crashes with the dreaded "Illegal attempt to establish a relationship 'cdPatient' between objects in different contexts"

I'm struggling to understand why this is happening as both Patient and ParameterMeasurement entities are created using the same managed object context which is passed to the function by the caller.

I've already tried to store the objectID of the Patient (created before instantiating ParameterMeasurement instances) and then creating a local copy of the Patient instance inside the batch insert closure (code in place below but commented out) but this does not resolve the issue. I've also checked my model (all OK, relationships are good), deleted the app and reset the sim but still no joy.

Finally, I've stuck in print statements to check the MOCs associated with both entities at the point of instantiation and the MOC passed to the function. As expected, the memory addresses match which makes it look like the error message is a red herring.

Can anyone point me in the right direction? This seems to have been a common issue in the past (lots of posts 5y+ ago with ObjC but little in Swift) but the examples on don't deal with this specific scenario.

I have the following tensorflow model:

I'm currently trying to understand mpi4py. I set mpi4py.rc.initialize = False and mpi4py.rc.finalize = False because I can't see why we would want initialization and finalization automatically. The default behavior is that MPI.Init() gets called when MPI is being imported. I think the reason for that is because for each rank a instance of the python interpreter is being run and each of those instances will run the whole script but that's just guessing. In the end, I like to have it explicit.

Now this introduced some problems. I have this code

I want to implement my own custom datagenerator for at multi-input keras model i have built using the functional api from keras.

I have read alot about the sequence class and how i can extend it's functionality i various ways.

My dataset i heavily unbalanced containing 3 classes.

What i want to achieve is build a custom datagenerator which uses flowfromdataframe. This dataframe contains paths to images. By constraining the number of image paths from the overrepresented class directory i can successfully undersample and thereby balance the dataset.

Dataframe structure:

However the remaining images i leave out still contain rich information i want my model to learn.

Is it possible to use something like a a callback "onepochend" that calls a function in my imagedatagenerator which swaps out the old paths in dataframe and replace it with random selected new paths?

Callback keras docs: https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback

Generator class docs: https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence

Sketched my idea:

Or do tensorflow/keras has something that achieves this?