H5F | Deprecated , please use hyperform instead https

Your code is looking in the current directory which is not where the file is.

Based on the error message, it looks like you are on windows. Is the file 'train_catvnoncat.h5' in your Downloads folder? Find that file on your system and copy the full path. You can then update this:

Okay. So, the problem is the size mismatch which occurs because np expects us to have values which fill all the dimensions. A quick fix is making the values in your 1st-3rd dimensions, for Mes_Press as 0 (Default).

A sample code is as follows:

The error indicates that dataset 'data' doesn't exist in your HDF5 file. So, we have to figure out why it's not there. :-) You didn't say where you found the example you are running. The one I found in the pyqtgraph/examples repository has code to create the file in function def createFile(finalSize=2000000000):.
I assume you ran this code to create test.hdf5?
If you didn't create the file with the example code, where did you get test.hdf5?
Either way, here is some code to interrogate your HDF5 file. It will give us dataset names and attributes (shape and dtype). With that info, we can determine the next steps.

Yes, that behavior is expected. Take a look at this answer to see more detailed example of the same behavior: How does HDF handle the space freed by deleted datasets without repacking. Note that the space will be reclaimed/reused if you add new datasets.

To reclaim the unused space in the file, you have to use a command line utility. There are 2 choices: ptrepack and h5repack: Both are used for a number of external file operations. To reduce file size after object deletion, create a new file from the old one as shown below:

• ptrepack utility delivered with PyTables.
  • Reference here: PyTables ptrepack doc
  • Example: ptrepack file1.h5 file2.h5 (creates file2.h5 from file1.h5)
• h5repack utility from The HDF Group.
  • Reference here: HDF5 h5repack doc
  • Example: h5repack [OPTIONS] file1.h5 file2.h5 (creates file2.h5 from file1.h5)

Both have options to use a different compression method when creating the new file, so are also handy if you want to convert from compressed to uncompressed (or vice versa)

As it was already pointed in the comments - your from_pretrained param should be either id of a model hosted on huggingface.co or a local path:

A path to a directory containing model weights saved using save_pretrained(), e.g., ./my_model_directory/.

See documentation

Looking at your stacktrace it seems like your code is run inside:

/content/drive/My Drive/msc-project/code/model.py so unless your model is in: /content/drive/My Drive/msc-project/code/input/bert-base-multilingual-cased/ it won't load.

I would also set the path to be similar to documentation example ie:

bert = TFBertModel.from_pretrained("./input/bert-base-multilingual-cased/")

The weights file extensions should be changed from .hdf5.index and .hdf5.data . . . to just .index and .data . . ..

The call for inference should then be modified accordingly to exclude the .hdf5 part, so e.g.

Your model will forget previous dataset as you train on new dataset.

I read in reinforcement learning, when game are used to train Deep Reinforcement Learning (DRL), then you have to create memory replay, which collect data from different rounds of game, because each round of game has different data, then randomly some of that data is chosen to train model. that way DRL model can learn to play different rounds of game without forgetting previous rounds.

You can try to create a single dataset by taking some random samples from each dataset.

When you train model on new dataset that make sure data from all previous rotation are in current rotation.

Also in transfer learning, when you train model on new dataset, you have to freeze previous layers so that model don`t forget previous training. you are not using transfer learning but still when you start training on 2nd dataset your 1st dataset will slowly be removed from memory of weights.

you can try freezing initial layers of decoder so that they are not updated when extracting feature, assuming all of the dataset contain similar images, that way your model will not forget previous training as in transfer learning. but still when you train on new dataset previous will be forgotten.

Hi first of all do you need to store your model or your model weights? To know the difference between that, model.save() save you weights and structure model and ... but model.save_weights() just save your weight model, I suggest you see this link for more information.

If you want to save the model, I suggest using model.save("test.hd5") or model.save(test.hdf5") and use tensorflow.kears.models.load_model("test.hd5") to load the model.

That's a big matrix (300GB if all ints). Likely you will have to write incrementally. (I don't have enough RAM on my system to do it all at one.)

Without seeing your data types, it's hard to give specific advice.
First question: do you really want to create a Table or will an Array suffice? PyTables has both types. What's the difference?
An Array holds homogeneous data (like a NumPy ndarray) and can have any dimension. An Table is typically used to hold heterogeneous data (like a NumPy recarray) and is always 2d (really a 1d array of structured types). Tables also support complex queries with the PyTables API.

The key when creating a Table is to either use the description= or obj= parameter to describe the structured types (and field names) for each row. I recently posted an answer that shows how to create a Table. Please review. You may find you don't want to create 200000 fields/columns to define the Table. See this answer: different data types for different columns of an array

If you just want to save a matrix of 200000x200000 homogeneous entities, an array is easier. (Given the data size, you probably need to use an EArray, so you can write the data in increments.) I wrote a simple example that creates an EArray with 2000x200000 entities, then adds 3 more sets of data (each 2000 rows; total of 8000 rows).

• The shape=(0,nrows) parameter indicates the first axis can be extended, and creates ncols columns.
• The expectedrows=nrows parameter is important in large datasets to improvie I/O performance.

The resulting HDF5 file is 6GB. Repeat earr.append(arr) 99 times to get 200000 rows. Code below: