ttree | C # implementation of a T-Tree data structure | Dataset library

There are a lot of HDF5 I/O issues to consider. I will try to cover each.

From my tests, time spent doing I/O is primarily a function of the number of reads/writes and not how much data (in MB) you read/write. Read this SO post for more details: pytables writes much faster than h5py. Why? Note: it shows I/O performance for a fixed amount of data with different I/O write sizes for both h5py and PyTables. Based on this, it makes sense that most of the time is spent in the last line -- that's where you are reading the data from disk to memory as NumPy arrays (v1, v2, v3, v4, v5, v6).

Regarding your questions:

1. There's a reason there is no difference between reading d2['SimSignal_X'][0] and d2['SimSignal_X'][:]. Both read the entire dataset into memory (all ~1000 dataset values). If you only want to read a slice of the data, you need to use slice notation. For example, d2['SimSignal_X'][0:100] only reads the first 100 values (assumes d2['SimSignal_X'] only has a single axis -- shape=(1000,)). Note; reading a slice will reduce required memory, but won't improve I/O read time. (In fact, reading slices will probably increase read time.)
2. I am not familiar with CERN ROOT, so can't comment about performance of h5py vs ROOT. If you want to use Python, here are several things to consider:
   • You are reading the HDF5 data into memory (as a NumPy array). You don't have to do that. Instead of creating arrays, you can create h5py dataset objects. This reduces the I/O initially. Then you use the objects "as-if" they are np.arrays. The only change is your last line of code -- like this: v1, v2, v3, v4, v5, v6 = d2['SimSignal_X'], d2['SimSignal_Y'], d2['SimSignal_Z'], d2['SimEfield_X'], d2['SimEfield_Y'], d2['SimEfield_Z']. Note how the slice notation is not used ([0] or [:]).
   • I have found PyTables to be faster than h5py. Your code can be easily converted to read with PyTables.
   • HDF5 can use "chunking" to improve I/O performance. You did not say if you are using this. It might help (hard to say since your datasets aren't very large). You have to define this when you create the HDF5 file, so might not be useful in your case.
   • Also you did not say if a compression filter was used when the data was written. This reduces the on-disk file size, but has the side effect of reducing I/O performance (increasing read times). If you don't know, check to see if compression was enabled at file creation.

You're not doing something wrong: it's a NotImplementedError because memberwise serialization has not been implemented in Uproot. That's Issue #38, which has been getting a lot of attention recently.

Other people finding this question, years later: check to see if Issue #38 has been resolved.

This is actually a question about Uproot. In this line:

This will be better documented when it's ported to Uproot 4, but the best documentation we have on writing jagged arrays in Uproot 3 right now is the pull request and associated issues (all linked to each other):

https://github.com/scikit-hep/uproot3/pull/477

Here is an example from the tests:

why the second statement works, meanwhile the first causes an error. If the & operator works on variables

Because the & operator works on variables, and more generally on lvalue expressions, which designate objects, which have addresses. It does not work on rvalue expressions, which represent values without (necessarily) any persistent storage or address -- for example, 12 or x + 2. Function return values are rvalues.

No, this is not possible.

The first difficulty is opening an arbitrary ROOT file and adding any content to it in a robust way (i.e. the result can be read back by ROOT and yet more can be added by ROOT, without segfaults). If you want the long story, it's in scikit-hep/uproot#381, but the short story is that if Uproot makes the file itself, Uproot controls the initial conditions and can put the file into a known good state. We (developers of Uproot) can know some good states without understanding 100% of the state space. But if we want to accept files from ROOT, modify them, and give them back to ROOT, then we have to understand all the possible states that its filesystem-inside-a-file defragmentation algorithm can get into before we can trust our modifications of those states to stay within the space of good states. Understanding that state space would require a lot of study because (1) it's a whole filesystem that can reclaim free space and everything, and (2) there isn't a specification for this aspect of the format. It's defined by what the C++ code does.

The second difficulty is adding a TBranch to an existing TTree. That's not implemented, but much more manageable: we'd just make a new copy of the TTree metadata with the extra TBranch and link the new metadata to the old TBaskets (data). The main difficulty is adding any object to a preexisting ROOT file.

You can use root only with the c++ standard with which your root version has been compiled. This is a known inconvenience and I believe the developers would be happy if/when they get this sorted out. The build configuration root-config --cflags should set the appropriate c++ standard. This might very well be c++17 (it is for me)

If the data are fixed-size but stored as vector, then they're treated as though they were not fixed-size. Uproot would always read them as jagged arrays, and therefore the asarray method described in the other question is unavailable.

That said, if you have more knowledge than your file's metadata and are willing to try an unsupported hack, you can force the interpretation of your vector to always have three elements. Here's an example—first, we need a suitable ROOT file:

I believe you're mistaking Branch for SetBranchAddress in the code you posted. The documentation of Branch says (e.g. https://root.cern/doc/master/classTTree.html#ab47499eeb7793160b20fa950f4de716a) "Add a new branch". While you want to read from an existing branch. https://root.cern/doc/master/classTTree.html#a39b867210e4a77ef44917fd5e7898a1d.

So I think if you replace