datacube | Multidimensional data storage with rollups

I have a datacube opened with Xarray, which has several variables and a time vector ("mid_date", dimension 18206) in format datetime64.

The variables are 18206 x 334 x 333. The issue is that the time vector is not sorted at all, and I would like to sort it in ascending order (oldest to most recent). And at the same time, reorganize my variables' arrays.

Then, I would like to select part of a variable (for example: "vy"), between 2 dates (so I can do calculations on only a part of my data). I can sort the date vector but can't apply that sorting on the other variables. How could I do that ?

Here is the information of the dataset:

I am working with a datacube such as data[x,y,z]. Each point is a velocity through time, and the [x,y] grid corresponds to coordinates. If I pick a point of coordinates x and y, it is likely that the timeseries is incomplete (with some NaNs). I created a function which searches for the closest neighbor with a value, and replaces the NaN of my xy point with it. However I want to know if there is a more efficient way to code something which does the same ?

Joined to this message is a photo of how the function evaluates the neighbors. The number of each point represents its rank (5 is the 5th neighbor evaluated).

I tried something like this:

Let's say that I have a datacube of 10x10x100 (100 is the timeseries):

I get an

ImportError: DLL load failed: The specified procedure could not be found.

error when trying to import geopandas in python 3.6. Specifically, I get the error when using ipython but not when using python. Also, this affects Windows (a Windows Server 2016 virtual machine) and not Linux. I’ve found a few previous posts on this, or very similar issues, but I’m rejecting their suitability as they either don’t clearly resolve the problem or else conflate it with pip installs.

This post from nearly two years ago, for example, reports a similar error, but concludes with a “Never mind, I did a pip install of geopandas”.

This post from just over a couple of years ago has an accepted answer despite the original poster commenting that it didn’t work for them! There’s a mention of a blog post from Geoff Boeing that I’ve seen before as providing a working method, despite that blog post providing more than one approach (a conda install and a more manual sequence of steps) and the comment not clarifying what worked for them.

There’s this post from nearly two and a half years ago that conflates conda and pip install methods and doesn’t have an accepted answer. There’s a suggestion in a comment that, for the commenter, it was an issue with gdal on conda-forge. There’s an answer that refers to Geoff Boeing’s blogpost again. The implication may be that the install of gdal via conda can be problematic and, if it is, then the manual sequence of steps is required. I am not persuaded this is my issue.

My problem occurs specifically on a Windows Server 2016 virtual machine and when specifying only the conda-forge channel. Also, pertinently, it only occurs in ipython (and thus Jupyter notebooks) and not in python, thus:

Create environment specifying the conda defaults channel, specifying python 3.6, ipython, and geopandas:

I've written a through focus STEM acquisition script that reads in an image using the DSAcquire function, where I specify the signal to be read in with DSAcquireData(img, signalindex, etc.).

The nice thing about the above is that I can read in the image without it appearing on screen, copy it into a datacube, and then acquire the next one in the series, etc.

If I want to use two signals instead of one (eg HAADF and BF), it looks like the only way to do this is to use DSStartAcquisition after setting the digiscan parameters?

How should I go about copying signals into two preallocated image stacks (stack1, stack2)? Preferably without tens of images cluttering the screen (but ideally with some measure of progress?)

I have a stored procedure that generates a report. The actual report is a bit complex so I will try to explain myself as simple as possible with table examples:

My main table has the following data:

table1

I have a stored procedure that returns a set with different clauses, this is the part I'm having a little problem with:

I have a "seed" GeoDataFrame (GDF)(RED) which contains a 0.5 arc minutes global grid ((180*2)*(360*2) = 259200). Each cell contains an absolute population estimate. In addition, I have a "leech" GDF (GREEN) with roughly 8250 adjoining non-regular shapes of various sizes (watersheds).

I wrote a script to allocate the population estimates to the geometries in the leech GDF based on the overlapping area between grid cells (seed GDF) and the geometries in the leech GDF. The script works perfectly fine for my sample data (see below). However, once I run it on my actual data, it is very slow. I ran it overnight and the next morning only 27% of the calculations had been performed. I will have to run this script many times and waiting for two days each time, is simply not an option.

After doing a bit of literature research, I already replaced (?) for loops with for index i in df.iterrows() (or is this the same as "conventional" python for loops) but it didn't bring about the performance imporvement I had hoped for.

Any suggestion son how I can speed up my code? In twelve hours, my script only processed only ~30000 rows out of ~200000.

My expected output is the column leech_df['leeched_values'].