ndindex | A Python library for manipulating indices of ndarrays

I have a dataframe and since I have to perform many calculations on it I figured I'd give Numpy a try, so I'm just learning how to use it. This is my dataframe

My apologies if the title seems vague, but I tried my best. In any case, I have a dataframe with three columns, one containing datetime values (for time of observation), another containing the range (distance from instrument at which the observation was made), and the last containing the intensity of the observations. The scatter plot for this data is shown below:

I need to filter out the random isolated 'salt and pepper' observations, and plan to use a median filter to do this. However, I'm not sure how to do this. I've tried to create a 2D array containing intensity values indexed according to the time and range. So 00:00 UT corresponds to row 0 and 0 km corresponds to column 0 and so on... empty positions contain NaNs. I then apply the median filter (scipy's medfilt: scipy.ndimage.median_filter) to this 2D array.

My issue is that it seems inefficient, as I'm having to loop over large series of data to create the array. And, of course, converting the filtered 2D array to a corresponding 1D series is difficult.

Here's the code I am using to obtain a 2D array

Jacobian of matrix with respect to itself

I am implementing an in-house automatic differentiation module using only native functions of NumPy, and for any kind of matrix operations, constructing a 4D array from a 2D array like the one in the picture seems to show up in different places.

My current approach is quite simple: if I'm given a k-by-d matrix called a, I am doing something like

Physical Background

I'm working on a function that calculates some metrics for each vertical profile in an up to four dimensional temperature field (time, longitude, latitude, pressure as height measure). I have a working function that takes the pressure and temperature at a single location and returns the metrics (tropopause information). I want to wrap it with a function that applies it to every vertical profile in the data passed.

Technical Description of the Problem

I want my function to apply another function to every 1D array corresponding to the last dimension in my N-dimensional array, where N <= 4. So I need an efficient loop over all dimensions but the last one without knowing the number of dimensions beforehand.

Why I Open a New Question

I am aware of several questions (e.g., iterating over some dimensions of a ndarray, Iterating over the last dimensions of a numpy array, Iterating over 3D numpy using one dimension as iterator remaining dimensions in the loop, Iterating over a numpy matrix with unknown dimension) asking how to iterate over a specific dimension or how to iterate over an array with unknown dimensions. The combination of these two problems is new as far as I know. Using numpy.nditer for example I haven't found out how to exclude just the last dimension regardless of the number of dimensions left.

EDIT

I tried to do a minimal, reproducible example:

I have an array A with the shape (3,3) which can be thought of as the sliding window view of an unkown array with the shape (5,). I want to compute the inverse of windowing the array with the shape (5,). The adjoint operation of this will be summation. What I mean is that I want to accumulate the values in each corresponding window with the related position in the array with the shape (5,). Ofcourse, my expected output of this inverse function and the input A are not related and are just ordinary arrays. I have two examples which I hope explains this better.

I have 2d array and dataframe df with x and y coordinates, I want to map values of 2D array to corresponding x and y coordinates in the dataframe in a new column

Array

I am writing an implementation of Bridson's Poisson disc sampling (https://www.cs.ubc.ca/~rbridson/docs/bridson-siggraph07-poissondisk.pdf) for Python.

A feature of this sampling is that there is a set minimal distance between samples, and it avoids clustering. The neighbour lookup for new candidate points exploits this feature, and uses a background grid to speed up searches.

The background grid consists of a boolean grid M that stores True for non-empty cells and False otherwise, and a grid of points P that stores the exact coordinates. Both of them are implemented as n-dimensional numpy arrays.

The grid size cellsize is selected in such a way that there is at most one sample in each grid cell, and then you only have to check a few of the nearest rows and columns.

For now, I use the following procedure for checking if the point p is close to any of the existing points:

I'm pulling in a 4000 wide x 2000 high flat map of the earth (src_filename) and have pre-calculated the pixel relocation x, y values which are stored in a numpy.ndarray (transform_array, 2000, 2000, 2).

The relocated pixels are supposed to be copied from a srcimg_array to a 2000x2000 polarimg_array which gets saved out to an image.

I have a for loop that iterates through the transform_array, reads the location value in each location, and then copies pixel RGB value from the source to the polar array - ie: transform_array [0, 0] contains [1414, 1500] which means the source image color values at [0, 0] get copied to [1414, 1500] in the polarimg_array, [0, 1] value gets copied to [1413, 1500], etc.

This process takes about 30 seconds per image and I'm trying to speed that process up.

How can this be accomplished using Python? (ie: not C or Cython or the like).

I'm trying to do a grid search over a model I've trained. So, producing a mesh, then predicting that mesh with the model to find a maximum. I'm producing the mesh with: