vaex | Core hybrid Apache Arrow/NumPy DataFrame

Given a list of indexes (offset values) according which splitting a numpy array, I would like to adjust it so that the splitting does not occur on duplicate values. This means duplicate values will be in one chunk only.

I have worked out following piece of code, which gives the result, but I am not super proud of it. I would like to stay in numpy world and use vectorized numpy functions as much as possible. But to check the indexes (offset values) I use a for loop, and store the result in a list.

Do you have any idea how to vectorize the 2nd part?

If this can help, ar is an ordered array. (I am not using this info in below code).

I tried to open a parquet on an Azure data lake gen 2 storage using SAS URL generated (with the datetime limit and token embedded in the url) using vaex by doing:

vaex.open(sas_url)

and I got the error

ERROR:MainThread:vaex:error opening 'the path which was also the sas_url(can't post it for security reasons)' ValueError: Do not know how to open (can't publicize the sas url) , no handler for https is known

How do I get vaex to read the file or is there another azure storage that works better with vaex?

I am trying to tokenize natural language for the first sentence in wikipedia in order to find 'is a' patterns. n-grams of the tokens and left over text would be the next step. "Wellington is a town in the UK." becomes "town is a attr_root in the country." Then find common patterns using n-grams.

For this I need to replace string values in a string column using other string columns in the dataframe. In Pandas I can do this using

My data come from BigQuery exported to GCS bucket as CSV file and if the file size is quite massive, BigQuery will automatically split the data into several chunk. With time series in mind, the time series might be scattered across different files. I have a custom function that I want to applied to each TimeseriesID.

Here's some constraint of the data:

• The data is sorted by TimeseriesID and TimeID
• The number of row of each files is may vary, but at minimum 1 row (which is very unlikely)
• The starting of TimeID is not always 0
• The length of each time series may vary but at maximum it will only scattered across 2 files. No time series scatter in 3 different files.

Here's the initial setup to illustrate the problem:

I am trying to load a dataframe(vaex) to dash datatable and getting following error.

Invalid argument data passed into DataTable with ID "table". Expected an array. Was supplied type object.

Tried the following, is it possible to load vaex dataframe to dash datatable like pandas dataframe.

I'm looking to use a distributed cache in python. I have a fastApi application and wish that every instance have access to the same data as our load balancer may route the incoming requests differently. The problem is that I'm storing / editing information about a relatively big data set from a arrow feather file and processing it with Vaex. The feather file automaticaly loads the correct types for the data. The data structure I need to store will use a user id as a key and the value will be a large array of arrays of numbers. I've looked at memcache and redis as possible caching solutions, but both seem to store entries as strings / simple values. I'm looking to avoid parsing strings and extra processing on a large amount data. Is there a distributed caching stategy that will let me persist types?

One solution we came up with is to store the data in mutliple feather files in a directory that is accessible to all instances of the app but this seems to be messy as you would need to clean up / delete the files after each session.

Suppose we have a dataset.

The code is as following the input dataframe is

I was trying to compute the pandas.plotting.scatter_matrix() values for very large pandas.DataFrame() (relatively speaking for this specific operation, most libraries either run OOM most of the time or implement a row count check of 50000, see vaex-scatter).

The 'Time series' DataFrame shape I have is (10000000, 41). Every value is either a float or an integer.

Q1: So the first thing I would already like to ask is how do I do that memory and space efficiently.

• I tried to do it typically (like in the examples in the documentation) using matplotlib and modin.pandas.DataFrames looping over each pair, so the indexing and operations/calculations I want to do are relatively fast including the to_numpy() method. How ever as you might have already seen from the image 1 pair takes 18.1 secs at least and 41x41 pairs are too difficult to handle in my task and I feel there is a relatively faster way of doing things. :)

• I tried using the pandas scatter plot function which is also too slow and crashes my memory. This is done using the native pandas package and not the modin.pandas. This was done by first converting the modin.pandas.DataFrame to pandas.DataFrame via the private modin.pandas.DataFrame._to_pandas() accessor. This approach is too slow too. I stopped waiting after I ran out of memory 1 hour later.

• I tried plotting with vaex. This was the fastest but I ran into other errors which arent related to the question.

• please do not suggest seaborn's pair plot. Tried and it takes around 5 mins to generate a pairplot() for a pandas.DataFrame of shape (1000x8), also is cantered around pandas.

• I am plotting a scatter matrix of all the features sampled 10000 times. so modin.DataFrame.sample(10000) since it kind of is okay to view at the general trend but i do not wish to do this if there is a better option.
• Converting it to pandas.DataFrame and using pandas.plotting.scatter_matrix like this, so that i dont have to wait for it to be rendered onto the jupyter notebook.