dataframes | A library for working with tabular data in Luna

Situation: I have two dataframes df1 and df2, where df1 has a datetime index based on days, and df2 has two date columns 'wk start' and 'wk end' that are weekly ranges as well as one data column 'statistic' that stores data corresponding to the week range.

What I would like to do: Add to df1 a column for 'statistic' whereby I lookup each date (on a daily basis, i.e. each row) and try to find the corresponding 'statistic' depending on the week that this date falls into.

I believe the answer would require merging df2 into df1 but I'm lost as to how to proceed after that.

Appreciate any help you might provide! Thanks!

df1: (note: I skipped the rows between 2019-06-12 and 2019-06-16 to keep the example short.)

df2:

Desired output:

code for the dataframes d1 and d2

I have a a few dataframes, a few thousand rows each that look similar to this :

This example has been tested with Spark 2.4.x. Let's consider 2 simple dataframes:

I have a dataframe:

I would like to calculate mean values of a specific column from many similarly formatted dataframes, which are named dataframe_1 - dataframe_100. I have been trying a for loop as below:

I grouped a dataframe test_df2 by frequency 'B' (by business day, so each name of the group is the date of that day at 00:00) and am now looping over the groups to calculate timestamp differences and save them in the dict grouped_bins. The data in the original dataframe and the groups looks like this:

What I want is to calculate the difference between each row's timestamp, for example of rows 7 and 0, since they have the same externalId.

What I did for that purpose is the following.

I have a dataframe like this:

I have a couple of arcs dataframes with a very similar structure to these:

I need to find a way to create a new dataframe that merges both, following this structure:

or this structure:

Currently I have a piece of code that can do something similar to that but instead of a pandas dataframe (which is what I want to use) I'm using dictionaries (the reason behind that is that each "route" has different caracteristics that makes them unique from each other so a dictionary is useful and at the time I was just learning Python) but the issue is that it takes too much time and uses a lot of memory so I'm trying to find a way to make it a little bit quicker, avoiding 'for' loops and trying to apply Pandas to create the merged dataframe.

This is an extract of the structure of my current piece of code, for this example, consider the 'A' dataframe as the one that holds every combination possible of arcs so the 'if' condition makes sure that a connection exists before creating the route.

Hello I have a list of dataframes where I want to add new columns to each of those dataframe. My current for-loop approach gets the job done, however I was looking for an elegant approach, something from apply family of functions.

Here is a reprex-