metric2 | service BI tool using responsive web based dashboards

I am trying to make a new slide that contains a table with some data in it using the Google Slides API. I'm getting an invalid JSON payload error.

What i've tried to do is create a function that makes the new table request.

I am running a simulation study where my results are stored in a nested list structure. The first level of the list represents the different hyperparameters generated by the model. The second level is the number of replications of the same model (changing the seed).

In the example below, I list the output of a model that is governed by two hyperparameters (hyperpar1 and hyperpar2) where both can take 2 different values, leading to 4 different combinations of the resulting model. Additionally, each of the 4 possible combinations was run twice (different seeds), leading to eight possible combinations (as can be seen below with str(res, max = 2)). Finally, two performance metrics were recovered from each possible iteration of the models (metric1 and metric2) and also the value of the two parameters of the model beta = list(b1 = value, b2 = value).

My problem is related to the last part. I want to flatten the list beta and get each component inside of the list as a component of the upper list that contains it, but keeping the whole structure described in the beginning unchanged.

Below an example:

Comming from R/dplyr, I'm used to the piping concept to chain transformation steps during data analysis and have taken this to pandas in a sometimes similar, sometimes better but also sometimes worse fashion (see this article for reference). This is an example of a worse situation.

I'm conducting an analysis of some objects and want to understand the behavior by some grouping variable and for steps further dwon the line (which are not relevant here), I have to have the calculated metrics per grouping in seperate columns. Hence, I'm chaining agg() with pivot() and end up with a multiindex, which I'd like to collapse or flatten.

What I do:

I need a query in MongoDB where I turn a list into True or False depending on whether a value exists in the list or not.

Here's a summarized view of the schema.

I have a table that has JSON data stored and I'm using json_exists functions in the query. Below is my sample data from the column for one of the rows.

I have following pipeline in HDFS which I am processing in spark

input table : batch, team, user, metric1, metric2

This table can has user level metrics in hourly batches. In same hour a user can have multiple entries.

level 1 aggregation : this aggregation to get latest entry per user per batch

agg(metric1) as user_metric1, agg(metric2) as user_metric2 (group by batch, team, user)

level 2 aggregation : get team level metrics

agg(user_metric1) as team_metric1, agg(user_metric2) as team_metric2 (group by batch, team)

Input table is 8gb (snappy parquet format) in size in HDFS. My spark job is showing shuffle write to 40gb and at least 1 gb per executor shuffle spill.

In order to minimize this, if I repartition input table on user level before performaing aggregation,

I am struggling to modify my code to define a specific range of the secondary x-axis. Below is a snippet of the relevant code for creating 2 x-axes, and the output it generates:

I have a log that looks like this:

having records in test table like so (metrics column is jsonb type):