assume | like assertions | Assertion library

I'm normally OK on the joining and appending front, but this one has got me stumped.

I've got one dataframe with only one row in it. I have another with multiple rows. I want to append the value from one of the columns of my first dataframe to every row of my second.

df1:

df2:

Output I'm seeking:

df2

I figured that this was along the right lines, but it listed out NaN for all rows:

I have created a GCP service account with org viewer permissions (I assume therefore having read rights in all projects)

I have timecodes with this structure hh:mm:ss.SSS for which i have a own Class, implementing the Temporal Interface. It has the custom Field TimecodeHour Field allowing values greater than 23 for hour. I want to parse with DateTimeFormatter. The hour value is optional (can be omitted, and hours can be greater than 24); as RegEx (\d*\d\d:)?\d\d:\d\d.\d\d\d

For the purpose of this Question my custom Field can be replaced with the normal HOUR_OF_DAY Field.

My current Formatter

I read this answer, which clarified a lot of things, but I'm still confused about how I should go about designing my primary key.

First off I want to clarify the idea of WCUs. I get that WCU is the write capacity of max 1kb per second. Does it mean that if writing a piece of data takes 0.25 seconds, I would need 4 of those to be billed 1 WCU? Or each time I write something it consumes 1 WCU, but I could also write X times within 1 second and still be billed 1 WCU?

Usage

I want to create a table that stores the form data for a set of gyms (95% will be waivers, the rest will be incidents reports). Most of the time, each forms will be accessed directly via its unique ID. I also want to query the forms by date, form, userId, etc..

We can assume an average of 50k forms per gym

Options

• First option is straight forward: having the formId be the partition key. What I don't like about this option is that scan operations will always filter out 90% of the data (i.e. the forms from other gyms), which isn't good for RCUs.

• Second option is that I would make the gymId the partition key, and add a sort key for the date, formId, userId. To implement this option I would need to know more about the implications of having 50k records on one partition key.

• Third option is to have one table per gyms and have the formId as partition key. This seems to be like the best option for now, but I don't really like the idea of having a a large number of tables doing the same thing in my account.

Is there another option? Which one of the three is better?

Edit: I'm assuming another option would be SimpleDB?

I am trying to create scatter plots of all the combinations for the columns: insulin, sspg, glucose (mclust, diabetes dataset, in R) with class as the colo(u)r. By that I mean insulin with sspg, insulin with glucose and sspg with glucose.

And I would like to do that with tidyverse, purrr, mappings and pipe operations. I can't quite get it to work, since I'm relatively new to R and functional programming.

When I load the data I've got the columns: class, glucose, insulin and sspg. I also used pivot_longer to get the columns: attr and value but I was not able to plot it and don't know how to create the combinations.

I assume that there will be an iwalk() or map2() function at the end and that I might have to use group_by() and nest() and maybe combn(., m=2) for the combinations or something like that. But it will probably have some way simpler solution that I can not see myself.

My attempts have amounted to this:

Here is the question:

I have a generator object, that loads quite big amount of data and hogs the I/O of the system. The data is too big to fit into memory all at once, hence the use of generator. And I have a consumer that all of the CPU to process the data yielded by generator. It does not consume much of other resources. Is it possible to interleave these tasks using threads?

For example I'd guess it is possible to run the simplified code below in 11 seconds.

The Question

How do I best execute memory-intensive pipelines in Apache Beam?

Background

I've written a pipeline that takes the Naemura Bird dataset and converts the images and annotations to TF Records with TF Examples of the required format for the TF object detection API.

I tested the pipeline using DirectRunner with a small subset of images (4 or 5) and it worked fine.

The Problem

When running the pipeline with a bigger data set (day 1 of 3, ~21GB) it crashes after a while with a non-descriptive SIGKILL. I do see a memory peak before the crash and assume that the process is killed because of a too high memory load.

I ran the pipeline through strace. These are the last lines in the trace:

I have a DataFrame like this