subset | End of life

I'm having trouble understanding why TypeScript is inferring a certain type for an array element when the type is a union type and the types 'overlap'. I've reduced it to this minimum repro:

...

When I use the following code to print all subsets of the string "abc", the code works as expected, printing : ab a b

...

I have a function that will be used under different modules. There are two functions that take different arguments but the function logic is similar. I am trying to unite func1 and func2 functions into one.

Is there a way I can use the python functionality to handle this case?

func1

...

I am trying to remove duplicates based on the column item_id from a dataframe df.

df :

...

The Question

How do I best execute memory-intensive pipelines in Apache Beam?

Background

I've written a pipeline that takes the Naemura Bird dataset and converts the images and annotations to TF Records with TF Examples of the required format for the TF object detection API.

I tested the pipeline using DirectRunner with a small subset of images (4 or 5) and it worked fine.

The Problem

When running the pipeline with a bigger data set (day 1 of 3, ~21GB) it crashes after a while with a non-descriptive SIGKILL. I do see a memory peak before the crash and assume that the process is killed because of a too high memory load.

I ran the pipeline through strace. These are the last lines in the trace:

...

I'm working with some data where I have hourly observations for patients. In some cases, some of the features for a specific patient are completely empty. I'm trying to find a way to impute the data by using constant average that's based off a population subset of 50 other patients who have the same gender and a similar age. I've given a simplified look at the data below:

HR O2Sat Temp Platelets Age Gender PatientID 80 98 36.5 NaN 52 1 A0 82 96 37.0 NaN 52 1 A0 82 100 36.3 160 53 1 A1 90 93 36.6 165 53 1 A1 83 95 35.9 140 23 0 A2 79 98 36.2 155 23 0 A2 88 92 36.6 163 60 0 A3 90 91 36.3 165 60 0 A3 81 95 37.1 NaN 20 0 A4 81 92 36.9 NaN 20 0 A4

I've reordered the dataframe by age and have this code so far

data = data.sort_values(['Age']).groupby(['PatientID','Gender']).apply(lambda x: x.fillna(x.mean()))

But I know that that's going to use all of the available data to find the mean but I'm not sure how to limit it to 50 patients of a similar age.

...

I would like to find minimum distance of each voxel to a boundary element in a binary image in which the z voxel size is different from the xy voxel size. This is to say that a single voxel represents a 225x110x110 (zyx) nm volume.

Normally, I would do something with scipy.ndimage.morphology.distance_transform_edt (https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.ndimage.morphology.distance_transform_edt.html) but this gives the assume that isotropic sizes of the voxel:

...

I have two data frames. df1 and df2. both with c columns
using a clustering method, I ended up with 10 clusters. same clusters for each df is true. this means for example the 4th row of both df s go to the same cluster.
I added a cluster column to both dfs, showing the assigned cluster for each row.

I want to create a list.
this list contains 10 matrices, such that.
matrix 1, is a 2*c matrix. its first row is obtained by colmeans of those rows of df1 which are in cluster 1. and its 2nd row is obtained by colmeans of those rows of df2 which are in cluster 1.
and matrix 2 , colmeans of cluster 2 and so on.
this is what I ve done. but I get the 10th matrix only and not a list of matrices 1 to 10.
I would appreciate any help with this.

...

I am using the SQL connector to capture CDC on a table that we only expose a subset of all columns on the table. The table has two unique indexes A & B on it. Neither index is marked as the PRIMARY INDEX but index A is logically the primary key in our product and what I want to use with the connector. Index B references a column we don't expose to CDC. Index B isn't truly used in our product as a unique key for the table and it is only marked UNIQUE as it is known to be unique and marking it gives us a performance benefit.

This seems to be resulting in the error below. I've tried using the message.key.columns option on the connector to specify index A as the key for this table and hopefully ignore index B. However, the connector seems to still want to do something with index B

1. How can I work around this situation?
2. For my own understanding, why does the connector care about indexes that reference columns not exposed by CDC?
3. For my own understanding, why does the connector care about any index besides what is configured on the CDC table i.e. see CDC.change_tables.index_name documentation

...