observations | Tools for loading standard data sets in machine learning | Machine Learning library

I have a dataframe with several groups and a different number of observations per group. I would like to create a new dataframe with no more than n observations per group. Specifically, for the groups that have a largen number I would like to select the n last observations. An example data set:

I have a dataset with the name of Danish ministers and their position from 1990 to 2020 (data comes from dataset called WhoGovern; https://politicscentre.nuffield.ox.ac.uk/whogov-dataset/). The dataset consists of the ministers name, the ministers position, the prestige of that position, and the year in which the minister had that given position.

My problem is that some ministers are counted twice in the same year (i.e., the rows aren't unique in terms of name and year). See the example in the picture below, where "Bertel Haarder" was both Minister of Health and Minister of Interior Affairs in 2010 and 2021.

I want to create a dataset, where all the rows are unique combinations of name and year. However, I do not want to remove any information from the dataset. Instead, I want to use the information in the prestige column to combine the duplicated rows into one. The observations with the highest prestige should be the main observations, where the other information should be added in a new column, e.g., position2 and prestige2. In the example with Bertel Haarder the data should look like this:

(PS: Sorry for bad presenting of the tables, but didn't know how to create a nice looking table...)

Here's the dataset for creating a reproducible example with observations from 2010-2020:

I'm working with some data where I have hourly observations for patients. In some cases, some of the features for a specific patient are completely empty. I'm trying to find a way to impute the data by using constant average that's based off a population subset of 50 other patients who have the same gender and a similar age. I've given a simplified look at the data below:

I've reordered the dataframe by age and have this code so far

data = data.sort_values(['Age']).groupby(['PatientID','Gender']).apply(lambda x: x.fillna(x.mean()))

But I know that that's going to use all of the available data to find the mean but I'm not sure how to limit it to 50 patients of a similar age.

From the “iris” dataset, how to find the number of observations whose “Sepal.Length” is greater than ‘6.5’ Using only loops or conditional statements

I am trying to make a next-word prediction model with LSTM + Mixture Density Network Based on this implementation(https://www.katnoria.com/mdn/).

Input: 300-dimensional word vectors*window size(5) and 21-dimensional array(c) representing topic distribution of the document, used to train hidden initial states.

Output: mixing coefficient*num_gaussians, variance*num_gaussians, mean*num_gaussians*300(vector size)

x.shape, y.shape, c.shape with an experimental 161 obserbations gives me such:

(TensorShape([161, 5, 300]), TensorShape([161, 300]), TensorShape([161, 21]))

I posted a similar question a week ago but I failed to identify the real problem. Therefore, the question was far from being correct.

Now, I clearly now what is going on but I cannot understand why it is happening. I also reviewed similar problems related with the same error but the solutions for these problems were not applicable to my case.

I am plotting the frequency distribution of a variable during the fieldwork progress of a survey. Therefore, it shows how the proportion of that variables has changed through time.

So, I have a variable (Startday) that tells which day the respondent took the survey, if he/she did not then it is NA. Then, I have the typical variables like sex or marital status.

This is the code to plot such graph

I work in databricks. I have a dataframe d which contains few columns with '?' string value. I want to covert these '?' values to NULL because I want to use dropna(['...']) function later to delete observations with NULL values. I have no idea how to do this, nothing works. I tried:

numpy:

TypeError: 'DataFrame' object does not support item assignment

I have a variable that contains the name conflict parties. Most of them are noted like:

"Government of Afghanistan" "Government of Peru" "Government of Liberia"

I wondered how I could drop the part "Government of" and keep "Afghanistan", "Peru" etc. Since the dataset contains about 1000 observations, it would be nice to find a solution that doesnt require to type the name of every country.

I have a dataset that identifies observations based on two variables: Time and Country. The variable of interest is dichotomous, and has the value 0 if the event didn't occur and 1 if it did. For some countries more than one observation is reported per year. The data can be summarized like this:

Is it possible two colapse these multiple observations into one observation per year and country with either outcome 0 (if the event never occured) or 1(if the event occured at least once)? Like this?:

Thank you in advance !