ohe | code behind App.net 's reference private messaging UI

Python beginner here...

Trying to understand how to use OneHotEncoder from the sklearn.preprocessing library. I feel pretty confident in using it in combination with fit_transform so that the results can also be fit to the test dataframe. Where I get confused is what to do with the resulting encoded array. Do you then convert the ohe results back to a dataframe and append it to the existing train/test dataframe?

The ohe method seems a lot more cumbersome than the pd.get_dummies method, but from my understanding using ohe with fit_transform makes it easier to apply the same transformation to the test data.

Searched for hours and having a lot of trouble trying to find a good answer for this.

Example with the widely used Titanic dataset:

I'm trying to train a model from a dataset of about a few thousands of entries with 51 numerical features and a labeled column, Example:

when training the model to predict the 3 labels (candidate, false positive, confirmed) the loss is always nan and the accuracy stabilizes very fast on a specific value. The code:

I am working on a machine learning problem, where I have a lot of zipcodes (~8k unique values) in my data set. Thus I decided to hash the values into a smaller feature space instead of using something like OHE.

The problem I encountered was a very small percentage (20%) of unique rows in my hash, which basically means from my understanding, that I have a lot of duplicates/collisions. Even though I increased the features in my hash table to ~200, I never got more than 20% of unique values. This does not make sense to me, since with a growing number of columns in my hash, more unique combinations should be possible

I used the following code to hash my zip codes with scikit and calculate the collisions based on unique vales in the last array:

I am trying to use properly pipelines and column transformers from sklearn but always end up with an error. I reproduced it in the following example.

I am trying to use KNN for imputing categorical variables in python.

In order to do so, a typical way is to one hot encode the variables before. However sklearn OneHotEncoder() doesn't handle NAs so you need to rename them to something which creates a seperate variable.

Small reproducible example:

I am trying to build a prediction model but currently keep getting an error: raise ValueError("Input contains NaN") ValueError: Input contains NaN. I tried to use np.any(np.isnan(dataframe)) and np.any(np.isnan(dataframe)), but I just keep getting new errors. For example, TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''.

Here is the code so far:

I am using sklearns' pipeline function, to one hot encode, and to model. Almost exactly as in this post.

After using a Pipeline, I am not able to get tree contributions anymore. Getting this error:

AttributeError: 'Pipeline' object has no attribute 'n_outputs_'

I tried to play around with the parameters of the treeinterpreter, but I am stuck.

Therefore my question: is there any way how we can get the contributions out of a Tree, when we are using sklearns Pipeline?

EDIT 2 - Real data as requested by Venkatachalam:

I have strings like these ones in a file:

I am trying to predict income (70000+) based on specific categorical fields (Sex and Highest Cert, dip, deg) based on python code below.

I created a range for the average income and then specified the specific range of income (70000+) I wanted to predict using (Sex and Highest Cert, dip, deg)

I have the following code. However, I get an error when I get to the One hot encoding part of the code. I am using python on visual studio. I have tried changing the categorical field to "Age", but it does not work. The code is below. Please how can I fix it? Thank you.