simple-ml | 机器学习经典算法/应用示例 | Machine Learning library

Within the transformers of ColumnTransformer, turns out you cannot use a list of column name strings for integer_features or categorical_features. If you change them to lists of the numeric column indices, meaning something like integer_features = [5,6] and categorical_features = [0, 1, 2, 3, 4], it should work.

MinMaxScaler in Spark works on each feature individually. From the documentation we have:

Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.

$$ Rescaled(e_i) = \frac{e_i - E_{min}}{E_{max} - E_{min}} * (max - min) + min $$

[...]

So each column in the features array will be scaled separately. In this case, the MinMaxScaler is set to have a minimum value of 5 and a maximum value of 10.

The calculation for each column will thus be:

1. In the first column, the min value is 1.0 and the maximum is 3.0. We have 1.0 -> 5.0, and 3.0 -> 10.0. 2.0 will there for become 7.5.
2. In the second column, the min value is 0.1 and the maximum is 10.1. We have 0.1 -> 5.0 and 10.1 -> 10.0. The only other value in the column is 1.1 which will become ((1.1-0.1) / (10.1-0.1)) * (10.0 - 5.0) + 5.0 = 5.5 (following the normal min-max formula).
3. In the third column, the min value is -1.0 and the maximum is 3.0. So we know -1.0 -> 5.0 and 3.0 -> 10.0. For 1.0 it's in the middle and will become 7.5.

The 5-th column is a structure representing sparse vectors in Spark. It has three components:

• vector length - in this case all vectors are of length 10 elements
• index array holding the indices of non-zero elements
• value array of non-zero values

So

I solved it by removing scala library from the build path, to do this, right click on the scala library container > build path > remove from build path not sure about the root cause though.