Spark-The-Definitive-Guide | Spark : The Definitive Guide 's Code Repository

Use aggregate instead of transform function to calculate the total price like this:

MinMaxScaler in Spark works on each feature individually. From the documentation we have:

Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.

$$ Rescaled(e_i) = \frac{e_i - E_{min}}{E_{max} - E_{min}} * (max - min) + min $$

[...]

So each column in the features array will be scaled separately. In this case, the MinMaxScaler is set to have a minimum value of 5 and a maximum value of 10.

The calculation for each column will thus be:

1. In the first column, the min value is 1.0 and the maximum is 3.0. We have 1.0 -> 5.0, and 3.0 -> 10.0. 2.0 will there for become 7.5.
2. In the second column, the min value is 0.1 and the maximum is 10.1. We have 0.1 -> 5.0 and 10.1 -> 10.0. The only other value in the column is 1.1 which will become ((1.1-0.1) / (10.1-0.1)) * (10.0 - 5.0) + 5.0 = 5.5 (following the normal min-max formula).
3. In the third column, the min value is -1.0 and the maximum is 3.0. So we know -1.0 -> 5.0 and 3.0 -> 10.0. For 1.0 it's in the middle and will become 7.5.

The 5-th column is a structure representing sparse vectors in Spark. It has three components:

• vector length - in this case all vectors are of length 10 elements
• index array holding the indices of non-zero elements
• value array of non-zero values

So

I solved it by removing scala library from the build path, to do this, right click on the scala library container > build path > remove from build path not sure about the root cause though.

Spark is a lazy evolution. However, that doesn't mean It can't verify if file exist of not while loading it.

Lazy evolution happens on DataFrame object, and in order to create dataframe object they need to first check if file exist of not.

Check the following code.