xgboost | Distributed Gradient Boosting

xgboost has min_child_weight, but outside of the ordinary regression task that is indeed different from minimum samples. I couldn't say why the additional parameter isn't included. Note though that in binary classification, the logloss hessian is p(1-p) and is between 0 and 1/4, with values near zero for the very confident predictions; so in effect setting min_child_weight is requiring many currently-uncertain rows in each leaf, which may be close enough to (or better than!) setting a minimum number of rows.

Seems you need to upgrade pip, see this question.

But there can be multiple pips in a Dataproc cluster, you need to choose the right one.

1. For init actions, at cluster creation time, /opt/conda/default is a symbolic link to either /opt/conda/miniconda3 or /opt/conda/anaconda, depending on which Conda env you choose, the default is Miniconda3, but in your case it is Anaconda. So you can run either /opt/conda/default/bin/pip install --upgrade pip or /opt/conda/anaconda/bin/pip install --upgrade pip.

2. For custom images, at image creation time, you want to use the explicit full path, /opt/conda/anaconda/bin/pip install --upgrade pip for Anaconda, or /opt/conda/miniconda3/bin/pip install --upgrade pip for Miniconda3.

So, you can simply use /opt/conda/anaconda/bin/pip install --upgrade pip for both init actions and custom images.

Changing Java install to FROM openjdk:15.0.2-jdk-slim has solved the issue

I am not an expert with scikit-learn but it seems that one of the requirements of various objects used by this framework is that they can be cloned by calling the sklearn.base.clone method. This appears to be something that the existing XGBRegressor class does, so is something your subclass of XGBRegressor must also do.

What may help is to pass any other unexpected keyword arguments as a **kwargs parameter. In your constructor, kwargs will contain a dict of all of the other keyword parameters that weren't assigned to other constructor parameters. You can pass this dict of parameters on to the call to the superclass constructor by referring to them as **kwargs again: this will cause Python to expand them out:

When using the XGBoost Python package you can choose between two different APIs to train your model. XGB's own Learning API and the Scikit-Learn API.
When using the Scikit-Learn API data is passed to the model as numpy array or pandas dataframes. When using the Learning API data is passed using the DMatrix.

Have a look at the python examples, to see both APIs used.

Basically you already found the "use of DMatrix instead of a Pandas Dataframe" in the docs: It is a data structure the XGBoost developers created for "memory efficiency and training speed" with their machine learning library.

A ! magic can be included in a function, but can't be performed via exec.

As noted in the pip page for dask-xgboost:

As stated in the XGBoost Docs

Parameter tuning is a dark art in machine learning, the optimal parameters of a model can depend on many scenarios.

You asked for suggestions for your specific scenario, so here are some of mine.

1. Drop the dimensions booster from your hyperparameter search space. You probably want to go with the default booster 'gbtree'. If you are interested in the performance of a linear model you could just try linear or ridge regression, but don't bother with it during your XGBoost parameter tuning.
2. Drop the dimension base_score from your hyperparameter search space. This should not have much of an effect with sufficiently many boosting iterations (see XGB parameter docs).
3. Currently you have 3200 hyperparameter combinations in your grid. Expecting to find a good one by looking at 50 random ones might be a bit too optimistic. After dropping the booster and base_score dimensions you would be down to

The save_config method noted here can be used to create a string representation of the model's configuration. This can be converted to a dict: