xgboost | Scalable, Portable and Distributed Gradient Boosting Library, for Python, R, Java, Scala, C and more

Error in .h2o.__checkConnectionHealth() : 
  H2O connection has been severed. Cannot connect to instance at http://localhost:54321/
getaddrinfo() thread failed to start

In addition: There were 13 warnings (use warnings() to see them)
Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = urlSuffix,  : 
  Unexpected CURL error: getaddrinfo() thread failed to start

openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)

Starting H2O JVM and connecting: ... Connection successful!

R is connected to the H2O cluster: 
    H2O cluster uptime:         1 seconds 896 milliseconds 
    H2O cluster timezone:       America/Chicago 
    H2O data parsing timezone:  UTC 
    H2O cluster version:        3.35.0.2 
    H2O cluster version age:    19 hours and 24 minutes  
    H2O cluster name:           H2O_started_from_R_jholderieath_glq667 
    H2O cluster total nodes:    1 
    H2O cluster total memory:   19.84 GB 
    H2O cluster total cores:    12 
    H2O cluster allowed cores:  12 
    H2O cluster healthy:        TRUE 
    H2O Connection ip:          localhost 
    H2O Connection port:        54321 
    H2O Connection proxy:       NA 
    H2O Internal Security:      FALSE 
    H2O API Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4 
    R Version:                  R version 4.1.1 (2021-08-10) 

Building wheels for collected packages: pynacl, libcst
  Building wheel for pynacl (PEP 517): started
  Building wheel for pynacl (PEP 517): still running...
  Building wheel for pynacl (PEP 517): finished with status 'done'
  Created wheel for pynacl: filename=PyNaCl-1.5.0-cp37-cp37m-linux_x86_64.whl size=201317 sha256=4e5897bc415a327f6b389b864940a8c1dde9448017a2ce4991517b30996acb71
  Stored in directory: /root/.cache/pip/wheels/2f/01/7f/11d382bf954a093a55ed9581fd66c3b45b98769f292367b4d3
  Building wheel for libcst (PEP 517): started
  Building wheel for libcst (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /opt/conda/anaconda/bin/python /opt/conda/anaconda/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpon3bonqi
       cwd: /tmp/pip-install-9ozf4fcp/libcst

gcloud dataproc clusters create cluster-test \
--enable-component-gateway \
--region us-east1 \
--zone us-east1-b \
--master-machine-type n1-highmem-32 \
--master-boot-disk-size 500 \
--num-workers 3 \
--worker-machine-type n1-highmem-16 \
--worker-boot-disk-size 500 \
--optional-components ANACONDA,JUPYTER,ZEPPELIN \
--image-version 1.5.54-ubuntu18 \
--tags <tag-name> \
--bucket '<cloud storage bucket>' \
--initialization-actions 'gs://goog-dataproc-initialization-actions-us-east1/connectors/connectors.sh','gs://goog-dataproc-initialization-actions-us-east1/python/pip-install.sh' \
--metadata='PIP_PACKAGES=wheel datalab xgboost==1.3.3 shap oyaml click apache-airflow apache-airflow-providers-google' \
--initialization-action-timeout 30m \
--metadata gcs-connector-version=2.1.1,bigquery-connector-version=1.1.1,spark-bigquery-connector-version=0.17.2 \
--project <project-name>

gcloud dataproc clusters create cluster-test \
--enable-component-gateway \
--region us-east1 \
--zone us-east1-b \
--master-machine-type n1-highmem-32 \
--master-boot-disk-size 500 \
--num-workers 3 \
--worker-machine-type n1-highmem-16 \
--worker-boot-disk-size 500 \
--optional-components ANACONDA,JUPYTER,ZEPPELIN \
--image-version 1.5.54-ubuntu18 \
--tags <tag-name> \
--bucket '<cloud storage bucket>' \
--initialization-actions 'gs://goog-dataproc-initialization-actions-us-east1/connectors/connectors.sh','gs://<bucket-path>/upgrade-pip.sh','gs://goog-dataproc-initialization-actions-us-east1/python/pip-install.sh' \
--metadata='PIP_PACKAGES=wheel datalab xgboost==1.3.3 shap oyaml click apache-airflow apache-airflow-providers-google' \
--initialization-action-timeout 30m \
--metadata gcs-connector-version=2.1.1,bigquery-connector-version=1.1.1,spark-bigquery-connector-version=0.17.2 \
--project <project-name>

water.exceptions.H2OIllegalArgumentException: Illegal argument: o of function: IcedWrapper: 
    at water.IcedWrapper.<init>(IcedWrapper.java:152) ~[h2o.jar:?]
    at water.util.TwoDimTable.set(TwoDimTable.java:254) ~[h2o.jar:?]
    at water.util.ReproducibilityInformationUtils.createNodeInformationTable(ReproducibilityInformationUtils.java:72) ~[h2o.jar:?]
    at hex.Model$Output.createReproducibilityInformationTable(Model.java:1199) ~[h2o.jar:?]
    at hex.Model$Output.<init>(Model.java:991) ~[h2o.jar:?]
    at hex.Model$Output.<init>(Model.java:973) ~[h2o.jar:?]
    at hex.tree.xgboost.XGBoostOutput.<init>(XGBoostOutput.java:16) ~[h2o.jar:?]
    at hex.tree.xgboost.XGBoost$XGBoostDriver.buildModelImpl(XGBoost.java:419) ~[h2o.jar:?]
    at hex.tree.xgboost.XGBoost$XGBoostDriver.buildModel(XGBoost.java:393) ~[h2o.jar:?]
    at hex.tree.xgboost.XGBoost$XGBoostDriver.computeImpl(XGBoost.java:379) ~[h2o.jar:?]
    at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:246) ~[h2o.jar:?]
    at water.H2O$H2OCountedCompleter.compute(H2O.java:1652) ~[h2o.jar:?]
    at jsr166y.CountedCompleter.exec(CountedCompleter.java:468) ~[h2o.jar:?]
    at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263) [h2o.jar:?]
    at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974) [h2o.jar:?]
    at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477) [h2o.jar:?]
    at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [h2o.jar:?] 

class XGBoostQuantileRegressor(XGBRegressor):
    def __init__(
            self, quant_alpha, n_estimators=100, max_depth=3, base_score=0.5, gpu_id=None,
            booster='gbtree', colsample_bylevel=1, colsample_bynode=1, colsample_bytree=1, gamma=0,
            importance_type=None, interaction_constraints=None, n_jobs=1, random_state=0,
            tree_method='auto', missing=1, objective='reg:linear', learning_rate=0.1,
            max_delta_step=0, min_child_weight=1, monotone_constraints=None, num_parallel_tree=1,
            reg_alpha=0, reg_lambda=1, scale_pos_weight=1, subsample=0.5, validate_parameters=False,
            verbosity=0
    ):
        self.quant_alpha = quant_alpha
        super().__init__(
            n_jobs=n_jobs, random_state=random_state, tree_method=tree_method, missing=1,
            n_estimators=n_estimators, max_depth=max_depth, objective=objective,
            base_score=base_score, booster=booster, colsample_bylevel=colsample_bylevel,
            colsample_bynode=colsample_bynode, colsample_bytree=colsample_bytree, gamma=gamma,
            gpu_id=gpu_id, importance_type=importance_type, learning_rate=learning_rate,
            interaction_constraints=interaction_constraints, max_delta_step=max_delta_step,
            min_child_weight=min_child_weight, monotone_constraints=monotone_constraints,
            num_parallel_tree=num_parallel_tree, reg_alpha=reg_alpha, reg_lambda=reg_lambda,
            scale_pos_weight=scale_pos_weight, validate_parameters=validate_parameters,
            verbosity=verbosity, subsample=subsample)

    def fit(self, X, y):
        super().set_params(
            objective=partial(XGBoostQuantileRegressor.log_cosh_loss, alpha=self.quant_alpha))
        super().fit(X, y)
        return self

    def predict(self, X):
        return super().predict(X)

    @staticmethod
    def log_cosh_loss(y_true, y_pred, alpha):
        err = y_pred - y_true
        err = np.where(err < 0, alpha * err, (1 - alpha) * err)
        grad = np.tanh(err)
        hess = 1 / np.cosh(err)**2
        return grad, hess

class XGBoostQuantileRegressor(XGBRegressor):
    def __init__(self, quant_alpha, max_depth=3):
        self.quant_alpha = quant_alpha
        super().__init__(max_depth=max_depth)

    def fit(self, X, y):
        super().set_params(
            objective=partial(XGBoostQuantileRegressor.log_cosh_loss, alpha=self.quant_alpha))
        super().fit(X, y)
        return self

    def predict(self, X):
        return super().predict(X)

    @staticmethod
    def log_cosh_loss(y_true, y_pred, alpha):
        err = y_pred - y_true
        err = np.where(err < 0, alpha * err, (1 - alpha) * err)
        grad = np.tanh(err)
        hess = 1 / np.cosh(err)**2
        return grad, hess

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/temp.py in <module>
      230 z1 = xgboost_quantile_regressor(alpha=0.95)[0][1]
----> 231 z1.fit(x_trainval, y_trainval)
      232 pred_y1 = z1.predict(x_trainval)
      233 
      234 z2 = xgboost_quantile_regressor(alpha=0.05)[0][1]

/temp.py in fit(self, X, y)

~/.local/lib/python3.9/site-packages/optuna/integration/sklearn.py in fit(self, X, y, groups, **fit_params)
    873         )
    874 
--> 875         self.study_.optimize(
    876             objective, n_jobs=self.n_jobs, n_trials=self.n_trials, timeout=self.timeout
    877         )

~/.local/lib/python3.9/site-packages/optuna/study/study.py in optimize(self, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
    398             )
    399 
--> 400         _optimize(
    401             study=self,
    402             func=func,

~/.local/lib/python3.9/site-packages/optuna/study/_optimize.py in _optimize(study, func, n_trials, timeout, n_jobs, catch, callbacks, gc_after_trial, show_progress_bar)
     64     try:
     65         if n_jobs == 1:
---> 66             _optimize_sequential(
     67                 study,
     68                 func,

~/.local/lib/python3.9/site-packages/optuna/study/_optimize.py in _optimize_sequential(study, func, n_trials, timeout, catch, callbacks, gc_after_trial, reseed_sampler_rng, time_start, progress_bar)
    161 
    162         try:
--> 163             trial = _run_trial(study, func, catch)
    164         except Exception:
    165             raise

~/.local/lib/python3.9/site-packages/optuna/study/_optimize.py in _run_trial(study, func, catch)
    262 
    263     if state == TrialState.FAIL and func_err is not None and not isinstance(func_err, catch):
--> 264         raise func_err
    265     return trial
    266 

~/.local/lib/python3.9/site-packages/optuna/study/_optimize.py in _run_trial(study, func, catch)
    211 
    212     try:
--> 213         value_or_values = func(trial)
    214     except exceptions.TrialPruned as e:
    215         # TODO(mamu): Handle multi-objective cases.

~/.local/lib/python3.9/site-packages/optuna/integration/sklearn.py in __call__(self, trial)
    219     def __call__(self, trial: Trial) -> float:
    220 
--> 221         estimator = clone(self.estimator)
    222         params = self._get_params(trial)
    223 

~/.local/lib/python3.9/site-packages/sklearn/base.py in clone(estimator, safe)
     80     for name, param in new_object_params.items():
     81         new_object_params[name] = clone(param, safe=False)
---> 82     new_object = klass(**new_object_params)
     83     params_set = new_object.get_params(deep=False)
     84 

TypeError: __init__() got an unexpected keyword argument 'objective'

def get_aws_sagemaker_model(model_loc):
    """
    TO BE USED IN A JUPYTER NOTEBOOK
    
    extracts a sagemaker model that has ran and been completed
    
    deletes the copied items and leaves you with the model
    
    note that you will need to have the package installed with correct
    versioning for whatever model you have trained
    ie. if you are loading an XGBoost model, have XGBoost installed
    
    Args:
        model_loc (str) : s3 location of the model including file name
        
    Return:
        model: unpacked and loaded model
    """ 
    
    import re
    import tarfile
    import os
    import pickle as pkl

    # extract the filename from beyond the last backslash
    packed_model_name = re.search("(.*\/)(.*)$" , model_loc)[2]
    
    # copy and paste model file locally
    command_string = "!aws s3 cp {model_loc} ."
    exec(command_string)
    
    # use tarfile to extract
    tar = tarfile.open(packed_model_name)
    
    # extract filename from tarfile
    unpacked_model_name = tar.getnames()[0]
    
    tar.extractall()
    tar.close()
    
    model = pkl.load(open(unpacked_model_name, 'rb'))
    
    # cleanup copied files and unpacked model
    os.remove(packed_model_name)
    os.remove(unpacked_model_name)
    
    return model

Traceback (most recent call last):

  File "/home/ec2-user/anaconda3/envs/env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "/tmp/ipykernel_10889/996524724.py", line 1, in <module>
    model = get_aws_sagemaker_model("my-model-loc")

  File "/home/ec2-user/SageMaker/env/src/utilities/model_helper_functions.py", line 167, in get_aws_sagemaker_model
    exec(command_string)

  File "<string>", line 1
    !aws s3 cp my-model-loc .
    ^
SyntaxError: invalid syntax

from dask_ml.model_selection import train_test_split
import dask
import xgboost
import dask_xgboost
from dask.distributed import Client
import dask_ml.model_selection as dcv

#split the data
x_train,x_test,y_train,y_test=train_test_split(x,y,test_size=0.33,random_state=42)

client = Client(n_workers=10, threads_per_worker=1)

#Trying to do hyperparamter running
model_xgb = xgb.XGBRegressor(seed=42,verbose=True)


params={
    'learning_rate':[0.1,0.01,0.05],
    'max_depth':[1,5,8],
    'gamma':[0,0.5,1],
    'scale_pos_weight':[1,3,5]
}

grid_search = GridSearchCV(model_xgb, params, cv=3, scoring='neg_mean_squared_error')

grid_search.fit(x_train, y_train)

#train data with best paraeters
bst = dask_xgboost.train(client, grid_search.best_params_, x_train, y_train, num_boost_round=10)

#predict data
dask_xgboost.predict(client, bst, x_test).persist()

dask_xgboost.predict(client, bst, x_test).persist().compute()
>>>ValueError: Data must be 1-dimensional

model = XGBRegressor()
model.fit(X_train,y_train["speed"])
y_predict_speed = model.predict(X_test)

from sklearn.metrics import r2_score
print("R2 score:", r2_score(y_test["speed"],y_predict_speed, multioutput='variance_weighted'))
R2 score: 0.3540656307310167

booster=['gbtree','gblinear']
base_score=[0.25,0.5,0.75,1]

## Hyper Parameter Optimization
n_estimators = [100, 500, 900, 1100, 1500]
max_depth = [2, 3, 5, 10, 15]
booster=['gbtree','gblinear']
learning_rate=[0.05,0.1,0.15,0.20]
min_child_weight=[1,2,3,4]

# Define the grid of hyperparameters to search
hyperparameter_grid = {
    'n_estimators': n_estimators,
    'max_depth':max_depth,
    'learning_rate':learning_rate,
    'min_child_weight':min_child_weight,
    'booster':booster,
    'base_score':base_score
    }

# Set up the random search with 4-fold cross validation
random_cv = RandomizedSearchCV(estimator=regressor,
            param_distributions=hyperparameter_grid,
            cv=5, n_iter=50,
            scoring = 'neg_mean_absolute_error',n_jobs = 4,
            verbose = 5, 
            return_train_score = True,
            random_state=42)

random_cv.fit(X_train,y_train["speed"])

random_cv.best_estimator_

XGBRegressor(base_score=0.5, booster='gblinear', colsample_bylevel=None,
             colsample_bynode=None, colsample_bytree=None, gamma=None,
             gpu_id=-1, importance_type='gain', interaction_constraints=None,
             learning_rate=0.15, max_delta_step=None, max_depth=15,
             min_child_weight=3, missing=nan, monotone_constraints=None,
             n_estimators=500, n_jobs=16, num_parallel_tree=None,
             random_state=0, reg_alpha=0, reg_lambda=0, scale_pos_weight=1,
             subsample=None, tree_method=None, validate_parameters=1,
             verbosity=None)

regressor = XGBRegressor(base_score=0.5, booster='gblinear', colsample_bylevel=None,
             colsample_bynode=None, colsample_bytree=None, gamma=None,
             gpu_id=-1, importance_type='gain', interaction_constraints=None,
             learning_rate=0.15, max_delta_step=None, max_depth=15,
             min_child_weight=3, monotone_constraints=None,
             n_estimators=500, n_jobs=16, num_parallel_tree=None,
             random_state=0, reg_alpha=0, reg_lambda=0, scale_pos_weight=1,
             subsample=None, tree_method=None, validate_parameters=1,
             verbosity=None)

regressor.fit(X_train,y_train["speed"])
y_pred = regressor.predict(X_test)

from sklearn.metrics import r2_score
print("R2 score:", r2_score(y_test["speed"],y_pred, multioutput='variance_weighted'))

R2 score: 0.14258774171629718

bst = xgb.train(params, dtrain)
bst.params # does not work!