estimators | Machine Learning Versioning made Simple | Machine Learning library

There is no one such resource from AWS that provides the comprehensive view of how to use SageMaker SDK Estimator to train and save models.

I put a diagram and brief explanation to get the overview on how SageMaker Estimator runs a training.

1. SageMaker sets up a docker container for a training job where:

   • Environment variables are set as in SageMaker Docker Container. Environment Variables.
   • Training data is setup under /opt/ml/input/data.
   • Training script codes are setup under /opt/ml/code.
   • /opt/ml/model and /opt/ml/output directories are setup to store training outputs.

There is no contradiction, because estimators_ is not used when training the metaclassifier. After the cross-val-predictions are made, you don't actually have fitted base estimators (or rather, you have multiple copies of each, depending on your cv parameter). For predicting on new data, you need a single fitted copy of each base estimator; those are obtained by fitting on the full X, and are stored in the attribute estimators_.

Thanks for your questions. Here are answers:

• SageMaker PySpark SDK https://sagemaker-pyspark.readthedocs.io/en/latest/ does the opposite of what you want: being able to call a non-spark (or spark) SageMaker job from a Spark environment. Not sure that's what you need here.

• Running Spark in SageMaker jobs. While you can use SageMaker Notebooks to connect to a remote EMR cluster for interactive coding, you do not need EMR to run Spark in SageMaker jobs (Training and Processing). You have 2 options:

  • SageMaker Processing has a built-in Spark Container, which is easy to use but unfortunately not connected to SageMaker Model Tuning (that works with Training only). If you use this, you will have to find and use a third-party, external parameter search library ; for example Syne Tune from AWS itself (that supports bayesian optimization)

  • SageMaker Training can run custom docker-based jobs, on one or multiple machines. If you can fit your Spark code within SageMaker Training spec, then you will be able to use SageMaker Model Tuning to tune your Spark code. However there is no framework container for Spark on SageMaker Training, so you would have to build your own, and I am not aware of any examples. Maybe you could get inspiration from the Processing container code here to build a custom Training container

Your idea of using the Training job as a client to launch an EMR cluster is good and should work (if SM has the right permissions), and will indeed allow you to use SM Model Tuning. I'd recommend:

• each SM job to create a new transient cluster (auto-terminate after step) to keep costs low and avoid tuning results to be polluted by inter-job contention that could arise if running everything on the same cluster.
• use the cheapest possible instance type for the SM estimator, because it will need to stay up during all duration of your EMR experiment to collect and print your final metric (accuracy, duration, cost...)

In the same spirit, I once used SageMaker Training myself to launch Batch Transform jobs for the sole purpose of leveraging the bayesian search API to find an inference configuration that minimizes cost.

Imo the point here is the following. On one side, NN models do support multi-output regression tasks on their own, which might be solved defining an output layer similar to the one you built, namely with a number of nodes equal to the number of outputs (though, with respect to your construction, I would specify a linear activation with activation=None rather than a sigmoid activation).

The point of refit is that the model will be refitted using the best parameter set found before and the entire dataset. To find the best parameters, cross-validation is used which means that the dataset is always split into a training and a validation set, i.e. not the entire dataset is used for training here.

When you define multiple metrics, you have to tell scikit-learn how it should determine what best means for you. For convenience, you can just specify any of your scorers to be used as the decider so to say. In that case, the parameter set that maximizes this metric will be used for refitting.

If you want something more sophisticated, like taking the parameter set that returned the highest mean of all scorers, you have to pass a function to refit that given all the created metrics returns the index of the corresponding best parameter set. This parameter set will then be used to refit the model.

Those metrics will be passed as a dictionary of strings as keys and NumPy arrays as values. Those NumPy arrays have as many entries as parameter sets that have been evaluated. You find a lot of things in there. What is probably the most relevant is mean_test_*scorer-name*. Those arrays contain for each tested parameter set the mean scorer-name-scorer computed across the cv splits.

In code, to get the index of the parameter set, that returns the highest mean across all scorers, you can do the following

Referring to this example you could just make a list of dictionaries. One containing sfm and its related parameters and the other one not using "passthrough".

The problem is the {\iffalse}\fi in the abc bib entry. This syntax makes no sense, just remove it.

Now I know why people do this sort of work in R: Because R is fantastic for this kind of work. If anyone comes across this in the future, go get R. It's a compact, easy-to-use, easy-to-learn language with a great IDE.

If you've got a Postgres server where you can install PL/R, so much the better. PL/R is written to use the DBI and RPostgreSQL R packages to connect with Postgres. Meaning, you should be able to develop your code in RStudio, and then add the bits of wrapping required to make it run in PL/R within your Postgres server.

For outliers, I'm happy with univOutl (Univariate Outliers) so far, which provides 10 common, and less common, methods.

I am not an expert with scikit-learn but it seems that one of the requirements of various objects used by this framework is that they can be cloned by calling the sklearn.base.clone method. This appears to be something that the existing XGBRegressor class does, so is something your subclass of XGBRegressor must also do.

What may help is to pass any other unexpected keyword arguments as a **kwargs parameter. In your constructor, kwargs will contain a dict of all of the other keyword parameters that weren't assigned to other constructor parameters. You can pass this dict of parameters on to the call to the superclass constructor by referring to them as **kwargs again: this will cause Python to expand them out: