Autotuner | repo contains the code | Machine Learning library

Hi thanks for using mlr3proba. I have actually just finished writing a tutorial that answers exactly this question! It covers training, tuning, and evaluating the neural networks in mlr3proba. For your specific question, the relevant part of the tutorial is this:

To see whether tuning has an effect, you can just add an untuned learner to the benchmark. Otherwise, the conclusion could be that tuning alpha is sufficient for your example.

I adapted the code so that it runs with an example task.

Repeated hyperparameter tuning (alpha and lambda) of glmnet can be done using the SECOND mlr3 approach as stated above. The coefficients can be extracted with stats::coef and the stored values in the AutoTuner

The result shows that the 3 hyperparameters chosen from 3 inner resampling are not garantee to be the same.

It sounds like you want to fit a final model with the hyperparameters selected in the inner resamplings. Nested resampling is not used to select hyperparameter values for a final model. Only check the inner tuning results for stable hyperparameters. This means that the selected hyperparameters should not vary too much.

1. Yes, you are comparing the aggregated performance of all outer resampling test sets (rr$aggregate()) with the performances estimated on the inner resampling test sets (lapply(rr$learners, function(x) x$tuning_result)).

2. The aggregated performance of all outer resampling iterations is the unbiased performance of a ranger model with optimal hyperparameters found by grid search. You can run at$train(task) to get a final model and report the performance estimated with nested resampling as the unbiased performance of this model.

The reason why you can't access $importance of the at variable is that it is an AutoTuner, which does not directly offer variable importance and only "wraps" around the actual Learner being tuned.

The trained GraphLearner is saved inside your AutoTuner under $learner:

You can use getBMRModels() to get the models, which will tell you what hyperparameters were used to fit them. See the benchmark section of the documentation.

As ?AutoTuner tells, this class fits a model with the best hyperparameters found during the tuning. This model is then used for prediction, in your case to newdata when calling its method .$predict_newdata().

Also in ?AutoTuner you see the documentation linked to ?TuningInstance. This then tells you what the $tune_x and params slots represent. Try to look up the help pages next time - that's what they are there for ;)

This seems to initiate tuning again.

Why again? It does it in the first place, on all observations of task. I assume you might confuse yourself by the common misconception between "train/predict" vs. "resample". Read more about the theoretical differences of both to understand what both are doing. They have completely different aims and are not connected. Maybe the following reprex makes it more clear.

I finally found a workaround. I took the idea to connect via the public IP of the CloudSQL instance. For that you needed to allow connections to your CloudSQL instance from every IP:

1. Go to the overview page of your CloudSQL instance in GCP
2. Click on the Authorization tab
3. Click on Add network and add 0.0.0.0/0 (!! this will allow every IP address to connect to your instance !!)

To add security to the process, I used SSL keys and only allowed SSL connections to the instance:

1. Click on SSL tab
2. Click on Create a new certificate to create a SSL certificate for your server
3. Click on Create a client certificate to create a SSL certificate for you client
4. Click on Allow only SSL connections to reject all none SSL connection attempts

After that I stored the certificates in a Google Cloud Storage bucket and load them before connecting within the Dataflow job, i.e.:

I think that I've found the answer to what I'm looking for. In brief, what I'd like to do is:

Create a graph pipeline with multiple learners. I'd like some of the learners to be inserted with fixed hyperparameters, while for others I'd like to have their hyperparameters tuned. Then, I'd like to benchmark them and select the 'best' one. I'd also like the benchmarking of learners to happen under different class balancing strategies, namely, do nothing, up-sample and down-sample. The optimal parameter settings for the up/down-sampling (e.g. ratio) would also be determined during tuning.

Two examples below, one that almost does what I want, the other doing exactly what I want.

Example 1: Build a pipe that includes all learners, that is, learners with fixed hyperparameters, as well as learners whose hyperparameters require tuning

As will be shown, it seems like a bad idea to have both kinds of learners (i.e. with fixed and tunable hyperparameters), because tuning the pipe disregards the learners with tunable hyperparameters.