mlr | Machine Learning in R | Machine Learning library

I would suggest conditional aggregation:

Combine Get-ChildItem (or, since you're not recursing, just Get-Item) with * and -Include, which (unlike -Filter) accepts an array of wildcard patterns:^[1]

Hi thanks for using mlr3proba! The reason for this is because the parameter names change when wrapped in the pipeline, you can see this in the example below. There are a few options to solve this, you could change the parameter ids to match the new names after wrapping in PipeOps (Option 1 below), or you could specify the tuning ranges for the learner first then wrap it in the PipeOp (Option 2 below), or you could use an AutoTuner and wrap this in the PipeOps. I use the final option in this tutorial.

As far as I understand it, LeDell2015 proposes and evaluate a general strategy that optimizes AUC as a black-box function by learning optimal weights. They do not really propose a best strategy or any concrete defaults so I looked into the defaults of the SuperLearner package's AUC optimization strategy.

Assuming I understood the paper correctly:

The LearnerClassifAvg basically implements what is proposed in LeDell2015 namely, it optimizes the weights for any metric using non-linear optimization. LeDell2015 focus on the special case of optimizing AUC. As you rightly pointed out, by setting the measure to "classif.auc" you get a meta-learner that optimizes AUC. The default with respect to which optimization routine is used deviates between mlr3pipelines and the SuperLearner package, where we use NLOPT_LN_COBYLA and SuperLearner ... uses the Nelder-Mead method via the optim function to minimize rank loss (from the documentation).

So in order to get exactly the same behaviour, you would need to implement a Nelder-Mead bbotk::Optimizer similar to here that simply wraps stats::optim with method Nelder-Mead and carefully compare settings and stopping criteria. I am fairly confident that NLOPT_LN_COBYLA delivers somewhat comparable results, LeDell2015 has a comparison of the different optimizers for further reference.

Thanks for spotting the error in the documentation. I agree, that the description is a little unclear and I will try to improve this!

Hi thanks for using mlr3proba. I have actually just finished writing a tutorial that answers exactly this question! It covers training, tuning, and evaluating the neural networks in mlr3proba. For your specific question, the relevant part of the tutorial is this:

The key issues with your code are the following:

1. While it is necessary to add a column of ones to the features matrix x_data before running the regression with statsmodels, this is not necessary when running the regression with tensorflow. This means that you are passing 3 features to tensorflow instead of 2, where the additional feature (the first column of x_data) is constant.
2. You are normalizing x_data after the first column of ones has already been added with x_data = sm.add_constant(x_data). As a column of ones has zero variance, after normalization you get a column of nan (as you are dividing by zero). This means that the first of the 3 features that you are passing to tensorflow is completely missing (i.e. it's always nan).
3. While statsmodels takes as inputs at first y and then X, tensorflow takes as inputs at first X and then y. This means that you have switched the features and target when running the regression in tensorflow.

I included a complete example below.

Try

According to jakob-r's comment, this function is not yet available in mlr3. It might become available in mlr3.

While the documentation of the AUC filter states that it is analogous to mlr3measures::auc, it does not actually use that function. Instead, it uses its own implementation which calculates the Area Under the ("Receiver Operating Characteristic") Curve, which does not require a probability value, only a continuous value that can be used to divide samples with a cutoff.

The Filters, as I understand them, are primarily used to calculate filter scores (see the documentation of the Filter base class). They are mostly useful in combination with the PipeOpFilter from the mlr3pipelines package, which is also what the example from the book that you cite is doing. PipeOps are generally useful to integrate a filtering step (or any kind of preprocessing) into a learning process (the book chapter on mlr3pipelines is probably a good point to learn about that). However, these can also be used to compute steps, e.g. filter out columns from a Task: