caretEnsemble | caret models all the way down turtle | Machine Learning library

No, you can only do a stacked ensemble of H2O models, i.e. the models must have been trained on H2O cluster.

You are almost there. One of the things to remember is that when you want to use caretEnsemble is that in trainControl you have to set the resample index via the 'index' option in trainControl. If you run caretList it tends to set this itself, but it is better to do this yourself. This is especially true when you run different models outside of caretList. You need to make sure the resampling is the same. You can also see this in the example on github you refer to.

trainControl does not by default generate you the indices, it acts as a way of passing all the parameters to each model you are training.

When we search github issues regarding the error we can find this particular issue.

You need to make sure that every model is fit with the EXACT same resampling folds. caretEnsemble builds the ensemble by merging together the test sets for each cross-validation fold, and you will get incorrect results if each fold has different observations in it.

Before you fit your models, you need to construct a trainControl object, and manually set the indexes in that object.

E.g. myControl <- trainControl(index=createFolds(y, 10)).

We are working on an interface to caretEnsemble that handles constructing the resampling strategy for you and then fitting multiple models using those resamples, but it is not yet finished.

To reiterate, that check is there for a reason. You need to set the index argument in trainControl, and pass the EXACT SAME indexes to each model you wish to ensemble.

So what that means is when you specify number = 5 and repeats = 3 the models aren't actually getting a predetermined index for what samples belong to each fold but are rather generating their own independently.

Therefore to ensure that the models are consistent with one another regarding which samples belong to which folds you must specify index = createFolds(iris$Species, 5) in your trainControl object

ok this is actually addressed well in the docs.

A bit of debugging and the error comes from a function called bestPreds. This is a not exported function and looks in the model_lists for the saved predictions (all or final) in the control object. This you have not set in your control object. If you add this, everything will run fine. I do admit that an error message would be nice in this place instead of just throwing an error.

There's a note in the caret documentation that num.round needs to be set by the user outside the tune_grid: http://topepo.github.io/caret/train-models-by-tag.html

This code:

Just use the newdata parameter and it will work