ROC | Distributed Multi-GPU GNN Framework | GPU library

Function roc_curve expects array with true labels y_true and array with probabilities for positive class y_score (which usually means class 1). Therefore what you need is not

I think the problem is that y_hat = rf1.predict(X_test) is returning binary classification output (0 and 1). For the ROC AUC you need a probability or score. Instead, you should use predict_proba:

A ROC curve is based on soft predictions, i.e. it uses the predicted probability of an instance to belong to the positive class rather than the predicted class. For example with sklearn one can obtain the probabilities with predict_proba instead of predict (for the classifiers which provide it, example).

Note: OP used the tag multiclass-classification, but it's important to note that ROC curves can only be applied to binary classification problems.

One can find a short explanation of ROC curves here.

Here is complete code to plot all ROC curves of a data set df with the same structure of the data set in the question. I first first create a data set because the one in the question only as one class (label is always 0). Then,

1. Get the current directory, create a temporary directory to save the graphics files;
2. in the for loop, compute the predictions and their performance;
3. open a graphics device with png();
4. plot the performance, saving it to disk and close the device.

There are now as many "Perf_X?.png" files as variables "X?"in the data.frame. Thesepng` file related instructions can be removed but with 80 plots, it's better to saved them and see them one by one later.

The order of your factor levels is ignored by geom_roc. Notice that whichever way round your assign your levels = c('R', 'M'), you get the warning:

The issue lies in your squeeze. You don't need to specify the index when using squeeze. squeeze flattens the array into 1D. If you pick [:,0,:], it's only 1 entry and hence the error.

Simply do

You should stop trying to invent the wheel, and instead start to leverage the built-in capabilities of Azure Databricks. Because Apache Spark (and Databricks) is the distributed system, machine learning on it should be also distributed. There are two approaches to that:

1. Training algorithm is implemented in the distributed fashion - there is a number of such algorithms packaged into Apache Spark and included into Databricks Runtimes

2. Use machine learning implementations designed to run on a single node, but train multiple models in parallel - that what typically happens during hyper-parameters optimization. And what is you're trying to do

Databricks runtime for machine learning includes the Hyperopt library that is designed for the efficient finding of best hyper-parameters without trying all combinations of the parameters, that allows to find them faster. It also include the SparkTrials API that is designed to parallelize computations for single-machine ML models such as scikit-learn. Documentation includes a number of examples of using that library with single-node ML algorithms, that you can use as a base for your work - for example, here is an example for scikit-learn.

P.S. When you're running the code with multiprocessing, then the code is executed only on the driver node, and the rest of the cluster isn't utilized at all.

To solve this problem, probability = TRUE need to be specified both when constructing the classifier and making predictions:

Did you look at the probabilities versus the fitted values? You can read about how probability works with SVM here.

If you want to look at the performance you can use the library DescTools and the function Conf or with the library caret and the function confusionMatrix. (They provide the same output.)