NDCG | cat label_qid_score

The Vespa index has no user documents here, so most likely the user and news embeddings have not been fed to the system. After they are calculated in the previous step (https://docs.vespa.ai/en/tutorials/news-4-embeddings.html), be sure to feed them to Vespa:

I do not know what the val_ndcg_metric is but apparently you want it to increase as the model trains. In the callback you set mode='auto'. Try setting mode='max'. This will halt training if the value of the val_ndcg_metric stops increasing for a patience number of epochs.

Evidently, you have overfitted your model. You do not share with us how you initially evaluated your model and achieved 0.78 NDCG, but I hope you did everything as you should.

You do not share a lot of information concerning your data. For example, do you have enough samples? How many features do you have? Maybe you have more features than samples and that is why you try to perform Feature Selection. You could also check how different is your validation set (the one your teacher provided) and your training set. Also check what happens if you use this validation set as part of your training set by training the model using Cross-Validation. I would check what are the performances across folds and the variance of those performances. If they vary a lot then the problem might stem from the data.

Despite this, I would advise you not to perform hyper-parameter tuning manually and on a single validation set. The main reason is because you will simply overfit on this validation set and when the test set comes along your performance will not be as you have anticipated. For that reason, you can use Randomised Search using Cross-Validation after you carefully set your hyper-parameter space.sklearn has a really nice and easy to use implementation. You can checkout other techniques like Halving Randomised Search; also implemented by sklearn.

Even if you perform hyper-parameter tuning correctly, the performance improvement will not be as high as you are hoping. Hyper-parameter tuning normally boosts your performance by 1-5%. Therefore, I would recommend you to check your features. Maybe you can generate new ones from the current feature space or create cross-features, discard collinear features etc.

Remove space from 'binary:logistic' and it should work. According to this this documentation there is no space in between.

Yes, this is possible. You would want to apply a listwise learning to rank approach instead of the more standard pairwise loss function.

In pairwise loss, the network is provided with example pairs (rel, non-rel) and the ground-truth label is a binary one (say 1 if the first among the pair is relevant, and 0 otherwise).

In the listwise learning approach, however, during training you would provide a list instead of a pair and the ground-truth value (still a binary) would indicate if this permutation is indeed the optimal one, e.g. the one which maximizes nDCG. In a listwise approach, the ranking objective is thus transformed into a classification of the permutations.

For more details, refer to this paper.

Obviously, the network instead of taking features as input may take BERT vectors of queries and the documents within a list, similar to ColBERT. Unlike ColBERT, where you feed in vectors from 2 docs (pairwise training), for listwise training u need to feed in vectors from say 5 documents.

1. If you add labels to your plot functions, then you won't have to supply legend() with handles and labels - this is more convenient.

2. I would recommend using a loop structure instead of multiple if statements.

3. Regarding the legend, using ncol parameter is going to help you a lot here. You may find the matplotlib documentation legend tutorial helpful

4. If you're working with multiple subplots with different sizes, then I'd recommend using gridspec, otherwise just use plt.subplots() with ncols and nrows parameters. For example:

   fig, axes = plt.subplots(ncols=2, nrows=5, figsize=(12,12))
   axes = axes.flatten() #this results in a 1d array of 10 axes

I simulated your data and implemented what I think you are looking for below.

Trying to obtain such metrics which include ranking (see the docs) for single true-predicted pairs does not make any sense (although admittedly the error message is not very informative here); you need at least two pairs:

Suppose you have N observations in y_train. You have to transform y_train to a matrix of N rows and 12 columns.

The model is an attribute of tf.keras.callbacks.Callback, so you can access it directly with self.model. For accessing the value of the loss, you can use the "logs" object that is passed to the methods of tf.keras.callbacks.Callback, that will contain a key named "loss".

If you need to access to other variables (that won't change during the training), then you can set them as instance variables of your callback, and add them during the construction of the callback by defining the __init__ function.