Detectron.pytorch | pytorch implementation of Detectron | Computer Vision library

When training a multi-object detector, you usually have (at least) two types of losses:
1. loss_bbox a loss that measure how "tight" the bounding box the model predicted around the true object (usually a regression loss, L1, smoothL1 etc.).
2. loss_cls a loss that measure the correctness of the classification of each predicted bounding box: each box may contain an object class, or a "background". This loss is usually cross entropy loss.

When training a detector, the model predict quite a few (~1K) possible boxes per image. Most of them are empty (i.e. belongs to "background" class). The loss function associate each of the predicted boxes with the ground truth boxes annotation of the image.
If a predicted box has a significant overlap with a ground truth box then loss_bbox and loss_cls are computed to see how well the model is able to predict the ground truth box.
On the other hand, if a predicted box has no overlap with any ground truth box, than only loss_cls is computed for the "background" class.
However, if there is only very partial overlap with ground truth the predicted box is "discarded" and no loss is computed. I suspect, for some reason, this is the case for your training session.
I suggest you check the parameters that determines the association between predicted boxed and ground truth annotations. Moreover, look at the parameters of your "anchors": these parameters determines the scale and aspect ratios of the predicted boxes.