BBox-Label-Tool | A simple tool for labeling object bounding boxes in images | Data Labeling library

I'm trying to train my yolo model to identify fire extinguishers and to label it as "Fire Safety". Currently is either I get a overfit or underfit images(see below).

My sample images size with annotations is around ~1500

yolo-new.cfg config of width=608 and height=608

And I have trained using the following command:

python flow --model cfg/yolo-new.cfg --labels one_label.txt --train --trainer adam --dataset "C://Users//G//Desktop//Development//ML//YOLO//BBox-Label-Tool//Images//002" --annotation "C://Users//G//Desktop//Development//ML//YOLO//BBox-Label-Tool//AnnotationsXML//002" --batch 4 --gpu 0.8

So after 13000 steps:

So I went to validate my results and this is what I get(Checkpoint 13000):

So perhaps I thought this might be a case of severe overfitting, thus I iterate through the checkpoints to see which has the closest fit.

This is what I get using checkpoint 6500

This is what I get using checkpoint 6000

This is what I get using checkpoint 5500

So, as you can see checkpoint 6000 is the best possible result in my case but it isn't good enough. How do I improve on this? Increase batch size ?(My GPU 1070Ti cant handle. Cuda out of memory occurs) Any Ideas to solve this?