TensorFlow-cn | 简单粗暴 TensorFlow A Concise Handbook | Machine Learning library

I am using tensorflow's object detection api with faster_rcnn_resnet101 and get the following error when trying to train:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Input to reshape is a tensor with 36 values, but the requested shape requires a multiple of 16

[[{{node Reshape_13}}]]

[[IteratorGetNext]]

[[IteratorGetNext/_7243]]

(1) Invalid argument: Input to reshape is a tensor with 36 values, but the requested shape requires a multiple of 16

[[{{node Reshape_13}}]]

[[IteratorGetNext]]

0 successful operations. 0 derived errors ignored.

I am using a slightly modified version of the pets-train.sh file to run the training (only paths have been altered). I am trying to train on tf.record files containing jpg images of size (1280, 720) and have made no changes to the network architecture (I have confirmed that all images in the record are of this size).

Curiously, I can successfully run inference on these images when I do something equivalent to what's in the tutorial file detect_pets.py. This makes me think something is wrong with the way that I've created the tf.record files (code below) rather than anything to do with the shape of the images, despite the error having to do with reshape. However,I've successfully trained on tf.records created in the same way before (from images of size (600, 600), (1024, 1024), and (720, 480), all with the same network). Moreover, I've previously encountered a similar error (only the numbers were different but the error was still with node Reshape_13) on a different data set of images with size (600, 600).

I am using python 3.7, tf version 1.14.0, cuda 10.2, Ubuntu 18.04

I've looked extensively at various other posts (here, here, here, here, and here) but I wasn't able to make any progress.

I've tried adjusting the keep_aspect_ratio_resizer parameters (originally min_dimension=600, max_dimension=1024 but I've also tried min, max = (720, 1280) and have tried pad_to_max_dimension: true with both of these min/max choices as well).

This is the code I'm using to create the tf.record file (apologies or indentations being off here):