conv_opt | Python package for linear and quadratic programming | Robotics library

I think the reason is that when you add the tf.train.AdamOptimizer(0.005).minimize(cost) op, it is implicitly assumed that you optimize over all trainable variables (because you didn't specify otherwise). Therefore, you need to know the values of these variables and of all the intermediate tensors which take part in the calculation of cost, including the gradients (which are tensors too and are implicitly added to the computational graph). Now lets count the variables and tensors from perceptron:

1. W
2. b
3. tf.reshape(x, [-1,N])
4. tf.matmul( ..., W)
5. its gradient with respect to the first argument.
6. its gradient with respect to the second argument.
7. tf.add(..., b, name="y")
8. its gradient with respect to the first argument.
9. its gradient with respect to the second argument.
10. tf.nn.sigmoid(y, name="sigmoid")
11. its gradient.
12. tf.reshape(act, [-1, 64, 64, 3], name="yhat")

I'm not actually 100% sure that this is how the accounting is done, but you get the idea of where the number 12 could have come from.

Just as an exercise, we can see that this type of accounting also explains where the number 9 comes from in your chart:

1. x - yhat
2. its gradient with respect to the first argument
3. its gradient with respect to the second argument
4. np.square(...)
5. its gradient
6. tf.reduce_mean(..., axis=1)
7. its gradient
8. tf.reduce_mean( sq_error, name="cost" )
9. its gradient