pytorch-lr-finder | A learning rate range test implementation in PyTorch | Machine Learning library

The code looks like it's using the implementation correctly. To answer your last question,

Can see the training accuracy is much lower 84.09833333333333 versus 9.93 . Should the learning rate finder find a learning rate that allows to achieve greater training set accuracy ?

Not really. A few points

1. You are using Adam, which scales the learning rate adaptively for each parameter in the network. The initial learning rate will matter less, as opposed to traditional SGD, for example. The original authors of Adam write

   The hyper-parameters have intuitive interpre-tations and typically require little tuning. [1]

2. A well tuned learning rate should make your network converge faster (i.e in less epochs). It might still find the same local minima as a higher learning rate, but faster. The risk with too high learning rates is that you overshoot the local minima and instead find a poor one. With a tiny learning rate you should get the best training accuracy, but it will take very long.

3. You are training your model for only 2 epochs. If I had to guess, the algorithm has found that a small learning rate leads to good optima, but since it is small, it requires more time to converge. To test this theory, I would recommend running your training longer.

All that said, your time is probably better spent using Adam with default parameters and directing your attention elsewhere, such as modelling choices (layers, nodes, activations, etc). In my experience standard Adam works really well in most cases.

[1] https://arxiv.org/abs/1412.6980