linearlr | free learning rate schedule for general and low-budget | Machine Learning library
kandi X-RAY | linearlr Summary
kandi X-RAY | linearlr Summary
This repository implements a tuning-free learning rate schedule for training deep neural networks and it is studied in Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints. This linearly decaying schedule outperforms the popular step decay on many benchmark tasks and is very useful in a low-budget setting (e.g. a small number of epochs).
Support
Quality
Security
License
Reuse
Top functions reviewed by kandi - BETA
- Gets the learning rate
- Returns the closed form l
linearlr Key Features
linearlr Examples and Code Snippets
Lines of Code : 6License : Permissive (MIT)
Copy@article{Li2020BudgetTrain,
title={Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints},
author={Li, Mengtian and Yumer, Ersin and Ramanan, Deva},
journal={ICLR},
year={2020}
}
from linearlr import LinearLR
schedule = LinearLR(optimizer, T)
Community Discussions
Trending Discussions on linearlr
QUESTION
Using implementation of lr_finder from https://github.com/davidtvs/pytorch-lr-finder based on paper https://arxiv.org/abs/1506.01186
Without the learning rate finder :
...ANSWER
Answered 2019-Feb-06 at 16:27The code looks like it's using the implementation correctly. To answer your last question,
Can see the training accuracy is much lower 84.09833333333333 versus 9.93 . Should the learning rate finder find a learning rate that allows to achieve greater training set accuracy ?
Not really. A few points
You are using Adam, which scales the learning rate adaptively for each parameter in the network. The initial learning rate will matter less, as opposed to traditional SGD, for example. The original authors of Adam write
The hyper-parameters have intuitive interpre-tations and typically require little tuning. [1]
A well tuned learning rate should make your network converge faster (i.e in less epochs). It might still find the same local minima as a higher learning rate, but faster. The risk with too high learning rates is that you overshoot the local minima and instead find a poor one. With a tiny learning rate you should get the best training accuracy, but it will take very long.
You are training your model for only 2 epochs. If I had to guess, the algorithm has found that a small learning rate leads to good optima, but since it is small, it requires more time to converge. To test this theory, I would recommend running your training longer.
All that said, your time is probably better spent using Adam with default parameters and directing your attention elsewhere, such as modelling choices (layers, nodes, activations, etc). In my experience standard Adam works really well in most cases.
Community Discussions, Code Snippets contain sources that include Stack Exchange Network
Vulnerabilities
No vulnerabilities reported
Install linearlr
You can use linearlr like any standard Python library. You will need to make sure that you have a development environment consisting of a Python distribution including header files, a compiler, pip, and git installed. Make sure that your pip, setuptools, and wheel are up to date. When using pip it is generally recommended to install packages in a virtual environment to avoid changes to the system.
Support
Reuse Trending Solutions
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from over 650 million Knowledge Items
Find more librariesStay Updated
Subscribe to our newsletter for trending solutions and developer bootcamps
Share this Page
