deep-q-learning | PyTorch Implementation of Deep Q | Reinforcement Learning library

In Reinforcement Learning (RL) there is often a lot of CPU computation required for each sample step (of course dependent on environment, some environments can use GPU too). The RL-model has a hard time understanding the rewards and what action caused that specific reward, since a good reward could be dependent on a way earlier action. Therefore we want a simple model-architectures (shallow and fewer weights) while doing RL, else the training time will be way to slow. Hence your systems bottle neck is likely gathering samples rather than training the data. Also Note that not all Tensorflow-architectures scale equally well with GPU. Deep models with high numbers of weights like most Images cases scales super well (like CNN and MLP network with MNIST), while time-dependent RNN has less speedup potential (see this stackexchange question). So set your expectation accordingly when using GPU.

Through my RL experience, I have figured some possible speedups I could share, and would love to see more suggestions!

1. Single sample step, can be speed up by creating multiple environment run in parallel, equal to the number of CPU cores (there are packages for parallel processing in python you can use fore this). This can potential speed up sampling data proportional to the number of CPU cores.

2. Between sampling you have to do model predictions for next action. Instead of calling model.predict at each step, you can call a single model.predict for all your parallel states (using a batch_size equal to the number of parallel environments). This will speed up prediction time, as there is more optimization options.

3. The change from updating model weights to prediction is surprisingly slow. Hopefully this will be speed up in the future? But while the change is as slow as today, you can speed up training by holding the model constant and do lots of sample and prediction (example a whole episode, or multiple steps within an episode), then train the model on all the newly gathered data afterwards. In my case this resulted in periodically high GPU utilization.

4. Since sampling is most likely the bottle neck, you can make a historical repo of state, action, rewards. Than at training you can sample randomly data from this repo and train it together with the newly gathered data. This is known as "Experience Replay" in RL.

5. Maybe the most fun, and highest potential for improvements is by using more advance RL-learning architectures. Example changing the loss function (check out PPO for example), using and tuning the "generalized advantage estimation" calculated by the rewards. Or changing the model by for example including time dependencies with RNN, VAC or combining them all like here.

Hopefully this help you speed up the training time, and maybe get more utilization of your GPU.