ddpg | Implements the DDPG model from Lillicrap et al | Machine Learning library

The issue has been resolved thanks to some simple but helpful advice I received on Reddit. I was disrupting the tracking of my variables by making changes using my custom for-loop. I should have used a TensorFlow function instead. The following changes fixed the problem for me:

You can always create your own/custom policy network then you have full control over the layers and also the initialization of the weights.

If you want to use the default model you have the following params to adapt it to your needs:

As you mentioned in your question, PPO, DDPG, TRPO, SAC, etc. are indeed suitable for handling continuous action spaces for reinforcement learning problems. These algorithms will give out a vector of size equal to your action dimension and each element in this vector will be a real number instead of a discrete value. Note that stochastic algorithms like PPO will give a multivariate probability distribution from which you sample the actions.

Most of the robotic environments in Mujoco-py, PyBullet, Robosuite, etc. are environment with multiple continuous action spaces. Here the action spaces can be of the form [torque_for_joint_1, torque_for_join_2, ..., torque_for_joint_n] where torque_for_joint_i can be a real valued number determining by how much would that joint move.

Regarding implementations for solving these environments, robosuite does offer sample solutions for benchmarking the environments with different algorithms. You could also look up stable-baselines or one of the standard RL libraries.

Yes, you shouldn't do it like that. What you should do instead, is propagating through parts of the graph.

Now, graph contains both actor and critic. If the computations pass through the same part of graph (say, twice through actor), it will raise this error.

• And they will, as you clearly use actor and critic joined with loss value (this line: loss_actor = -self.critic(state, action))

• Different optimizers do not change anything here, as it's backward problem (optimizers simply apply calculated gradients onto models)

• This is how to fix it in GANs, but not in this case, see Actual fix paragraph below, read on if you are curious about the topic

If part of a neural network (critic in this case) does not take part in the current optimization step, it should be treated as a constant (and vice versa).

To do that, you could disable gradient using torch.no_grad context manager (documentation) and set critic to eval mode (documentation), something along those lines:

The actor usually is a neural network, and the reason of actor's action restrict in [-1,1] is usually because the output layer of the actor net using activation function like Tanh, and one can process this outputs to let action belong to any range.

The reason of actor can choose the good action depending on environment, is because in MDP(Markov decision process), the actor doing trial and error in the environment, and get reward or penalty for actor doing good or bad, i.e the actor net get gradients towards better action.

Note algorithms like PPG, PPO, SAC, DDPG, can guarantee the actor would select the best action for all states in theory! (i.e assume infinite learning time, infinite actor net capacity, etc.) in practice, there usually no guarantee unless action space is discrete and environment is very simple.

Understand the idea behind RL algorithms will greatly help you understand source codes of those algorithms, after all, code is implementation of the idea.

net.zero_grad() sets the gradients of all its parameters (including parameters of submodules) to zero. If you call optim.zero_grad() that will do the same, but for all parameters that have been specified to be optimised. If you are using only net.parameters() in your optimiser, e.g. optim = Adam(net.parameters(), lr=1e-3), then both are equivalent, since they contain the exact same parameters.

You could have other parameters that are being optimised by the same optimiser, which are not part of net, in which case you would either have to manually set their gradients to zero and therefore keep track of all the parameters, or you can simply call optim.zero_grad() to ensure that all parameters that are being optimised, had their gradients set to zero.

Moreover, what happens if I do both?

Nothing, the gradients would just be set to zero again, but since they were already zero, it makes absolutely no difference.

If I do none, then the gradients get accumulated, but what does that exactly mean? do they get added?

Yes, they are being added to the existing gradients. In the backward pass the gradients in respect to every parameter are calculated, and then the gradient is added to the parameters' gradient (param.grad). That allows you to have multiple backward passes, that affect the same parameters, which would not be possible if the gradients were overwritten instead of being added.

For example, you could accumulate the gradients over multiple batches, if you need bigger batches for training stability but don't have enough memory to increase the batch size. This is trivial to achieve in PyTorch, which is essentially leaving off optim.zero_grad() and delaying optim.step() until you have gathered enough steps, as shown in HuggingFace - Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU & Distributed setups.

That flexibility comes at the cost of having to manually set the gradients to zero. Frankly, one line is a very small cost to pay, even though many users won't make use of it and especially beginners might find it confusing.

I solved this problem completely by rewriting the code and adding the learning function in a separate session

So I figured out that huskarl is only compatible with tensorflow==2.0.0a0. I found that out by uninstall tf and reinstalling it and catching an error. :/

Eventually I got this working. I was running tensorflow 1.4 when I thought I was using tensorflow 2 and thus eager execution was disabled by default. Once I enabled eager execution I didn't need to do anything special on the other thread, it just worked.

I created a separate copy of the model on the main thread, and copied the weights from the training thread in a lock right before execution.