dqn | Basic DQN implementation | Machine Learning library

The main problem I think is the discount factor, gamma. You are setting it to 1.0, which mean that you are giving the same weight to the future rewards as the current one. Usually in reinforcement learning we care more about the immediate reward than the future, so gamma should always be less than 1.

Just to give it a try I set gamma = 0.99 and run your code:

Fixed! I was just doing it in the wrong way.

well is not hard to do it the MSLE equation as the photo below shows

now, as some user on the PyTorch form suggested

you can be added as a class like this

since you are calling a tensor that contains a matrix, you need to specify which indices your calling in your case just adding [0] to the forward statement will solve the problem and in the [actions], replace it with [actions.item()]

The input shape should be (1,15) in your case. This is because the actual input shape is increased by an additional dimension to what you specify in shape as keras processes inputs in batches, with the batch size being the first parameter.

The error message is telling you that inputs of shape (1,1,15) are being passed to the model ( i.e. batch of 1, and input shape (1,15), so ndims=3), but you only have ndims=2. Despite only passing (15,) as the input shape, notice that ndims = 2. This is because Keras is adding an additional dimension for the batch.

So to fix, set shape=(1,15) which will then be processed as (1,1,15) which is what Keras expects.

Destructure the array, and take the keys (1st item), and the values (the rest). Map the values array, and then map each sub-array of values, take the respective key by the value, and return a pair of [key, value]. Convert the array pairs to an object with Object.fromEntries():

The DqnAgent expects a TFPyEnvironment but you're implementing the environment as an PyEnvironment.

To fix this error you should convert the environment into the TensorFlow implementation before you are creating the agent.

You can do it this way:

As you've already noted in your comment, DQN is not compatible with continuous action spaces because of how DQN works; argmax of "a" for Q(s,a) - It's impossible to check Q(s,a) for all a when a is continuous.

Having said that, when applying this to Policy Gradient methods (which are compatible with continuous action spaces) you will run into the same issue in your question because with policy gradient you need to provide a probability for each action that you take. Something like this could work:

• Actor (a neural net in this case) gives 3 outputs.
• First 2 outputs are probabilities of each discrete value.
• 3rd output is your continuous value.

Take the softmax of the first two outputs which gives you your discrete value and then take the third output which is continuous and this will give you your action. You then need to derive the probability of that action which is given by the combined probability of all of your outputs.

You would not want to add in unnecessary features which are repetitive in the state representation as it might hamper your RL agent convergence later when you would want to scale your model to larger input sizes(if that is in your plan).

Also, the decision of how much of information you would want to give in the state representation is mostly experimental. The best way to start would be to just give in a single value as the battery state. But if the model does not converge, then maybe you could try out the other options you have mentioned in your question.