Deep-Reinforcement-Learning | several Deep Reinforcement Learning techniques ( Deep Q | Machine Learning library

Using an A2C agent from this article, how to get numerical values of value_loss, policy_loss and entropy_loss when weights are being updated?

The model I'm using is double-headed, both heads share the same trunk. The policy head output shape is [number of actions, batch size] and value head has a shape of [1, batch_size]. Compiling this model returns a size incompatibility error, when these loss functions are given as metrics:

I'm learning about Distributional RL from 'Deep Reinforcement Learning Hands On' code. And there is a method in model class:

Following this example and this article on reinforcement learning. I finally manage to create a similar Q-learning that learns to play another game environment. The only problem i have is with the last output layer of the neural network which represents the input actions of the game.

The mechanics/logic of the game environment are not relevant to this question, but the game environment require 2 types of inputs at the same time for each given action:

1. Input number 1 represents a single key press between 3 possible keys. So basically the layer should output 3 classes where the probabilities sum is 1. I will then pick the class with the highest probability out of these three.
2. Input number 2 represents a percentage ranging from 0 to 1. And should be independent of the first three classes.

I really do not see how can i create this last output layer so that it has a total of 4 output classes. The first 3 classes should give probabilities between each other with a total sum of 1. And the last class should be independent of the first three and should ranging from 0 to 1.

Can somebody point me in the right direction on how to achieve this? How do i structure such a layer?

I am thinking of something like this for the first input: