q-learning | Q-learning implementation for Machine Learning Course | Machine Learning library

Learning curves in RL are typically plots of returns over time, not Q-losses or anything like this. So you should run your environment, compute the total reward (aka return) and plot it at a corresponding time.

Well, I figured out the problem; I wasn't assigning any memory to targetNW.

• Calculate the mean for each group, and then add them to the existing ax with a seaborn.lineplot
• Set dodge=False in the seaborn.boxplot
• Remember that the line in the boxplot is the median, not the mean.
  • Add the means to boxplot with showmeans=True, and then remove marker='o' from the lineplot, if desired.
• As pointed out JohanC's answer:
  • sns.pointplot(data=dfm, x='variable', y='value', hue='parametrized_factor', ax=ax) can be used without the need for calculating dfm_mean, however there isn't a legend=False parameter, which then requires manually managing the legend.
  • Also, I think it's more straightforward to use dodge=False than to calculate the offsets.
  • Either answer is viable, depending on your requirements.

• The dataframe can be melted into a long format with pandas.DataFrame.melt, and then plotted with seaborn.boxplot or seborn.catplot and specifying the hue parameter.

If your dataset is formed, for example, of time series, you can set each instant of time as your state. Then, you can make your agent to explore the data series for learning a policy over it.

If your dataset is already labeled with actions, you can train the agent over it for learning the a police underlying those actions.

The trick is to feed your agent with each successive instant of time, as if it were exploring it on real time.

Of course, you need to model the different states from the information in each instant of time.

You can find the answer in research paper about those algorithms, because when a new algorithm been proposed we usually need the experiments to show the evident that it have advantage over other algorithm.

The most commonly used evaluation method in research paper about RL algorithms is average return (note not reward, return is accumulated reward, is like the score in game) over timesteps, and there many way you can average the return, e.g average wrt different hyperparameters like in Soft Actor-Critic paper's comparative evaluation average wrt different random seeds (initialize the model):

Figure 1 shows the total average return of evaluation rolloutsduring training for DDPG, PPO, and TD3. We train fivedifferent instances of each algorithm with different randomseeds, with each performing one evaluation rollout every1000 environment steps. The solid curves corresponds to themean and the shaded region to the minimum and maximumreturns over the five trials.

And we usually want compare the performance of many algorithms not only on one task but diverse set of tasks (i.e Benchmark), because algorithms may have some form of inductive bias for them to better at some form of tasks but worse on other tasks, e.g in Phasic Policy Gradient paper's experiments comparison to PPO:

We report results on the environments in Procgen Benchmark (Cobbe et al.,2019). This benchmark was designed to be highly diverse, and we expect improvements on this benchmark to transfer well to many other RL environment

In gym's environments (e.g. FrozenLake) discrete actions are usually encoded as integers.

It looks like the error is caused by a non-standard way that this environment represents actions.

I've annotated what I assume the types might be when the action variable is set:

After you get close enough to convergence, a stochastic environment would make it impossible to converge if the learning rate is too high.

Think of it like a ball rolling into a funnel. The speed at which the ball is rolling is like the learning rate. Because it's stochastic, the ball will never directly go into the hole, it will always just miss it. Now, if the learning rate is too high, then just missing is disastrous. It will shoot right past the hole.

That is why you want to steadily decrease the learning rate. It is like the ball losing velocity due to friction, which will always allow it to drop into the hole no matter which direction it's coming from.

data.table is fast for doing lookups and manipulations in very large tables of data, but it's not going to be fast at adding rows one by one like python dictionaries. I'd expect it would be copying the whole table each time you add a row which is clearly not what you want.

You can either try to use environments (which are something like a hashmap), or if you really want to do this in R you may need a specialist package, here's a link to an answer with a few options.