Reinforcement-Learning | 强化学习、DQN、linUCB -

TensorFlow has 2 execution modes: eager execution, and graph mode. TensorFlow default behavior, since version 2, is to default to eager execution. Eager execution is great as it enables you to write code close to how you would write standard python. It's easier to write, and it's easier to debug. Unfortunately, it's really not as fast as graph mode.

So the idea is, once the function is prototyped in eager mode, to make TensorFlow execute it in graph mode. For that you can use tf.function. tf.function compiles a callable into a TensorFlow graph. Once the function is compiled into a graph, the performance gain is usually quite important. The recommended approach when developing in TensorFlow is the following:

• Debug in eager mode, then decorate with @tf.function.
• Don't rely on Python side effects like object mutation or list appends.
• tf.function works best with TensorFlow ops; NumPy and Python calls are converted to constants.

I would add: think about the critical parts of your program, and which ones should be converted first into graph mode. It's usually the parts where you call a model to get a result. It's where you will see the best improvements.

You can find more information in the following guides:

• Better performance with tf.function
• Introduction to graphs and tf.function

So, there are at least two things you can change in your code to make it run quite faster:

1. The first one is to not use model.predict on a small amount of data. The function is made to work on a huge dataset or on a generator. (See this comment on Github). Instead, you should call the model directly, and for performance enhancement, you can wrap the call to the model in a tf.function.

Model.predict is a top-level API designed for batch-predicting outside of any loops, with the fully-features of the Keras APIs.

1. The second one is to make your training step a separate function, and to decorate that function with @tf.function.

So, I would declare the following things before your training loop:

I fixed the problem by specifying TensorFlow version:

Alternativey, use

I found the answer to my problem. In Keras, the metrics built-in functionality provides an interface for measuring performance and losses of the model, be it a custom or standard one.

When compiling a model as follows:

Yes, states can be represented by anything you want, including vectors of arbitrary length. Note, however, that if you are using a tabular version of Q-learning (or SARSA as in this case), you must have a discrete set of states. Therefore, you need a way to map the representation of your state (for example, a vector of potentially continuous values) to a set of discrete states.

Expanding on the example you have given, imagine that you have three states represented by vectors:

After many days of research I found out that the resources were not getting initialized and hence it was saying var does not exist. So, I found few lines of codes that I pasted after graph.as_default() and before saving the model. These are the lines of codes:-

Yes, you are right. Usually you define a dictionary containing a map between integers and every action your agent can make. You can see that in the function n_actions is used exactly to sample a random action index when you don't select the optimal one.

It will call the __call__ method on the instance. See this demo: