gridworld | Gridworlds and Markov Decision Processes in Python

I believe the comment from @TimRoberts above correctly diagnoses the issue.

These arrays currently reference the same object, so any update to one updates the other.

When you initialize arr2 = arr1, it creates a reference to the same object in memory. This makes it such that when one object is updated, so is the values of the other.

To create an array without this pointer style reference you can use:

In order to initialize a list of n duplicate values, you can use the multiplication operator, like this:

TensorFlow has 2 execution modes: eager execution, and graph mode. TensorFlow default behavior, since version 2, is to default to eager execution. Eager execution is great as it enables you to write code close to how you would write standard python. It's easier to write, and it's easier to debug. Unfortunately, it's really not as fast as graph mode.

So the idea is, once the function is prototyped in eager mode, to make TensorFlow execute it in graph mode. For that you can use tf.function. tf.function compiles a callable into a TensorFlow graph. Once the function is compiled into a graph, the performance gain is usually quite important. The recommended approach when developing in TensorFlow is the following:

• Debug in eager mode, then decorate with @tf.function.
• Don't rely on Python side effects like object mutation or list appends.
• tf.function works best with TensorFlow ops; NumPy and Python calls are converted to constants.

I would add: think about the critical parts of your program, and which ones should be converted first into graph mode. It's usually the parts where you call a model to get a result. It's where you will see the best improvements.

You can find more information in the following guides:

• Better performance with tf.function
• Introduction to graphs and tf.function

So, there are at least two things you can change in your code to make it run quite faster:

1. The first one is to not use model.predict on a small amount of data. The function is made to work on a huge dataset or on a generator. (See this comment on Github). Instead, you should call the model directly, and for performance enhancement, you can wrap the call to the model in a tf.function.

Model.predict is a top-level API designed for batch-predicting outside of any loops, with the fully-features of the Keras APIs.

1. The second one is to make your training step a separate function, and to decorate that function with @tf.function.

So, I would declare the following things before your training loop:

You can use the ActorWorld.add() method with the Location argument to place the actor at a specific location:

add

public void add(Location loc, Actor occupant)

Adds an actor to this world at a given location.

In your case it would be something like:

On your first time through the loop, policy_converged is set to False. After that, nothing will ever set it to True, so the break is never reached, and it loops forever.

DeepQLearning does not require to enumerate the state space and can handle continuous space problems. DeepQLearning.jl only uses the generative interface of POMDPs.jl. As such, you do not need to implement the states function but just gen and initialstate (see the link on how to implement the generative interface).

However, due to the discrete action nature of DQN you also need POMDPs.actions(mdp::YourMDP) which should return an iterator over the action space.

By making those modifications to your implementation you should be able to use the solver.

The neural network in DQN takes as input a vector representation of the state. If your state is a m dimensional vector, the neural network input will be of size m. The output size of the network will be equal to the number of actions in your model.

In the case of the grid world example, the input size of the Flux model is 2 (x, y positions) and the output size is length(actions(mdp))=4.

Provide default constructor of the class as well.

While it is not clear what lines of code you need explained above, I believe your question is, about List Comprehensions in python, as that is focus of the question title.