mcts | A simple Monte Carlo Tree Search library | Reinforcement Learning library

I'm trying to implement MCTS on Openai's atari gym environments, which requires the ability to plan: acting in the environment and restoring it to a previous state. I read that this can be done with the ram version of the games:

recording the current state in a snapshot: snapshot = env.ale.cloneState()

restoring the environment to a specific state recorded in snapshot: env.ale.restoreState(snapshot)

so I tried using the ram version of breakout:

I've been recently trying to play with MCTS implementation for simple board game. I'd like to make AI play with itself to gather some sample playthroughs. I'd figure out I could make them use the same MCTS tree (for better performance). Or so it looks like.

But would that be valid ? Or I need 2 separate trees for both AI with separate win/plays data to behave correctly ?

How can I use related generic types in Rust?

Here's what I've got (only the first line is giving me trouble):

I am looking for a method of simulating my game till win or lose, to feed into a Monte Carlo Tree Search algorithm.

My game is a turn based , tile based tactical RPG similar to Final Fantasty Tactics, Fire Emblem etc.

The idea is that the AI would perform thousands of playouts (or up until a threshold) until they determine the optimal next move.

Each AI and Player agent would make a random valid move, until the game is over.

Why a MCTS Simulation? Why not minmax?

I need to simulate the game as closely to the real thing for several reasons:

1. Encoding the game state into a lower-dimensional structure is impossible as most actions are tightly coupled to Unity constructs like Colliders and Rays.
2. It is fairly difficult, if not impossible to statically evaluate the game state at X moves ahead - without any knowledge of previous moves. Therefore I would need to carry out each move sequentially on a game state, to produce the next game state before anything can be evaluated.

To expand on point 2: Using a simple minmax approach, and statically evaluating the game state by looking at something like the Current Health of all players, would be useful but not accurate. As not every action will provide an immediate change to health.

Example:

Which produces a higher (max damage) dealt over 2 turns:

1. Move infront of player, attack -> Move behind player, attack

OR

1. Move infront of player, Use attack buff -> Attack for x4 damage

In this example, the minmax approach would never result in the 2nd option, even though it does more damage over 2 turns, due to its static evaluation of the buff move resulting in 0, or perhaps even negatively.

In order for it to select the 2nd option, it would need to retain knowledge of previous actions. Ie. it would need to simulate the game almost perfectly.

When we add in other elements like: Stage Traps, destructible environment and status effects. It becomes pretty much impossible to use a static evaluation

Time.timeScale

This allows me to speed up physics and other interactions, which is exactly what I need. However - this is a global property, so the game will appear to run at super speed for a fraction of a second when the AI is "thinking".

Increasing the speed of NavMesh Agents

All my movements take place on a NavMesh - so the only perceivable way of making these movements "instant" is to increase the speed. This is problematic as the movements are not fast enough - and it causes physics issues due to the increased velocity, sometimes characters spin out of control and fly off the map.

For reference here is a screenshot of my (in active development) game.

What I need is a method for "playing" my game extremely quickly.

I just need to be able to run these simulations quickly and efficiently before every AI move.

I would love to hear from someone with some experience doing something like this - but any input would be greatly appreciated!

Thanks

I am working on macOS Catalina and I want to compile some c++ code using CMake.

This code needs the library boost that I installed via Brew (brew install boost). In my code I added #include to use it.

When I compile I got the following error:

From many blogs and this one https://web.archive.org/web/20160308070346/http://mcts.ai/about/index.html We know that the process of MCTS algorithm has 4 steps.

1. Selection: Starting at root node R, recursively select optimal child nodes until a leaf node L is reached.

What does leaf node L mean here? I thought it should be a node representing the terminal state of the game, or another word which ends the game. If L is not a terminal node (one end state of the game), how do we decide that the selection step stops on node L?

1. Expansion: If L is a not a terminal node (i.e. it does not end the game) then create one or more child nodes and select one C.

From this description I realise that obviously my previous thought incorrect. Then if L is not a terminal node, it implies that L should have children, why not continue finding a child from L at the "Selection" step? Do we have the children list of L at this step?
From the description of this step itself, when do we create one child node, and when do we need to create more than one child nodes? Based on what rule/policy do we select node C?

1. Simulation: Run a simulated playout from C until a result is achieved.

Because of the confusion of the 1st question, I totally cannot understand why we need to simulate the game. I thought from the selection step, we can reach the terminal node and the game should be ended on node L in this path. We even do not need to do "Expansion" because node L is the terminal node.

1. Backpropagation: Update the current move sequence with the simulation result.

Fine. Last question, from where did you get the answer to these questions?

Thank you

BTW, I also post the same question https://ai.stackexchange.com/questions/16606/how-to-understand-the-4-steps-of-monte-carlo-tree-search

In order to make it easy for others to help me I have put all the codes here https://pastebin.com/WENzM41k it will starts as 2 agents compete each other.

I'm trying to implement Monte Carlo tree search to play 9-board tic-tac-toe in Python. The rules of the game is like the regular tic-tac-toe but with 9 3x3 sub-boards. The place of the last piece is placed decides which sub-board to place your piece. It's kind like ultimate tic-tac-toe but if one of the sub-board is won the game ends.

I'm trying to learn MCTS and I found some code on here: http://mcts.ai/code/python.html

I used node class and UCT class on the website and added my 9 board tic-tac-toe game state class and some other codes. All codes are here:

When I run my program I get this error:

I would like to parallelize my MCTS program. There are several ways to do this:

1. Leaf parallelization, where each leaf is expanded and simulated in parallel.
2. Root parallelization, where each thread/process creates a separate tree and when some number of simulations are finished, trees are combined to give a better statistic
3. Tree parallelization, where all threads/processes share the same tree and each thread/process explores different parts of the tree.

(If my explanation is unclear, checkout this review paper on MCTS. On page 25, different methods on parallelizing MCTS are described in detail.)

Question:

Since multiprocessing in Python has to create separate subprocesses, 2. Root parallelization fits quite nicely, whereas I am assuming that 3. Tree parallelization is not feasible. (Since for tree parallelization, all subprocesses would have to share the same tree - which is difficult to do in Python)

Am I correct? I skimmed through the multiprocessing documentation and if I understood correctly, it seems like it is possible to pass information back and forth between subprocesses for some basic data types, but are highly discouraged due to speed etc.

If so, tree parallelization in Python would be a bad idea right?