强化学习TD从后状态学习

Question

我正在制作一个程序，通过强化学习和基于后状态的时间差分学习方法（TD（λ）），教两名玩家玩一个简单的棋盘游戏。学习是通过训练神经网络来实现的。我使用萨顿的非线性TD/Backprop神经网络）我很想听听你对我以下困境的看法。在两个对手之间进行回合的基本算法/伪代码如下

WHITE.CHOOSE_ACTION(GAME_STATE); //White player decides on its next move by evaluating the current game state ( TD(λ) learning)

GAME_STATE = WORLD.APPLY(WHITE_PLAYERS_ACTION);  //We apply the chosen action of the player to the environment and a new game state emerges

 IF (GAME STATE != FINAL ){ // If the new state is not final (not a winning state for white player), do the same for the Black player

    BLACK.CHOOSE_ACTION(GAME_STATE)

GAME_STATE = WORLD.APPLY(BLACK_PLAYERS_ACTION) // We apply the chosen action of the black player to the environment and a new game state emerges.
}

每个玩家应在何时调用其学习方法玩家。学习（GAME\u状态）。这是难题。

选项A.在每个玩家移动后，在新的后状态出现后，如下所示：

WHITE.CHOOSE_ACTION(GAME_STATE);
GAME_STATE = WORLD.APPLY(WHITE_PLAYERS_ACTION);
WHITE.LEARN(GAME_STATE)    // White learns from the afterstate that emerged right after his action
IF (GAME STATE != FINAL ){
    BLACK.CHOOSE_ACTION(GAME_STATE)
    GAME_STATE = WORLD.APPLY(BLACK_PLAYERS_ACTION)
    BLACK.LEARN(GAME_STATE) // Black learns from the afterstate that emerged right after his action

选项B.在每个玩家移动之后，在新的后状态出现之后，也在对手移动之后，如果对手做出获胜的移动。

WHITE.CHOOSE_ACTION(GAME_STATE);
GAME_STATE = WORLD.APPLY(WHITE_PLAYERS_ACTION);
WHITE.LEARN(GAME_STATE)
IF (GAME_STATE == FINAL ) //If white player won
    BLACK.LEARN(GAME_STATE) // Make the Black player learn from the White player's winning afterstate
IF (GAME STATE != FINAL ){ //If white player's move did not produce a winning/final afterstate
    BLACK.CHOOSE_ACTION(GAME_STATE)
    GAME_STATE = WORLD.APPLY(BLACK_PLAYERS_ACTION)
    BLACK.LEARN(GAME_STATE)
    IF (GAME_STATE == FINAL) //If Black player won
        WHITE.LEARN(GAME_STATE) //Make the White player learn from the Black player's winning afterstate

我认为方案B更合理。

Answer 1

通常，通过TD学习，代理将具有3个功能：

• 开始（观察）→ 行动
• 步骤（观察、奖励）→ 行动
• 完成（奖励）

行动与学习相结合，游戏结束时也会有更多的学习。