环境：总共有25个回合。有两种类型的操作：构建 CS 和构建 CI。
目标：找到可以在使用专门机器学习/强化学习给出的总回合数中建造的最大 CI（建筑物）数量。
注意：尽管 CS 在技术上是建筑物，但我并未将其计数计入建筑物总数。当阅读我的代码中的“建筑物”意味着仅构建 CI 时，请务必注意这一点。
公式：BPT（每回合建筑数）= CS/4 + 5 每建造 4 个 CS，您的 CI 增加 1。（您从 5 开始）
For example:
turn 1: build 5 CI (bpt: 5) (total buildings: 5)
turn 2: build 1 CS (bpt: 5) (total buildings: 5)
turn 3: build 1 CS (bpt: 5) (total buildings: 5)
turn 4: build 1 CS (bpt: 5) (total buildings: 5)
turn 5: build 1 CS (bpt: 6) (total buildings: 5)
turn 6: build 6 CI (bpt: 6) (total buildings: 11) (increased by BPT 6)
我的总体目标是到 25 岁，看看可以构建的最大 CI 数量。除此之外，我想知道我需要采取哪些步骤以及以何种顺序采取这些行动以最大化最佳情况。
我下面的代码似乎实现了这一点，但是当我尝试使用经过训练的模型时它失败了。在我完成所有剧集之后，我的理解是，我的 q_values 表将能够绘制出最佳路径。
不幸的是，我的最终 q_values 表似乎具有所有相同的值，并且 np.argmax 的使用只是为所有决策选择第 0 个索引。我注意到的是，在训练过程中，我的模型正确识别了最佳解决方案，但由于某种原因，我最终的 q_values 表没有反映它。
一个重要的注意事项：在第 25 回合，如果正确完成，最大建筑物应该是 126。前 4 个回合应该是 CS，其余的应该是 CI，以最大限度地提高可能性。
import numpy as np
import math
import pdb
class AI:
def __init__(self, turns: int, learning_rate: int, discount_factor: int, actions: list, q_values: list):
'''
turns: max number of turns an agent can take,
learning_rate: the rate in which an agent should learn,
discount_factor: the decayed reward amount
actions: the actions which the agent can take,
q_values: a mapping of probabilities which suggests which action should be taken at any given state
history_cs: state - number of cs built
history_ci: state - number of ci built (buildings)
'''
# default values
self.state = 0
self.cs = 0
self.buildings = 0
self.max_buildings = 0
self.history_cs = []
self.history_ci = []
self.turns = turns
self.learning_rate = learning_rate
self.discount_factor = discount_factor
self.actions = actions
self.q_values = q_values
def reset(self):
''' Resets the default values back to their original values '''
self.state = 0
self.cs = 0
self.buildings = 0
self.history_cs = []
self.history_ci = []
def get_reward(self) -> int:
''' The reward will be based on the number of buildings created '''
return self.buildings
def is_game_over(self) -> bool:
''' Determines if all turns have been used '''
return self.state == self.turns
def get_bpt(self, cs: int) -> int:
''' Determines the current buildings per turn '''
return (math.floor(cs/4)) + 5
def get_next_action(self, epsilon: float) -> int:
'''
Returns the most likely successful action with some probability that an inferior action may happen occasionally.
'''
if np.random.random() < epsilon:
return np.argmax(self.q_values[self.state])
else:
return np.random.randint(2)
def get_next_state(self, action_index: int) -> int:
''' Executes next action and returns the next state '''
if self.actions[action_index] == "build ci":
new_buildings = self.get_bpt(self.cs)
self.buildings += new_buildings
self.history_ci.append({self.state: new_buildings})
elif self.actions[action_index] == "build cs":
self.cs += 1
self.history_cs.append({self.state : 1})
self.state += 1
return self.state
def print_best_path(self):
self.reset()
while not ai.is_game_over():
action_index = self.get_next_action(1.)
if action_index == 0:
print(f"build ci")
else:
print(f"build cs")
self.get_next_state(action_index)
print(f"total construction sites: {self.cs}")
print(f"total buildings: {self.buildings}")
TURNS = 25
ai = AI(turns=TURNS,
learning_rate=0.9,
discount_factor=0.9,
actions=["build ci", "build cs"],
q_values=np.zeros((TURNS+1, 1, 2)))
for episode in range(100000):
ai.reset()
action_index = None
while not ai.is_game_over():
action_index = ai.get_next_action(.9)
old_state = ai.state
next_state = ai.get_next_state(action_index)
if ai.buildings < ai.max_buildings:
reward = -10
else:
reward = -1
old_q_value = ai.q_values[old_state, 0, action_index]
temporal_difference = reward + (ai.discount_factor * np.max(ai.q_values[next_state])) - old_q_value
new_q_value = old_q_value + (ai.learning_rate * temporal_difference)
ai.q_values[old_state, 0, action_index] = new_q_value
if ai.buildings > ai.max_buildings:
ai.max_buildings = ai.buildings
print(f"\nepisode: {episode}")
print(ai.history_cs)
print(ai.history_ci)
print(f"total construction sites: {ai.cs}")
print(f"total buildings: {ai.buildings}")
#if ai.buildings == 126:
# print(ai.q_values)
#pdb.set_trace()
#ai.print_best_path()