1

我正在使用 DQN 进行资源分配,代理应将到达请求分配给最佳虚拟机。我正在修改 Cartpole 代码如下:

import random
import gym
import numpy as np
from collections import deque
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import Adam
import os 

class DQNAgent:
    def __init__(self, state_size, action_size):
        self.state_size = state_size
        self.action_size = action_size
        self.memory = deque(maxlen=2000)
        self.gamma = 0.95 
        self.epsilon = 1.0 
        self.epsilon_decay = 0.995 
        self.epsilon_min = 0.01 
        self.learning_rate = 0.001 
        self.model = self._build_model()
    
    def _build_model(self):
        model = Sequential()
        model.add(Dense(24, input_dim=self.state_size, activation='relu')) 
        model.add(Dense(24, activation='relu')) 
        model.add(Dense(self.action_size, activation='linear')) 
        model.compile(loss='mse', optimizer=Adam(lr=self.learning_rate))
        return model
    
    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done)) 

    def act(self, state):
        if np.random.rand() <= self.epsilon: 
            return random.randrange(self.action_size)
        act_values = self.model.predict(state) 
        return np.argmax(act_values[0])

    def replay(self, batch_size):
        minibatch = random.sample(self.memory, batch_size) 
        for state, action, reward, next_state, done in minibatch: 
            target = reward 
            if not done: 
                target = (reward + self.gamma * np.amax(self.model.predict(next_state)[0])) 
            target_f = self.model.predict(state) 
            target_f[0][action] = target
            self.model.fit(state, target_f, epochs=1, verbose=0)

        if self.epsilon > self.epsilon_min:
            self.epsilon *= self.epsilon_decay

    def load(self, name):
        self.model.load_weights(name)

    def save(self, name):
        self.model.save_weights(name)

Cartpole 状态作为 Q 网络的输入是由环境给定的。

0   Cart Position
1   Cart Velocity       -Inf    Inf
2   Pole Angle          ~ -41.8°    ~ 41.8°
3   Pole Velocity At Tip

问题是在我的代码中 Q 网络的输入是什么?因为代理应该根据到达请求的大小采取最好的行动,但这不是由环境给出的。我应该通过这个输入值,大小来输入 Q 网络吗?

4

1 回答 1

1

Deep Q-Network 架构的输入由重放内存提供,代码如下:

def remember(self, state, action, reward, next_state, done):
    self.memory.append((state, action, reward, next_state, done))

该系统的动态如原始论文Deepmind 论文所示,是您与系统交互,将转换存储在重放内存中,然后将其用于训练步骤。在上面的行中,您正在存储这些经验。

基本上,网络的输入是状态并输出 Q 值。在您的代码中,没有与环境的交互,此时您可以获得这些转换(体验)来提供重放内存。因此,如果您无法提取环境中的某些信息以表示为状态,那么您将无法对此做出假设。

于 2020-10-08T13:39:50.550 回答