在 GPU 上训练 PyTorch 模型几个小时后，程序失败并出现错误
RuntimeError：cuDNN 错误：CUDNN_STATUS_INTERNAL_ERROR
培训条件
nn.LSTM神经网络： PyTorch 4 层nn.Linear输出- Deep Q 网络代理（带有重放内存的 Vanilla DQN）
state传入forward()具有形状(32, 20, 15)，32批量大小在哪里- 每集 50 秒
- 错误发生在大约 583 集（8 小时）或 1,150,000 步之后，其中每一步都涉及通过 LSTM 模型的前向传递。
我的代码在训练开始之前还设置了以下值
torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)
我们如何解决这个问题？由于这发生在培训的 8 小时内，因此一些有根据的猜测在这里会非常有帮助！
谢谢！
更新：
注释掉这两torch.backends.cudnn...行没有用。CUDNN_STATUS_INTERNAL_ERROR仍然会发生，但要早得多，大约在第 300 集（585,000 步）左右。
torch.manual_seed(0)
#torch.backends.cudnn.deterministic = True
#torch.backends.cudnn.benchmark = False
np.random.seed(0)
系统
- PyTorch 1.6.0.dev20200525
- CUDA 10.2
- cuDNN 7604
- 蟒蛇 3.8
- 视窗 10
- 英伟达 1080 GPU
错误回溯
RuntimeError Traceback (most recent call last)
<ipython-input-18-f5bbb4fdfda5> in <module>
57
58 while not done:
---> 59 action = agent.choose_action(state)
60 state_, reward, done, info = env.step(action)
61 score += reward
<ipython-input-11-5ad4dd57b5ad> in choose_action(self, state)
58 if np.random.random() > self.epsilon:
59 state = T.tensor([state], dtype=T.float).to(self.q_eval.device)
---> 60 actions = self.q_eval.forward(state)
61 action = T.argmax(actions).item()
62 else:
<ipython-input-10-94271a92f66e> in forward(self, state)
20
21 def forward(self, state):
---> 22 lstm, hidden = self.lstm(state)
23 actions = self.fc1(lstm[:,-1:].squeeze(1))
24 return actions
~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
575 result = self._slow_forward(*input, **kwargs)
576 else:
--> 577 result = self.forward(*input, **kwargs)
578 for hook in self._forward_hooks.values():
579 hook_result = hook(self, input, result)
~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\nn\modules\rnn.py in forward(self, input, hx)
571 self.check_forward_args(input, hx, batch_sizes)
572 if batch_sizes is None:
--> 573 result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,
574 self.dropout, self.training, self.bidirectional, self.batch_first)
575 else:
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
更新：try... except在我的代码上尝试了发生此错误的位置，除此之外RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR，我们还获得了该错误的第二次回溯RuntimeError: CUDA error: unspecified launch failure
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
<ipython-input-4-e8f15cc8cf4f> in <module>
61
62 while not done:
---> 63 action = agent.choose_action(state)
64 state_, reward, done, info = env.step(action)
65 score += reward
<ipython-input-3-1aae79080e99> in choose_action(self, state)
58 if np.random.random() > self.epsilon:
59 state = T.tensor([state], dtype=T.float).to(self.q_eval.device)
---> 60 actions = self.q_eval.forward(state)
61 action = T.argmax(actions).item()
62 else:
<ipython-input-2-6d22bb632c4c> in forward(self, state)
25 except Exception as e:
26 print('error in forward() with state:', state.shape, 'exception:', e)
---> 27 print('state:', state)
28 actions = self.fc1(lstm[:,-1:].squeeze(1))
29 return actions
~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\tensor.py in __repr__(self)
152 def __repr__(self):
153 # All strings are unicode in Python 3.
--> 154 return torch._tensor_str._str(self)
155
156 def backward(self, gradient=None, retain_graph=None, create_graph=False):
~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in _str(self)
331 tensor_str = _tensor_str(self.to_dense(), indent)
332 else:
--> 333 tensor_str = _tensor_str(self, indent)
334
335 if self.layout != torch.strided:
~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in _tensor_str(self, indent)
227 if self.dtype is torch.float16 or self.dtype is torch.bfloat16:
228 self = self.float()
--> 229 formatter = _Formatter(get_summarized_data(self) if summarize else self)
230 return _tensor_str_with_formatter(self, indent, formatter, summarize)
231
~\AppData\Local\Continuum\anaconda3\envs\rl\lib\site-packages\torch\_tensor_str.py in __init__(self, tensor)
99
100 else:
--> 101 nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
102
103 if nonzero_finite_vals.numel() == 0:
RuntimeError: CUDA error: unspecified launch failure