System setup:
Ubuntu 16.04, Tesla V100 on AWS p3-2xlarge, Nvidia driver 396.54, Cuda 9.0.176_384.81, CuDNN 9.0
Tensorflow GPU 1.9.0, Python 3.6 using pyenv
我对 Google Quickdraw 游戏很好奇，并且正在研究他们如何训练模型。
我跟着文件
https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/train_model.py
运行以下命令
python train_model.py \
--training_data train_data \
--eval_data eval_data \
--model_dir /tmp/quickdraw_model/ \
--cell_type cudnn_lstm
训练和评估数据是使用生成的
https://github.com/tensorflow/models/blob/master/tutorials/rnn/quickdraw/create_dataset.py
并在此处使用文件：https ://console.cloud.google.com/storage/browser/quickdraw_dataset/full/simplified
然后程序在出现以下错误后停止：
2019-02-01 06:41:15.770071: E tensorflow/stream_executor/cuda/cuda_dnn.cc:943] CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor with state memory size: 3932160 bytes.
2019-02-01 06:41:15.770123: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at cudnn_rnn_ops.cc:1214 : Unknown: CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor with state memory size: 3932160 bytes.
Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/abc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/ubuntu/.pyenv/versions/abc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/.pyenv/versions/abc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: CUDNN_STATUS_EXECUTION_FAILED Failed to set dropout descriptor with state memory size: 3932160 bytes.
[[Node: cudnn_lstm/CudnnRNN = CudnnRNN[T=DT_FLOAT, direction="bidirectional", dropout=0.3, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](transpose, cudnn_lstm/zeros, cudnn_lstm/zeros, cudnn_lstm/opaque_kernel/read)]]
[[Node: OptimizeLoss/clip_by_global_norm/mul_1/_239 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_354_OptimizeLoss/clip_by_global_norm/mul_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
经过一番研究，似乎错误是通过调用函数 cudnnSetDropoutDescriptor 给出的。
https://github.com/tensorflow/tensorflow/blob/r1.9/tensorflow/stream_executor/cuda/cuda_dnn.cc#L932
检查 API 文档后，似乎 CUDNN_STATUS_EXECUTION_FAILED 可能是由库错误或错误安装引起的。
我通过运行 mnist 测试检查了安装，它通过了。
顺便说一句，我还尝试在没有 cell_type 参数的情况下运行上述命令，这意味着它将在 CPU 上运行。它能够毫无问题地运行。另外，我尝试使用以下设置运行相同的东西，它给出了相同的错误。
Ubuntu 18.04, Tesla V100 on AWS p3.2xlarge, NVidia Driver 410.79, Cuda 10.0.130_410.48, CuDNN 10.0,
Tensorflow GPU 12.0/10.0, Python 3.6 using pyenv
有没有人尝试过这样做并遇到类似的问题？