2

df.to_csv()作为multiprocessing进程的一部分运行以下行时：

filtered_chunk.to_csv(os.path.join(output_dir, "filtered_{}".format(csv_filename)), mode='a+')

我收到以下错误。我不知道问题出在哪里（下面的片段是针对一个multiprocessing进程的）：

Process ForkPoolWorker-13:
Traceback (most recent call last):
  File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
  File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/multiprocessing/process.py", line 108, in run
  File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/multiprocessing/pool.py", line 109, in worker
  <insert stacktrace for code I wrote...>
    filtered_chunk.to_csv(os.path.join(output_dir, "filtered_{}".format(csv_filename)), mode='a+')
  File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/pandas/core/generic.py", line 3170, in to_csv
  File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/pandas/io/formats/csvs.py", line 185, in save
  File "/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8/site-packages/pandas/io/common.py", line 418, in get_handle
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 914, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1342, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1314, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1443, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1483, in _fill_cache
OSError: [Errno 127] Key has expired: '/home/User1/miniconda3/envs/part_ii_dev-conda/lib/python3.8'

在每个进程都因上面的堆栈跟踪而失败后，我得到一个Bus error (core dumped)打印输出。

“密钥已过期”让我想到需要验证或提供限时访问的东西（文件指针？访问服务器的密钥/票证？）。我在服务器上的会话中运行脚本tmux，所以我不认为访问服务器会成为问题（一些错误 127 的谷歌搜索指出我遇到了越来越多的问题，但我认为这不是问题）。由于脚本在get_handle基本上返回文件句柄的方法中失败，文件指针似乎是问题，但我不知道为什么。

如果有帮助，我正在使用multiprocessing并行化一个函数，但是不同的进程写入不同的 csv 文件，所以这不应该是一个问题。

4

0 回答 0