我正在尝试使用以下代码使用 papermill 自动生成几个笔记本:
template = "preprocessing_template.ipynb"
parameters = {"allowed_values":[0,1], "epsilon":0.01, "to_csv":True}
kernel_name = "my_env"
grid_folders_csv_names_outputs = [("dir_a", "a.csv","output_a.ipynb"),\
("dir_b", "b.csv","output_b.ipynb"),\
("dir_c", "c.csv","output_c.ipynb"),
[
def execute_notebook(template,output_file,parameters,kernel_name,grid_folder,csv_name):
parameters["grid_folder"]=grid_folder
parameters["csv_name"]=csv_name
nb = pm.execute_notebook(template,
output_file,
parameters=parameters,
kernel_name=kernel_name)
#nbs = []
#nbs.append(nb)
return nb
for grid_folder, csv_name, output_file in tqdm(grid_folders_csv_names_outputs):
execute_notebook(template,output_file,parameters,kernel_name,grid_folder,csv_name)
不过,我收到一条错误消息
DeadKernelError: Kernel died
这似乎是因为内存完全被淹没了,因为我在运行代码时通过监控系统进行了检查。显然,上面的代码片段生成了几个进程来运行参数化的笔记本,一旦迭代完成,这些进程就不会被杀死,从而在内存中累积。我能做些什么来解决这个问题?我使用的是 Ubuntu 18.04,代码在 anaconda 2019.10 环境中运行。