1

我正在使用 HTcondor 生成一些数据（txt、png）。通过运行我的程序，它会在 .sub 文件旁边创建一个名为 datasets 的目录，其中存储了数据集。不幸的是，condor 完成后并没有将这些创建的数据还给我。换句话说，我的目标是在 .sub 文件旁边的“Datasets”子文件夹中获取创建的数据。

我试过：1）不把数据放在datasets子文件夹下，我想得到它们。但是，这不是一个顺利的解决方案，因为我生成了大约 100 个文件，这些文件现在与 .sub 文件和所有其他文件混合在一起。

2）我也尝试在子文件中进行设置，导致：

notification = Always
should_transfer_files = YES
RunAsOwner = True
When_To_Transfer_Output = ON_EXIT_OR_EVICT
getenv = True

transfer_input_files = main.py
transfer_output_files = Datasets

universe      = vanilla
log           = log/test-$(Cluster).log
error         = log/test-$(Cluster)-$(Process).err
output        = log/test-$(Cluster)-$(Process).log
executable    = Simulation.bat

queue

这次我收到错误，即找不到数据集。拼写已经检查过了。

3) 另一种选择是将所有内容打包成一个 zip，但由于我必须运行数百个作业，因此我不想在之后解压缩所有这些文件。

我希望有人提出如何解决这个问题的好主意。

4

1 回答 1

1

Just for the record here: HTCondor does not transfer created directories at the end of the run or its contents. The best way to get the content back is to write a wrapper script that will run your executable and then compress the created directory at the root of the working directory. This file will be transferred with all other files. For example, create run.exe:

./Simulation.bat
tar zcf Datasets.tar.gz Datasets

and in your condor submission script put:

executable    = run.exe

However, if you do not want to do this and if HTCondor is using a common shared space like an AFS you can simply copy the whole directory out:

./Simulation.bat
cp -r Datasets <AFS location>

The other alternative is to define an initialdir as described at the end of: https://research.cs.wisc.edu/htcondor/manual/quickstart.html

But one must create the directory structure by hand.

also, look around pg. 65 of: https://indico.cern.ch/event/611296/contributions/2604376/attachments/1471164/2276521/TannenbaumT_UserTutorial.pdf

This document is, in general, a very useful one for beginners.

于 2019-03-06T22:28:52.687 回答