0

My problem is that: I have a bash script that do something and then call 800 bsub jobs like this:

pids=
rm -f ~/.count-*
for i in `ls _some_files_`; do
    of=~/.count-${i}
    bsub -I "grep _something_ $i > $of" &
    pids="${!} ${pids}"
done
wait ${pids}

Then the scripts process the output files $of and echo the results.

The trouble is that I got a lot of lines like:

Job <7536> is submitted to default queue <interactive>.
<<Waiting for dispatch ...>>
<<Starting on hostA>>

It's actually 800 times the 3 lines above. Is there a way of suppressing this LSF lines?

I've tried in the loop above:

bsub -I "grep _something_ $i > $of" &> /dev/null

I does remove the LSF verbosity but instead of submitting almost all 800 jobs at once and then take less than 4 min to run, it submits just few jobs at a time and I have to wait more than an hour for the script to finish.

AFAIK lsf bsub doesn't seem to have a option to surpress all this verbosity. What can I do here?

4

2 回答 2

2

您可以通过将环境变量 BSUB_QUIET 设置为 bsub 之前的任何值（包括空值）来抑制此输出。所以，在你的循环说你可以添加：

export BSUB_QUIET=

然后，如果您想将其恢复正常，您可以使用以下命令清除变量：

unset BSUB_QUIET

希望对您有所帮助。

于 2013-09-11T16:09:47.443 回答
0

您是否考虑过使用作业依赖项并对日志文件进行后处理？

1）运行每个“子”作业（删除“-Is”）并将IO输出到单独的输出文件。每个作业都应使用作业名提交（请参阅 -J）。作业名可以形成一个数组。

2) 你最后的工作将取决于孩子们的完成情况（见-w）。

除了跨集群并发运行之外，这种方法的另一个优点是您的整个进程不会受到 IO 问题的影响。

于 2013-07-09T22:08:29.390 回答