Well, I am trying to run serial MPI jobs masked as a one job on our supercomputer. The main submission script basically looks like that:
#!/bin/bash -l
#PBS -l nodes=4:ppn=8,walltime=24:00:00
cat $PBS_NODEFILE | uniq | tr '\\012' ' ' > tmp-$PBS_JOBID
read -a NODE < tmp-$PBS_JOBID
rm tmp-$PBS_JOBID
inode=-1
ijob=0
for ((K=1;K<=8;K++))
do
[ $((ijob++ % 2)) -eq 0 ] && ((inode++))
ssh ${NODE[inode]} _somepath_/RUN$K/sub.script &
done
wait
exit 0
Each sub.script looks like:
#!/bin/bash -l
#PBS -l walltime=24:00:00,nodes=1:ppn=4
module load intel
module load ompi
export FORT_BUFFERED=1
*run executable*
wait
exit 0
And sometimes I encounter an error for each sub.script (jobs die immediately):
/bin/bash: -
: invalid option
Usage: /bin/bash [GNU long option] [option] ...
/bin/bash [GNU long option] [option] script-file ...
*etc.*
The most interesting thing is that it is a random error meaning if I run the same script for the second (or 3rd etc.) time it will run without any problems. Sometimes I'm lucky, sometimes I'm not... Removing -l won't help because in that case modules cannot be loaded and mpirun won't work. Any suggestions how to fix it?
Thanks a lot in advance!