5

我很少在论坛上发布问题，但这个问题让我很难过。我很好奇是什么原因造成的（解决方案也很好，但主要是，我想知道我为什么会遇到这个问题）：

我最近编写了一个 python 脚本，用于包装由 PBS 作业启动的远程命令的调用：

#! /usr/bin/env python
#
# Copyright (c) 2009 Maciej Brodowicz
# Copyright (c) 2011 Bryce Lelbach
#
# Distributed under the Boost Software License, Version 1.0. (See accompanying
# file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

from datetime import datetime

from string import letters, digits

from types import StringType

from optparse import OptionParser

from threading import Thread

# subprocess instantiation wrapper. Unfortunately older Python still lurks on
# some machines.
try:
  from subprocess import Popen, STDOUT, PIPE
  from types import StringType

  class process:
    _proc = None
    _exec = None

    def __init__(self, cmd):
      self._proc = Popen(cmd, stderr = STDOUT, stdout = PIPE,
        shell = (False, True)[type(cmd) == StringType])

    def poll(self):
      return self._proc.poll()

    def pid(self):
      return self._proc.pid

    def _call(self):
      # annoyingly, KeyboardInterrupts are transported to threads, while most
      # other Exceptions aren't in python
      try:
        self._proc.wait()
      except Exception, err:
        self._exec = err

    def wait(self, timeout=None):
      if timeout is not None:
        thread = Thread(target=self._call)
        thread.start()

        # wait for the thread and invoked process to finish
        thread.join(timeout)

        # be forceful
        if thread.is_alive():
          self._proc.terminate()
          thread.join()

          # if an exception happened, re-raise it here in the master thread 
          if self._exec is not None:
            raise self._exec

          return (True, self._proc.returncode)

        if self._exec is not None:
          raise self._exec

        return (False, self._proc.returncode)

      else:
        return (False, self._proc.wait())

    def read(self):
      return self._proc.stdout.read()

except ImportError, err:
  # no "subprocess"; use older popen module
  from popen2 import Popen4
  from signal import SIGKILL
  from os import kill, waitpid, WNOHANG

  class process:
    _proc = None

    def __init__(self, cmd):
      self._proc = Popen4(cmd)

    def poll(self):
      return self._proc.poll()

    def pid(self):
      return self._proc.pid

    def _call(self):
      # annoyingly, KeyboardInterrupts are transported to threads, while most
      # other Exceptions aren't in python
      try:
        self._proc.wait()
      except Exception, err:
        self._exec = err

    def wait(self, timeout=None):
      if timeout is not None:
        thread = Thread(target=self._call)
        thread.start()

        # wait for the thread and invoked process to finish
        thread.join(timeout)

        # be forceful
        if thread.is_alive():
          kill(self._proc.pid, SIGKILL)
          waitpid(-1, WNOHANG)
          thread.join()

          # if an exception happened, re-raise it here in the master thread 
          if self._exec is not None:
            raise self._exec

          return (True, self._proc.wait())

        if self._exec is not None:
          raise self._exec

        return (False, self._proc.wait())

      else:
        return (False, self._proc.wait())

    def read(self):
      return self._proc.fromchild.read()

def run(cmd, timeout=3600):
  start = datetime.now() 
  proc = process(cmd)
  (timed_out, returncode) = proc.wait(timeout)
  now = datetime.now()

  output = ''

  while True:
    s = proc.read()

    if s:
      output += s
    else:
      break

  return (returncode, output, timed_out)

def rstrip_last(s, chars):
  if s[-1] in chars:
    return s[:-1]
  else:
    return s

# {{{ main
usage = "usage: %prog [options]" 

parser = OptionParser(usage=usage)

parser.add_option("--timeout",
                  action="store", type="int",
                  dest="timeout", default=3600,
                  help="Program timeout (seconds)")

parser.add_option("--program",
                  action="store", type="string",
                  dest="program",
                  help="Program to invoke") 

(options, cmd) = parser.parse_args()

if None == options.program:
  print "No program specified"
  exit(1)

(returncode, output, timed_out) = run(options.program, options.timeout)

if not 0 == len(output):
  print rstrip_last(output, '\n')

if timed_out:
  print "Program timed out"

exit(returncode)
# }}}

另一个 python 脚本根据 PBS 报告的可用资源将命令行参数放在一起，类似于 mpirun。我使用 python-paramiko 通过 SSH 启动远程命令。最初我只是直接执行命令，但是当远程运行的进程之一以信号（例如 SIGSEGV）退出时，我未能接收到正确的退出代码。因此，需要上面的脚本。

当在我的开发集群上运行这个脚本时，我注意到这个脚本在我的 4 核 Debian GNU/Linux 节点上运行失败，但它在我的 48 核 RHEL/Linux 节点上运行：

在 Debian 节点上：

wash@hermione0:~/sandbox$ python --version
Python 2.6.7
wash@hermione0:~/sandbox$ uname -a
Linux hermione0 2.6.32-5-amd64 #1 SMP Wed Jan 12 03:40:32 UTC 2011 x86_64 GNU/Linux
wash@hermione0:~/sandbox$ time ./hpx_invoke.py --program='sleep 30' --timeout=5
Program timed out

real 0m30.025s
user 0m0.016s
sys  0m0.012s
wash@hermione0:~/sandbox$

在 RHEL 节点上：

[22:08:23]:wash@vega:/home/wash/sandbox$ python --version
Python 2.6.6
[22:09:28]:wash@vega:/home/wash/sandbox$ uname -a
Linux vega 2.6.32-131.4.1.el6.x86_64 #1 SMP Fri Jun 10 10:54:26 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[22:09:30]:wash@vega:/home/wash/sandbox$ time ./hpx_invoke.py --program='sleep 30' --timeout=5
Program timed out

real 0m5.053s
user 0m0.040s
sys  0m0.020s
[22:09:41]:wash@vega:/home/wash/sandbox$

这可能是什么原因造成的？

PS 我是这些盒子的系统管理员。

4

2 回答 2

1

我猜想可用包的不同导致在任何一台机器上都使用“子进程实例化包装器”的不同分支。在一个分支中，您将使用 SIGTERM（terminate()调用），在另一个分支中使用 SIGKILL。

话虽如此，sleep在任何一个信号下似乎都过早地结束了。可能还有其他差异，但很难说。你最好放入一些调试代码，看看在什么机器上会发生什么。

于 2011-07-02T11:37:45.987 回答
0

问题原来是作为 shell 调用子进程（两台机器都有 subprocess 包）。在 RHEL 节点上，当 /bin/sh 被杀死时，调用的程序也被杀死。在 Debian 节点上，只有 /bin/sh 进程被杀死，它被调用的程序仍然存在。

我通过将脚本更改为不再使用 shell=True 来解决此问题。

于 2011-07-02T18:45:16.177 回答