1

我构建了这个线程类来使用 TensorRT 运行推理：

class GPUThread(threading.Thread):

  def __init__(self, engine_path):
    threading.Thread.__init__(self)
    self.engine_path = engine_path
    self.engine = self.open_engine(engine_path)

  def run(self):
    cuda.init()
    #self.dev = cuda.Device(0)
    #self.ctx = self.dev.make_context()
    self.rt_run()
    #self.ctx.pop()
    #del self.ctx
    return

  def rt_run(self):
    with self.engine.create_execution_context() as context:
      inputs, outputs, bindings, stream = self.allocate_buffers(self.engine)
      # ...  Retrieve image
      self.load_input(inputs[0].host, image)
      [output] = self.do_inference(
        context,
        bindings=bindings,
        inputs=inputs,
        outputs=outputs,
        stream=stream
      )
    return

  def load_input(self, pagelocked_buffer, image):
    # ... Image transformations ...
    # Copy to the pagelocked input buffer
    np.copyto(pagelocked_buffer, crop_img)
    return

  def allocate_buffers(self, engine):
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()
    for binding in engine:
      size = trt.volume(engine.get_binding_shape(binding)) * engine.max_batch_size
      dtype = trt.nptype(engine.get_binding_dtype(binding))
      # Allocate host and device buffers
      host_mem = cuda.pagelocked_empty(size, dtype)
      device_mem = cuda.mem_alloc(host_mem.nbytes)
      # Append the device buffer to device bindings.
      bindings.append(int(device_mem))
      # Append to the appropriate list.
      if engine.binding_is_input(binding):
        inputs.append(HostDeviceMem(host_mem, device_mem))
      else:
        outputs.append(HostDeviceMem(host_mem, device_mem))
    return inputs, outputs, bindings, stream

  def run_inference(self, context, bindings, inputs, outputs, stream, batch_size=1):
    # Transfer input data to the GPU.
    [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
    # Run inference.
    context.execute_async(batch_size=batch_size, bindings=bindings, stream_handle=stream.handle)
    # Transfer predictions back from the GPU.
    [cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs]
    # Synchronize the stream
    stream.synchronize()
    # Return only the host outputs.
    return [out.host for out in outputs]

运行上面的代码时，我得到错误：stream = cuda.Stream() pycuda._driver.LogicError: explicit_context_dependent failed: invalid device context - no currently active context?这个函数在上面cuda.Stream()被调用。allocate_buffers

所以我然后尝试下面的run（注意这是上面注释掉的代码）：

self.dev = cuda.Device(0)
self.ctx = self.dev.make_context()
self.rt_run()
self.ctx.pop()
del self.ctx

这会导致我的系统在调用rt_run's时完全冻结。create_execution_context我猜在创建 PyCuda 上下文和创建 TensorRT 执行上下文之间存在冲突？我在 Jetson Nano 上运行它。

如果我删除create_execution_context代码，我可以分配缓冲区，并且似乎上下文处于活动状态并在工作线程中找到。但是，如果没有 TensorRT执行上下文，我就无法运行推理。execute_async不是self.ctx上面的方法。

请注意，从主线程运行时不会出现这些问题。我可以使用 PyCuda 的 autoinit 并像上面的代码一样创建一个执行上下文。

所以总而言之，在工作线程中，除非我调用，否则我无法分配缓冲区，self.dev.make_context但这会导致create_execution_context调用崩溃系统。如果我不调用self.dev.make_context，我将无法在执行上下文中分配缓冲区，因为invalid device context调用时cuda.Stream()出现错误allocate buffers。

我正在运行的内容：

  • 张量RT 6
  • PyCuda 1.2
  • 杰森纳米 2019 (A02)
4

0 回答 0