0

我正在使用 eventlet 类和并发设置为 8 运行标准 celery worker。当这种情况发生时,这些是非常忙碌的工作人员(但在不忙时可能会发生,很难说)。

我知道我的任务没有任何泄漏,并且以每个子项设置的最大任务运行它以防万一。

不过,我遇到了这些错误,我不知道为什么。据我所知,这是随机的。我的环境中有什么东西吗?已知的问题?除非我有误解,否则这似乎不是由于任务本身的错误。我错了吗?任何想法都会很棒!

此外,作为 prefork 运行时,我没有收到任何这些错误。

错误一:

2015-05-02 06:13:37,452: WARNING/MainProcess] /opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py:364:
 RuntimeWarning: Exception raised outside body: SystemError('error return without exception set',):
Traceback (most recent call last):
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py", line 283, in trace_task
    uuid, retval, SUCCESS, request=task_request,
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/base.py", line 248, in store_result
    request=request, **kwargs)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/base.py", line 481, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/cache.py", line 126, in set
    return self.client.set(key, value, self.expires)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/memcache.py", line 584, in set
    return self._set("set", key, val, time, min_compress_len)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/memcache.py", line 835, in _set
    return _unsafe_set()
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/memcache.py", line 827, in _unsafe_set
    return(server.expect("STORED", raise_exception=True)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/memcache.py", line 1196, in expect
    line = self.readline(raise_exception)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/memcache.py", line 1182, in readline
    data = recv(4096)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/greenio.py", line 325, in recv
    timeout_exc=socket.timeout("timed out"))
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/greenio.py", line 200, in _trampoline
    mark_as_closed=self._mark_as_closed)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/hubs/__init__.py", line 159, in trampoline
    return hub.switch()
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/hubs/hub.py", line 293, in switch
    return self.greenlet.switch()
SystemError: error return without exception set

错误2:

Traceback (most recent call last):
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/worker/__init__.py", line 227, in _process_task
    req.execute_using_pool(self.pool)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/worker/job.py", line 263, in execute_using_pool
    correlation_id=uuid,
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/concurrency/base.py", line 156, in apply_async
    **options)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/concurrency/eventlet.py", line 144, in on_apply
    self.getpid)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/greenpool.py", line 106, in spawn_n
    self.sem.acquire()
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/semaphore.py", line 96, in acquire
    hubs.get_hub().switch()
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/hubs/hub.py", line 293, in switch
    return self.greenlet.switch()
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/greenpool.py", line 93, in _spawn_n_impl
    self._spawn_done(coro)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/eventlet-0.16.0.dev-py2.7.egg/eventlet/greenpool.py", line 125, in _spawn_done
    self.coroutines_running.remove(coro)
KeyError: <greenlet.greenlet object at 0xc759e0>

错误 3:

[2015-05-02 08:41:48,786: WARNING/MainProcess] /opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py:364:
 RuntimeWarning: Exception raised outside body: TypeError('sequence item 1: expected string, NoneType found',):
Traceback (most recent call last):
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py", line 253, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py", line 201, in on_error
    R = I.handle_error_state(task, eager=eager)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py", line 85, in handle_error_state
    }[self.state](task, store_errors=store_errors)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/app/trace.py", line 118, in handle_failure
    req.id, exc, einfo.traceback, request=req,
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/base.py", line 121, in mark_as_failure
    traceback=traceback, request=request)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/base.py", line 248, in store_result
    request=request, **kwargs)
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/base.py", line 481, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/opt/app/thisapp/software/python/lib/python2.7/site-packages/celery-3.1.16-py2.7.egg/celery/backends/base.py", line 406, in get_key_for_task
    self.task_keyprefix, key_t(task_id), key_t(key),
TypeError: sequence item 1: expected string, NoneType found
4

1 回答 1

0

尝试降低 Celery 的并发性。

对于 Solaris,我们没有一个好的并发实现(集线器)。除了不真正支持的 pyevent 。所以它会使用 select 或 poll 并且仅限于观看 1024 个文件描述符。一项任务可能使用零个或多个文件描述符,所以我建议调整得比现在低,没有特定的数字。

升级到最新的 Eventlet 版本也总是一个好主意。今天 2015-05-04 它是0.17.3.

于 2015-05-04T05:15:00.893 回答