8

从 celery 2.4.5 升级后,我开始让 celery 随机关闭。
我在 centOS 机器上使用 celery 3.0.12、boto 2.6 和 amazon sqs 和 django 1.2.7(pip freeze dump 在底部)

我在跑步

service celerybeat start
service celeryd start

在我启动 celery 几秒钟后,它停止(关闭),如果我查看其中一个 celery 日志,我总是会看到:

[2012-12-31 10:13:40,275: INFO/MainProcess] Task patrol.tasks.test[270f1558-bcc2-441b-8961 e1f21a2dbd27] succeeded in 0.318082094193s: None
[2012-12-31 10:13:40,424: INFO/MainProcess] child process calling self.run()
[2012-12-31 10:13:40,428: INFO/MainProcess] Got task from broker: patrol.tasks.myTask[d9a5ab26-71ca-448b-a4da-40315570f219]
[2012-12-31 10:13:40,666: INFO/MainProcess] Got task from broker: tasks.test[99edb7e2-caff-4892-a95b-c18a9d7f5c51]
[2012-12-31 10:13:41,114: WARNING/MainProcess] Restoring 2 unacknowledged message(s).
[2012-12-31 10:13:41,115: WARNING/MainProcess] UNABLE TO RESTORE 2 MESSAGES: (TypeError('<boto.sqs.message.Message instance at 0x3269758> is not JSON serializable',), TypeError('<boto.sqs.message.Message instance at 0x32697e8> is not JSON serializable',))
[2012-12-31 10:13:41,116: WARNING/MainProcess] EMERGENCY DUMP STATE TO FILE -> /tmp/tmppO4Bbp <-
[2012-12-31 10:13:41,116: WARNING/MainProcess] Cannot pickle state: TypeError('a class that defines __slots__ without defining __getstate__ cannot be pickled',). Fallback to pformat.

我使用 maxtaskperchild 的低值来快速重新创建关机。如果我给出更高的值,那么在关机发生之前需要更长的时间。

编辑

在尝试隔离问题时,我删除了所有定期任务。现在我只有一个周期性任务和一个基本上什么都不做的任务,但我仍然可以每次都重现这个错误。

@task
def myTask():
    print 1
    return

class test(PeriodicTask):
    run_every = datetime.timedelta(seconds=3)
    def run(self, **kwargs):
            myTask.delay()
            print '2'

我的 /init.d/celeryd

芹菜

我的 /default/celeryd

# Name of nodes to start, here we have a single node
# or we could have three nodes:
CELERYD_NODES="w1 w2"

CELERYD_LOG_LEVEL="INFO"

# Where to chdir at start.
CELERYD_CHDIR="/var/myproject"

# How to call "manage.py celeryd_multi"
CELERYD_MULTI="python $CELERYD_CHDIR/manage.py celeryd_multi"

# How to call "manage.py celeryctl"
CELERYCTL="python $CELERYD_CHDIR/manage.py celeryctl"

MAXTASKPERCHILD=2 # this is low on purpose to recreate the shutdown fast
CELERY_CONC=5
EXPRESS_CONC=2
# Extra arguments to celeryd
CELERYD_OPTS="-Q:w1 celery,backup -c:w1 $CELERY_CONC -Q:w2 express -c:w2 $EXPRESS_CONC --time-limit=3600 --maxtasksperchild=$MAXTASKPERCHILD -E"

# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celeryd/%n.log"
CELERYD_PID_FILE="/var/run/celeryd/%n.pid"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"

# Path to celerybeat
CELERYBEAT="python $CELERYD_CHDIR/manage.py celerybeat"

# Extra arguments to celerybeat.  This is a file that will get
# created for scheduled tasks.  It's generated automatically
# when Celerybeat starts.
CELERYBEAT_OPTS="--schedule=/var/run/celerybeat-schedule"

# Log level. Can be one of DEBUG, INFO, WARNING, ERROR or CRITICAL.
CELERYBEAT_LOG_LEVEL="INFO"

# Log file locations
CELERYBEAT_LOGFILE="/var/log/celeryd/celerybeat.log"
CELERYBEAT_PIDFILE="/var/run/celeryd/celerybeat.pid"

我的点冻结

Django==1.2.7
M2Crypto==0.20.2
MySQL-python==1.2.3c1
amqp==1.0.6
amqplib==1.0.2
anyjson==0.3.3
billiard==2.7.3.19
boto==2.1.1
celery==3.0.12
certifi==0.0.6
distribute==0.6.10
django-celery==3.0.11
django-kombu==0.9.4
django-picklefield==0.3.0
ghettoq==0.4.5
importlib==1.0.2
iniparse==0.3.1
ipython==0.12
kombu==2.5.4
lxml==2.3.4
mixpanel-celery==0.5.0
netaddr==0.7.6
numpy==1.6.2
odict==1.4.4
ordereddict==1.1
pycrypto==2.6
pycurl==7.19.0
pygooglechart==0.3.0
pygpgme==0.1
python-dateutil==1.5
python-memcached==1.48
pytz==2012h
requests==0.9.0
six==1.2.0
urlgrabber==3.9.1
yum-metadata-parser==1.1.2
4

3 回答 3

1

我想您有应用程序将非 json 格式的内容放入队列中。以短信为例。您可以尝试在 celery 设置中更改队列,看看它是如何工作的

于 2012-12-27T06:32:32.760 回答
0

将 Kombu 升级到 2.5.4 之后的任何版本,因为它是 Kombu 中的一个错误。它应该在 Celery >= 3.0.20 的任何版本中修复(这取决于 kombu>=2.5.5)。

http://kombu.readthedocs.org/en/latest/changelog.html#version-2-5-5

于 2014-05-08T20:35:15.467 回答
0

在我的情况下,我使用了错误版本的 python python3.8,但是我需要一个旧版本。

所以我更新了版本:

pyenv install 3.6.7 
pyenv local 3.6.7
pyenv local
virtualenv -p python3.6 venv 
source venv/bin/activate
pip install -r requirements.txt
于 2020-11-25T18:02:24.430 回答