2

我已经在我的 Django 1.10 项目中安装了 Celery[sqs] 和 django-celery-beat。我尝试使用 Supervisor on 和 Elastic Beanstalk 实例同时运行它们(worker 和 beat)。

使用以下脚本动态创建 Supervisor 配置:

#!/usr/bin/env bash

# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}

# create celery beat config script
celerybeatconf="[program:celery-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A phsite --loglevel=DEBUG --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# create celery worker config script
celeryworkerconf="[program:celery-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A phsite --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999

environment=$celeryenv"

# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf

# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update

之后运行以下 ebextension:

container_commands:
  01_create_celery_beat_configuration_file:
    command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
  02_chmod_supervisor_sock:
    command: "chmod 777 /opt/python/run/supervisor.sock"
  03_create_logs:
    command: "touch /var/log/celery-beat.log /var/log/celery-worker.log"
  04_chmod_logs:
    command: "chmod 777 /var/log/celery-beat.log /var/log/celery-worker.log"
  05_start_celery_worker:
    command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celery-worker"
  06_start_celery_beat:
    command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf start celery-beat"

登录Instance,运行时

/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf status

当 celery-worker 运行时,celery-beat 已经“未启动”(带有一个空的日志文件)。最奇怪的部分是,如果我手动运行它(例如

/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf start celery-beat

它运行没有错误。

任何人都知道如何调试它?为什么稍后加载时它不会在 eb_extension 中加载?也许这与 Django 尚未启动并且正在使用django_celery_beat.schedulers:DatabaseScheduler配置的事实有关?

4

1 回答 1

1

所以简单的原因是,在 eb_extension 中创建的 shell 脚本:

container_commands:
  01_create_celery_beat_configuration_file:
    command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"

在 appdeploy/post 目录中创建,因此在执行以下命令后运行(基本上是部署后)。start/restart 命令没有做任何事情,因为 shell 脚本还没有注册这些服务。‍♂️</p>

于 2019-01-16T08:45:18.790 回答