我使用 docker 镜像在 ecs fargate 上部署了 webserver、scheduler、flower 和 worker。我正在使用最新的气流版本 2.1.2。当我运行 dag 时,工作节点抛出错误,说明如下:
[2021-08-13 11:38:45,323: ERROR/ForkPoolWorker-7] Task airflow.executors.celery_executor.execute_command[c22087fe-52e7-402d-bc89-d341e37f56e9] raised unexpected: AirflowException('Celery command failed on host: ip-172-30-1-180.ec2.internal')
Traceback (most recent call last):
File "/root/.local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task
R = retval = fun(*args, **kwargs)
File "/root/.local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__
return self.run(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 88, in execute_command
_execute_in_fork(command_to_exec)
File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 99, in _execute_in_fork
raise AirflowException('Celery command failed on host: ' + get_hostname())
airflow.exceptions.AirflowException: Celery command failed on host: ip-172-XX-X-XX.ec2.internal