0

我有一个气流任务,其日志显示运行成功,但相同的日志也显示 SIGTERM。

[2021-11-30 00:52:25,481] {taskinstance.py:1087} INFO - 
--------------------------------------------------------------------------------
[2021-11-30 00:52:25,481] {taskinstance.py:1088} INFO - Starting attempt 1 of 11
[2021-11-30 00:52:25,482] {taskinstance.py:1089} INFO - 
--------------------------------------------------------------------------------
[2021-11-30 00:52:25,492] {taskinstance.py:1107} INFO - Executing <Task(PythonOperator): > on 2021-11-29T00:30:00+00:00
[2021-11-30 00:52:25,500] {standard_task_runner.py:52} INFO - Started process 11633 to run task

[2021-11-30 00:52:25,509] {standard_task_runner.py:77} INFO - Job 1580251: Subtask 
[2021-11-30 00:52:25,609] {logging_mixin.py:104} INFO - Running <TaskInstance: DWH_D_V1.1.0.transform 2021-11-29T00:30:00+00:00 [running]> on host 
[2021-11-30 00:52:25,718] {taskinstance.py:1300} INFO - Exporting the following env vars:

AIRFLOW_CTX_EXECUTION_DATE=2021-11-29T00:30:00+00:00
AIRFLOW_CTX_DAG_RUN_ID=scheduled__2021-11-29T00:30:00+00:00
[2021-11-30 00:52:26,105] {transform.py:55} INFO - Empty 
[2021-11-30 00:52:26,755] {transform.py:58} INFO - Inserting values
[2021-11-30 01:07:54,098] {db_table.py:561} INFO - Datadog metrics push
[2021-11-30 01:07:54,099] {hostname.py:60} INFO - No agent or invalid configuration file found
[2021-11-30 01:07:54,113] {db_table.py:573} INFO - Datadog push: Number of rows: 47951835 
[2021-11-30 01:07:54,405] {api_client.py:138} INFO - 202 POST 
[2021-11-30 01:07:54,405] {datadog_logger.py:65} INFO - Datadog push: {'status': 'ok'}
[2021-11-30 01:07:54,405] {db_table.py:586} INFO - Datadog push: Execution time: 928.146014213562 s
[2021-11-30 01:07:54,477] {api_client.py:138} INFO - 202 POST https://api.datadoghq.com/api/v1/series (71.0046ms)
[2021-11-30 01:07:54,477] {datadog_logger.py:65} INFO - Datadog push: {'status': 'ok'}
[2021-11-30 01:07:54,477] {transform_dim_simple.py:66} INFO - Done
[2021-11-30 01:07:54,478] {python.py:151} INFO - Done. Returned value was: None
[2021-11-30 01:07:54,501] {taskinstance.py:1204} INFO - Marking task as SUCCESS. dag_id=DWH_D_V1.1.0, task_id=trasnform, execution_date=20211129T003000, start_date=20211130T005225, end_date=20211130T010754
[2021-11-30 01:07:54,705] {local_task_job.py:197} WARNING - State of this instance has been externally set to success. Terminating instance.
[2021-11-30 01:07:54,707] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 11633
[2021-11-30 01:07:54,753] {taskinstance.py:1284} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-11-30 01:07:54,753] {logging_mixin.py:104} WARNING - Exception ignored in: <function _collection_gced at 0x7f4e1ca08f70>
[2021-11-30 01:07:54,753] {logging_mixin.py:104} WARNING - Traceback (most recent call last):
[2021-11-30 01:07:54,753] {logging_mixin.py:104} WARNING -   File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/event/registry.py", line 53, in _collection_gced
[2021-11-30 01:07:54,754] {logging_mixin.py:104} WARNING -     def _collection_gced(ref):
[2021-11-30 01:07:54,754] {logging_mixin.py:104} WARNING -   File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1286, in signal_handler
[2021-11-30 01:07:54,755] {logging_mixin.py:104} WARNING -     raise AirflowException("Task received SIGTERM signal")
[2021-11-30 01:07:54,755] {logging_mixin.py:104} WARNING - airflow.exceptions.AirflowException: Task received SIGTERM signal
[2021-11-30 01:07:54,802] {taskinstance.py:1265} INFO - 1 downstream tasks scheduled from follow-on schedule check
[2021-11-30 01:07:54,840] {process_utils.py:66} INFO - Process psutil.Process(pid=11633, status='terminated', exitcode=0, started='00:52:25') (11633) terminated with exit code 0

我知道任务已经成功,因为 datadog 在那天显示了该过程的指标推送。在气流收到外部触发器或类似的东西之后,但我确信没有人手动将任务标记为成功,我们没有任何进程可以改变该进程的状态。

这只随机发生了两次。

这是一种已知的行为还是有任何需要调查的提示。我虽然在增加AIRFLOW_CORE_KILLED_TASK_CLEANUP_TIME,但由于它是随机的,而且它发生了几次,甚至不知道如何检查它是否是解决方案,以防它工作正常。

4

0 回答 0