1

从 Airflow 使用 DockerOperator 时出现错误：

raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to DockerOperator (task_id: etl_in_ch). Invalid arguments were:
**kwargs: {'volumes': ['./CH_ETL/src:/usr/src/copy_data', './pyproject.toml:pyproject.toml']}

我的代码：

from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.docker_operator import DockerOperator
from airflow.operators.dummy_operator import DummyOperator

default_args = {
'owner'                 : 'airflow',
'description'           : 'Extract data from different sources into CH and train model with it',
'depend_on_past'        : False,
'start_date'            : datetime(2021, 7, 20),
'email_on_failure'      : False,
'email_on_retry'        : False,
'retries'               : 1,
'retry_delay'           : timedelta(minutes=5)
}

with DAG('docker_operator_demo', default_args=default_args, schedule_interval="00 23 * * *", catchup=False) as dag:
    start_dag = DummyOperator(
        task_id='start_dag'
        )

    end_dag = DummyOperator(
        task_id='end_dag'
        )

    t1 = DockerOperator(
        task_id='etl_in_ch',
        image='python:3.9.2-slim',
        container_name='etl_in_ch',
        api_version='auto',
        auto_remove=True,
        command="apt-get update && apt-get install -y cron && apt-get install -y libxml2 libxslt-dev wget bzip2 gcc \
                && pip install --no-cache-dir  --upgrade pip \
                && pip install --no-cache-dir poetry==1.1.5 \
                && poetry config virtualenvs.create false\
                && poetry install --no-interaction --no-ansi \
                && chmod +x /src/__main__.py \
                %% python __main__.py",
        docker_url="unix://var/run/docker.sock",
        network_mode="bridge",
        environment={"PYTHONDONTWRITEBYTECODE": 1, "PYTHONUNBUFFERED": 1},
        working_dir="/usr/src/copy_data",
        volumes=['./CH_ETL/src:/usr/src/copy_data', './pyproject.toml:pyproject.toml']
    )


    start_dag >> t1

    t1 >> end_dag

我在这里做错了什么？我正在尝试使用 Apache Airflow 运行 dockerized python 服务，老实说，我看不出为什么突然“volumes”应该是错误的论点。

4

1 回答 1

1

Docker Provider (2.0.0) 的最新（主要）版本有一些重大更改（主要版本更改正确指示）：https ://airflow.apache.org/docs/apache-airflow-providers-docker/stable/index .html#id1 - 您需要将“volumes”替换为“mounts”

于 2021-07-20T19:00:56.623 回答