0

我正在尝试在 Airflow 中使用 DockerOperator 来运行容器。

当 dag 运行时，我收到以下错误

2021-05-12 18:45:59,417] {taskinstance.py:1128} ERROR - Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

图像存储在 AWS ECR 中。我使用以下命令为 ecr 生成密码：

aws ecr get-login-password --region <region-name>

我的任务的配置如下：

  • 变量和默认参数
docker_conn_string = "DOCKER_HUB_CONNECTION_STRING"
docker_hub_url = Variable.get("DOCKER_HUB_URL")
docker_image = docker_hub_url + '/image-name'

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'retry_delay': timedelta(minutes=1),
    "max_active_runs": 1,
    "retries": 0
}
  • 有向无环图
dag = DAG(
   'do_some_work',
   default_args = default_args,
   schedule_interval = "0 0 * * *",
   catchup = False,
)
  • 码头工人
run_image = DockerOperator(
    api_version = 'auto',
    docker_conn_id = docker_conn_string,
    auto_remove = True,
    # force_pull = True,
    command = 'command to be run',
    image = docker_image,
    task_id = 'do_work',
    dag = dag,
    start_date = datetime(2021, 5, 5)
)

我尝试过的事情，

  • 重新检查连接字符串以使用 https 和 tcp
  • 1.21版本前后手动添加docker版本
  • 当我使用 docker 登录凭据运行 docker pull 时，可以拉取图像，因此 EMR 密码不是问题
  • 当我在本地气流实例中运行它时，如果图像已经在本地下载，它就可以工作

我的评估是，在拉取 docker 操作员用来确定 docker 版本的图像之前还有额外的 api，这是失败的原因，因此无法拉取图像。我认为这可能与某些 ECR 设置有关。

这里的任何帮助将不胜感激。

谢谢

4

0 回答 0