1

我正在运行在 Docker 上运行的 MWAA 实例的本地环境。一切正常,我尝试将 SecretsManager 设置为仅用于连接的替代后端。

我已将此添加到airflow.cfg文件中:

[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix" : "airflow/connections", "variables_prefix" : null, "config_prefix" : null}

然后,我aws_default在 Airflow Connections 中配置了一个连接:类型:Amazon Web Services

名称:aws_default

登录:<aws_access_key>

密码:<aws_secret_access_key>

我通过使用使用此连接 ID 启动 AWSHook 的函数验证了凭证是否正常工作。

我向凭证所属的用户添加了 SecretsManager 读写策略。

然后,我尝试使用 SnowflakeOperator 对 Snowflake 运行查询,以便从 SecretsManager 获取连接,但我收到此错误,就好像后端 SecretsManager 服务无法找到使用 API 和查询密钥的凭据一样。xxxx_snowflake_operator 和 xxxx_snowflake_hook 只是对snowflake_operatorand的包装snowflake_hook,做同样的事情,没有任何改变(只有 UI 的颜色)

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
    result = self._execute_task(context, task_copy)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
    result = task_copy.execute(context=context)
  File "/usr/local/airflow/plugins/operators/xxxx_snowflake_operator.py", line 84, in execute
    raise ex
  File "/usr/local/airflow/plugins/operators/xxxx_snowflake_operator.py", line 78, in execute
    parameters=self.parameters)
  File "/usr/local/lib/python3.7/site-packages/airflow/hooks/dbapi.py", line 173, in run
    with closing(self.get_conn()) as conn:
  File "/usr/local/lib/python3.7/site-packages/airflow/providers/snowflake/hooks/snowflake.py", line 215, in get_conn
    conn_config = self._get_conn_params()
  File "/usr/local/airflow/plugins/hooks/xxxx_snowflake_hook.py", line 22, in _get_conn_params
    conn = self.get_connection(self.snowflake_conn_id)
  File "/usr/local/lib/python3.7/site-packages/airflow/hooks/base.py", line 67, in get_connection
    conn = Connection.get_connection_from_secrets(conn_id)
  File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 351, in get_connection_from_secrets
    conn = secrets_backend.get_connection(conn_id=conn_id)
  File "/usr/local/lib/python3.7/site-packages/airflow/secrets/base_secrets.py", line 64, in get_connection
    conn_uri = self.get_conn_uri(conn_id=conn_id)
  File "/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/secrets_manager.py", line 115, in get_conn_uri
    return self._get_secret(self.connections_prefix, conn_id)
  File "/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/secrets_manager.py", line 153, in _get_secret
    SecretId=secrets_path,
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 162, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 373, in add_auth
    raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials

SecretsManager Airflow 后端为什么找不到凭证?它不使用默认的 AWS 连接 ID 来运行其 botocore API 请求吗?

更新:我通过在 ~/.aws/config 中设置一个包含 aws 访问和密钥的配置文件来使其工作。我想知道我是否可以让它使用 Airflow 中的 aws_default 连接。

4

1 回答 1

1

您可以airflow.cfg使用以下参数更新您的:

backend_kwargs = {"connections_prefix" : "airflow/connections", "variables_prefix" : "airflow/variables", "aws_access_key_id":"XXXXXXXXXXXXXX", "aws_secret_access_key":"XXXXXXXXXXXXXX","region_name":"XXXXXXXXXXXXXX"}

重建映像

./mwaa-local-env build-image

然后再次启动气流

./mwaa-local-env start

请注意,当您在 AWS 上设置 MWAA 环境时,您不aws_access_key_id应该设置aws_secret_access_keyregion_name

于 2021-11-29T20:57:14.530 回答