我正在运行在 Docker 上运行的 MWAA 实例的本地环境。一切正常,我尝试将 SecretsManager 设置为仅用于连接的替代后端。
我已将此添加到airflow.cfg
文件中:
[secrets]
backend = airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend
backend_kwargs = {"connections_prefix" : "airflow/connections", "variables_prefix" : null, "config_prefix" : null}
然后,我aws_default
在 Airflow Connections 中配置了一个连接:类型:Amazon Web Services
名称:aws_default
登录:<aws_access_key>
密码:<aws_secret_access_key>
我通过使用使用此连接 ID 启动 AWSHook 的函数验证了凭证是否正常工作。
我向凭证所属的用户添加了 SecretsManager 读写策略。
然后,我尝试使用 SnowflakeOperator 对 Snowflake 运行查询,以便从 SecretsManager 获取连接,但我收到此错误,就好像后端 SecretsManager 服务无法找到使用 API 和查询密钥的凭据一样。xxxx_snowflake_operator 和 xxxx_snowflake_hook 只是对snowflake_operator
and的包装snowflake_hook
,做同样的事情,没有任何改变(只有 UI 的颜色)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1138, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/airflow/plugins/operators/xxxx_snowflake_operator.py", line 84, in execute
raise ex
File "/usr/local/airflow/plugins/operators/xxxx_snowflake_operator.py", line 78, in execute
parameters=self.parameters)
File "/usr/local/lib/python3.7/site-packages/airflow/hooks/dbapi.py", line 173, in run
with closing(self.get_conn()) as conn:
File "/usr/local/lib/python3.7/site-packages/airflow/providers/snowflake/hooks/snowflake.py", line 215, in get_conn
conn_config = self._get_conn_params()
File "/usr/local/airflow/plugins/hooks/xxxx_snowflake_hook.py", line 22, in _get_conn_params
conn = self.get_connection(self.snowflake_conn_id)
File "/usr/local/lib/python3.7/site-packages/airflow/hooks/base.py", line 67, in get_connection
conn = Connection.get_connection_from_secrets(conn_id)
File "/usr/local/lib/python3.7/site-packages/airflow/models/connection.py", line 351, in get_connection_from_secrets
conn = secrets_backend.get_connection(conn_id=conn_id)
File "/usr/local/lib/python3.7/site-packages/airflow/secrets/base_secrets.py", line 64, in get_connection
conn_uri = self.get_conn_uri(conn_id=conn_id)
File "/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/secrets_manager.py", line 115, in get_conn_uri
return self._get_secret(self.connections_prefix, conn_id)
File "/usr/local/lib/python3.7/site-packages/airflow/providers/amazon/aws/secrets/secrets_manager.py", line 153, in _get_secret
SecretId=secrets_path,
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 663, in _make_api_call
operation_model, request_dict, request_context)
File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 682, in _make_request
return self._endpoint.make_request(operation_model, request_dict)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
operation_name=operation_model.name)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
return self._emitter.emit(aliased_event_name, **kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
return self._emit(event_name, kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
response = handler(**kwargs)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 162, in sign
auth.add_auth(request)
File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 373, in add_auth
raise NoCredentialsError()
botocore.exceptions.NoCredentialsError: Unable to locate credentials
SecretsManager Airflow 后端为什么找不到凭证?它不使用默认的 AWS 连接 ID 来运行其 botocore API 请求吗?
更新:我通过在 ~/.aws/config 中设置一个包含 aws 访问和密钥的配置文件来使其工作。我想知道我是否可以让它使用 Airflow 中的 aws_default 连接。