google-cloud-dataproc - GCP Dataproc 的 cloud-sql-proxy 失败，nc：连接到 localhost 端口 3306 (tcp) 失败：连接被拒绝

Question

我正在使用 Terraform 创建一个使用 GCP cloudsql 实例作为 hivemetastore 的 dataproc 集群，terrafrm 项目创建集群及其所有先决条件（网络、服务帐户、cloudsql 实例和用户等）。提供帮助以帮助解决此问题，但是当集群创建失败并出现错误
cloud-sql-proxy.sh时，我无法使其工作：cloud-sql-proxy.sh

nc：连接到 localhost 端口 3306 (tcp) 失败：连接被拒绝

我把头撞在墙上，试图找出原因，但无法深入了解，所以希望这里有人能提供帮助。

我在https://github.com/jamiekt/democratising-dataproc托管了 terraform 项目。重现问题非常简单，请按照下列步骤操作：

1. 如果您还没有安装 terraform
2. gcloud如果您还没有安装
3. 创建一个新的 GCP 项目
4. 为您的新项目启用Cloud Dataproc API
5. gcloud auth application-default login #creates a file containing credentials that terraform will use
6. git clone git@github.com:jamiekt/democratising-dataproc.git && cd democratising-dataproc
7. export GCP_PROJECT=name-of-project-you-just-created
8. make init
9. make apply

这应该成功地启动一个网络、子网、cloudsql 实例、几个存储桶（其中一个包含 cloud-sql-proxy.sh）、一个服务帐户、一个防火墙，然后在尝试创建 dataproc 集群时失败。

如果有人可以看看并告诉我为什么这会失败，我将非常感激。

Answer 1

您似乎没有在cloud-sql-proxy.sh.tmpl模板 ( diff )中使用最新版本的cloud-sql-proxy.sh脚本。

您可能想尝试使用来自 Dataproc Cloud SQL I/O 和 Hive Metastore初始化操作的最新脚本版本更新您的模板。

Answer 2

这里有很多问题现在已经解决了：

• service_account_scopes 需要指定和hive:hive.metastore.warehouse.dir属性需要设置
• Terraform 从 MySQL 中删除了默认的 root 用户，因此必须重新创建它。我缺少（除其他外）主机属性（host = '%'）
• 其他各种东西

发布此消息时的 repo 状态将按预期工作（即使用 Terraform 创建一个使用共享 hive 元存储的 dataproc 集群）。

感谢@igor-dvorzhak 的回复，您指向关于配置 Hive Metastore 以使用 Cloud SQL 的文章的链接让我走上了正轨。

Answer 3

按照这里的教程，我遇到了同样的问题

我看不出你给出的修复方法是什么。似乎 hive 尝试通过 localhost 访问数据库我不明白为什么这是来自 dataproc 的日志：

Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mysql
Removed /etc/systemd/system/multi-user.target.wants/mysql.service.
2020-10-07 14:05:49 URL:https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 [16361206/16361206] -> "cloud_sql_proxy.linux.amd64" [1]
Created symlink /etc/systemd/system/multi-user.target.wants/cloud-sql-proxy.service → /usr/lib/systemd/system/cloud-sql-proxy.service.
About to run 'nc -zv localhost 3306' with retries...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 1 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 2 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 3 failed! Sleeping 2.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 4 failed! Sleeping 3.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 5 failed! Sleeping 5.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 6 failed! Sleeping 8.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 7 failed! Sleeping 13.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 8 failed! Sleeping 21.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 9 failed! Sleeping 34.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 10 failed! Sleeping 55.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 11 failed! Sleeping 89.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 12 failed! Sleeping 144.
Final attempt of 'nc -zv localhost 3306'...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused