1

我有一个配置有单个工作人员的单节点火花集群（4 个 cpu 核心和 15GB 内存）。我可以访问 Web UI 并查看工作节点。但是，我无法使用 spark-submit 提交作业。我有几个问题。

  1. 我在集群中存储了一个 uber-jar 文件。我使用以下命令提交作业spark-submit --class Main --deploy-mode cluster --master spark://cluster:7077 uber-jar.jar。这会启动作业，但会立即失败并显示以下日志消息。
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/11/13 01:19:47 INFO SecurityManager: Changing view acls to: admin
19/11/13 01:19:47 INFO SecurityManager: Changing modify acls to: admin
19/11/13 01:19:47 INFO SecurityManager: Changing view acls groups to: 
19/11/13 01:19:47 INFO SecurityManager: Changing modify acls groups to: 
19/11/13 01:19:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(admin); groups with view permissions: Set(); users  with modify permissions: Set(admin); groups with modify permissions: Set()
19/11/13 01:19:48 INFO Utils: Successfully started service 'driverClient' on port 46649.
19/11/13 01:19:48 INFO TransportClientFactory: Successfully created connection to cluster/10.10.10.10:7077 after 37 ms (0 ms spent in bootstraps)
19/11/13 01:19:48 INFO ClientEndpoint: Driver successfully submitted as driver-20191113011948-0010
19/11/13 01:19:48 INFO ClientEndpoint: ... waiting before polling master for driver state
19/11/13 01:19:53 INFO ClientEndpoint: ... polling master for driver state
19/11/13 01:19:53 INFO ClientEndpoint: State of driver-20191113011948-0010 is FAILED
19/11/13 01:19:53 INFO ShutdownHookManager: Shutdown hook called
19/11/13 01:19:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-4da02cd2-5cfc-4a2a-ad10-41a594569ea1

我做错了什么以及如何正确提交作业。

  1. 如果我的 uber-jar 文件在我的本地计算机中，如何正确使用 spark-submit 将使用 uber-jar 文件的 spark 作业从本地计算机提交到集群。我已经尝试在本地计算机上运行 spark-shell，方法是使用spark-shell --master spark:\\cluster:7077. 这会在我的本地计算机中启动一个 spark shell，我可以看到（在 spark Web UI 中）worker 在集群中获得了分配给它的内存。但是，如果我尝试在 shell 中执行任务，我会收到以下错误消息。 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
4

0 回答 0