4

运行以下命令将作业提交给纱线，但任务管理器的分配数量不是预期的。集群有足够的资源<220vcores, 720G mem>

/data/clusterserver/flink-1.7.2/bin/flink run -m yarn-cluster -yn 4 -ys 2  /data/clusterserver/flink-1.7.2/examples/batch/WordCount.jar --input hdfs:///tmp/wordcount.txt --output hdfs:///tmp/count.result

2019-04-10 15:04:57,397 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,397 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,405 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,405 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,554 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}
2019-04-10 15:04:58,017 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/data/clusterserver/flink-1.7.2/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-04-10 15:05:01,419 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1554348641734_0105
2019-04-10 15:05:01,446 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1554348641734_0105
2019-04-10 15:05:01,447 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-04-10 15:05:01,450 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-04-10 15:05:05,747 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
Starting execution of program

从 INFO 日志中我们可以看到，发送给 yarn 的参数是 {masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}，但是 flink WebUI 中只有一个 taskManager。

使用 yarn-session.shwith-n也会遇到同样的问题，即从命令行控制任务管理器的数量不起作用。

  • flink 版本 1.7.2
  • hadoop 2.8.5 版（容量调度程序）
  • 爪哇：1.8.0_91
4

1 回答 1

6

从 Flink 开始1.5，Flink 支持完全的资源弹性。这意味着 FlinkTaskManagers在 Yarn 上运行时能够动态启动和停止。结果是您不再需要为TaskManagersYarn 上的 Flink 集群指定容器数。这就是 Flink 不再尊重-n/-yn选项的原因。

相反，Flink 会考虑作业的并行度来计算执行作业所需的插槽数，然后启动TaskManagers满足插槽要求所需的插槽数。

您可以在此处和此处找到有关部署更改的更多信息。

于 2019-04-10T11:38:49.077 回答