0

我正在使用 Apache Hadoop、Spark 和 DL4J 开展分布式深度学习项目。

我的主要问题是在 spark 上启动我的应用程序时，它会进入运行状态并且永远不会超过 10% 的进度我收到此警告

2019-08-23 20:55:49,198 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1161
2019-08-23 20:55:49,224 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[5] at saveAsTextFile at BaseTrainingMaster.java:211) (first 15 tasks are for partitions Vector(0, 1))
2019-08-23 20:55:49,226 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 2 tasks
2019-08-23 20:56:04,286 WARN cluster.YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-08-23 20:56:17,526 WARN cluster.YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
2019-08-23 20:56:23,135 WARN cluster.YarnClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

这最后 3 行不停地运行

实际上我只有 1 个主节点和 1 个从节点，安装了Hadoop和Spark

  • Master的内存为 8GB，配备英特尔 i5 6500
  • Slave是 4GB RAM 和 intel i3 4400

检查HDFS的 WebUI 和日志文件后，我可以看到HDFS工作没有问题 Yarn WebUI 和日志还显示 Yarn 与 1 DATANODE一起工作正常

在这里你可以检查我的代码，看看它卡在哪里

VoidConfiguration config = VoidConfiguration.builder()
            .unicastPort(40123)
            .networkMask("192.168.0.0/42")   
            .controllerAddress("192.168.1.35")  
            .build();

    log.log(Level.INFO,"==========After voidconf");

    //      Create the TrainingMaster instance
    TrainingMaster trainingMaster = new SharedTrainingMaster.Builder(config, 1)
            .batchSizePerWorker(10) 
            .workersPerNode(1)      
            .build();

    log.log(Level.INFO,"==========after training master");
    SparkDl4jMultiLayer sparkNet = new SparkDl4jMultiLayer(sc, conf, trainingMaster);


    log.log(Level.INFO,"==========after sparkMultilayer");
    //      Execute training:
    log.log(Level.INFO,"==========Starting training");
    for (int i = 0; i < 100; i++) {
        log.log(Level.INFO,"Epoch : " + i); // this is the Last line from my code that is printed in the Log
        sparkNet.fit(rddDataSetClassification); //it gets stuck here 
        log.log(Level.INFO,"Epoch : " + i + " / " + i);
    }
    log.log(Level.INFO,"after training");
    //      Dataset Evaluation
    Evaluation eval = sparkNet.evaluate(rddDataSetClassification);
    log.log(Level.INFO, eval.stats());

纱线站点.xml

<property>
            <name>yarn.acl.enable</name>
            <value>0</value>
    </property>

    <property>
            <name>yarn.resourcemanager.hostname</name>
            <value>192.168.1.35</value>
    </property>

    <property>
            <name>yarn.nodemanager.aux-services</name>
            <value>mapreduce_shuffle</value>
    </property>

<property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>3072</value>
</property>

<property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>3072</value>
</property>

<property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>256</value>
</property>

<property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
</property>

spark-defult.conf：

spark.master        yarn
spark.driver.memory     2500m
spark.yarn.am.memory    2500m
spark.executor.memory   2000m
spark.eventLog.enabled      true
spark.eventLog.dir      hdfs://hadoop-MS-7A75:9000/spark-logs
spark.history.provider            org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory     hdfs://hadoop-MS-7A75:9000/spark-logs
spark.history.fs.update.interval  10s
spark.history.ui.port             18080

我怀疑有任何资源问题，所以我尝试设置属性，例如将spark.executor.cores和 s park.executor.instances设置为1 我还尝试更改 yarn 和 spark 上下的内存分配（我不确定它是如何工作的）

来自 spark.deploy.master....的日志

2019-08-23 20:18:33,669 INFO master.Master: I have been elected leader! New state: ALIVE
2019-08-23 20:18:40,771 INFO master.Master: Registering worker 192.168.1.37:42869 with 4 cores, 2.8 GB RAM

来自 spark.deploy.worker....的日志

19/08/23 20:18:40 INFO Worker: Connecting to master hadoop-MS-7A75:7077...
19/08/23 20:18:40 INFO TransportClientFactory: Successfully created connection to hadoop-MS-7A75/192.168.1.35:7077 after 115 ms (0 ms spent in bootstraps)
19/08/23 20:18:40 INFO Worker: Successfully registered with master spark://hadoop-MS-7A75:7077
4

1 回答 1

0

通过添加另一个奴隶解决了这个问题我不知道它为什么以及如何工作但是当我添加另一个奴隶时它工作了

于 2019-08-24T12:54:38.847 回答