2

我尝试设置 2 个队列 - queue1、queue2。我将这些队列的名称添加到 mapred-site.xml

    <property>
        <name>mapred.queue.names</name>
        <value>queue1,queue2</value>
    </property>

我配置了CapacityScheduler.xml，如下所示。

    <?xml version="1.0"?>
    <configuration>
         <property>
            <name>mapred.capacity-scheduler.maximum-system-jobs</name>
            <value>3000</value>   
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue1.capacity</name>
            <value>100</value>   
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.capacity</name>
            <value>100</value>   
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.maximum-capacity</name>
            <value>-1</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.maximum-capacity</name>
            <value>-1</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.supports-priority</name>
            <value>false</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.supports-priority</name>
            <value>false</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.minimum-user-limit-percent</name>
            <value>100</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.minimum-user-limit-percent</name>
            <value>100</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.user-limit-factor</name>
            <value>1</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.user-limit-factor</name>
            <value>1</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.maximum-initialized-active-tasks</name>
            <value>200000</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.maximum-initialized-active-tasks</name>
            <value>200000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.maximum-initialized-active-tasks-per-user</name>
            <value>100000</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.maximum-initialized-active-tasks-per-user</name>
            <value>100000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.queue.queue1.init-accept-jobs-factor</name>
            <value>10</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.queue.queue2.init-accept-jobs-factor</name>
            <value>10</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-supports-priority</name>
            <value>false</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-minimum-user-limit-percent</name>
            <value>100</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-user-limit-factor</name>
            <value>1</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-maximum-active-tasks-per-queue</name>
            <value>200000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-maximum-active-tasks-per-user</name>
            <value>100000</value>
         </property>

         <property>
            <name>mapred.capacity-scheduler.default-init-accept-jobs-factor</name>
            <value>10</value>
         </property>

         <!-- Capacity scheduler Job Initialization configuration parameters -->
         <property>
            <name>mapred.capacity-scheduler.init-poll-interval</name>
            <value>5000</value>
         </property>
         <property>
            <name>mapred.capacity-scheduler.init-worker-threads</name>
            <value>5</value>
         </property>
    </configuration>

bin/start-all.sh 启动以下服务。

    17083 DataNode
    17557 TaskTracker
    17373 JobTracker
    16902 NameNode
    17279 SecondaryNameNode
    17703 Jps

我能够查看 Jobtracker 的 WEB UI

    http://localhost:50030/

Tasktracker 的 WEB 用户界面

    http://localhost:50060/

显示“无法连接”。但几秒钟后，jobtracker 和 tasktracker 就会关闭。终端上的 jps 命令只显示

    17083 DataNode
    16902 NameNode
    17279 SecondaryNameNode
    17703 Jps

可能是什么解决方案。

4

2 回答 2

2

both of your queues have a capacity of 100 , which makes the capacity scheduler to think there are couple of queues that each have a capacity of 100%. I suggest you change the setting to :

<?xml version="1.0"?>
<configuration>
     <property>
        <name>mapred.capacity-scheduler.maximum-system-jobs</name>
        <value>3000</value>   
     </property>
     <property>
        <name>mapred.capacity-scheduler.queue.queue1.capacity</name>
        <value>80</value> <!-- change here -->
     </property>
     <property>
        <name>mapred.capacity-scheduler.queue.queue2.capacity</name>
        <value>20</value> <!-- change here -->   
     </property>

     <property>
        <name>mapred.capacity-scheduler.queue.queue1.maximum-capacity</name>
        <value>-1</value>
     </property>
     <property>
        <name>mapred.capacity-scheduler.queue.queue2.maximum-capacity</name>
        <value>-1</value>
     </property>
于 2013-09-16T16:58:14.787 回答
0

所有队列的总和必须始终且仅为 100（即 100%），您可以有两个队列分别为 100% 和 0% - 这是有效的。

此外，我认为始终有一个“默认”队列是一个好习惯，至少有一些分配。我不知道如果您在没有默认值的情况下不指定队列名称，调度程序会做什么。

于 2014-04-16T22:37:32.367 回答