我一直在寻找解决方案,但一无所获。也许这里有人遇到了这样的问题并且有一些想法。
基本上,我使用 hadoop 尝试做一些 mapReduce 任务。一个主节点和两个从节点。配置似乎没问题,因为我可以从每个节点 ssh 到每个节点。我已经设置/etc/hosts
和 hadoop 的配置,例如:mapred-site.xml
, hdfs-site.xml
.
任务执行后一切似乎都很好。没有错误或警告,但是当我grep WARN
查看日志文件时,有些条目让我担心。
在tasktracker-master.log
我发现:
2013-06-03 14:14:47,525 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:15:00,680 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:17:44,352 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:17:59,182 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:23:49,419 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:23:55,156 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:27:40,619 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:27:46,382 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:34:15,498 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:34:22,103 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 14:36:38,551 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031434_0002 being deleted.
2013-06-03 14:37:17,724 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031434_0003 being deleted.
2013-06-03 14:59:42,395 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:59:48,158 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.`
同样的事情在slave1/slave2
:
tasktracker-slave1.log:
2013-06-03 14:59:34,439 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:59:45,954 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 15:01:29,380 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031459_0001 being deleted.
2013-06-03 15:03:14,848 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031459_0003 being deleted.
tasktracker-slave2.log:
2013-06-03 14:59:37,642 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-06-03 14:59:45,805 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2013-06-03 15:02:29,489 WARN org.apache.hadoop.mapred.TaskTracker: Unknown job job_201306031459_0002 being deleted.
任何人都看过这些警告并且知道它们是什么意思?
还有一件事,我怎样才能知道任务是否真的在所有假定的节点上执行master, slave1, slave2
?这些警告是否意味着slave1/slave2
不起作用?
我希望我已经清楚地描述了我的问题。谢谢你的帮助。