1

在执行 hadoop jar 命令运行默认字数统计程序后，我开始关注 stderr。 

java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService and  Could not find the
main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster

我相信我不必明确设置 hadoop 类路径。因为我已经设置了环境。变量如下：或者我在这里遗漏了什么？

%HADOOP_HOME% =c:\hadoop
HADOOP_COMMON_HOME =%HADOOP_HOME%
HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop
HADOOP_HDFS_HOME =%HADOOP_HOME%
HADOOP_MAPRED_HOME =%HADOOP_HOME%

我正在执行的命令是

hadoop jar c:/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output

The output @ sdk command prompt is 

13/12/20 16:26:00 INFO mapreduce.Job: Job job_1387536911324_0001 failed with sta
te FAILED due to: Application application_1387536911324_0001 failed 2 times due
to AM Container for appattempt_1387536911324_0001_000002 exited with  exitCode:
1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
        at org.apache.hadoop.util.Shell.run(Shell.java:379)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:
589)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.la
unchContainer(DefaultContainerExecutor.java:195)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.C
ontainerLaunch.call(ContainerLaunch.java:283)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.C
ontainerLaunch.call(ContainerLaunch.java:79)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:918)
        at java.lang.Thread.run(Thread.java:662)

        1 file(s) moved.

ANd the stderr file is

java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    ... 12 more
Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster.  Program will exit.
Exception in thread "main"
4

3 回答 3

1

我正在运行 Linux，我遇到了完全相同的问题。通过将其添加到 yarn-site.xml 来解决：

        <property>
                <description>Classpath for typical applications.</description>
                <name>yarn.application.classpath</name>
                <value>
                        $HADOOP_CONF_DIR,
                        $HADOOP_COMMON_HOME/share/hadoop/common/*,
                        $HADOOP_COMMON_HOME/share/hadoop/common/lib/*,
                        $HADOOP_HDFS_HOME/share/hadoop/hdfs/*,
                        $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,
                        $HADOOP_YARN_HOME/share/hadoop/yarn/*,
                        $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
                </value>
        </property>
于 2014-01-07T21:45:48.203 回答
0

问题在于 yarn.application.classpath 属性。默认值使用像 $HADOOP_HOME 这样的 Linux 风格的环境变量引用，而不是像 %HADOOP_HOME% 这样的 Windows 风格的引用，因此您必须通过添加类似的属性来覆盖 %HADOOP_HOME%\etc\hadoop\yarn-site.xml 中的默认值这个：

<属性>
    <description>用于 YARN 应用程序的 CLASSPATH。以逗号分隔的 CLASSPATH 条目列表</description>
    <name>yarn.application.classpath</name>
    <值>
        %HADOOP_HOME%\etc\hadoop,
        %HADOOP_HOME%\share\hadoop\common\*,
        %HADOOP_HOME%\share\hadoop\common\lib\*,
        %HADOOP_HOME%\share\hadoop\hdfs\*,
        %HADOOP_HOME%\share\hadoop\hdfs\lib\*,
        %HADOOP_HOME%\share\hadoop\mapreduce\*,
        %HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
        %HADOOP_HOME%\share\hadoop\yarn\*,
        %HADOOP_HOME%\share\hadoop\yarn\lib\*
    </值>
</属性>
于 2014-01-04T05:13:55.927 回答
0

我发现 Hadoop 在迭代 YarnConfiguration 属性时无法解析 $HADOOP_HOME 和 $YARN_HOME 环境变量。在您的 Yarn 客户端中运行以下命令将打印未解析的配置，例如，

$HADOOP_HOME/ , $HADOOP_HOME/lib/

  YarnConfiguration conf = new YarnConfiguration()
  for (String c : conf.getStrings(
                YarnConfiguration.YARN_APPLICATION_CLASSPATH,
                YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)) {
      System.out.println(c);
  }

因此，如果您提供 yarn.application.classpath 属性的完整路径，NoClassDefFoundError 问题就会得到解决。

  <property>
    <description>CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries</description>
    <name>yarn.application.classpath</name>
    <value>
        /etc/hadoop/conf,
        /usr/lib/hadoop/*,
        /usr/lib/hadoop/lib/*,
        /usr/lib/hadoop-hdfs/*,
        /usr/lib/hadoop-hdfs/lib/*,
        /usr/lib/hadoop-mapreduce/*,
        /usr/lib/hadoop-mapreduce/lib/*,
        /usr/lib/hadoop-yarn/*,
        /usr/lib/hadoop-yarn/lib/*
    </value>
  </property>
于 2014-05-05T04:56:43.770 回答