0

我正在使用 Mrjob 在 Hadoop 中运行 python 代码。我在单节点集群上使用带有虚拟机的 CDH 包。当我在本地测试代码时，我的 mrjob 运行正确，但是当我在 Hadoop 集群上运行时，它会抛出错误：

No module named mrjob

当我在 python 之前删除“sudo”命令时，我收到以下消息。

no configs found; falling back on auto-configuration
no configs found; falling back on auto-configuration
creating tmp directory /tmp/main_mrjob.cloudera.20131022.180113.820659
writing wrapper script to /tmp/main_mrjob.cloudera.20131022.180113.820659/setup-wrapper.sh
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName
STDERR:     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR:     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.util.PlatformName.  Program will exit.
STDERR: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FsShell
STDERR: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FsShell
STDERR:     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
STDERR:     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
STDERR:     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
STDERR: Could not find the main class: org.apache.hadoop.fs.FsShell.  Program will exit.
Traceback (most recent call last):
  File "main_mrjob.py", line 17, in <module>
    MRWordFrequencyCount.run()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/job.py", line 500, in run
    mr_job.execute()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/job.py", line 518, in execute
    super(MRJob, self).execute()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/launch.py", line 146, in execute
    self.run_job()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/launch.py", line 207, in run_job
    runner.run()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/runner.py", line 458, in run
    self._run()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/hadoop.py", line 236, in _run
    self._upload_local_files_to_hdfs()
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/hadoop.py", line 263, in _upload_local_files_to_hdfs
    self._mkdir_on_hdfs(self._upload_mgr.prefix)
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/hadoop.py", line 271, in _mkdir_on_hdfs
    self.invoke_hadoop(['fs', '-mkdir', path])
  File "/home/cloudera/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/mrjob/fs/hadoop.py", line 104, in invoke_hadoop
    raise CalledProcessError(proc.returncode, args)
subprocess.CalledProcessError: Command '['/usr/lib/hadoop-0.20-mapreduce/bin/hadoop', 'fs', '-mkdir', 'hdfs:///user/cloudera/tmp/mrjob/main_mrjob.cloudera.20131022.180113.820659/files/']' returned non-zero exit status 1

在没有 sudo 的情况下，它似乎无法在 hdfs 上“mkdir”，但使用 sudo 它无法找到 mrjob。我真的很困惑......

非常感谢！！

杨

4

1 回答 1

0

在使用 cloudera quickstart vm 时，我遇到了同样的问题。

解决方案是：

  1. 将 HADOOP_HOME 设置为“/usr/lib/hadoop”：

    export HADOOP_HOME=/usr/lib/hadoop

  2. 创建指向 hadoop-streaming.jar 的符号链接：

    sudo ln -s /usr/lib/hadoop-mapreduce/hadoop-streaming.jar /usr/lib/hadoop
于 2013-11-06T13:58:51.327 回答