4

I've written a simple LzoWordCount the following to my Gateway/hadoop-env.sh:

HADOOP_CLASSPATH=/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/hadoop-lzo-cdh4-0.4.15-gplextras.jar
JAVA_LIBRARY_PATH=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.105/lib/hadoop/lib/native/

When I run the MR job, I get:

mapred.JobClient: Task Id : attempt_201307311800_0020_m_000002_2, Status : FAILED java.lang.RuntimeException: native-lzo library not available

Any ideas how to fix this issue? I did notice that 'hadoop classpath | grep native' returns nothing?

4

4 回答 4

3

问题原来是我们没有在数据节点上安装 lzop。我使用以下方法修复了它：

sudo apt-get install lzop
于 2013-08-07T19:09:51.640 回答
3

1.安装lzo

yum install lzo -y

2.安装hadoop-gpl-packaging

wget https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/hadoop-gpl-packing/hadoop-gpl-packaging-0.6.1-1.x86_64.rpm
rpm -i hadoop-gpl-packaging-0.6.1-1.x86_64.rpm

3.复制jar

cp /opt/hadoopgpl/lib/hadoop-lzo-*.jar $HADOOP_HOME/share/hadoop/common/

4 复制库

cp /opt/hadoopgpl/native/Linux-amd64-64/* $HADOOP_HOME/lib/native/
于 2016-06-15T14:00:09.423 回答
2

转到： https ://code.google.com/p/hadoop-gpl-packing/downloads/list

下载 jar 和 rpm 安装 rpm（当时：）hadoop-gpl-packaging-0.6.1-1.x86_64.rpm，然后

yum install install lzop
rpm -i hadoop-gpl-packaging-0.6.1-1.x86_64.rpm
sudo cp /opt/hadoopgpl/native/Linux-amd64-64/* /usr/lib/hadoop/lib/native/
sudo cp /opt/hadoopgpl/lib/*lzo* /usr/lib/hadoop/lib/

对上述命令感到厌烦，不要覆盖您不想覆盖的文件。因此，在运行此之前，请先备份目标文件夹。

于 2015-06-07T12:12:13.393 回答
0

所以这意味着你hadoop-lzo-***.jar不在你的HADOOP_CLASSPATH

然后，您可以将您的 jar 复制到您的hadoop/lib文件夹或添加路径hadoop/bin/hadoop-env.sh，您必须在所有节点上执行此操作

我希望它能解决它;-)

于 2013-08-07T07:35:27.170 回答