2

在 Mac OS X 上,当我使用压缩编解码器是 hadoop 时,我收到以下错误:

java.lang.IllegalArgumentException: SequenceFile doesn't work with GzipCodec without native-hadoop code!
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:386)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat.getRecordWriter(SequenceFileOutputFormat.java:61)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.(ReduceTask.java:569)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:638)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:260)

但是我已经配置了本机库。(我已经在本地构建了 hadoop 并将本机库复制到 hadoop 目录。

hadoop checknative -a 

16/06/06 13:16:54 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library system-native, will use pure-Java version
16/06/06 13:16:54 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libhadoop.dylib
zlib:    true /usr/lib/libz.1.dylib
snappy:  true /opt/hadoop-2.5.0-cdh5.3.3/lib/native/libsnappy.1.dylib
lz4:     true revision:99
bzip2:   false 
openssl: false org.apache.hadoop.crypto.OpensslCipher.initIDs()V
16/06/06 13:16:54 INFO util.ExitUtil: Exiting with status 1

所以看起来本机库还可以。

PS:当我使用 B2ZIPCodec 时,一切正常。

PSS:也许我的 .profile 有助于了解发生了什么?

export JAVA_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export LD_LIBRARY_PATH=/opt/hadoop-2.5.0-cdh5.3.3/lib/native
export HADOOP_HOME=/opt/hadoop-2.5.0-cdh5.3.3
export M2_HOME="/opt/maven/"
export MAVEN_OPTS="-Xms1024m -Xmx3000m -XX:MaxPermSize=228m -Dmaven.surefire.debug=-Xmx3000m"
export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.7.0_72.jdk/Contents/Home"
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:${HCAT_HOME}/bin:${M2_HOME}/bin:${PATH}

更新此配置在半年前运行良好。一些软件更新后,它坏了。

4

0 回答 0