我想在 MapReduce 中使用 LZO 压缩，但在运行 MapReduce 作业时出现错误。我正在使用带有 Java 程序的 Ubuntu。我只是想在我的本地机器上运行它。我最初的错误是
ERROR lzo.GPLNativeCodeLoader: Could not load native gpl library
下线
ERROR lzo.LzoCodec: Cannot load native-lzo without native-hadoop
接着
java.lang.RuntimeException: native-lzo library not available
我遵循了一些关于如何下载和配置文件以用于 LZO 压缩的在线和文本说明。在这里你可以在文件夹中看到我的 hadoop-lzo jarlib文件
我已经改变了我的配置。这是我的 core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>
</property>
<property>
<name>io.compression.codec.lzo.class</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
<name>mapred.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzopCodec</value>
</property>
</configuration>
和我的 mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapred.map.output.compression.codec</name>
<value>com.hadoop.compression.lzo.LzoCodec</value>
</property>
<property>
<name>mapred.child.env</name>
<value>JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:/home/matthew/hadoop/lib/native/lib/lib</value>
</property>
</configuration>
我还用这些行在同一个 conf 文件夹中修改了我的 hadoop-env.sh
export HADOOP_CLASSPATH=/home/hadoop/lib/hadoop-lzo-0.4.13.jar
export JAVA_LIBRARY=/home/hadoop/lib/native/lib/lib
如果你对什么感兴趣/home/hadoop/lib/native/lib/lib
对于它的价值，这是我的 Driver 类，它完成了所有的压缩
//import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
//import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
//import org.apache.hadoop.io.compress.CompressionCodec;
import com.hadoop.compression.lzo.LzopCodec;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
//import org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class FuzzyJoinDriver extends Configured implements Tool{
public static void main(String[] args) throws Exception {
int exitCode = ToolRunner.run(new Configuration(), new FuzzyJoinDriver(),args);
System.exit(exitCode);
}
@Override
public int run(String[] args) throws Exception {
if (args.length != 2) {
System.err.println("Usage: FuzzyJoinDriver <input path> <output path>");
System.exit(-1);
}
Configuration conf = new Configuration();
//Used to compress map output
//conf.setBoolean("mapred.compres.map.output", true);
//conf.setClass("mapred.map.output.compression.code", GzipCodec.class, CompressionCodec.class);
Job job = new Job(conf);
job.setJarByClass(FuzzyJoinDriver.class);
job.setJobName("Fuzzy Join");
//Distributed Cache
//DistributedCache.addCacheFile(new URI("/cache/ReducerCount.txt"), job.getConfiguration());
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.setMapperClass(FuzzyJoinMapper.class);
job.setReducerClass(FuzzyJoinReducer.class);
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(RelationTuple.class);
job.setPartitionerClass(JoinKeyPartitioner.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(Text.class);
//Used to compress output from reducer
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, LzopCodec.class);
//SequenceFileOutputFormat.setOutputCompressionType(job, org.apache.hadoop.io.SequenceFile.CompressionType.BLOCK);
return(job.waitForCompletion(true) ? 0 : 1);
}
}
它编译没有错误。
我担心我对配置和其他步骤缺乏了解会导致我走上错误的道路，也许对于那些比我有更好理解的人来说，我错过了一些简单的东西。谢谢你走到这一步。我知道这是一个很长的帖子。