3

I've developed a MapReduce with reduce-side join on Hadoop 0.20.2 version. All java classes compiles without any error. My program works as it has to. But this version of hadoop is on test virtual machine. On my real cluster I have another version of hadoop (hadoop 2.00-cdh4.1.2). When I want to compile java-classes it doesn't work. I get 100 errors like following:

JoinMapper.java:8: package org.apache.hadoop.filecache does not exist
import org.apache.hadoop.filecache.DistributedCache;
                                  ^
JoinMapper.java:9: package org.apache.hadoop.fs does not exist
import org.apache.hadoop.fs.Path;
                           ^
JoinMapper.java:10: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
JoinMapper.java:11: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
JoinMapper.java:12: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileInputFormat;
                               ^
JoinMapper.java:13: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.FileOutputFormat;
                               ^
JoinMapper.java:14: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.InputFormat;
                               ^
JoinMapper.java:15: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobClient;
                               ^
JoinMapper.java:16: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.JobConf;
                               ^
JoinMapper.java:17: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.MapReduceBase;
                               ^
JoinMapper.java:18: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Mapper;
                               ^
JoinMapper.java:19: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.OutputCollector;
                               ^
JoinMapper.java:20: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.Reporter;
                               ^
JoinMapper.java:21: package org.apache.hadoop.mapred does not exist
import org.apache.hadoop.mapred.TextOutputFormat;
                               ^
JoinMapper.java:22: package org.apache.hadoop.mapreduce.lib.input does not exist
import org.apache.hadoop.mapreduce.lib.input.MultipleInputs;
                                            ^
JoinMapper.java:23: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.Tool;
                             ^
JoinMapper.java:24: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.ToolRunner;
                             ^
JoinMapper.java:26: cannot find symbol
symbol: class MapReduceBase
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Mapper
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                         ^
JoinMapper.java:26: cannot find symbol
symbol: class LongWritable
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                              ^
TextPair.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;
^
TextPair.java:4: cannot find symbol
symbol: class WritableComparable
public class TextPair implements WritableComparable<TextPair> {
                                 ^
TextPair.java:4: interface expected here
public class TextPair implements WritableComparable<TextPair> {
                                                   ^
JoinMapper.java:26: cannot find symbol
symbol: class Text
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                                                              ^
JoinMapper.java:26: interface expected here
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> {
                                                               ^
NcdcRecordParser.java:2: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.*;

I understand this version doesn't have packages that I need. But I don't know how to solve my problem. Can anyone help me?

4

3 回答 3

2

据我所知，这些 java 包应该仍然存在于最新版本的 Hadoop 中。

我认为您的问题是 hadoop jar 不在您的类路径中。当我尝试在没有它们的情况下编译我的 map reduce 项目时，我遇到了类似的错误。

如果您使用的是 javac，请尝试以下操作（这使用我的 Ubuntu CDH4 开发机器上的路径位置）：

javac -classpath /usr/lib/hadoop/*:/usr/lib/hadoop-0.20-mapreduce/*:/usr/lib/hadoop-hdfs/* JoinMapper.java TextPair.java NcdcRecordParser.java
于 2012-12-26T17:59:23.020 回答
1

Apache Hadoop 在 0.20 和 0.23（也称为 2.0.0）之间对其内部包进行了重大重新命名。这包括重命名org.apache.hadoop.mapred为org.apache.hadoop.mapreduce.

Hadoop 的 Cloudera 发行版具有单独的 Maven 工件，您可以依赖这些工件来使用旧版本进行构建。工件的名称列在CDH4 Maven 页面上；你想要带有 version 的工件2.0.0-mr1-cdh4.1.2。

另请参阅使用 mapred 或 mapreduce 包创建 Hadoop 作业是否更好？有关是否要移植到较新 API 的一些详细信息。

于 2012-12-26T18:48:46.537 回答
0

为 Maven 尝试 param -Dhadoop.profile=23。另外，请参阅https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098596#comment-13098596

于 2012-12-26T17:46:01.033 回答