2

im just a beginner in hadoop.im getting null pointer exception while performing seconday sort

This is my mapper class 

public void map(LongWritable key, Text value,
            OutputCollector<Text, Employee> outputCollector, Reporter reporter)
            throws IOException {
        // TODO Auto-generated method stub
        String employeeId = value.toString().split(",")[0];
        String employeeName= value.toString().split(",")[1];
        String employeeDept= value.toString().split(",")[2];
        String employeejoinDate= value.toString().split(",")[3];
        String employeSalary= value.toString().split(",")[4];
        //System.out.println(employeSalary);
        Employee employee=new Employee(Integer.parseInt(employeeId),employeeName,employeeDept,employeejoinDate,Integer.parseInt(employeSalary));
        outputCollector.collect(new Text(employeeName),employee);

    }

This is my reducer 

public void reduce(Text arg0, Iterator<Employee> arg1,
            OutputCollector<NullWritable,IntWritable> arg2, Reporter arg3)
            throws IOException {
        // TODO Auto-generated method stub

        System.out.println("inside reducer");
        while(arg1.hasNext()){
            arg2.collect(NullWritable.get(),new IntWritable(arg1.next().getEmployeeSalary()));
        }

this is my employee class 

public class Employee  implements WritableComparable<Employee>{

    private int employeeId;
    private String employeeName;
    private String employeeDept;
    private String employeeJoinDt;
    private int employeeSalary;
    public Employee(int employeeId,String employeeName,String employeeDept,String employeeJoinDt,int employeeSalary){
        this.employeeId=employeeId;
        this.employeeName=employeeName;
        this.employeeDept=employeeDept;
        this.employeeJoinDt=employeeJoinDt;
        this.employeeSalary=employeeSalary;
    }
    public int getEmployeeId() {
        return employeeId;
    }

    public void setEmployeeId(int employeeId) {
        this.employeeId = employeeId;
    }

    public String getEmployeeName() {
        return employeeName;
    }

    public void setEmployeeName(String employeeName) {
        this.employeeName = employeeName;
    }

    public String getEmployeeDept() {
        return employeeDept;
    }

    public void setEmployeeDept(String employeeDept) {
        this.employeeDept = employeeDept;
    }

    public String getEmployeeJoinDt() {
        return employeeJoinDt;
    }

    public void setEmployeeJoinDt(String employeeJoinDt) {
        this.employeeJoinDt = employeeJoinDt;
    }

    public int getEmployeeSalary() {
        return employeeSalary;
    }

    public void setEmployeeSalary(int employeeSalary) {
        this.employeeSalary = employeeSalary;
    }


    @Override
    public void readFields(DataInput input) throws IOException {
        // TODO Auto-generated method stubt
        this.employeeId=input.readInt();
        this.employeeName=input.readUTF();
        this.employeeDept=input.readUTF();
        this.employeeJoinDt=input.readUTF();
        this.employeeSalary=input.readInt();

    }

    @Override
    public void write(DataOutput output) throws IOException {
        // TODO Auto-generated method stub
        output.writeInt(this.employeeId);
        output.writeUTF(this.employeeName);
        output.writeUTF(this.employeeDept);
        output.writeUTF(this.employeeJoinDt);
        output.writeInt(this.employeeSalary);   
    }
    public int compareTo(Employee employee) {
        // TODO Auto-generated method stub
        if(this.employeeSalary>employee.getEmployeeSalary())
            return 1;
        else if(this.employeeSalary<employee.getEmployeeSalary())
            return -1;
        else 
            return 0;
    }
}

this is my sort comparator class

public class SecondarySortComparator extends WritableComparator {


    public SecondarySortComparator(){

        super(Employee.class);
        System.out.println("sort");
    }
    @Override
    public int compare(WritableComparable a, WritableComparable b) {
        // TODO Auto-generated method stub
        Employee employee1 = (Employee)a;
        Employee employee2 = (Employee)b;
        int i = employee1.getEmployeeSalary()>employee2.getEmployeeSalary()?1:-1;
        return i;
    }

this is my groupo comparator class 

public class SecondarySortGroupingComparator extends WritableComparator{

public SecondarySortGroupingComparator(){

    super(Employee.class,true);
    System.out.println("group");

}
@Override
    public int compare(WritableComparable a, WritableComparable b) {
        // TODO Auto-generated method stub
        Employee employee1 = (Employee)a;
        Employee employee2 = (Employee)b;
        return employee1.getEmployeeName().compareTo(employee2.getEmployeeName());
    }

}

this is the error iam getting 

13/09/01 19:13:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/09/01 19:13:47 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/01 19:13:47 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
13/09/01 19:13:47 INFO mapred.FileInputFormat: Total input paths to process : 1
13/09/01 19:13:47 INFO mapred.JobClient: Running job: job_local_0001
13/09/01 19:13:47 INFO util.ProcessTree: setsid exited with exit code 0
13/09/01 19:13:47 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b3f8f6
13/09/01 19:13:47 INFO mapred.MapTask: numReduceTasks: 1
13/09/01 19:13:47 INFO mapred.MapTask: io.sort.mb = 100
13/09/01 19:13:48 INFO mapred.JobClient:  map 0% reduce 0%
13/09/01 19:13:48 INFO mapred.MapTask: data buffer = 79691776/99614720
sort13/09/01 19:13:48 INFO mapred.MapTask: record buffer = 262144/327680

1
1
1
1
13/09/01 19:13:49 INFO mapred.MapTask: Starting flush of map output
13/09/01 19:13:49 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
    at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
    at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:70)
    at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1298)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
13/09/01 19:13:49 INFO mapred.JobClient: Job complete: job_local_0001
13/09/01 19:13:49 INFO mapred.JobClient: Counters: 0
13/09/01 19:13:49 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
    at secondarysort.JobRunner.main(JobRunner.java:31)

any suggestions on how to solve this problem?

thanks in advance

4

1 回答 1

2

这条线似乎导致了问题。

context.write(new Text(employeeName),  employee);

您将员工对象（Employee 类型）作为值而不是键发出，并且 SecondarySortComparator 和 SecondarySortGroupingComparator 都在您的键而不是值上工作。

因此，主要问题是您将 Text 作为键传递，这导致了问题。您可能会考虑将员工对象作为 Key 而不是 Text 传递，以便两个 Comparator 实际工作。

您可能还想在 Employee 类中放置一个默认构造函数 -

public Employee() { }
于 2013-09-01T22:07:27.050 回答