1

我扩展了 WritableComparable 并希望将其存储为映射器作为映射器值。

public class SenderRecieverPair implements WritableComparable<BinaryComparable> {

    Set<InternetAddress> pair = new TreeSet<InternetAddress>(new Comparator<InternetAddress>() {

        @Override
        public int compare(InternetAddress add1, InternetAddress add2) {
            return add1.getAddress().compareToIgnoreCase(add2.getAddress());
        }

    });

    public SenderRecieverPair() {
        super();
    }

    public SenderRecieverPair(InternetAddress add1, InternetAddress add2) {
        super();
        pair.add(add1);
        pair.add(add1);
    }


    public Set<InternetAddress> getPair() {
        return pair;
    }

    @Override
    public void write(DataOutput out) throws IOException {
        for (Iterator<InternetAddress> iterator = pair.iterator(); iterator.hasNext();) {
            InternetAddress email = (InternetAddress) iterator.next();
            String mailAddress = email.getAddress();
            if(mailAddress == null) {
                mailAddress = "";
            }
            byte[] address = mailAddress.getBytes("UTF-8");
            WritableUtils.writeVInt(out, address.length);
            out.write(address, 0, address.length);
            String displayName = email.getPersonal();
            if(displayName == null) {
                displayName = "";
            }
            byte[] display = displayName.getBytes("UTF-8");
            WritableUtils.writeVInt(out, display.length);
            out.write(display, 0, display.length);
        }

    }

    @Override
    public void readFields(DataInput in) throws IOException {
        for (int i = 0; i < 2; i++) {
            int length = WritableUtils.readVInt(in);
            byte[] container = new byte[length];
            in.readFully(container, 0, length);
            String mailAddress = new String(container, "UTF-8");
            length = WritableUtils.readVInt(in);
            container = new byte[length];
            in.readFully(container, 0, length);
            String displayName = new String(container, "UTF-8");
            InternetAddress address = new InternetAddress(mailAddress, displayName);
            pair.add(address);
        }

    }

    @Override
    public int compareTo(BinaryComparable o) {
        // TODO Auto-generated method stub
        return 0;
    }

}

但是我收到以下错误。请帮助我理解并纠正这一点

2013-07-29 06:49:26,753 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
    2013-07-29 06:49:26,891 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
    2013-07-29 06:49:27,004 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100
    2013-07-29 06:49:27,095 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720
    2013-07-29 06:49:27,095 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680
    2013-07-29 06:49:27,965 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
    2013-07-29 06:49:27,988 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
    2013-07-29 06:49:27,991 WARN org.apache.hadoop.mapred.Child: Error running child
    java.lang.RuntimeException: java.io.EOFException
        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:128)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:967)
        at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
        at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
        at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1253)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1154)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:581)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
    Caused by: java.io.EOFException
        at java.io.DataInputStream.readByte(DataInputStream.java:250)
        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:299)
        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:320)
        at com.edureka.sumit.enron.datatype.SenderRecieverPair.readFields(SenderRecieverPair.java:68)
        at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:122)
        ... 14 more
    2013-07-29 06:49:27,993 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

谢谢

4

3 回答 3

2

这是故意的吗？

public SenderRecieverPair(InternetAddress add1, InternetAddress add2) {
    super();
    pair.add(add1);
    pair.add(add1);
}

您添加 add1 两次，因此在写入循环中，您只从集合中获得 1 个元素，而不是两个

于 2013-07-30T06:22:07.440 回答
1

几个观察：

  • 如果您知道您使用的是一对，SenderRecieverPair那么我不会使用 Set - 将这两个对象显式存储为实例变量。该集合允许您无意中向集合中添加额外的值，并且您的 write 方法将根据集合大小写出 0、1、2 或更多（您的 readFields 方法在 for 循环中明确期望 2）。
  • 其次，如果你坚持使用集合，你应该知道 hadoop 在调用你的 map/reduce 任务之间重用了对象实例。这意味着每次调用 map / reduce 方法时，实际的对象引用都是相同的，只是底层内容将通过调用readFields. 在您的情况下，您不调用pair.clear() 作为 readFields 方法的第一部分，这意味着该集合将在调用之间继续增长。
  • 最后，在您的类中使用 Text 对象InternetAddress来存储电子邮件地址和显示名称，然后序列化要简单得多，因为您可以委托该对象，该对象可以委托给 Text 对象：

例如：

public class InternetAddress implements WritableComparable<InternetAddress> {
    protected Text emailAddress = new Text();
    protected Text displayName = new Text();

    // getter and setters for the above two fields
    // ..

    // compareTo method
    // ..

    @Override
    public void write(DataOutput out) throws IOException {
        emailAddress.write(out);
        displayName.write(out);
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        emailAddress.readFields(in);
        displayName.readFields(in);
    }
}

public class SenderRecieverPair implements WritableComparable<BinaryComparable> {
    protected Set<InternetAddress> pair = new TreeSet<InternetAddress>();

    // other methods omitted
    ..

    @Override
    public void write(DataOutput out) throws IOException {
        int safety = 0;
        for (Iterator<InternetAddress> iterator = pair.iterator(); iterator.hasNext();) {
          InternetAddress p1 = (InternetAddress) iterator.next();
          p1.write(out);

          p2 = (InternetAddress) iterator.next();
          p2.write(out);

          if (++safety == 3) {
              throw new IOException("More than two items in pair");
          }
        }
    }

    @Override
    public void readFields(DataInput in) throws IOException {
        pair.clear();

        // Note a more efficient method would be to re-use the objects already in the set (which is even easier to do if you don't use a set and just store the two objects as instance variables)

        InternetAddress a1 = new InternetAddress();
        a1.readFields(in);
        pair.add(a1);

        InternetAddress a2 = new InternetAddress();
        a2.readFields(in);
        pair.add(a2);
    }
}

哦，我没有看到方法 - 如果您使用（默认）并在映射器和化简器之间传递这些对象，hashCode则绝对应该覆盖这些方法。HashPartitioner

于 2013-07-29T11:11:38.397 回答
0

如果您尝试读取超出文件末尾的附加对象，则会引发java.io.EOFException异常。所以我认为因为你在 readFields 方法中循环，这可能是你的问题背后的原因。

于 2013-07-29T10:26:17.460 回答