hadoop - Hadoop :how to set more than one column as a key and more than one column as a value in mapreduce classes in Hadoop

Question

i want to set more than one column as a key and more than one column as a value in mapreduce "key-value pairs" classes in Hadoop using java and the file reads from contains 20 column .thank you

Answer 1

将您想要作为键和值发出的所有列组合成一个分隔字符串并将它们作为文本发出。

假设您的输入如下所示：

不，姓名，年龄，国家

1，塔里克，25，印度

2，萨米，25，xyz

并且您想发出“No+Age”作为键和“Name+Country”作为值。

public static class MyMapper extends Mapper<LongWritable, Text, Text, Text> {

        String line = "";
        String val = "";
        String[] parts;
        String key = "";

        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

            line = value.toString();
            parts = line.split(",");
            key = parts[0] + "." + parts[2];
            val = parts[1] + "." + parts[3];
            context.write(new Text(key), new Text(value));
        }
}

Answer 2

您可以制作一个复合对象，implements WritableComparable<YourClassName>以简洁的形式将键存储在一起。请参阅此链接以获取一个很好的示例。

但是，鉴于您需要 20 个组件，我可能会建议只使用一个Text对象并在适合那么多组件时对其进行解析。我经常使用制表符分隔值并使用自定义 TSV 解析器解析它们，但仅Text.toString()使用合适的分隔符拆分它们char就足够了。