0

How to write the contents of mapper into file. Is this fine.

public class MyMapper extends
        Mapper<Object, Text, Text, MatrixWritable > {
public void map(Object key, Text value, Context context)
            throws IOException, InterruptedException {

 Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(conf);

  Path inputfile = new Path("in/map");
  BufferedWriter getdatabuffer = new BufferedWriter(new OutputStreamWriter(fs.create(inputfile)));
  if(value.toString()!= null){
             getdatabuffer.write(value.toString());
         }
             getdatabuffer.close();

If my inputfile is splited whether the above code works fine?

In reducer i am combining all the mapper data.

EDIT

        Path inputfile = new Path("in/map");
             FSDataOutputStream out = fs.create(inputfile);
         if(value.toString()!= null){
            out.writeBytes(value.toString());
         }
            out.close();
4

1 回答 1

1

Mapper task run concurrently on multiple nodes in the Hadoop cluster. Your method of writing with normal Java Writer classes will not work just because you need to use HDFS API to write the data.

Instead in the map method use context.write() to write the data to HDFS files.

于 2013-11-13T06:04:24.517 回答