2

我正在尝试在 MapReduce 上实现 K-means。我已将初始质心文件上传到分布式缓存

在司机班

DistributedCache.addCacheFile(new URI("GlobalCentroidFile"),conf);

在我的映射器类

Path[] localFiles = DistributedCache.getLocalCacheFiles(job);
File file = new File(localFiles[0].getName());
System.out.println(" File read is "+localFiles[0].getName());
BufferedReader bufferedReader = new BufferedReader(new FileReader(file));       
System.out.println("Goin in while loop");
    ....
   // some code omitted
    ....
    } catch (IOException e) {
        System.out.println("\n"+e);
    }

$HADOOP_HOME/logs/ 中的输出是

File read is localhostGlobalCentroidFile
java.io.FileNotFoundException: localhostGlobalCentroidFile (No such file or directory)

当我做

ganesh@ganesh-PC:~/Desktop$ hadoop fs -ls

Warning: $HADOOP_HOME is deprecated.


Found 4 items

-rw-r--r--   1 ganesh supergroup         26 2013-04-02 16:15 /user/ganesh
/GlobalCentroidFile

-rw-r--r--   1 ganesh supergroup         18 2013-04-02 16:16 /user/ganesh
/GlobalCentroidFile1

-rw-r--r--   1 ganesh supergroup        672 2013-04-02 16:15 /user/ganesh/input

drwxr-xr-x   - ganesh supergroup          0 2013-04-02 16:16 /user/ganesh/output

ganesh@ganesh-PC:~/Desktop$ hadoop fs -cat GlobalCentroidFile

Warning: $HADOOP_HOME is deprecated.


2.3    4.3

34.4    33.3

45.5    34

可能是什么问题？

4

1 回答 1

1

以下应该是您的代码。您正在尝试使用不起作用的普通 java 文件读取结构读取 HDFS 中可用的文件。

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
int bytesRead = 0;
byte[] buffer = new byte[2048];
Path inFile = new Path(argv[0]);
if (fs.exists(inFile)) {
    FSDataInputStream in = fs.open(inFile);
    while ((bytesRead = in.read(buffer)) > 0) {
        // Read Logic
    }
    in.close();
}
于 2013-04-02T12:00:24.367 回答