58

我想在 HDFS 中创建一个文件并在其中写入数据。我使用了这段代码：

Configuration config = new Configuration();     
FileSystem fs = FileSystem.get(config); 
Path filenamePath = new Path("input.txt");  
try {
    if (fs.exists(filenamePath)) {
        fs.delete(filenamePath, true);
    }

    FSDataOutputStream fin = fs.create(filenamePath);
    fin.writeUTF("hello");
    fin.close();
}

它会创建文件，但不会在其中写入任何内容。我搜索了很多，但没有找到任何东西。我的问题是什么？我需要任何权限才能在 HDFS 中写入吗？

谢谢。

4

4 回答 4

74

@Tariq 的替代方法，您可以在获取文件系统时传递 URI

import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.conf.Configuration
import java.net.URI
import org.apache.hadoop.fs.Path
import org.apache.hadoop.util.Progressable
import java.io.BufferedWriter
import java.io.OutputStreamWriter

Configuration configuration = new Configuration();
FileSystem hdfs = FileSystem.get( new URI( "hdfs://localhost:54310" ), configuration );
Path file = new Path("hdfs://localhost:54310/s2013/batch/table.html");
if ( hdfs.exists( file )) { hdfs.delete( file, true ); } 
OutputStream os = hdfs.create( file,
    new Progressable() {
        public void progress() {
            out.println("...bytes written: [ "+bytesWritten+" ]");
        } });
BufferedWriter br = new BufferedWriter( new OutputStreamWriter( os, "UTF-8" ) );
br.write("Hello World");
br.close();
hdfs.close();
于 2013-07-23T19:41:40.070 回答
24

将环境变量定义HADOOP_CONF_DIR到 Hadoop 配置文件夹或在代码中添加以下 2 行：

config.addResource(new Path("/HADOOP_HOME/conf/core-site.xml"));
config.addResource(new Path("/HADOOP_HOME/conf/hdfs-site.xml"));

如果不添加，客户端将尝试写入本地 FS，从而导致权限被拒绝异常。

于 2013-04-14T22:50:14.877 回答
0

这应该可以解决问题

import org.apache.commons.io.IOUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

import java.io.*;
import java.nio.charset.StandardCharsets;


public static void writeFileToHDFS() throws IOException {
        Configuration configuration = new Configuration();
        configuration.set("fs.defaultFS", "hdfs://localhost:9000");
        configuration.addResource(new Path("/HADOOP_HOME/conf/core-site.xml"));
        configuration.addResource(new Path("/HADOOP_HOME/conf/hdfs-site.xml"));
        FileSystem fileSystem = FileSystem.get(configuration);
        //Create a path
        String fileName = "input.txt";
        Path hdfsWritePath = new Path("/user/yourdesiredpath/" + fileName);
        FSDataOutputStream fsDataOutputStream = fileSystem.create(hdfsWritePath,true);

        BufferedWriter bufferedWriter = new BufferedWriter(new OutputStreamWriter(fsDataOutputStream,StandardCharsets.UTF_8));
        bufferedWriter.write("Java API to write data in HDFS");
        bufferedWriter.close();
        fileSystem.close();
    }
于 2021-02-09T09:59:40.597 回答
-2

请尝试以下方法。

FileSystem fs = path.getFileSystem(conf);
SequenceFile.Writer inputWriter = new SequenceFile.Writer(fs, conf, path, LongWritable.class, MyWritable.class);
inputWriter.append(new LongWritable(uniqueId++), new MyWritable(data));
inputWriter.close();
于 2013-04-14T16:33:45.663 回答