我正在尝试将一个 HDFS 数据复制到另一个 HDFS 位置。
我可以使用“distcp”命令实现相同的目的
hadoop distcp hdfs://mySrcip:8020/copyDev/* hdfs://myDestip:8020/copyTest
但我想尝试使用 Java Api。经过长时间的搜索，找到了一段代码并执行了。但它没有将我的 src 文件复制到目的地。
public class TouchFile {
/**
* @param args
* @throws Exception
*/
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
//create configuration object
Configuration config = new Configuration();
config.set("fs.defaultFS", "hdfs://mySrcip:8020/");
config.set("hadoop.job.ugi", "hdfs");
/*
* Distcp
*/
String sourceNameNode = "hdfs://mySrcip:8020/copyDev";
String destNameNode = "hdfs://myDestip:8020/copyTest";
String fileList = "myfile.txt";
distFileCopy(config,sourceNameNode,destNameNode,fileList);
}
/**
* Copies files from one cloud to another using Hadoop's distributed copy features. Uses
* input to build DISTCP configuration settings.
*
* param config Hadoop configuration
* param sourceNameNode full HDFS path to parent source directory
* param destNameNode full HDFS path to parent destination directory
* param fileList Comma separated string of file names in sourceNameNode to be copied to destNameNode
* returns Elapsed time in milliseconds to copy files
*/
public static long distFileCopy( Configuration config, String sourceNameNode, String destNameNode, String fileList ) throws Exception {
System.out.println("In dist copy");
StringTokenizer tokenizer = new StringTokenizer(fileList,",");
ArrayList<String> list = new ArrayList<>();
while ( tokenizer.hasMoreTokens() ){
String file = sourceNameNode + "/" + tokenizer.nextToken();
list.add( file );
}
String[] args = new String[list.size() + 1];
int count = 0;
for ( String filename : list ){
args[count++] = filename;
}
args[count] = destNameNode;
System.out.println("args------>"+Arrays.toString(args));
long st = System.currentTimeMillis();
DistCp distCp=new DistCp(config,null);
distCp.run(args);
return System.currentTimeMillis() - st;
}
}
我做错什么了吗。请建议