1

有谁知道使用 Amazon 的 S3Distcp 工具和在 EMR 上运行的 MapR 是否存在问题?我正在尝试使用它,但在 /mnt/var/log/hadoop/steps 中不断出现以下异常:

Exception in thread "main" java.lang.RuntimeException: Unable to delete directory hdfs:/tmp/e9333a37-f400-4982-9687-326e33d9b37d/files
at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.deleteRecursive(S3DistCp.java:606)
at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:464)
at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.run(S3DistCp.java:216)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.amazon.external.elasticmapreduce.s3distcp.Main.main(Main.java:12)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs:/tmp/e9333a37-f400-4982-9687-326e33d9b37d/files
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:85)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1416)
at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:69)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:1450)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1432)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:232)
at com.amazon.external.elasticmapreduce.s3distcp.S3DistCp.deleteRecursive(S3DistCp.java:603)

我用来提交作业步骤的命令行是:

elastic-mapreduce --jobflow $JOB_ID --jar s3://us-east-1.elasticmapreduce/libs/s3distcp/1.latest/s3distcp.jar \
--args '--src,s3n://PVData/raw, \
--dest,/PVData/raw'

对于 --dest 参数,我也尝试了 maprfs:///PVData/raw 和 hdfs:///PVData/raw ,但它们也不起作用。

4

1 回答 1

2

我在 MapR 论坛 (http://bit.ly/S7gzcv) 上得到了这个问题的答案。问题是我需要使用 s3distcp 的 --tmpDir 参数将临时目录指定为 maprfs:///tmp

于 2012-12-12T01:43:14.683 回答