1

我创建了自己的 Mapreduce jar 文件，并使用 HDFS shell 命令成功地在 Cosmos 的旧 Hadoop 集群中进行了测试。下一步是在新集群中测试同一个 jar，所以我将它上传到 新集群的 HDFS中，到我的主文件夹 (user/my.username)。

当我尝试使用下面的 curl 帖子开始 Mapreduce 作业时，

curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/my.username/jobs" -d '{"jar":"dt.jar","class_name":"DistanceTest","lib_jars":"dt.jar","input":"input","output":"output"}' -H "Content-Type: application/json" -H "X-Auth-Token: xxxxxxxxxxxxxxxxxxx"

我得到：

{“成功”：“假”，“错误”：255}

我为 jar 尝试了不同的路径值，得到了相同的结果。我是否必须将我的 jar 上传到其他地方，或者我错过了一些必要的步骤？

4

1 回答 1

1

代码中有一个错误，已在 FIWARE Lab 的全局实例中修复。

我已经对此进行了测试：

$ curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/myuser/jobs" -d '{"jar":"mrjars/hadoop-mapreduce-examples.jar","class_name":"wordcount","lib_jars":"mrjars/hadoop-mapreduce-examples.jar","input":"testdir","output":"outputdir"}' -H "Content-Type: application/json" -H "X-Auth-Token: mytoken"
{"success":"true","job_id": "job_1460639183882_0016"}

请注意在这种情况下，mrjars/hadoop-mapreduce-examples.jar是 . 下的相对路径hdfs:///user/myuser。始终在此操作中使用相对路径。

$ curl -X GET "http://storage.cosmos.lab.fiware.org:14000/webhdfs/v1/user/myuser/mrjars?op=liststatus&user.name=myuser" -H "X-Auth-Token: mytoken"
{"FileStatuses":{"FileStatus":[{"pathSuffix":"hadoop-mapreduce-examples.jar","type":"FILE","length":270283,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464702215233,"modificationTime":1464702215479,"blockSize":134217728,"replication":3}]}}

结果：

$ curl -X GET "http://storage.cosmos.lab.fiware.org:14000/webhdfs/v1/user/myuser/outputdir?op=liststatus&user.name=myuser" -H "X-Auth-Token: mytoken"
{"FileStatuses":{"FileStatus":[{"pathSuffix":"_SUCCESS","type":"FILE","length":0,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464706333691,"modificationTime":1464706333706,"blockSize":134217728,"replication":3},{"pathSuffix":"part-r-00000","type":"FILE","length":128,"owner":"myuser","group":"myuser","permission":"644","accessTime":1464706333264,"modificationTime":1464706333460,"blockSize":134217728,"replication":3}]}}

我会尽快将修复上传到 GitHub 中的 Cosmos 存储库。

于 2016-05-31T15:03:43.520 回答