1

我在完成写入谷歌存储的 mapreduce 作业时遇到了一个奇怪的错误,以前有人见过吗?

工作“158354152558......”的最终结果是“成功”

……

File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduc/handlers.py", line 539, in _finalize_job
mapreduce_spec.mapper.output_writer_class().finalize_job(mapreduce_state)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/output_writers.py", line 571, in finalize_job
files.finalize(create_filename)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 568, in finalize
f.close(finalize=True)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 291, in close
self._make_rpc_call_with_retry('Close', request, response)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 427, in _make_rpc_call_with_retry
_make_call(method, request, response)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 252, in _make_call
_raise_app_error(e)
File "/base/data/home/apps/s~app/bqmapper.360899047207944804/libs/mapreduce/lib/files/file.py", line 186, in _raise_app_error
raise UnknownError()
UnknownError
4

1 回答 1

0

玩了之后,我发现云存储上的打开文件必须在不到 1 小时内完成,否则它会因这个可爱的 UnknownError 而失败。

我缓解了增加分片数量以加快映射速度的问题,并将 output_sharding 策略更改为“输入”,每个分片创建一个文件。

于 2012-08-10T13:38:01.650 回答