0

我的 hadoop 作业的映射器任务总是挂在那里,即使它的进度是 100%,它的状态仍在运行。而且我看到在映射器任务日志中完成了很多合并操作。我已经将参数 io.sort.mb 和 io.sort.factor 更改为更大的数字。但这并没有解决我的问题。我有大约 2000 个 mapper 任务,mapper 任务的总输出大小为 200G,这意味着每个 mapper 任务的输出大小约为 100M,这对我来说应该不是很大的大小。所以我不明白为什么mapper任务做这么多合并操作

关于这个问题的任何想法?谢谢

这是我的日志(顺便说一句,下面的大小数字是什么意思?):

[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44335 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45076 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44057 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45369 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45388 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45816 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44765 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45761 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45379 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44456 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45581 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45939 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45418 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 45241 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44809 bytes
[INFO] Merger - Merging 2 sorted segments
[INFO] Merger - Down to the last merge-pass, with 2 segments left of total size: 44736 bytes
[INFO] Merger - Merging 2 sorted segments
4

0 回答 0