exception - 清理失败的地图

Question

我的映射器会将一些数据写入本地磁盘并在映射器完成时将其清理。但是，如果发生错误（发生异常），则不会调用 cleanup() 方法。我可以在我的映射器中捕获异常，但我无法处理在我的映射器中未调用的异常（例如：作业跟踪器故障转移到备用节点）。

当映射器失败时，有什么方法可以清理吗？

Answer 1

You can override the run method of mapper to include a try / catch around the iteration of input keys from the context and ensure that cleanup is called:

@Override
public void run() {
  setup(context);

  try {
    while (context.nextKeyValue()) {
      map(context.getCurrentKey(), context.getCurrentValue(), context);
    }
  } finally {
    cleanup(context);
  }
}

You'll need to make sure that your cleanup method doesn't have any logic in it to try and output records, or set a flag in your mapper to denote that an error occurred.

This may not protect against all types of task failure (JVM crash for example), for which i don't think you have any other method, other than to maybe run a job after the original job whose role is to ensure the resources used are properly cleaned up.

Answer 2

使用作业类，如果作业完成，您绝对可以删除一些文件夹，即使目录在本地文件系统中，使用FileSystem类

更多关于 hadoop 中的文件系统