hadoop - Hadoop map reduce - access missing data

Question

Lets say I have a client script that pulls a large size of data from hadoop. What functionality in hadoop gives me advantage of looking at the retrieved data and ask for (point out) a missing part of data, to make a specific request just to read that missing part? Is this functionality a part of datanode map or reduce?

Thanks

Answer 1

没有直接的方法可以实现这一点。一旦您的脚本提取数据并将其写入 HDFS，它只是另一条数据。它与您的其他数据无关。您必须将其与要与之进行比较的数据一起阅读，并通过编写一些适合您需要的比较逻辑自己进行比较。

首先，您可以查看MultipleInputs。

PS：如果你能找到适合你的东西，请与我们分享。这将具有很大的价值。非常感谢。