hadoop - 在 FOREACH 中计算 SUM

Question

假设我有以下

DATA = foreach INPUT {
   //..
   generate group, count(name) as total;
}

我最终会得到一个键按名称分组的关系

('mike', 'someprop', 10)
('mike', 'otherprop', 3)
('doug', 'xprop', 5)
...

我想得到每个名字的前 10 名的总和：

ALIAS = group DATA by name;
RESULT = foreach ALIAS {
   SORTED = ORDER DATA by total desc;
   TOP10 = LIMIT SORTED 10;

   //doesn't work! can't have GROUP inside FOREACH
   AGG = group TOP10 ALL;
   TOPTOTAL = foreach AGG generate SUM(AGG.total);

   generate group, TOPTOTAL;
}

如何计算 a 中关系的值 (SUM,COUNT, ETC )foreach？目前没有办法GROUP ALL在 foreach 中应用 a 。

Answer 1

SUM只是一个以包为参数的函数，您可以通过投影来创建此包TOP10：

ALIAS = group DATA by name;
RESULT = foreach ALIAS {
   SORTED = ORDER DATA by total desc;
   TOP10 = LIMIT SORTED 10;
   generate group, SUM(TOP10.total);
}