apache-pig - 在 pig 中访问类似数组的元素

Question

我有以下形式的数据：id,val1,val2

例子

1,0.2,0.1
1,0.1,0.7
1,0.2,0.3
2,0.7,0.9
2,0.2,0.3
2,0.4,0.5

所以首先我想按 val1 以降序对每个 id 进行排序..so somethng like

1,0.2,0.1
1,0.2,0.3
1,0.1,0.7
2,0.7,0.9
2,0.4,0.5
2,0.2,0.3

然后为每个 id 选择第二个元素 id,val2 组合所以例如：

  1,0.3
  2,0.5

我该如何处理？

谢谢

Answer 1

Pig 是一种脚本语言，而不是像 SQL 那样的关系语言，它非常适合与嵌套在 FOREACH 中的运算符的组一起工作。以下是解决方案：

A = LOAD 'input' USING PigStorage(',') AS (id:int, v1:float, v2:float);
B = GROUP A BY id; -- isolate all rows for the same id
C = FOREACH B { -- here comes the scripting bit
    elems = ORDER A BY v1 DESC; -- sort rows belonging to the id
    two = LIMIT elems 2; -- select top 2
    two_invers = ORDER two BY v1 ASC; -- sort in opposite order to bubble second value to the top
    second = LIMIT two_invers 1;
    GENERATE FLATTEN(group) as id, FLATTEN(second.v2);
};
DUMP C;

在您的示例中，id 1 有两行 v1 == 0.2 但 v2 不同，因此 id 1 的第二个值可以是 0.1 或 0.3

Answer 2

A = LOAD 'input' USING PigStorage(',') AS (id:int, v1:int, v2:int);
B = ORDER A BY id ASC, v1 DESC;
C = FOREACH B GENERATE id, v2;
DUMP C;