apache-pig - Pig Latin 中的计数任务

Question

假设我有一个情侣(id, value)列表和一个potentialIDs.

对于每一个potentialIDs我想计算ID出现在第一个列表中的次数。

例如

couples:
1 a
1 x
2 y

potentialIDs
1
2
3

Result:
1 2
2 1
3 0

我正在尝试这样做，PigLatin但这似乎并不简单。

你能给我一些提示吗？

Answer 1

总体计划是：您可以按 id 对情侣进行分组并执行 a COUNT，然后对潜在ID 和COUNT. 从那里您可以根据需要对其进行格式化。代码应该更详细地解释如何做到这一点。

注意：如果您需要我更详细地告诉我，但我认为评论应该很好地解释发生了什么。

-- B generates the count of the number of occurrences of an id in couple
B = FOREACH (GROUP couples BY id) 
    -- Output and schema of the group is:
    -- {group: chararray,couples: {(id: chararray,value: chararray)}}
    -- (1,{(1,a),(1,x)})
    -- (2,{(2,y)})

    -- COUNT(couples) counts the number of tuples in the bag
    GENERATE group AS id, COUNT(couples) AS count ;

-- Now we want to do a LEFT join on potentialIDs and B since it will
-- create nulls for IDs that appear in potentialIDs, but not in B
C = FOREACH (JOIN potentialIDs BY id LEFT, B BY id) 
    -- The output and schema for the join is:
    -- {potentialIDs::id: chararray,B::id: chararray,B::count: long}
    -- (1,1,2)
    -- (2,2,1)
    -- (3,,)

    -- Now we pull out only one ID, and convert any NULLs in count to 0s
    GENERATE potentialIDs::id, (B::count is NULL?0:B::count) AS count ;

架构和输出为C：

C: {potentialIDs::id: chararray,count: long}
(1,2)
(2,1)
(3,0)

如果您不想在 中使用消歧运算符（ :: ）C，您可以将该GENERATE行更改为：

GENERATE potentialIDs::id AS id, (B::count is NULL?0:B::count) AS count ;