3

在执行看起来像是两个级别的分组时，我有一个关于 Pig 的问题。例如，假设我有一些示例输入数据，例如：

email_id:chararray    from:chararray        to:bag{recipients:tuple(recipient:chararray)}
e1                    user1@example.com     {(friend1@example.com),(friend2@example.com),(friend3@myusers.com)}
e2                    user1@example.com     {(friend1@example.com),(friend4@example.com)}
e3                    user1@example.com     {(friend5@example.com)}
e4                    user2@example.com     {(friend2@example.com),(friend4@example.com)}

因此，每一行都是从用户“发件人”到用户“发件人”的电子邮件。

我最终想要一个所有发件人和他们发送电子邮件的所有人的列表，包括为每个人发送的电子邮件数量，从高到低排序，例如：

user1@example.com     {(friend1@example.com, 2), (friend2@example.com, 1), (friend3@example.com, 1), (friend4@example.com, 1), (friend5@example.com, 1)}
user2@example.com     {(friend2@example.com, 1), (friend4@example.com, 1)}

将不胜感激有关在 Pig 中解决此问题的最佳方法的想法！

4

1 回答 1

6

这是脚本的一个版本：

inpt = load '/pig_data/pig_fun/input/from_senders.txt' as (email_id:chararray, from:chararray, to:bag{recipients:tuple(recipient:chararray)});

pivot = foreach inpt generate from, FLATTEN(to);
pivot = foreach pivot generate from, to::recipient as recipient;
dump pivot;
/*
(user1@example.com,friend1@example.com)
(user1@example.com,friend2@example.com)
(user1@example.com,friend3@myusers.com)
(user1@example.com,friend1@example.com)
(user1@example.com,friend4@example.com)
(user1@example.com,friend5@example.com)
(user2@example.com,friend2@example.com)
(user2@example.com,friend4@example.com)
*/

grp = group pivot by (from, recipient);
with_count = foreach grp generate FLATTEN(group), COUNT(pivot) as count;
dump with_count;
/*
(user1@example.com,friend1@example.com,2)
(user1@example.com,friend2@example.com,1)
(user1@example.com,friend3@myusers.com,1)
(user1@example.com,friend4@example.com,1)
(user1@example.com,friend5@example.com,1)
(user2@example.com,friend2@example.com,1)
(user2@example.com,friend4@example.com,1)
*/

to_bag = group with_count by from;
result = foreach to_bag {
    order_by_count = order with_count by count desc;
    generate group as from, order_by_count.(recipient, count);
};
dump result;
/*
(user1@example.com,{(friend1@example.com,2),(friend2@example.com,1),(friend3@myusers.com,1),(friend4@example.com,1),(friend5@example.com,1)})
(user2@example.com,{(friend2@example.com,1),(friend4@example.com,1)})
*/

希望能帮助到你。

于 2012-07-09T15:37:16.557 回答