8

假设我有一组餐厅评论数据：

User,City,Restaurant,Rating
Jim,New York,Mecurials,3
Jim,New York,Whapme,4.5
Jim,London,Pint Size,2
Lisa,London,Pint Size,4
Lisa,London,Rabbit Whole,3.5

我想按用户和城市的平均评论生成一个列表。即输出：

User,City,AverageRating
Jim,New York,3.75
Jim,London,2
Lisa,London,3.75

我可以编写一个 Pig 脚本如下：

Data = LOAD 'data.txt' USING PigStorage(',') AS (
    user:chararray, city:chararray, restaurant:charray, rating:float
);

PerUserCity = GROUP Data BY (user, city);

ResultSet = FOREACH PerUserCity {
    GENERATE group.user, group.city, AVG(Data.rating);
}

但是我很好奇我是否可以先将更高级别的组（用户）分组，然后再将下一个级别（城市）分组：即

PerUser = GROUP Data BY user;

Intermediate = FOREACH PerUser {
    B = GROUP Data BY city;
    GENERATE group AS user, B;
}

我得到：

Error during parsing.
Invalid alias: GROUP in {
  group: chararray,
  Data: {
    user: chararray,
    city: chararray,
    restaurant: chararray,
    rating: float
  }
}

有没有人成功地尝试过这个？是否根本不可能在 FOREACH 中进行 GROUP？

我的目标是做类似的事情：

ResultSet = FOREACH PerUser {
    FOREACH City {
        GENERATE user, city, AVG(City.rating)
    }
}
4

5 回答 5

9

目前允许的操作是FOREACH内的DISTINCT, FILTER,LIMIT和。ORDER BY

现在，按（用户，城市）直接分组是按照您所说的好方法。

于 2011-02-11T18:17:47.453 回答
2

Pig 0.10 版的发行说明建议现在支持嵌套的 FOREACH 操作。

于 2012-12-27T15:45:11.553 回答
1

尝试这个：

Records = load 'data_rating.txt' using PigStorage(',') as (user:chararray, city:chararray, restaurant:chararray, rating:float);
grpRecs = group Records By (user,city);
avgRating_Byuser_perCity = foreach grpRecs generate AVG(Records.rating) as average; 
Result = foreach avgRating_Byuser_perCity generate flatten(group), average;
于 2014-05-18T18:37:23.587 回答
0 
awdata = load 'data' using PigStorage(',') as (user:chararray , city:chararray , restaurant:chararray , rating:float);
data = filter rawdata by user != 'User';

groupbyusercity = group data by (user,city);

--describe groupbyusercity;
--groupbyusercity: {group: (user: chararray,city: chararray),data: {(user: chararray,city: chararray,restaurant: chararray,rating: float)}}

average = foreach groupbyusercity {
    generate group.user,group.city,AVG(data.rating);
}

dump average;
于 2014-04-29T07:20:42.073 回答
0

按两个键分组，然后展平结构会导致相同的结果：

像你一样加载数据

Data = LOAD 'data.txt' USING PigStorage(',') AS (
    user:chararray, city:chararray, restaurant:charray, rating:float);

按用户和城市分组

 ByUserByCity = GROUP Data BY (user, city);

添加组的平均评分（您可以添加更多，例如 COUNT(Data) as count_res）然后将组结构展平为原始结构。

ByUserByCityAvg = FOREACH ByUserByCity GENERATE
FLATTEN(group) AS (user, city),
AVG(Data.rating) as user_city_avg;

结果是：

Jim,London,2.0
Jim,New York,3.75
Lisa,London,3.75
User,City,
于 2014-05-21T11:07:35.747 回答