1

我有两组数据，我想在 Pig 中进行比较。两者都具有相同的唯一 ID，第二组数据中的名称随机更改。逻辑如下：

  • 加载 empl1 原始数据
  • 加载 empl2 原始数据
  • 选择“名称不同”和“emplno 相等”的行

我已经做好了：

A1=  LOAD 'cassandra://employees_pig1/employees_cf' USING CassandraStorage() AS (key, columns: bag {T: tuple(name, value)});

B1=  LOAD 'cassandra://employees_pig2/employees_cf' USING CassandraStorage() AS (key, columns: bag {T: tuple(name, value)});

A2 = FOREACH A1 GENERATE key, FLATTEN(columns);

B2 = FOREACH B1 GENERATE key as key2, FLATTEN(columns);

呵呵，不能在论坛发图。这是说明 A2、B2 的链接 https://picasaweb.google.com/lh/photo/SU3QgKsbA4nmq83cdnhiVdMTjNZETYmyPJy0liipFm0?feat=directlink

现在需要一些帮助，我是否正确地接近这个？

C1 = join A2 by key, B2 by key2;

D1= filter C1 by A2.key==B2.key2 -- cannot do a A2.first_name!=B2.first_name;

想做 选择行，其中“名称不相同”和“emplno 相等”但不完全确定如何。请给个建议。

感谢你

更新： - 而不是加入我做了一个 cogroup C3= COGROUP A2 by key, B2 by key2;

https://picasaweb.google.com/lh/photo/_lkEqW4BvIgbnZSHKDCJGNMTjNZETYmyPJy0liipFm0?feat=directlink

接下来，我正在考虑做

D1= FOREACH C3 GENERATE group, A2.first_name as fn1, B2.first_name as fn2

该组返回所需的结果（即empno），但'A2.first_name，B2.first_name'不正确。需要知道如何访问 A2 和 B2 包/元组中的数据。

然后我就可以通过 fn1==fn2 进行过滤。

4

2 回答 2

1

解决了 ：）

步骤： - 下载 pygmalion https://github.com/jeromatron/pygmalion/downloads

快速测试：

register '/usr/share/dse/pygmalion/pygmalion-1.0.0.jar';
define FromCassandraBag org.pygmalion.udf.FromCassandraBag();
define ToCassandraBag org.pygmalion.udf.ToCassandraBag();

A1=  LOAD 'cassandra://employees_pig1/employees_cf' USING CassandraStorage() AS (key,
columns: bag {T: tuple(name, value)});
B1=  LOAD 'cassandra://employees_pig2/employees_cf' USING CassandraStorage() AS (key, 
columns: bag {T: tuple(name, value)});

A2 = foreach A1 generate key,
flatten(org.pygmalion.udf.FromCassandraBag('first_name', columns))
as (first_name: chararray);

B2 = foreach B1 generate key,
flatten(org.pygmalion.udf.FromCassandraBag('first_name', columns))
as (first_name: chararray);

C1 = join A2 by key, B2 by key;
D1= filter C1 BY A2::first_name != B2::first_name;
于 2012-11-29T04:00:07.077 回答
1

By doing a JOIN (at least, an inner join, which is what you have done above), you already take care of ensuring that the emplnos from A and B are equal. Then all you have to do is filter by whether the names are the same.

C1 = join A2 by key, B2 by key;
D1 = filter C1 by A2::name != B2::name;
于 2012-11-28T19:34:16.033 回答