0

我有 TB 的 HTTP 访问日志，我正在尝试分析这些日志以供历史参考。我正在尝试获取每日唯一访问者 (IP) 数量和每日点击次数。

如果这是在 MySQL 中，我会做这样的事情：

SELECT COUNT(1) from access_logs union
SELECT COUNT(DISTINCT(ip)) from access_logs

有什么办法可以用一份工作做类似的事情，所以我不必第二次做地图工作？

我知道这行不通，但这是我正在寻找的功能。

X = (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
Y = COUNT(X);
Z = COUNT(DISTINCT(X.IP);
OUT = UNION Y,Z;
STORE OUT into ...
4

1 回答 1

0

如果要计算输入中的所有记录，则需要使用 GROUP ALL，它会创建一个包。同样出于性能原因，它使用累加器 DISTINCT 函数 org.apache.pig.builtin.Distinct

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
GRP = group IPs all;
OUT = foreach GRP generate COUNT(IPs) as all_cnt, COUNT(org.apache.pig.builtin.Distinct(IPs.ip)) as distinct_cnt;

如果您有太多 IP 并且遇到与内存相关的异常，则可以执行以下操作：

X = load 'path' as (fqdn:chararray,ip:chararray,date:chararray,time:chararray,uri:chararray,ua:chararray);
IPs = FOREACH X GENERATE ip; // project early for performance reasons
Dist_IPs = distinct IPs;
GRP_DIST = group Dist_IPs all;
DIST = foreach GRP_DIST generate COUNT(GRP_DIST) as cnt, 'dist' as category;

GRP_ALL = group IPs all;
ALL = foreach GRP_ALL generate COUNT(GRP_ALL)as cnt, 'all' as category;

OUT = union DIST, ALL;
于 2013-10-04T08:46:21.177 回答