3

在 S3 存储桶中写入数据时出现奇怪的错误。我不会经常收到此错误。因此，无法弄清楚究竟是什么问题。仅供参考，我每次都保持我的 EMR 配置相同。此外，s3 存储桶中的文件夹是NOT写保护的。

    insert overwrite directory 's3://logs/apr'
select f.cookie,sum(f.pgvw) as pageview, count(distinct(f.cookie)) as visits from
(
SELECT a.cookie,a.session,count(distinct(a.date_time)) as pgvw from
(
SELECT extcookie(cs_Cookie) as cookie,extsession(cs_cookie) as session,concat(logdate,' ',time) as date_time FROM apr_1 where (uri like '%.aspx%' or uri like '%.html%') and (not(uri like '/lts%')) 
and (extcookie(cs_Cookie)!='-' and extcookie(cs_Cookie)!='  ')
and (extsession(cs_Cookie)!='-' and extsession(cs_Cookie)!='  ')
group by extcookie(cs_cookie),extsession(cs_cookie),logdate,time
)a
group by a.cookie,a.session
)f
where f.pgvw>1
group by f.cookie;

作业失败的日志：

FINISH_TIME="1373886754825" HOSTNAME="10\.144\.95\.241" ERROR="java\.lang\.RuntimeException: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row {\"_col0\":\"CWC\=4LD8UplOIb7rD5X3uinVAWd7H\",\"_col1\":7,\"_col2\":1}
        at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.map(ExecMapper\.java:166)
        at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:50)
        at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:441)
        at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:377)
        at org\.apache\.hadoop\.mapred\.Child$4\.run(Child\.java:255)
        at java\.security\.AccessController\.doPrivileged(Native Method)
        at javax\.security\.auth\.Subject\.doAs(Subject\.java:396)
        at org\.apache\.hadoop\.security\.UserGroupInformation\.doAs(UserGroupInformation\.java:1132)
        at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:249)
    Caused by: org\.apache\.hadoop\.hive\.ql\.metadata\.HiveException: Hive Runtime Error while processing row {\"_col0\":\"CWC\=4LD8UplOIb7rD5X3uinVAWd7H\",\"_col1\":7,\"_col2\":1}
        at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.process(MapOperator\.java:550)
        at org\.apache\.hadoop\.hive\.ql\.exec\.ExecMapper\.map(ExecMapper\.java:148)
        \.\.\. 8 more
    Caused by: java\.lang\.IndexOutOfBoundsException: Index: 1, Size: 1
        at java\.util\.ArrayList\.RangeCheck(ArrayList\.java:547)
        at java\.util\.ArrayList\.get(ArrayList\.java:322)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.StandardStructObjectInspector\.init(StandardStructObjectInspector\.java:118)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.StandardStructObjectInspector\.<init>(StandardStructObjectInspector\.java:106)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.ObjectInspectorFactory\.getStandardStructObjectInspector(ObjectInspectorFactory\.java:274)
        at org\.apache\.hadoop\.hive\.serde2\.objectinspector\.ObjectInspectorFactory\.getStandardStructObjectInspector(ObjectInspectorFactory\.java:259)
        at org\.apache\.hadoop\.hive\.ql\.exec\.ReduceSinkOperator\.initEvaluatorsAndReturnStruct(ReduceSinkOperator\.java:188)
        at org\.apache\.hadoop\.hive\.ql\.exec\.ReduceSinkOperator\.processOp(ReduceSinkOperator\.java:197)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:471)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:762)
        at org\.apache\.hadoop\.hive\.ql\.exec\.TableScanOperator\.processOp(TableScanOperator\.java:83)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.process(Operator\.java:471)
        at org\.apache\.hadoop\.hive\.ql\.exec\.Operator\.forward(Operator\.java:762)
        at org\.apache\.hadoop\.hive\.ql\.exec\.MapOperator\.process(MapOperator\.java:531)
        \.\.\. 9 more
4

1 回答 1

-1

当您的 SELECT 或 GROUP BY 子句包含两个或多个仅在大小写上有所不同的字段或聚合时（例如，extsession(cs_cookie) 和 extsession(cs_Cookie)），优化器将尝试组合这些字段，即使 ExecMapper 没有。这会导致您看到的错误。

您可以通过将“cs_cookie”的所有实例转换为“cs_Cookie”（反之亦然）并再次尝试相同的查询来确认或否认这是您的问题的原因。如果您不再收到错误，那是由于此优化问题。

于 2014-05-20T20:12:09.983 回答