0

运行以下 Hive 查询会返回特殊字符：

SELECT t6.amt amt2,t6.color color
FROM(
 SELECT t5.color color, t5.c1 amt
 FROM(
  SELECT t1.c1 c1, t1.c2 AS color 
  from(
   SELECT  7716 AS c1, "Red" AS c2 UNION 
   SELECT  6203 AS c1, "Blue" AS c2
  ) t1
 ) t5
order by color) t6
ORDER BY color

它返回结果为

amt color
4   �
3   �

它是一个已知的蜂巢虫吗？

解释计划

    Map 5 <- Union 2 (CONTAINS)
Reducer 3 <- Union 2 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (SIMPLE_EDGE)

Stage-0
   Fetch Operator
      limit:-1
      Stage-1
         Reducer 4
         File Output Operator [FS_331359]
            compressed:false
            Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
            table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
            Select Operator [SEL_331358]
            |  outputColumnNames:["_col0","_col1"]
            |  Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
            |<-Reducer 3 [SIMPLE_EDGE]
               Reduce Output Operator [RS_331357]
                  key expressions:_col1 (type: int)
                  sort order:+
                  Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                  value expressions:_col0 (type: string)
                  Select Operator [SEL_331351]
                     outputColumnNames:["_col0","_col1"]
                     Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                     Group By Operator [GBY_331350]
                     |  keys:KEY._col0 (type: int), KEY._col1 (type: string)
                     |  outputColumnNames:["_col0","_col1"]
                     |  Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                     |<-Union 2 [SIMPLE_EDGE]
                        |<-Map 1 [CONTAINS]
                        |  Reduce Output Operator [RS_331349]
                        |     key expressions:_col0 (type: int), _col1 (type: string)
                        |     Map-reduce partition columns:_col0 (type: int), _col1 (type: string)
                        |     sort order:++
                        |     Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                        |     Group By Operator [GBY_331348]
                        |        keys:_col0 (type: int), _col1 (type: string)
                        |        outputColumnNames:["_col0","_col1"]
                        |        Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                        |        Select Operator [SEL_331342]
                        |           outputColumnNames:["_col0","_col1"]
                        |           Statistics:Num rows: 1 Data size: 91 Basic stats: COMPLETE Column stats: COMPLETE
                        |           TableScan [TS_331341]
                        |              alias:_dummy_table
                        |              Statistics:Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: COMPLETE
                        |<-Map 5 [CONTAINS]
                           Reduce Output Operator [RS_331349]
                              key expressions:_col0 (type: int), _col1 (type: string)
                              Map-reduce partition columns:_col0 (type: int), _col1 (type: string)
                              sort order:++
                              Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                              Group By Operator [GBY_331348]
                                 keys:_col0 (type: int), _col1 (type: string)
                                 outputColumnNames:["_col0","_col1"]
                                 Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                                 Select Operator [SEL_331344]
                                    outputColumnNames:["_col0","_col1"]
                                    Statistics:Num rows: 1 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
                                    TableScan [TS_331343]
                                       alias:_dummy_table
                                       Statistics:Num rows: 1 Data size: 1 Basic stats: COMPLETE Column stats: COMPLETE

在这里禁用或启用配置参数可以帮助我吗？

如果我反转最外层选择中的列顺序，则查询返回预期结果。我本来希望结果是

颜色量

蓝色6203

红色7716

4

1 回答 1

1

我在我的 Hive 2.3 上使用 MR 和 Tez 尝试了相同的查询，结果与您的相同。我关闭了所有查询优化、统计数据收集和 rcp，但结果保持不变。问题是 Hiveorder by在单个 reducer 上进行，并且由于您有两个连续order by的 Hive 将它们合并到单个 reduce 阶段（很容易查看您是否查看和扩展或格式化查询计划）。更准确地说，Hive_col0, _col1对列别名使用等，在t5子查询中你的键是_col0，但这t6就是_col1为什么在选择运算符中你看到

expressions:: "_col1 (type: string), _col0 (type: int)"

并在减少输出运算符

key expressions:: "_col1 (type: int)"

So Hive some how switched key type when the select columns are swapped. 如果 t5 和 t6 中的类型顺序相同，则没有问题

key expressions:: "_col0 (type: string)"

如何避免这种情况——我真的不知道order by在单个减速器中进行顺序不是由于额外的优化。

于 2018-10-04T08:55:14.653 回答