6

我们正在使用 Hive (0.13.0) 测试多节点 hadoop 集群 (2.4.0)。集群工作正常，但是当我们在 hive 中运行查询时，mapred 作业总是在本地执行。例如：

没有 hive-site.xml（实际上，除了默认值之外没有任何配置文件）我们设置 mapred.job.tracker：

hive> SET mapred.job.tracker=192.168.7.183:8032;

并运行查询：

hive> select count(1) from suricata;

Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
OpenJDK 64-Bit Server VM warning: You have loaded library /hadoop/hadoop-2.4.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
14/04/29 12:48:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/04/29 12:48:02 WARN conf.Configuration: file:/tmp/hadoopuser/hive_2014-04-29_12-47-57_290_2455239450939088471-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
Execution log at: /tmp/hadoopuser/hadoopuser_20140429124747_badfcce6-620e-4718-8c3b-e4ef76bdba7e.log
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2014-04-29 12:48:05,450 null map = 0%,  reduce = 0%
.......
.......
2014-04-29 12:52:26,982 null map = 100%,  reduce = 100%
Ended Job = job_local1983771849_0001
Execution completed successfully
**MapredLocal task succeeded**
OK
266559841
Time taken: 270.176 seconds, Fetched: 1 row(s)

我们缺少什么？

4

2 回答 2

2

将hive.exec.mode.local.auto 设置为 false这将禁用 Hive 中的本地模式执行

于 2015-03-20T08:44:37.690 回答
1

对于每个查询，编译器都会生成 map-reduce 作业的 DAG。如果作业在本地模式下运行，请检查以下属性：

mapreduce.framework.name=local;
hive.exec.mode.local.auto=false;

如果启用了自动选项，则配置单元在本地模式下运行作业，如果

Total input size < hive.exec.mode.local.auto.inputbytes.max
Total number of map tasks < hive.exec.mode.local.auto.tasks.max
Total number of reduce tasks =< 1 or 0

这些选项从 0.7 开始可用

于 2017-09-01T06:50:48.970 回答