2

我将 Hadoop 0.20.2 与 Hive 0.11 一起使用。我已经成功地将一些单独的表中的 csv 文件插入到 hive/hdfs 中。选择和加入工作完美无缺。在尝试分析一些数据时，我需要利用 hive 的内置功能，例如：

  • 子串
  • 迄今为止
  • 兰特
  • 等等

例如：

select sid, request_id, to_date(times), to_unix_timestamp(times) from contents where sid = '5000000032066010373';

sid和request id在这里是字符串，times是时间戳列不幸的是，我在使用这些函数时只会收到错误（总是相同的错误堆栈）：

java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
        at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 5 more
Caused by: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
        ... 10 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 13 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
        at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
        ... 18 more
Caused by: java.lang.NoClassDefFoundError: org/codehaus/jackson/JsonFactory
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDTFJSONTuple.<clinit>(GenericUDTFJSONTuple.java:56)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:526)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.registerGenericUDTF(FunctionRegistry.java:520)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.<clinit>(FunctionRegistry.java:423)
        at org.apache.hadoop.hive.ql.exec.DefaultUDFMethodResolver.getEvalMethod(DefaultUDFMethodResolver.java:59)
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.initialize(GenericUDFBridge.java:154)
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:111)
        at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:141)
        at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:970)
        at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:996)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
        at org.apache.hadoop.hive.ql.exec.FilterOperator.initializeOp(FilterOperator.java:78)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
        at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
        at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:543)
        at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
        at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:100)
        ... 18 more
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.JsonFactory

我在这里做错了什么？

SHOW FUNCTIONS;

告诉我这些功能在那里......

4

3 回答 3

0

尝试使用 HIVE_HOME/lib 中的更新 HADOOP_HOME/lib 中的 jackson*.jar，重新启动 Hadoop 并尝试提交作业。

于 2013-09-10T06:24:04.837 回答
0

我在使用 hive + apache hadoop 0.20.2 时遇到了同样的问题

问题是 hadoop-0.20.2 在 lib 文件夹中没有这些 jackson jar，因此您每次都在 hive 控制台中手动添加它：

add jars $HIVE_HOME/lib/jackson-core-asl-1.8.8.jar $HIVE_HOME/lib/jackson-jaxrs-1.8.8.jar $HIVE_HOME/lib/jackson-mapper-asl-1.8.8.jar $HIVE_HOME/lib/jackson-xc-1.8.8.jar;

或者您将所有这些 jar 复制到集群中的每台机器上？

于 2013-10-14T16:36:26.150 回答
0

尝试使用 HIVE_HOME/lib 中的更新 HADOOP_HOME/lib 中的 jackson*.jar，重新启动 Hadoop 并尝试提交作业。

谢谢它的工作

于 2013-10-26T16:12:26.273 回答