0

我构建了druid，在其中加载批量数据时出错，如下：

        2016-10-19T07:06:08,656 INFO [task-runner-0-priority-0] org.apache.hadoop.mapreduce.Job - Counters: 12
        Job Counters 
            Failed map tasks=4
            Launched map tasks=4
            Other local map tasks=3
            Data-local map tasks=1
            Total time spent by all maps in occupied slots (ms)=13910
            Total time spent by all reduces in occupied slots (ms)=0
            Total time spent by all map tasks (ms)=13910
            Total vcore-seconds taken by all map tasks=13910
            Total megabyte-seconds taken by all map tasks=14243840
        Map-Reduce Framework
            CPU time spent (ms)=0
            Physical memory (bytes) snapshot=0
            Virtual memory (bytes) snapshot=0
    2016-10-19T07:06:08,657 ERROR [task-runner-0-priority-0] io.druid.indexer.DetermineHashedPartitionsJob - Job failed: job_1476065812127_0012
    2016-10-19T07:06:08,657 INFO [task-runner-0-priority-0] io.druid.indexer.JobHelper - Deleting path[var/druid/hadoop-tmp/wikiticker/2016-10-19T070532.438Z/9d778c51830e449f831a3966a679df11]
    2016-10-19T07:06:08,682 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[HadoopIndexTask{id=index_hadoop_wikiticker_2016-10-19T07:05:32.450Z, type=index_hadoop, dataSource=wikiticker}]
    java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
        at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:204) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        at io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:175) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_79]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_79]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_79]
        at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
        at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        ... 7 more
    Caused by: com.metamx.common.ISE: Job[class io.druid.indexer.DetermineHashedPartitionsJob] failed!
        at io.druid.indexer.JobHelper.runJobs(JobHelper.java:343) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
        at io.druid.indexer.HadoopDruidDetermineConfigurationJob.run(HadoopDruidDetermineConfigurationJob.java:91) ~[druid-indexing-hadoop-0.9.1.1.jar:0.9.1.1]
        at io.druid.indexing.common.task.HadoopIndexTask$HadoopDetermineConfigInnerProcessing.runTask(HadoopIndexTask.java:291) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
        at io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:201) ~[druid-indexing-service-0.9.1.1.jar:0.9.1.1]
        ... 7 more
    2016-10-19T07:06:08,687 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_hadoop_wikiticker_2016-10-19T07:05:32.450Z] status changed to [FAILED].
    2016-10-19T07:06:08,688 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
      "id" : "index_hadoop_wikiticker_2016-10-19T07:05:32.450Z",
      "status" : "FAILED",
      "duration" : 33220
    }

然后按照官方的解决方案，做了一些操作，但也没什么用：

  • 编辑 Druid 的 pom.xml 依赖项以匹配我的 Hadoop 版本中的 Jackson 版本并重新编译 Druid 

    Exception in thread "main" java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.annotations()Ljava/lang/Iterable;

  • 使用 maven-shade-plugin 制作一个胖罐 druid-services-0.9.1.1-selfcontained.jar

    java -Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath ~/druid-0.9.1.1/tmp/druid-services-0.9.1.1-selfcontained.jar:~/druid-0.9.1.1/conf/druid/_common/:~/jackson-* -Ddruid.extensions.loadList=[] io.druid.cli.Main index hadoop --coordinate "org.apache.hadoop:hadoop-client:2.6.0-cdh5.7.1" ../quickstart/wikiticker-index.json

执行这个命令，我得到：

    INFO: An exception was caught and reported. Message: java.lang.NullPointerException
    java.lang.NullPointerException
        at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
        at com.google.inject.spi.Elements.getElements(Elements.java:103)
        at com.google.inject.spi.Elements.getElements(Elements.java:94)
        at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
        at com.google.inject.spi.Elements.getElements(Elements.java:103)
        at com.google.inject.spi.Elements.getElements(Elements.java:94)
        at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:153)
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
        at com.google.inject.spi.Elements.getElements(Elements.java:103)
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
        at com.google.inject.Guice.createInjector(Guice.java:96)
        at com.google.inject.Guice.createInjector(Guice.java:73)
        at com.google.inject.Guice.createInjector(Guice.java:62)
        at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367)
        at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60)
        at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108)
        at io.druid.cli.Main.main(Main.java:105)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115)
        at io.druid.cli.Main.main(Main.java:105)

    2016-10-20T05:05:52,401 ERROR [main] io.druid.cli.CliHadoopIndexer - failure!!!!
    java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
        at io.druid.cli.CliHadoopIndexer.run(CliHadoopIndexer.java:115) [druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.Main.main(Main.java:105) [druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
    Caused by: com.google.inject.CreationException: Guice creation errors:

    1) An exception was caught and reported. Message: null
      at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173)

    2) Binding to null instances is not allowed. Use toProvider(Providers.of(null)) if this is your intended behaviour.
      at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:93)

    3) Could not find a suitable constructor in io.druid.metadata.MetadataStorageTablesConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
      at io.druid.metadata.MetadataStorageTablesConfig.class(MetadataStorageTablesConfig.java:35)
      at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95)

    3 errors
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.Main.main(Main.java:105) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        ... 6 more
    Caused by: java.lang.NullPointerException
        at io.druid.cli.CliInternalHadoopIndexer$1.configure(CliInternalHadoopIndexer.java:95) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements.getElements(Elements.java:94) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:173) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements.getElements(Elements.java:94) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.util.Modules$RealOverriddenModuleBuilder$1.configure(Modules.java:153) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.spi.Elements.getElements(Elements.java:103) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.Guice.createInjector(Guice.java:96) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.Guice.createInjector(Guice.java:73) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at com.google.inject.Guice.createInjector(Guice.java:62) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:367) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:60) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.CliInternalHadoopIndexer.run(CliInternalHadoopIndexer.java:108) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        at io.druid.cli.Main.main(Main.java:105) ~[druid-services-0.9.1.1-selfcontained.jar:0.9.1.1]
        ... 6 more

有没有人有同样的问题。任何人都可以帮忙吗？

4

1 回答 1

1

我能够运行索引作业。不需要有阴影的罐子。以下是步骤。

1) 下载与您的 CDH 匹配的 Hadoop 客户端。在 druid 分发目录中运行此命令。它将 Hadoop 客户端下载到hadoop-dependencies目录中。

 ./bin/pull-deps -r https://repository.cloudera.com/content/repositories/releases/ -h org.apache.hadoop:hadoop-client:2.6.0-mr1-cdh5.7.1

2) 在 middleManger 配置中设置客户端版本。

hadoopDependencyCoordinates=["org.apache.hadoop:hadoop-client:2.6.0-mr1-cdh5.7.1"]

3) 我使用索引服务来运行 Hadoop 作业。创建 JSON，然后jobProperties像这样放置。

"jobProperties": {
  "mapreduce.job.classloader": "true",
  "mapreduce.job.classloader.system.classes": "-javax.validation.,java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop.,org.xerial.snappy."
}

如果您仍然看到一些错误并且知道它存在于 CDH 和 Druid 中，请根据需要将类添加到列表中。

4) HTTP POST JSON 到 Overlord

于 2017-01-29T05:45:17.117 回答